tseを使って未投稿があるQiita Advent Calendarをさらす

## はじめに

この記事は、[クローラー／Webスクレイピング Advent Calendar 2016](http://qiita.com/advent-calendar/2016/crawler) 8日目の記事です。
出遅れてしまってすみませんでしたorz

## ここでやること

自分が出遅れたことは棚に上げて、他のAdvent Calendarにも未投稿者がいないかどうか調べてみようと思います。

Pythonによるテキスト整形ツール[tse](https://pypi.python.org/pypi/tse)と[beautifulsoup4](https://pypi.python.org/pypi/beautifulsoup4)を使って、Qiitaからデータを抽出します。

## tseとは？

詳しくはPyCon JP 2015で石本さんが発表されたスライドを見れば、理解できるかと思います。
[tse - Pythonによるテキスト整形ユーティリティ](http://www.slideshare.net/atsuoishimoto/tse-python)

今回使用するtse用例

	-ms  -- <モジュール名 import * 形式>でインポートする
	-b   -- 起動直後に実行するアクション
	{{}} -- インデントするブロックを指定

## 環境
Python 2.7.10

## 実際にやってみる

### カテゴリ一覧
```bash
$ curl http://qiita.com/advent-calendar/2016 | tse -ms 'bs4' -b 'for a in BeautifulSoup(sys.stdin.read(), "lxml").find_all("a", class_="adventCalendarCard_block_showAll"):{{print(a.get("href"))}}' > categories.txt
```

```categories.txt
/advent-calendar/2016/categories/to_be_decided
/advent-calendar/2016/categories/programming_languages
/advent-calendar/2016/categories/libraries
/advent-calendar/2016/categories/databases
/advent-calendar/2016/categories/web_technologies
/advent-calendar/2016/categories/mobile
/advent-calendar/2016/categories/devops
/advent-calendar/2016/categories/iot
/advent-calendar/2016/categories/os
/advent-calendar/2016/categories/editors
/advent-calendar/2016/categories/academic
/advent-calendar/2016/categories/services
/advent-calendar/2016/categories/company
/advent-calendar/2016/categories/miscellaneous
```

### カレンダー一覧
```bash
$ cat categories.txt | while read line; do curl http://qiita.com$line | tse -ms 'bs4' -b 'for td in BeautifulSoup(sys.stdin.read(), "lxml").find_all("td", class_="adventCalendarList_calendarTitle"):{{print(td.find_all("a")[-1].get("href"))}}' >> calendars.txt; done
```

```calendars.txt
cat calendars.txt
/advent-calendar/2016/chat-bot
/advent-calendar/2016/idd
/advent-calendar/2016/enough
/advent-calendar/2016/rite
/advent-calendar/2016/befool
~
/advent-calendar/2016/bookthankyou
/advent-calendar/2016/muscle
/advent-calendar/2016/ue4_inside
/advent-calendar/2016/job
/advent-calendar/2016/job2
```

### 未投稿カレンダー一覧
```bash
$ cat calendars.txt | while read line; do curl http://qiita.com$line | tse -ms 'bs4' -b 'bs=BeautifulSoup(sys.stdin.read(), "lxml"){{}}if bs.find("div", class_="adventCalendarItem_cancelButton"):print("[" + bs.title.string.split("- Qiita")[0].strip().encode("utf-8") + "](http://qiita.com" + bs.find(id="redirect_path").get("value")  + ")")' >> not_post_calendars.md; done
```

※ 12/10 5:00時点

[儀式 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/rite)
[Dart Advent Calendar 2016](http://qiita.com/advent-calendar/2016/dartlang)
[F# Advent Calendar 2016](http://qiita.com/advent-calendar/2016/fsharp)
[Java Puzzlers  Advent Calendar 2016](http://qiita.com/advent-calendar/2016/java_puzzlers)
[JavaScript Advent Calendar 2016](http://qiita.com/advent-calendar/2016/javascript)
[Lisp Advent Calendar 2016](http://qiita.com/advent-calendar/2016/lisp)
[Perl 5 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/perl5)
[PHP Advent Calendar 2016](http://qiita.com/advent-calendar/2016/php)
[PHPで何かを作る Advent Calendar 2016](http://qiita.com/advent-calendar/2016/make-system-by-php)
[Pythonでマイクラ風FPS AOSを遊び倒す Advent Calendar 2016](http://qiita.com/advent-calendar/2016/ace_of_spades)
[Smalltalk Advent Calendar 2016](http://qiita.com/advent-calendar/2016/smalltalk)
[Clojure Advent Calendar 2016](http://qiita.com/advent-calendar/2016/clojure)
[D言語 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/dlang)
[Go Advent Calendar 2016](http://qiita.com/advent-calendar/2016/go)
[Go (その3) Advent Calendar 2016](http://qiita.com/advent-calendar/2016/go3)
[Haskell Advent Calendar 2016](http://qiita.com/advent-calendar/2016/haskell)
[mruby Advent Calendar 2016](http://qiita.com/advent-calendar/2016/mruby)
[R Advent Calendar 2016](http://qiita.com/advent-calendar/2016/r-rstudio)
[Rust その2 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/rust-lang-2)
[Scala Advent Calendar 2016](http://qiita.com/advent-calendar/2016/scala)
[Swift その2 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/swift2)
[一人PHP総復習 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/php_beginner)
[A-Frame Advent Calendar 2016](http://qiita.com/advent-calendar/2016/a-frame)
[Django Advent Calendar 2016](http://qiita.com/advent-calendar/2016/django)
[Docker2 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/docker2)
[Grimoire.js Advent Calendar 2016](http://qiita.com/advent-calendar/2016/grimoire)
[GStreamer Advent Calendar 2016](http://qiita.com/advent-calendar/2016/gstreamer)
[phina.js Advent Calendar 2016](http://qiita.com/advent-calendar/2016/phinajs)
[React Native Advent Calendar 2016](http://qiita.com/advent-calendar/2016/react-native)
[ROS Advent Calendar 2016](http://qiita.com/advent-calendar/2016/ros)
[RxJava Advent Calendar 2016](http://qiita.com/advent-calendar/2016/rxjava)
[RxSwift Advent Calendar 2016](http://qiita.com/advent-calendar/2016/rxswift)
[Angular Advent Calendar 2016](http://qiita.com/advent-calendar/2016/angular)
[Laravel Advent Calendar 2016](http://qiita.com/advent-calendar/2016/laravel)
[Ruby on Rails Advent Calendar 2016](http://qiita.com/advent-calendar/2016/ruby_on_rails)
[Unity Assets Advent Calendar 2016](http://qiita.com/advent-calendar/2016/unity-asset)
[Vue.js Advent Calendar 2016](http://qiita.com/advent-calendar/2016/vue)
[Firebird Advent Calendar 2016](http://qiita.com/advent-calendar/2016/firebird)
[MySQL Casual Advent Calendar 2016](http://qiita.com/advent-calendar/2016/mysql-casual)
[eZ Publish / eZ Platform Advent Calendar 2016](http://qiita.com/advent-calendar/2016/ezpublish)
[jupyter notebook Advent Calendar 2016](http://qiita.com/advent-calendar/2016/jupyter)
[Markdown Advent Calendar 2016](http://qiita.com/advent-calendar/2016/markdown)
[Microservices Advent Calendar 2016](http://qiita.com/advent-calendar/2016/microservices)
[Webサイトパフォーマンス Advent Calendar 2016](http://qiita.com/advent-calendar/2016/website-performance)
[Windows Device Portal Advent Calendar 2016](http://qiita.com/advent-calendar/2016/wdp)
[アドエビスエンジニアの集い Advent Calendar 2016](http://qiita.com/advent-calendar/2016/technology-of-adebis)
[フロントエンドエンジニア Advent Calendar 2016](http://qiita.com/advent-calendar/2016/frontend-engineers)
[CSS Advent Calendar 2016](http://qiita.com/advent-calendar/2016/css)
[TECHSCORE Advent Calendar 2016 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/techscore)
[クローラー／Webスクレイピング Advent Calendar 2016](http://qiita.com/advent-calendar/2016/crawler)
[一人PostCSS Advent Calendar 2016](http://qiita.com/advent-calendar/2016/postcss)
[Appcelerator Titanium Advent Calendar 2016](http://qiita.com/advent-calendar/2016/titanium)
[IBM MobileFirst Advent Calendar 2016](http://qiita.com/advent-calendar/2016/ibmmobilefirst)
[Androidその２ Advent Calendar 2016](http://qiita.com/advent-calendar/2016/android_second)
[iOS Advent Calendar 2016](http://qiita.com/advent-calendar/2016/ios)
[iOS その2 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/ios2)
[iOS その3 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/ios3)
[Xamarin Advent Calendar 2016](http://qiita.com/advent-calendar/2016/xamarin)
[PowerShell Advent Calendar 2016](http://qiita.com/advent-calendar/2016/powershell)
[SORACOM  Advent Calendar 2016](http://qiita.com/advent-calendar/2016/soracom)
[JavaScript Robotics Advent Calendar 2016](http://qiita.com/advent-calendar/2016/javascriptrobotics)
[myThings Advent Calendar 2016](http://qiita.com/advent-calendar/2016/mythings)
[Raspberry Pi による乳児モニタリングシステムの開発 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/newborn-baby-monitoring)
[Windows 10 IoT Core Advent Calendar 2016](http://qiita.com/advent-calendar/2016/ms-iot)
[NetBSD Advent Calendar 2016](http://qiita.com/advent-calendar/2016/netbsd)
[Solaris Advent Calendar 2016](http://qiita.com/advent-calendar/2016/solaris)
[Linux Advent Calendar 2016](http://qiita.com/advent-calendar/2016/linux)
[Emacs Advent Calendar 2016](http://qiita.com/advent-calendar/2016/emacs)
[Visual Studio Code Advent Calendar 2016](http://qiita.com/advent-calendar/2016/vscode)
[オーバレイネットワーク Advent Calendar 2016](http://qiita.com/advent-calendar/2016/overlay-network)
[なんでもセキュリティ Advent Calendar 2016](http://qiita.com/advent-calendar/2016/security)
[ヒューマンコンピュータインタラクション論文紹介 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/hci)
[Machine Learning Advent Calendar 2016](http://qiita.com/advent-calendar/2016/machine-learning)
[機械学習に必要な高校数学やり直しアドベントカレンダー Advent Calendar 2016](http://qiita.com/advent-calendar/2016/math_for_ml)
[言語実装 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/lang_dev)
[DevRel Advent Calendar 2016](http://qiita.com/advent-calendar/2016/devrel)
[DIVE INTO CODE Advent Calendar 2016](http://qiita.com/advent-calendar/2016/diveintocode)
[GitLab Advent Calendar 2016](http://qiita.com/advent-calendar/2016/gitlab)
[IDCF Cloud Advent Calendar 2016](http://qiita.com/advent-calendar/2016/idcfcloud)
[Google Cloud Platform(1) Advent Calendar 2016](http://qiita.com/advent-calendar/2016/gcp)
[Kubernetes Advent Calendar 2016](http://qiita.com/advent-calendar/2016/kubernetes)
[Microsoft Azure Advent Calendar 2016](http://qiita.com/advent-calendar/2016/azure)
[クソアプリ Advent Calendar 2016](http://qiita.com/advent-calendar/2016/kusoapp)
[BRIGHT VIE エンジニア Advent Calendar 2016](http://qiita.com/advent-calendar/2016/brightvie)
[dotstudio Advent Calendar 2016](http://qiita.com/advent-calendar/2016/dotstudio)
[Engraphia Advent Calendar 2016](http://qiita.com/advent-calendar/2016/engraphia)
[Fablic Advent Calendar 2016](http://qiita.com/advent-calendar/2016/fablic)
[Fujitsu extended Advent Calendar 2016](http://qiita.com/advent-calendar/2016/fujitsu_ext)
[Hortonworks  Advent Calendar 2016](http://qiita.com/advent-calendar/2016/hortonworks)
[ids-info (東京大学教養学部学際科学科総合情報学コース) Advent Calendar 2016](http://qiita.com/advent-calendar/2016/ids2016)
[iRidge Advent Calendar 2016](http://qiita.com/advent-calendar/2016/iridge)
[KAYAC Advent Calendar 2016](http://qiita.com/advent-calendar/2016/kayac)
[KOIL CSC Advent Calendar 2016](http://qiita.com/advent-calendar/2016/koil-csc)
[Morning Project Samurai Advent Calendar 2016](http://qiita.com/advent-calendar/2016/mpsamurai)
[N高 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/n-highschool)
[Okinawa.go  Advent Calendar 2016](http://qiita.com/advent-calendar/2016/okinawago)
[Okinawa.rb Advent Calendar 2016](http://qiita.com/advent-calendar/2016/okinawarb)
[Perl入学式 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/perl-entrance)
[Scoville Engineers Advent Calendar 2016](http://qiita.com/advent-calendar/2016/scoville)
[Seattle Consulting Advent Calendar 2016](http://qiita.com/advent-calendar/2016/seattleconsulting)
[TokyoSW Advent Calendar 2016](http://qiita.com/advent-calendar/2016/tokyosw)
[Wondershake Advent Calendar 2016](http://qiita.com/advent-calendar/2016/wondershake)
[アラタナ Advent Calendar 2016](http://qiita.com/advent-calendar/2016/aratana)
[ランサーズ Advent Calendar 2016](http://qiita.com/advent-calendar/2016/lancers)
[情報科学専門学校 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/isc)
[数理システム Advent Calendar 2016](http://qiita.com/advent-calendar/2016/mathematical_systems)
[日本情報クリエイト Engineers Advent Calendar 2016](http://qiita.com/advent-calendar/2016/njc)
[東京理科大学 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/tokyo_university_of_science)
[Atrae Advent Calendar 2016](http://qiita.com/advent-calendar/2016/atrae)
[eureka Advent Calendar 2016](http://qiita.com/advent-calendar/2016/eureka)
[Goodpatch Advent Calendar 2016](http://qiita.com/advent-calendar/2016/goodpatch)
[HAL Advent Calendar 2016](http://qiita.com/advent-calendar/2016/hal)
[IT勉強会/コミュニティ運営 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/event-management)
[mixiグループ Advent Calendar 2016](http://qiita.com/advent-calendar/2016/mixi)
[Opt Technologies Advent Calendar 2016](http://qiita.com/advent-calendar/2016/opt-technologies)
[Sansan Advent Calendar 2016](http://qiita.com/advent-calendar/2016/sansan)
[Yahoo! JAPAN Tech Advent Calendar 2016](http://qiita.com/advent-calendar/2016/yahoojapan-tech)
[インタープリズムの面々が、普段の業務に役立つ記事を丹精込めて書き上げる！ Advent Calendar 2016](http://qiita.com/advent-calendar/2016/interprism)
[フューチャーアーキテクト Advent Calendar 2016](http://qiita.com/advent-calendar/2016/future)
[炎の型工房 ScratchとDartでゲームプログラム入門 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/firefirestyle-studio)
[CloudAnalytics Advent Calendar 2016](http://qiita.com/advent-calendar/2016/analytics)
[Hunachi^shoujin0^algorithm0 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/hunachi-shoujin0-algorithm0)
[Product Manager Advent Calendar 2016](http://qiita.com/advent-calendar/2016/productmanager)
[Product Owner Advent Calendar 2016](http://qiita.com/advent-calendar/2016/productowner)
[Stream Processing Advent Calendar 2016](http://qiita.com/advent-calendar/2016/streaming)
[Student Advent Calendar 2016](http://qiita.com/advent-calendar/2016/student)
[test Advent Calendar 2016](http://qiita.com/advent-calendar/2016/test)
[レイトレ Advent Calendar 2016](http://qiita.com/advent-calendar/2016/raytracing)
[下請けCOBOLERとして生きていた頃のノウハウ Advent Calendar 2016](http://qiita.com/advent-calendar/2016/cobol)
[個人開発者 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/individual-developer)
[可視化 / Data Visualization Advent Calendar 2016](http://qiita.com/advent-calendar/2016/dataviz)
[絵文字 / Emoji Advent Calendar 2016](http://qiita.com/advent-calendar/2016/emoji)
[acairojuni Advent Calendar 2016](http://qiita.com/advent-calendar/2016/acairojuni)
[Aizu Advent Calendar 2016](http://qiita.com/advent-calendar/2016/aizu)
[FLASHer Advent Calendar 2016](http://qiita.com/advent-calendar/2016/flashers)
[HoloLens Advent Calendar 2016](http://qiita.com/advent-calendar/2016/hololens)
[Serverless(2) Advent Calendar 2016](http://qiita.com/advent-calendar/2016/serverless2)
[チャットボット Advent Calendar 2016](http://qiita.com/advent-calendar/2016/chatbot)
[ひとりVRカレンダー Advent Calendar 2016](http://qiita.com/advent-calendar/2016/vr)
[ポエム Advent Calendar 2016](http://qiita.com/advent-calendar/2016/poet)
[技術書献本大感謝 Advent Calendar 2016](http://qiita.com/advent-calendar/2016/bookthankyou)

## おわりに

| 全て | 未投稿 |
|:---:|:---:|
|509|141|

結果はこのようになりました。

現時点では全体の28%です。
2016年を気持ちよく終えられるよう、後日投稿・代理投稿を駆使して未投稿カレンダーの割合を減らせるよう皆さん張り切っていきましょう！