Scrapyでノーベル章受賞者の情報を取得する(PJDV 6.4章) - Scrapy shell

#概要
- [PJDV 5.7～5.7.1](https://qiita.com/int_main_void/items/be86128d7694a13cdde3)、[PJDV 5.7.3](https://qiita.com/int_main_void/items/981407a23cfad704befb#xpath%E8%A8%98%E8%BF%B0%E7%AE%87%E6%89%80)の続き。
- [O'Reilly Japan - PythonとJavaScriptではじめるデータビジュアライゼーション](https://www.oreilly.co.jp/books/9784873118086/)(PJDV)の勉強用
- [PJDV 6.4「最初のScrapyスパイダー」(nwinners_list_spider.py)](https://qiita.com/int_main_void/items/981407a23cfad704befb#xpath%E8%A8%98%E8%BF%B0%E7%AE%87%E6%89%80)のXPathをscrapy shellで具体的に見ていきます。

#準備
##webページの読み込み
以下のいずれかで読み込んでおきます。

- ```scrapy shell http://en.wikipedia.org/wiki/List_of_Nobel_laureates_by_country```で起動
    - 参考 : PJDV 6.3.1章 or [scrapy shellでスクレイピングする - Qiita][scrapy_shell_qiita]
- ```scrapy shell```(引数のurlなし)で起動後、```fetch("http://en.wikipedia.org/wiki/List_of_Nobel_laureates_by_country")```
    - 参考 : [scrapy shellでスクレイピングする - Qiita][scrapy_shell_qiita]
- scrapy crawlからscrapy shellを起動する
    - 後述


```view(response)``` してブラウザを起動させておくと便利です。


![fetch.PNG](https://qiita-image-store.s3.amazonaws.com/0/34777/4eca70f5-5cf5-99ba-1e46-33eb51b2d122.png)




#XPath記述箇所
[nwinners_list_spider.py](
https://github.com/Kyrand/dataviz-with-python-and-js/blob/master/Ch06_Heavyweight_Scraping_with_Scrapy/nobel_winners/spiders/nwinners_list_spider.py)

##h2s = response.xpath('//h2')

###説明
|省略構文|完全な構文                           |意味    |参考    |
|:-------|:------------------------------------|:-------|:-------|
|//h2    |/descendant-or-self::node()/child::h2|全h2要素|[参考1](https://qiita.com/rllllho/items/cb1187cec0fb17fc650a#%E3%82%92%E7%94%A8%E3%81%84%E3%81%A6%E9%80%94%E4%B8%AD%E3%81%BE%E3%81%A7%E3%81%AE%E3%83%91%E3%82%B9%E3%82%92%E7%9C%81%E7%95%A5)<BR>[参考2](http://www.xmldb.jp/tech/xpath_1.php)|


###調査

何が受け取れているか確認してみます。


![h2.PNG](https://qiita-image-store.s3.amazonaws.com/0/34777/8e18bf87-01f4-0e69-fe83-a8a31fc61ec5.png)


![h2b1.png](https://qiita-image-store.s3.amazonaws.com/0/34777/067abcb8-8f68-5285-9d0e-fe0894a73843.png)


![h2b.png](https://qiita-image-store.s3.amazonaws.com/0/34777/38dd30d6-e881-d75d-bfa5-b9676be0eba0.png)



##country = h2.xpath('span[@class="mw-headline"]/text()').extract()



```python
winners = h2.xpath('following-sibling::ol[1]')
```

```python
for w in winners.xpath('li'):
```

```python
text = w.xpath('descendant-or-self::text()').extract()
```

#tips
##scrapy crawlからscrapy shellを起動する
https://doc.scrapy.org/en/latest/topics/shell.html#invoking-the-shell-from-spiders-to-inspect-responses

```python:nwinners_shell.py
# -*- coding: utf-8 -*-
import scrapy
class NWinnerShell(scrapy.Spider):
    name = 'nwinner_shell'
    allowed_domains = ['en.wikipedia.org']
    start_urls = [
        "http://en.wikipedia.org/wiki/List_of_Nobel_laureates_by_country"
    ]
    def parse(self, response):
        scrapy.shell.inspect_response(response, self)
```

#参考
[Scrapyの取得結果をブラウザで開く - Qiita][open_in_browser]
[scrapyでよく使うxpath, cssのセレクタ-Python Snippets][xpath_snippet]
[Python Tips：リストから重複した要素を削除したい - Life with Python][python_uniq]


<!--
http://vaaaaaanquish.hatenablog.com/entry/2017/06/25/202924
https://doc.scrapy.org/en/latest/topics/spiders.html?highlight=spider

mojiko-do
https://plaza.rakuten.co.jp/kugutsushi/diary/200805250000/

https://qiita.com/checkpoint/items/038b59b29df8e1e384a2

https://doc.scrapy.org/en/latest/topics/selectors.html

https://doc.scrapy.org/en/latest/topics/selectors.html?highlight=extract(#some-xpath-tips

https://www.tutorialspoint.com/scrapy/scrapy_shell.htm
-->
<!--LINK-->
[scrapy_shell_qiita]:https://qiita.com/te20/items/80879072defdff645b85#scrapy-shell%E3%81%A7web%E3%83%9A%E3%83%BC%E3%82%B8%E3%82%92%E8%AA%AD%E3%81%BF%E8%BE%BC%E3%82%80
[python_uniq]:https://www.lifewithpython.com/2013/11/python-remove-duplicates-from-lists.html
[xpath_snippet]:https://python.civic-apps.com/scrapy-xpath-css/
[open_in_browser]:https://qiita.com/crispy/items/d67fcf05b2e3916b369a
[open_in_scrapy_shell]:https://doc.scrapy.org/en/latest/topics/shell.html#invoking-the-shell-from-spiders-to-inspect-responses
[Crawler]:https://doc.scrapy.org/en/latest/topics/api.html