Scrapy - Command Line Tools

The Scrapy command line tool is used for controlling Scrapy, which is often referred to as 'Scrapy tool'. It includes the commands for various objects with a group of arguments and options.
Scrapy will find configuration settings in the scrapy.cfg file. Following are a few locations −
C:\scrapy(project folder)\scrapy.cfg in the system
C:\scrapy(project folder)\scrapy.cfg in the system
~/.config/scrapy.cfg ($XDG_CONFIG_HOME) and ~/.scrapy.cfg ($HOME) for global settings
~/.config/scrapy.cfg ($XDG_CONFIG_HOME) and ~/.scrapy.cfg ($HOME) for global settings
You can find the scrapy.cfg inside the root of the project.
You can find the scrapy.cfg inside the root of the project.
Scrapy can also be configured using the following environment variables −
SCRAPY_SETTINGS_MODULE
SCRAPY_PROJECT
SCRAPY_PYTHON_SHELL
The following structure shows the default file structure of the Scrapy project.
scrapy.cfg                - Deploy the configuration file
project_name/             - Name of the project
   _init_.py
   items.py               - It is project's items file
   pipelines.py           - It is project's pipelines file
   settings.py            - It is project's settings file
   spiders                - It is the spiders directory
      _init_.py
      spider_name.py
      . . .

The scrapy.cfg file is a project root directory, which includes the project name with the project settings. For instance −
[settings] 
default = [name of the project].settings  

[deploy] 
#url = http://localhost:6800/ 
project = [name of the project] 

Scrapy tool provides some usage and available commands as follows −
Scrapy X.Y  - no active project 
Usage: 
   scrapy  [options] [arguments] 
Available commands: 
   crawl      It puts spider (handle the URL) to work for crawling data 
   fetch      It fetches the response from the given URL

You can use the following command to create the project in Scrapy −
scrapy startproject project_name

This will create the project called project_name directory. Next, go to the newly created project, using the following command −
cd  project_name

You can control the project and manage them using the Scrapy tool and also create the new spider, using the following command −
scrapy genspider mydomain mydomain.com

The commands such as crawl, etc. must be used inside the Scrapy project. You will come to know which commands must run inside the Scrapy project in the coming section.
Scrapy contains some built-in commands, which can be used for your project. To see the list of available commands, use the following command −
scrapy -h

When you run the following command, Scrapy will display the list of available commands as listed −
fetch − It fetches the URL using Scrapy downloader.
fetch − It fetches the URL using Scrapy downloader.
runspider − It is used to run self-contained spider without creating a project.
runspider − It is used to run self-contained spider without creating a project.
settings − It specifies the project setting value.
settings − It specifies the project setting value.
shell − It is an interactive scraping module for the given URL.
shell − It is an interactive scraping module for the given URL.
startproject − It creates a new Scrapy project.
startproject − It creates a new Scrapy project.
version − It displays the Scrapy version.
version − It displays the Scrapy version.
view − It fetches the URL using Scrapy downloader and show the contents in a browser.
view − It fetches the URL using Scrapy downloader and show the contents in a browser.
You can have some project related commands as listed −
crawl − It is used to crawl data using the spider.
crawl − It is used to crawl data using the spider.
check − It checks the items returned by the crawled command.
check − It checks the items returned by the crawled command.
list − It displays the list of available spiders present in the project.
list − It displays the list of available spiders present in the project.
edit − You can edit the spiders by using the editor.
edit − You can edit the spiders by using the editor.
parse − It parses the given URL with the spider.
parse − It parses the given URL with the spider.
bench − It is used to run quick benchmark test (Benchmark tells how many number of pages can be crawled per minute by Scrapy).
bench − It is used to run quick benchmark test (Benchmark tells how many number of pages can be crawled per minute by Scrapy).
You can build a custom project command with COMMANDS_MODULE setting in Scrapy project. It includes a default empty string in the setting. You can add the following custom command −
COMMANDS_MODULE = 'mycmd.commands'

Scrapy commands can be added using the scrapy.commands section in the setup.py file shown as follows −
from setuptools import setup, find_packages  

setup(name = 'scrapy-module_demo', 
   entry_points = { 
      'scrapy.commands': [ 
         'cmd_demo = my_module.commands:CmdDemo', 
      ], 
   }, 
)
The above code adds cmd_demo command in the setup.py file.

    27 Lectures 
     3.5 hours 

 Attreya Bhatt
 Print
 Add Notes
 Bookmark this page