Package: scrapyd-client
Scrapyd-client is a client forScrapyd. It provides:Command line tools:scrapyd-deploy, to deploy your project to a Scrapyd serverscrapyd-client, to interact with your project once deployedPython client:ScrapydClient, to interact with Scrapyd within your python codescrapyd-deployDeploying your project to a Scrapyd server typically involves two steps:Eggifyingyour project. You’ll need to installsetuptoolsfor this. SeeEgg Caveatsbelow.Uploading the egg to the Scrapyd server through theaddversion.jsonendpoint.Thescrapyd-deploytool automates the process of building the egg and pushing it to the target
Scrapyd server.Including Static FilesIf the egg needs to include static (non-Python) files, edit thesetup.pyfile in your project.
Otherwise, you can skip this step.If you don’t have asetup.pyfile, create one with:scrapyd-deploy --build-egg=/dev/nullThen, set thepackage_datakeyword argument in thesetup()function call in thesetup.pyfile. Example (note:projectnamewould be your project’s name):fromsetuptoolsimportsetup,find_packagessetup(name='project',version='1.0',packages=find_packages(),entry_points={'scrapy':['settings = projectname.settings']},package_data={'projectname':['path/to/*.json']})Deploying a ProjectFirstcdinto your project’s root, you can then deploy your project with the following:scrapyd-deploy <target> -p <project>This will eggify your project and upload it to the target. If you have asetup.pyfile in your
project, it will be used, otherwise one will be created automatically.If successful you should see a JSON response similar to the following:Deploying myproject-1287453519 to http://localhost:6800/addversion.json
Server response (200):
{"status": "ok", "spiders": ["spider1", "spider2"]}To save yourself from having to specify the target and project, you can set the defaults in theScrapy configuration file.VersioningBy default,scrapyd-deployuses the current timestamp for generating the project version, as
shown above. However, you can pass a custom version using--version:scrapyd-deploy <target> -p <project> --version <version>The version must be comparable withLooseVersion. Scrapyd will use the greatest version unless
specified.If you use Mercurial or Git, you can useHGorGITrespectively as the argument supplied to--versionto use the current revision as the version. You can save yourself having to specify
the version parameter by adding it to your target’s entry inscrapy.cfg:[deploy]...version=HGLocal SettingsYou may want to keep certain settings local and not have them deployed to Scrapyd. To accomplish
this you can create alocal_settings.pyfile at the root of your project, where yourscrapy.cfgfile resides, and add the following to your project’s settings:try:fromlocal_settingsimport*exceptImportError:passscrapyd-deploydoesn’t deploy anything outside of the project module, so thelocal_settings.pyfile won’t be deployed.Egg CaveatsSome things to keep in mind when building eggs for your Scrapy project:Make sure no local development settings are included in the egg when you build it. Thefind_packagesfunction may be picking up your custom settings. In most cases you want to
upload the egg with the default project settings.Avoid using__file__in your project code as it doesn’t play well with eggs.
Consider usingpkgutil.get_datainstead. Instead of:path=os.path.dirname(os.path.realpath(__file__))# BADopen(os.path.join(path,"tools","json","test.json"),"rb").read()Use:importpkgutilpkgutil.get_data("projectname","tools/json/test.json")Be careful when writing to disk in your project, as Scrapyd will most likely be running under a
different user which may not have write access to certain directories. If you can, avoid writing
to disk and always usetempfilefor temporary files.Including dependenciesIf your project has additional dependencies, you can either install them on the Scrapyd server, or
you can include them in the project’s egg, in two steps:Create arequirements.txtfile at the root of the projectUse the--include-dependenciesoption when building or deploying your project:scrapyd-deploy --include-dependenciesscrapyd-clientFor a reference on each subcommand invokescrapyd-client<subcommand>--help.Where filtering with wildcards is possible, it is facilitated withfnmatch.
The--projectoption can be omitted if one is found in ascrapy.cfg.deployThis is a wrapper aroundscrapyd-deploy.projectsLists all projects of a Scrapyd instance:# lists all projects on the default target
scrapyd-client projects
# lists all projects from a custom URL
scrapyd-client -t http://scrapyd.example.net projectsscheduleSchedules one or more spiders to be executed:# schedules any spider
scrapyd-client schedule
# schedules all spiders from the 'knowledge' project
scrapyd-client schedule -p knowledge \*
# schedules any spider from any project whose name ends with '_daily'
scrapyd-client schedule -p \* \*_dailyspidersLists spiders of one or more projects:# lists all spiders
scrapyd-client spiders
# lists all spiders from the 'knowledge' project
scrapyd-client spiders -p knowledgeScrapydClientInteract with Scrapyd within your python code.fromscrapyd_clientimportScrapydClientclient=ScrapydClient()forprojectinclient.projects():print(client.jobs(project=project))Scrapy configuration fileTargetsYou can define a Scrapyd target in your project’sscrapy.cfgfile. Example:[deploy]url=http://scrapyd.example.com/api/scrapydusername=scrapypassword=secretproject=projectnameYou can now deploy your project without the<target>argument or-p<project>option:scrapyd-deployIf you have multiple targets, add the target name in the section name. Example:[deploy:targetname]url=http://scrapyd.example.com/api/scrapyd[deploy:another]url=http://other.example.com/api/scrapydIf you are working with CD frameworks, you do not need to commit your secrets to your repository. You can use environment variable expansion like so:[deploy]url=$SCRAPYD_URLusername=$SCRAPYD_USERNAMEpassword=$SCRAPYD_PASSWORDor using this syntax:[deploy]url=${SCRAPYD_URL}username=${SCRAPYD_USERNAME}password=${SCRAPYD_PASSWORD}To deploy to one target, run:scrapyd-deploy targetname -p <project>To deploy to all targets, use the-aoption:scrapyd-deploy -a -p <project>To list all available targets, use the-loption:scrapyd-deploy -lTo list all available projects on one target, use the-Loption:scrapyd-deploy -L exampleWhile your target needs to be defined with its URL inscrapy.cfg,
you can usenetrcfor username and password, like so:machine scrapyd.example.com
    username scrapy
    password secretHistory1.2.3 (2023-01-30)feat: Addscrapyd-client--usernameand--passwordoptions. (@mxdev88)feat: Expand environment variables in thescrapy.cfgfile. (@mxdev88)feat: AddScrapydClient: a python client to interact with Scrapyd. (@mxdev88)Add support for Python 3.10, 3.11. (@Laerte)1.2.2 (2022-05-03)fix: FixFileNotFoundErrorwhen usingscrapyd-deploy--deploy-all-targets.1.2.1 (2022-05-02)feat: Addscrapyd-deploy--include-dependenciesoption to install project dependencies from arequirements.txtfile. (@mxdev88)fix: Remove temporary directories created byscrapyd-deploy--deploy-all-targets.chore: Address deprecation warnings.chore: Add dependency onurllib3.1.2.0 (2021-10-01)Add support for Scrapy 2.5.Add support for Python 3.7, 3.8, 3.9, PyPy3.7.Drop support for Python 2.7, 3.4, 3.5.Removescrapyd_client.utils.get_config, which was a compatibility wrapper for Python 2.7.1.2.0a1 (2017-08-24)Installscrapyd-deployas a console script.Newscrapyd-clientCLI withdeploy,projects,spiders,
andschedulesubcommands.1.1.0 (2017-02-10)New-aoption to deploy to all targets.Fix returncode on egg deploy error.Add Python 3 support.Drop Python 2.6 support.1.0.1 (2015-04-09)Initial release.