Package: scraperx
ScraperX  SDKGetting StartedCreate a new directory where the scraper will live and add the following files:A config file:config.yaml(Example)The scraper file:your_scraper.py(Example)Next install this library from pypi:pip install scraperxRun the full scraper by runningpython your_scraper.py dispatchTo see the arguments for the command:python your_scraper.py dispatch -hSee all the commands available:python your_scraper.py -hSample scrapersSample scrapers can be found in theexamplesfolder of this repoDevelopingAny time the scraper needs to override the bases__init__, always pass in*args&**kwargslike so:def__init__(self,*args,**kwargs):super().__init__(*args,**kwargs)DispatchingTask dataThis is a dict of values that is passed to each step of the process. The scraper can put anything it wants here that it may need. But here are a few build in values that are not required, but are used if you do supply them:headers: Dict of headers to use each requestproxy: Full proxy string to be usedproxy_country: Used to get a proxy for this region, if this andproxyare not set, a random proxy will be used.device_type: used when setting a user-agent if one was not set. Options aredesktopormobileDownloadingUses arequests.Sessionto make get and post requests.
The__init__of theBaseDownloadclass can take the following args:task: Required. The task from the dispatcherheaders: (Named arg) dict to set headers for the whole session. default: random User-Agent for the device type, will use desktop if no device type is setproxy: (Named arg) Proxy string to use for the requestsignore_codes: (Named arg) List of HTTP Status codes to not retry on. If these codes are seen, it will treat the request as any other success.When using BaseDownloader, a requests session is created underself.session, so every get/post you make will use the same session per task.
Headers can also be set per call by passing the keyword args toself.request_get()andself.request_post(). Any kwargs you pass to self.request_get/post will be passed to the sessions get/post methods.When using BaseDownloader's get & post functions, it will use the requests session created ininitand a pythonrequestsresponse object.A request will retryntimes (3 by default) to get a successful status code, each retry it will try and trigger a function callednew_profile()where you have the chance to switch the headers/proxy the request is using (will only update for that request?). If that function does not exist, it will try again with the same data.There are a few custom arguments that can be passed into theself.request_*functions that this sdk will use. All others will be passed to therequestsmethods call.Named arguments:max_tries: Default=3. Type int. The number of tries a request will be tried, each try it will try and get a new proxy and User-Agentcustom_source_checks: Default=None. Type list of lists. Used to set the request to a set status code based on a regex that runs on the page source.This will look to see if the wordscaptchaare in the source page and set that response status code to a 403, with the status message beingCapacha Found. The status message is there so you know if it is a real 403 or your custom status.[(re.compile(r'captcha', re.I), 403, 'Capacha Found')]When usingself.request_*, it will return a normal requests.request response, If using custom source checks,response.reasonwill be set to the custom message passed in. This is useful if you have multiple ways a custom 403 happens and you need to do different actions depending on why.Saving the sourceThis is required for the extractor to run on the downloaded data. Inside ofself.download()just callself.save_request(r)on the request that was made. This will add the source file to a list of saved sources that will be passed to the extractor for parsing.Some keyword arguments that can be passed intoself.save_requesttemplate_values{dict}- Additional keys to use in the templatefilename{str}- Override the filename from the template_name in the configDownload ExceptionsThese exceptions will be raised when callingself.request_*. They will be caught safely so the scraper does not need to catch them. But if the scraper wanted to do something based on the exception, there can be atry/exceptaround the scrapersself.request_*.scraperx.exceptions.DownloadValueError: If there is an exception that is not caught by the othersscraperx.exceptions.HTTPIgnoreCodeError: When the status code of the request is found in theignore_codesargument of BaseDownloadrequests.exceptions.HTTPError: When the requests returns a non successful status code and was not found inignore_codesSetting headers/proxiesThe ones set in theself.request_get/request_postwill be combined with the ones set in the__init__and override if the key is the same.self.request_get/request_post kwargs headers/proxywill overrideself.task[headers/proxy]will overrideinitkwargs headers/proxyAny header/proxy set on the request (get/post/etc) will only be set for that single request. For those values to be set in the session they must be set from the init or be in the task data.ProxiesIf you have a list of proxies that the downloader should auto rotate between they can be saved in a csv in the following format:proxy,country
http://user:pass@some-server.com:5500,US
http://user:pass@some-server.com:5501,US
http://user:pass@some-server.com:6501,DESet the env varPROXY_FILEto the path of the above csv for the scraper to load it in.If you have not passed in a proxy directly in the task and this proxy csv exists, then it will pull a random proxy from this file. It will use theproxy_countryif set in the task data to select the correct country to proxy to.User-AgentIf you have not directly set a user-agent, a random one will be pulled based on thedevice_typein the task data.Ifdevice_typeis not set, it will default to use a desktop user-agent.
To set your own list of user-agents to choose from, create a csv in the following format:device_type,user_agent
desktop,"Some User Agent for desktop"
desktop,"Another User Agent for desktop"
mobile,"Now one for mobile"Set the env varUA_FILEto the path of the above csv for the scraper to load it in.ExtractingParsel documentationData extraction helpersself.pre_extract()User can override to do their own setup after the__init__and before any extraction happensself.find_css_elements(source, css_selectors)source- Parsel object to run the css selectors oncss_selectors- A list of css selectors to try and extract the dataReturns a Parsel element from the first css selector that returns data.This snippet would be in the scrapersMyScraperExtract(Extract)class, used in the method that is extracting the data.title_selectors=['h3','span.title',]result['title']=self.find_css_elements(element,title_selectors)\.xpath('string()').extract_first().strip()There are a few built in parsers that can assist with extracting some types of datafromscraperximportparsers#### Price#### This will parse the price out of a string and return the low and high values as floatsraw_p1_str='15,48€'p1=parsers.price(raw_p1_str)# p1 = {'low': 15.48, 'high': None}raw_p2_str='1,999'p2=parsers.price(raw_p2_str)# p2 = {'low': 1999.0, 'high': None}raw_p3_str='$49.95 - $99.99'p3=parsers.price(raw_p3_str)# p3 = {'low': 49.95, 'high': 99.99}#### Rating#### Parse the rating from a string# Examples: https://regex101.com/r/ChmgmF/3raw_r1_str='4.4 out of 5 stars'r1=parsers.rating(raw_r1_str)# r1 = 4.4raw_r2_str='An average of 4.1 star'r2=parsers.rating(raw_r2_str)# r2 = 4.1If there are more cases you would like these parsers to catch please open up an issue with the use case you are trying to parse.TestingWhen updating the extractors there is a chance that it will not work with the previous source files. So having a source and its QA'd data file is useful to test against to verify that data is still extracting correctly.Creating test filesRunpython your_scraper.py create-test path_to/metadata_source_fileThe input file is the*_metadata.jsonfile that gets created when you run the scraper and it downloads the source files.This will copy the metadata file and the sources into the directorytests/sample_data/your_scraper/using the time the source was downloaded (from the metadata) as the file name.It also creates extracted qa files for each of the sources based on your extractors.it extracts the data in json format to make it easy to qa and read.The QA files it created will have_extracted_(qa)_in the file name. What you have to do it confirm that all values are correct in that file. If everything looks good then fix the file name from having_extracted_(qa)_to_extracted_qa_. This will let the system know that the file has been checked and that is the data it will use to compare when testing.Create an empty filetests/__init__.py. This is needed for the tests to run.Next is to create the code that will run the tests. Create the filetests/tests.pywith the contents belowimportunittest# The testing frame work to usefromscraperx.testimportExtractorBaseTest# Does all the heavy lifting for the testfromyour_scraperimportscraperasmy_scraper# The scrapers Scraper class# If you have multiple scrapers, then import their extract classes here as well# This test will loop through all the test files for the scraperclassYourScraper(ExtractorBaseTest.TestCase):def__init__(self,*args,**kwargs):# The directory that the test files for your scraper are indata_dir='tests/sample_data/your_scraper'# ignore_keys will not test the qa values to the current extracted test value. This is most useful when dealing with timestamps or other values that will change on each time the data is extractedsuper().__init__(data_dir,my_scraper,ignore_keys=['time_extracted'],*args,**kwargs)# If you have multiple scrapers, then create a class for each# Feel free to include any other unit tests you may want to run as wellRunning the testspython -m unittest discover -vvConfig3 Ways of setting config values:CLI Argument: Will override any other type of config value. Use-hto see available optionsEnvironment variable: Will override a config value in the yamlYaml file: Will use these values if no other way is set for a keyConfig values_If you are using any aws service for any part of this, it will use theboto3 libraryand will try and get credentials from the system# config.yaml# This is a config file with all config values# Required fields are marked as suchdefault:dispatch:limit:5# Default None. Max number of tasks to dispatch. If not set, all tasks will runservice:# This is where both the download and extractor services will runname:local# (local, sns) Default: localsns_arn:sns:arn:of:service:to:trigger# Required if `name` is sns, if local this is not neededratelimit:type:qps# (qps, period) Required. `qps`: Queries per second to dispatch the tasks at. `period`: The time in hours to dispatch all of the tasks in.value:1# Required. Can be an int or a float. When using period, value is in hoursdownloader:save_metadata:true# (true, false) Default: true. If false, a metadata file will NOT be saved with the downloaded source.save_data:service:local# (local, s3) Default: local# Required if `service` is s3, if local these are not neededbucket_name:my-downloaded-data-bucketendpoint_url:https://s3.my-server.com# Used if not using AWS s3 bucket# Only needed if aws creds are not setup on the system or you want to not use the system credsaws_access_key_id:abcde# Auth key to access the s3 server.aws_secret_access_key:abcde123# Auth secret to access the s3 serverfile_template:test_output/{scraper_name}/{id}_source.html# Optional, Default is "output/extracted.json"extractor:save_data:service:local# (local, s3) Default: local# Required if `service` is s3, if local these are not neededbucket_name:my-extracted-data-bucketendpoint_url:https://s3.my-server.com# Used if not using AWS s3 bucket# Only needed if aws creds are not setup on the system or you want to not use the system credsaws_access_key_id:abcde# Auth key to access the s3 serveraws_secret_access_key:abcde123# Auth secret to access the s3 serverfile_template:test_output/{scraper_name}/{id}_extracted.json# Optional, Default is "output/source.html"If you are using thefile_templateconfig, a python.format()runs on this string so you can use{key_name}to make it dynamic. The keys you will have direct access to are the following:All keys in your task that was dispatchedAny thing you pass into thetemplate_values={}kwarg for the.save()fnAll values inscraper.log_extras. Currentlyscraper_name&run_idtime_downloaded: time (utc) passed from the downloader (in both the downloader and extractor)date_downloaded: date (utc) passed from the downloader (in both the downloader and extractor)time_extracted: time (utc) passed from the extractor (just in the extractor)date_extracted: date (utc) passed from the extractor (just in the extractor)Anything under thedefaultsection can also have its own value per scraper. So if we have a scraper namedsearchand we want it to use a different rate limit then all the other scrapers you can do:# Name of the python filesearch:dispatch:ratelimit:type:periodvalue:5To override thevaluein the above snippet using an environment variable, setDISPATCH_RATELIMIT_VALUE=1. This will override all dispatch ratelimit values in default and custom.IssuesIf you run into the errormay have been in progress in another thread when fork() was called.when running the scraper locally on a mac. Then set the env varexport OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YESThis is because of a security setting on macs when spawning threads from threadshttps://github.com/ansible/ansible/issues/32499#issuecomment-341578864