Package: scrapqd
ScrapQD - Scraper Query Definition (beta)ScrapQD consists of query definition created for scraping web data usingGraphQL-Corewhich is port ofGraphQL.js.Library intends to focus on how to locate data from website and eliminate backend process of crawling. So people can just have xpath and get data right away.It supports scraping usingrequestsfor traditional websites andseleniumfor modern websites (js rendering).
Under selenium it supports Google Chrome and FireFox drivers.ScrapQD library only useslxmlparser andxpathused to locate elements.Explore the docs »Source Code »Report Bug »Request Feature »Getting StartedHow to installpipinstallscrapqdHow to run the server standaloneYou can run scrapqd graphql server standalone without any additional code with below command.Flaskis used as server andlocalhost.python-mscrapqdFlask uses 5000 as default port. You can change the port and host with below options.python-mscrapqd--port5001--hostx.x.x.xQuerySample query is loaded to GraphQL UI andsample pageis available within the server to practice.querytest_query($url:String!,$name:GenericScalar!){result:fetch(url:$url){name:constant(value:$name)summary:group{total_emp_expenses:text(xpath:"//*[@id='emp-exp-total']",data_type:INT)total_shown_expenses:text(xpath:"//*[@id='exp-total']/span[2]",data_type:INT)total_approved_expenses:text(xpath:"//*[@id='emp-exp-approved']/span[2]",data_type:INT)}exp_details:list(xpath:"//div[@class='card']"){name:text(xpath:"//div[contains(@class,'expense-emp-name')]")amount:group{money:text(xpath:"//h6[contains(@class,'expense-amount')]/span[1]",data_type:INT)name:text(xpath:"//h6[contains(@class,'expense-amount')]/span[2]")}}}}query variable// url will be used in the above queryquery_variables={"url":"http://localhost:5000/scrapqd/sample_page/","name":"local-testing"}Result{"data":{"result":{"name":"local-testing","summary":{"total_emp_expenses":309,"total_shown_expenses":40,"total_approved_expenses":4},"exp_details":[{"name":"Friedrich-Wilhelm, Langern","amount":{"money":8800,"name":"egp"}},{"name":"Sebastian, Bien","amount":{"money":3365,"name":"mkd"}},{"name":"Rosa, Becker","amount":{"money":6700,"name":"xof"}},{"name":"Ines, Gröttner","amount":{"money":8427,"name":"npr"}}]}}}Executing with clientfromscrapqd.clientimportexecute_syncquery=r"""
        query test_query($url: String!, $name: GenericScalar!) {
          result: fetch(url: $url) {
            name: constant(value: $name)
            summary: group {
              total_shown_expenses: regex(xpath: "//*[@id='exp-total']", pattern: "(\\d+)")
            }
          }
        }"""query_variables={"url":"http://localhost:5000/scrapqd/sample_page/","name":"local-testing"}result=execute_sync(self.query,query_variables)Integrating with existing Flask appSample Flask appfromflaskimportFlaskname=__name__app=Flask(name)@app.route("/")defhello_world():return"<p>Hello, World!</p>"Integrating scrapqd with existing appfromscrapqd.appimportregister_scrapqdregister_scrapqd(app,register_sample_url=True,redirect_root=True)app:Flask applicationregister_sample_url:Falsewill not register sample page url to Flask application. Default isTrueredirect_root:Redirect root url to graphql ui if this is set toTrue. This will not reflect, if there is already root route defined as above example.Test (for development)Clone the github repositorygitclonehttps://github.com/dduraipandian/scrapqd.gitcreate virtual environment to workpip3installvirtualenvvirtualenvscrapqd_venvsourcescrapqd_venv/bin/activateinstall toxpipinstalltoxrun tox from the project root directorycurrent tox have four python version - py37,py38,py39,py310check your python versionpython3--version# Python 3.9.10once you get your version (example: use py39 for 3.9) to run toxtox-epy39FAQsHow to copy query from graphql ui to python code.you can normally copy code from ui to python code to execute using client.if you havregexquery, patterns needs to escaped in the python code. In such, use pythonraw strings, where backslashes are treated as literal characters, as above example.How to suppress webdriver logsIf you see webdriver logs like below, setWDM_LOG_LEVEL=0as environment variable and run[INFO][97002][2022-03-14T02:18:26+0530][SCRAPQD][/webdriver_manager/logger.py:log():26][WDM][Driver[/99.0.4844.51/chromedriver]...]How to change log level for scrapqd libraryERRORlevel is default logging. You can change this withSCRAPQD_LOG_LEVELenvironment variable.LicenseThis project is licensed under the MIT License - see theLICENSEfile for details