Package: dr-web-engine
Data Retrieval Web EngineContextMultiple technologies are used as web parsers, web scrapers, spider and so forth.
Comparative studies can be found inliteraturethat
categorise based on methods and technologies. We took a different perspective and looked atquerabilityfeature.
Our inspiration comes formOXPathwhere an extension of XPath is used to "query" and extract semi-structured data from the web.ObjectivesSimilarly toOXPath, our objective is to create a tool for data retrieval from the web based on a "query" mechanism.
We opted for using JSON constructs for our query definitions with augmented keywords, filters and actions.Technology stackThe tool is written in Python3 and can be included in other python projects by installing it from the python package index
usingpip3 install dr-web-engineor integrating with the tools command line interface by runningpython3 -m web_engine.runnerThe tool is build on top of several other packages which will be automatically installed. These are:SeleniumGeckodriver AutoinstallerLXMLPython InterfaceArgParseXVFBWrapperXVFB only works on Linux and if the parameter is True on a Windows or MacOX system you will get an error message.The Python Package page can be foundhereUsageTo use the integrated CLI runpython3 -m web_engine.runner. This will display the following help message:usage:runner.py[-h][-qQUERY][-e[ENGINE]][-ht[HEIGHT]][-wh[WIDTH]][-lat[LAT]][-lon[LON]][-img[IMG]][-l[LOG]][-xvfb[XVFB]]WebScrapEngineforsemi-structuredwebdataretrievalusingJSONqueryconstructs

optionalarguments:-h,--helpshowthishelpmessageandexit-qQUERY,--queryQUERYJSONquery-e[ENGINE],--engine[ENGINE]Engine:use[lxml]forparserengine(default),[selenium]foractionbasedwebscraping-ht[HEIGHT],--height[HEIGHT]specifythebrowserwindowheight(defaultis800,onlyusedwithSeleniumengine)-wh[WIDTH],--width[WIDTH]specifythebrowserwindowwidth(defaultis1280,onlyusedwithSeleniumengine)-lat[LAT],--lat[LAT]Latitude(notspecifiedbydefault)-lon[LON],--lon[LON]Longitude(notspecifiedbydefault)-img[IMG],--img[IMG]Loadimages-l[LOG],--log[LOG]SetflagtoTruetoseeverboseloggingoutput-xvfb[XVFB],--xvfb[XVFB]SetflagtoFalsetoseeFirefoxwhenusingSeleniumengineThere is only one required parameter:-q QueryFor example, to run the web data retrieval with the following JSON query (supposedly filetest.json):{"_doc":"https://www.google.com/search?q=Donald+Duck","links":[{"_base_path":"//div[@id='search'][1]//div[@class='g']","_follow":"//a[@id='pnnext'][1]/@href","link":"//div[@class='rc']/div[@class='r']/a/@href","title":"//h3/text()"}]}use the following command:python3 -m web_engine.runner -q test.json. The outcome will look like the following JSON result:{"links": [{"link": ["https://en.wikipedia.org/wiki/Donald_Duck"], 
           "title": ["Donald Duck - Wikipedia"]},
           {"link": ["https://cosleyzoo.org/white-pekin-duck/"],
            "title": ["White Pekin Duck – Cosley Zoo"]},
           {"link": ["https://www.cheatsheet.com/entertainment/donald-duck-turned-85-years-old.html/"],
            "title": ["Donald Duck Turned 85-Years-Old and Disney Fans Are Quacking ..."]},
           {"link": ["https://en.wikipedia.org/wiki/Daisy_Duck"],
            "title": ["Daisy Duck - Wikipedia"]},
           {"link": ["https://www.headstuff.org/culture/history/disney-studios-war-story-donald-duck-became-sgt/"],
            "title": ["Disney Studios At War - the story of how Donald Duck became a Sgt ..."]}In the JSON query provided, the items starting with_are keywords and can be filters, actions or instructions.
If we remove all the keywords the remaining JSON represents the structure of the expected output.In another more complex query we use some other keywords and actions:{"_doc":"https://www.checkatrade.com/trades/WayreHouseElectricalServices","data":{"ld_data":"//head/script[@type=\"application/ld+json\"][1]"},"reviews":[{"_doc":"https://www.checkatrade.com/trades/WayreHouseElectricalServices/reviews","_base_path":"//div[contains(@class, 'ReviewsPage__Content')]//div[contains(@class, 'ReviewsItem__Wrapper')]","_key":"review","_pre_remove":"//*[contains(@class,'alert-box')]","_follow":"//a[contains(@class,\"Chevrons__Wrapper\")][2]/@href","_follow_action":"//a[contains(@class,\"Chevrons__Wrapper\")][2]{click }","title":"//h3[contains(@class, 'ReviewsItem__Title')]","score":"//*[name()='svg']//text()[normalize-space()]","verified":"//div[contains(@class, 'ReviewsItem__Verified')]/text()[normalize-space()]","content":"//p[contains(@class, 'ReviewsItem__P')]","review_by":"//div[contains(@class, 'ReviewsItem__Byline')]/text()[normalize-space()]"}]}Keywords_doc: Represents the document to follow. Is usually a URL to a web page.  It is compulsory on the top level and can be provided on the lower levels of the hierarchical structure. The_dockeyword_base_path: To be used in an array extraction. Arrays are a lists of element and are defined in the query as an JSON array[].
When_base_pathis provided, all elements of the query in the array will be looked inside the HTML element as defined by_base_path._key: Use to assign each element of the array to assigned to the variable_key_pre_xxx: All actions that start with_pre_are to be executed before data extraction._pre_remove: Remove elements from page_follow: Follow the link if and when exists_follow_action: If element in follow exists, then perform action rather than following the link. The actions are defined as the last part of the XPath query and are always defined between carley brackets. In this case the action{click }means click on element.ExtendabilityThe package is intended to be easily extendable. For example the{click }action is defined in the query as follows:The corresponding is defined as follows:classClickAction(implements(Action)):def__init__(self,receiver,log:logging=None):self._log=logself._receiver=receiverdefexecute(self,*args):ifargsisNoneorlen(args)!=1:returnxpath_selector:str=args[0]wait=WebDriverWait(self._receiver.driver,10)elem=wait.until(EC.element_to_be_clickable((By.XPATH,xpath_selector)))elem.click()In the Scraper implementations, actions are registered against keywords as follows:click=ClickAction(self,self.log)filter_remove=FilterRemoveAction(self,self.log)self.register('click',click)self.register('remove',filter_remove)And invoked by simply matching the action keywords in the query as follows:defaction_get(self,actions:list):forxinactions:self.execute(x)returnself.get()defexecute(self,action_composite:str):action_name,action_path=SeleniumScraper.__get_action(action_composite)action_name=action_name.strip()ifaction_nameinself._actions.keys():self._history.append((time.time(),action_name))self._actions[action_name].execute(action_path)else:self.log.warn(f"Command [{action_name}] not recognised")@staticmethoddef__get_action(action_composite):pattern='{(.+?)}'matches=re.search(pattern,action_composite)ifnotmatches:returnNone,Noneaction_name=matches.group(1)action_xpath=re.sub(pattern,'',action_composite)returnaction_name,action_xpathFuture workThis work, whilst it is a working beta, is by no means complete and it's rather focused on a narrow specific problem. However, special effort has been made to keep the solution generic, universal and extendable for it to potentially grow into a mature Data Retrieval Web Engine based on JSON Queries.