Package: vanguardkit
Vanguard kitA convenient way to calculate the edit distance between html files to scrape with confidenceSometimes, scraping becomes a hard task, because the web sites are in continous changing.
What about if there was a way to prevent those changes before scrape a site?
Vanguard is a tool kit that provides a way to calculate the edit distance between
two html files by the Zhang-Shasha algorithm.
This package is based onzss.InstallationOS X & Linux:From PYPI$pip3installvanguardkitfrom the source$gitclonehttps://github.com/dany2691/vanguard-kit.git
$cdvanguard-kit
$python3setup.pyinstallUsage exampleWith vanguard, it is possible to convert html content into a tree (graph) of nodes.
The create_html_tree function is the responsible to do that, it returns an instance of the VanguardNode class that inherits from the zss.Node class:fromvanguardkitimportcreate_html_treewithopen("target_website.html")astarget_website:thml_tree=create_html_tree(target_website)It is possible to segment specific parts of an html file.By tag:withopen("target_website.html")astarget_website:html_tree=create_html_tree(html_file=target_website,specific_tag="footer")By tag and class:withopen("target_website.html")astarget_website:html_tree=create_html_tree(html_file=target_website,specific_tag="div",class_="main-div")By tag and id:withopen("target_website.html")astarget_website:html_tree=create_html_tree(html_file=target_website,specific_tag="div",id="super-div")Calculating distanceAs previously said, the used algorithm is the Zhang-Shasha, that computes the edit distance between the two given trees. Ths is possible with the zss package behind the scenes; vanguard only provides a way to convert html files into trees.fromvanguard_kitimportcreate_html_tree,calcuate_html_tree_distancewithopen("stored_target_website.html")asstored_file:withopen("current_target_website.html")ascurrent_file:previous_tree=create_html_tree(stored_file)current_tree=create_html_tree(current_file)print(calcuate_html_tree_distance(previous_tree,current_tree))# Prints 1Due to the VanguardNode class implements thesubdunder method, the next way to calculate the edit distance is possible:fromvanguard_kitimportcreate_html_tree,calcuate_html_tree_distancewithopen("stored_target_website.html")asstored_file:withopen("current_target_website.html")ascurrent_file:previous_tree=create_html_tree(stored_file)current_tree=create_html_tree(current_file)print(previous_tree-current_tree)# Prints 1Then, the next statement returns True:calcuate_html_tree_distance(previous_tree,current_tree)==previous_tree-current_treeDevelopment setupThis project usesPoetryfor dependecy resolution. It's a kind of mix between
pip and virtualenv. Follow the next instructions to setup the development enviroment.First of all, install Poetry:$pipinstallpoetry$gitclonehttps://github.com/dany2691/vanguard-kit.git
$cdvanguard_kit
$poetryinstallTo run the test-suite, inside the pybundler directory:$poetryrunpytesttest/-vvMetaDaniel Omar Vergara Pérez –@__danvergara __–daniel.omar.vergara@gmail.com--github.com/danvergaraValery Briz -@valerybriz--github.com/valerybrizContributingFork it (https://github.com/BentoBox-Project/vanguard-kit)Create your feature branch (git checkout -b feature/fooBar)Commit your changes (git commit -am 'Add some fooBar')Push to the branch (git push origin feature/fooBar)Create a new Pull Request