Package: warc2graph
warc2graphWarc2graph extracts a graph data structure from WARC files. The module was built to dig deeper into WARC files. It extracts (almost) all internal and external references from a WARC file by analyzing the WARC header and the payload. Multiple methods can be used for extraction, single or combined. Warc2graph has a CLI interface and can be used as a python module. The output when using the CLI interface consists of graph data in a standard graph XML format GEXF and several visualizations of that data using different visualization algorithms. We acknowledge that visualizations carry an epistemic value and thus need to be designed according to the analyzed objects and research questions. Warc2graph usesNetworkXas its graph data and analytics backend, so more involved graph data analytics can be realized when using warc2graph as a python module.The initial purpose of warc2graph was to analyze and visualize the textual structure of net literature works in theDLA corpus of net literature works and blogsdating from the early time of the web in the 1990s up to the 2000s. Development is part of theScience Data Center for Literatureresearch project. We now consider warc2graph as a tool for detailed WARC analytics regarding the referential structure of the archived sites and hope that it will be useful for the web archiving and web research community.Warc2graph is under active development.If you consider using warc2graph for a research project or in an archival context, please get in touch! We'd love to hear about your work.Warc2graph has been presented at the Electronic Literature Organization Conference 2020:| Overview and Video:https://elmcip.net/critical-writing/networks-net-literature-modelling-extracting-and-visualizing-link-based-networks| Conference Paper (PDF):https://elmcip.net/sites/default/files/media/critical_writing/attachments/claus-michael_schlesinger_mona_ulrich_pascal_hein_and_andre_blessing_networks_of_net_literature_-_modelling_extracting_and_visualizing_192.pdfInstallationwarc2graph requires Python >= 3.6.Use the package managerpipto install warc2graph.pipinstallwarc2graphAlternatively you can install manually using the python packagesetuptools.gitclonehttps://github.com/dla-marbach/warc2graph.gitcdwarc2graph
python3setup.pybuild
python3setup.pyinstall--userTo be able to use the dot algorithm to visualize the graph, make sure, to haveGraphVizinstalled.UsageYou can use the package in your python projects, or you can use the provided command line interface. While the former
offers more possibilities, the latter might be more intuitive.CLIThe installation of the package provides thewarc2graphcommand for your terminal. Callwarc2graph --helpto get an
overview over the available options.One warc fileIf you want to create a model for only one warc file simply callwarc2graphpath/to/warc.warc.gzIf the warc file is not on you file system, and you want it to be downloaded from the internet, you can pass an url. You
have to pass the parameterd.warc2graphurl/to/warc.warc.gzdList of warc filesIf you want to create a model using a list of warc files all together archiving one big website, first create a list of
all the warc files.lspath/to/warcs/*.warc.gz>>list_of_warcs.txtYou can also create the file manually, it should look as follows.path/to/warc1.warc.gz
path/to/warc2.warc.gz
path/to/warc3.warc.gz
path/to/warc4.warc.gzThen call warc2graph with the parameterwl, and the list as an input file.warc2graphlist_of_warcs.txtwlList of live webpagesYou can also model a website that is not archived. Create a plain text file containing the urls to all the webpages you
want to consider. This file should look as follows.url/to/webpage1.html
url/to/webpage2.htmThen call warc2graph with the parameterll, and the list as an input file.warc2graphlist_of_webpages.txtllFurther optionsmethods to usecreate visualisationblacklistPython packageYou can inspect theexamples.ipynbusingjupyter notebookfor some interactive examples.Our package relies heavily on thenetworkxpackage. Read its documentation for further
information about the possibilities and interfaces for the analysis of networkx graphs.Creating and plotting a modelimportwarc2graph# our packageimportmatplotlib.pyplotasplt# plot graphsimportnetworkxasnx# handle graphs# assign the path to a warc file to a variablewarc_path="tests/WEB-20210202165627638-00000-24143~clarin02~8443.warc.gz"# create a basic model with all resources as nodes and all links and embeddings as edgesbasic_model=warc2graph.create_graph(warc_path)# visualizing the graph using the graphviz "dot" algorithmfig,ax=plt.subplots(1,figsize=(8,4))pos=nx.drawing.nx_agraph.graphviz_layout(basic_model,prog="dot")nx.draw_networkx(basic_model,with_labels=False,pos=pos,ax=ax)plt.draw()Calculating different graph metricsimportwarc2graph# our packageimportnetworkxasnx# handle graphsfrompprintimportPrettyPrinter# print dicts nicelypp=PrettyPrinter()warc_path="tests/WEB-20210202165627638-00000-24143~clarin02~8443.warc.gz"basic_model=warc2graph.create_graph(warc_path)degree_centralities=nx.algorithms.centrality.degree_centrality(basic_model)pp.pprint(degree_centralities)Outputs:{'http://httpd.apache.org/':0.07692307692307693,'http://www.scientificlinux.org/':0.07692307692307693,'https://clarin09.ims.uni-stuttgart.de/':0.23076923076923078,'https://clarin09.ims.uni-stuttgart.de/icons/apache_pb2.gif':0.07692307692307693,'https://clarin09.ims.uni-stuttgart.de/sdc_warc/angular1.html':0.23076923076923078,'https://clarin09.ims.uni-stuttgart.de/sdc_warc/index.html':0.8461538461538463,'https://clarin09.ims.uni-stuttgart.de/sdc_warc/jquery.html':0.23076923076923078,'https://clarin09.ims.uni-stuttgart.de/sdc_warc/js/angular.min.js':0.07692307692307693,'https://clarin09.ims.uni-stuttgart.de/sdc_warc/js/jquery-1.11.3.min.js':0.07692307692307693,'https://clarin09.ims.uni-stuttgart.de/sdc_warc/page1.html':0.15384615384615385,'https://clarin09.ims.uni-stuttgart.de/sdc_warc/page2.html':0.15384615384615385,'https://clarin09.ims.uni-stuttgart.de/sdc_warc/page_target_ang1.html':0.07692307692307693,'https://clarin09.ims.uni-stuttgart.de/sdc_warc/page_target_jquery1.html':0.07692307692307693,'https://clarin09.ims.uni-stuttgart.de/sdc_warc/page_target_jquery2.html':0.07692307692307693}Enriched modelsYou can also enrich the models using the original data.importwarc2graph# our package# assign the path to a warc file to a variablewarc_path="tests/WEB-20210202165627638-00000-24143~clarin02~8443.warc.gz"# create an enriched model, structured like the basic model but containing the html content and counts of all tagsenriched_model=warc2graph.create_graph(warc_path,include_content=True,count_tags=True)index_node="https://clarin09.ims.uni-stuttgart.de/sdc_warc/index.html"print(enriched_model.nodes[index_node]["counted_tags"])# prints:# {'html': 1, 'head': 1, 'meta': 1, 'title': 1, 'body': 1, 'a': 4, 'br': 6}print(enriched_model.nodes[index_node]["content"])Prints:<!DOCTYPE html><html><head><metacharset="UTF-8"><title>Insert title here</title></head><body><ahref="page1.html">page1</a><br><br><ahref="page2.html">page2</a><br><br><ahref="angular1.html">angular1</a><br><br><ahref="jquery.html">jquery</a></body></html>ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.All contributed Code will be licensed under the GNU Lesser General Public License.By contributing you accept the following terms and conditions:You grant the rights for your contribution to be used, distributed and modified together with warc2graph and under
the same license.Your contribution consists of your work, no third party holds rights over it.You grant us the right to redistribute the software including your contribution under a different (permissive or
non-permissive) open source license.Licensewarc2graph is free software: you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.warc2graph is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU Lesser General Public License for more details.You should have received a copy of the GNU Lesser General Public License
along with warc2graph.  If not, seehttps://www.gnu.org/licenses/lgpl-3.0.html.ConsiderCOPYINGandCOPYING.LGPL.Acknowledgementwarc2graph makes heavy and critical use of following open source libraries:BeautifulSoupmatplotlibNetworkXSeleniumSetuptoolsWarciowebdriver-managertrafilatura