Package: ipwb
InterPlanetary Wayback (ipwb)Peer-To-Peer Permanence of Web ArchivesInterPlanetary Wayback (ipwb) facilitates permanence and collaboration in web archives by disseminating the contents ofWARCfiles into the IPFS network.IPFSis a peer-to-peer content-addressable file system that inherently allows deduplication and facilitates opt-in replication. ipwb splits the header and payload of WARC response records before disseminating into IPFS to leverage the deduplication, builds aCDXJ indexwith references to the IPFS hashes returned, and combines the header and payload from IPFS at the time of replay.InterPlanetary Wayback primarily consists of two scripts:ipwb/indexer.py- archival indexing script that takes the path to a WARC input, extracts the HTTP headers, HTTP payload (response body), and relevant parts of the WARC-response record header from the WARC specified and creates byte string representations. The indexer then pushes the byte strings into IPFS using a locally running IPFS daemon then creates aCDXJfile with this metadata for replay.py.ipwb/replay.py- rudimentary replay script to resolve requests for archival content contained in IPFS for replay in the browser.A pictorial representation of the ipwb indexing and replay process:An important aspect of archival replay systems is rewriting various resource references for proper memento reconstruction so that they are dereferenced properly from the archive from around the same datetime as of the root memento and not from the live site (in which case the resource might have changed or gone missing). Many archival replay systems perform server-side rewriting, but it has its limitations when URIs are generated using JavaScript. To handle this we useService Workerfor rerouting requests on the client-side when they are dereferenced to avoid any server-side rewiring. For this, we have implemented a separate library,Reconstructive, which is reusable and extendable by any archival replay system.Another important feature of archival replays is the inclusion of an archival banner in mementos. The purpose of an archival banner is to highlight that a replayed page is a memento and not a live page, to provide metadata about the memento and the archive, and to facilitate additional interactivity. Many archival banners used in different web archival replay systems are obtrusive in nature and have issues like style leakage. To eliminate both of these issues we have implemented aCustom HTML Element,as part of theReconstructivelibrary and used in the ipwb.InstallingInterPlanetary Wayback (ipwb) requires Python 3.8+. ipwb can also be used with Docker (see below).For conventional usage, the latest release of ipwb can be installed using pip:$ pip install ipwbThe latest development version containing changes not yet released can be installed from source:$ git clone https://github.com/oduwsdl/ipwb
$ cd ipwb
$ pip install ./SetupThe InterPlanetary File System (ipfs) daemon must be installed and running before starting ipwb. See theInstall IPFSpage to accomplish this. In the future, we hope to make this more automated. Once ipfs is installed, start the daemon:$ ipfs daemonIf you encounter a conflict with the default API port of 5001 when starting the daemon, running the following prior to launching the daemon will change the API port to access to one of your choosing (here, shown to be 5002):$ ipfs config Addresses.API /ip4/127.0.0.1/tcp/5002IndexingIn a separate terminal session (or the same if you started the daemon in the background), instruct ipwb to push contents of a WARC file into IPFS and create an index of records:$ ipwb index (path to warc or warc.gz)...for example, from the root of the ipwb repository:$ ipwb index samples/warcs/salam-home.warcThe ipwb indexer partitions the WARC into WARC Records and extracts the WARC Response headers, HTTP response headers, and the HTTP response bodies (payloads). Relevant information is extracted from the WARC Response headers, temporary byte strings are created for the HTTP response headers and payload, and these two bytes strings are pushed into IPFS. The resulting CDXJ data is written toSTDOUTby default but can be redirected to a file, e.g.,$ ipwb index (path to warc or warc.gz) >> myArchiveIndex.cdxjReplayingAn archival replay system is also included with ipwb to re-experience the content disseminated to IPFS. A CDXJ index needs to be provided and used by the ipwb replay system by specifying the path of the index file as a parameter to the replay system:$ ipwb replay <path/to/cdxj>ipwb also supports using an IPFS hash or any HTTP location as the source of the CDXJ:$ ipwb replay http://myDomain/files/myIndex.cdxj
$ ipwb replay QmYwAPJzv5CZsnANOTaREALhashYgPpHdWEz79ojWnPbdGOnce started, the replay system's web interface can be accessed through a web browser, e.g.,http://localhost:2016/by default.To run it under a domain name other thanlocalhost, the easiest approach is to use a reverse proxy that supports HTTPS. The replay system utilizesService Workerfor URL rerouting/rewriting to preventlive leakage (zombies). However, for security reason many web browsers have mandated HTTPS for the Service Worker API with only exception if the domain islocalhost.Caddy ServerandTraefikcan be used as a reverse-proxy server and are very easy to setup. They come with built-in HTTPS support and manage (install and update) TLS certificates transparently and automatically fromLet's Encrypt. However, any web server proxy that has HTTPS support on the front-end will work. To make ipwb replay aware of the proxy, use--proxyor-Pflag to supply the proxy URL. This way the replay will yield the supplied proxy URL as a prefix when generating various fully qualified domain name (FQDN) URIs or absolute URIs (for example, those in the TimeMap or Link header) instead of the defaulthttp://localhost:2016. This can be necessary when the service is running in a private network or a container and only exposed via a reverse-proxy. Suppose a reverse-proxy server is running and ready to forward all traffic on thehttps://ipwb.example.comto the ipwb replay server then the replay can be started as following:$ ipwb replay --proxy=https://ipwb.example.com <path/to/cdxj>Using DockerA pre-built Docker image is made available that can be run as following:$ docker container run -it --rm -p 2016:2016 oduwsdl/ipwbThe container will run an IPFS daemon, index a sample WARC file, and replay it using the newly created index. It will take a few seconds to be ready, then the replay will be accessible athttp://localhost:2016/with a sample archived page.To index and replay your own WARC file, bind mount your data folders inside the container using-v(or--volume) flag and run commands accordingly. The provided docker image has designated/datadirectory, inside which there arewarc,cdxj, andipfsfolders where host folders can be mounted separately or as a single mount point at the parent/datadirectory. Assuming that the host machine has a/path/to/datafolder under which there arewarc,cdxj, andipfsfolders and a WARC file at/path/to/data/warc/custom.warc.gz.$ docker container run -it --rm -v /path/to/data:/data oduwsdl/ipwb ipwb index -o /data/cdxj/custom.cdxj /data/warc/custom.warc.gz
$ docker container run -it --rm -v /path/to/data:/data -p 2016:2016 oduwsdl/ipwb ipwb replay /data/cdxj/custom.cdxjIf the host folder structure is something other than/some/path/{warc,cdxj,ipfs}then these volumes need to be mounted separately.To build an image from the source, run the following command from the directory where the source code is checked out. The name of the locally built image could be anything, but we useoduwsdl/ipwbto be consistent with the above commands.$ docker image build -t oduwsdl/ipwb .By default, the image building process also performs tests, so it might take a while to build the image. It ensures that an image will not be created with failing tests. However, it is possible to skip tests by supplying a build-arg--build-arg SKIPTEST=trueas illustrated below:$ docker image build --build-arg SKIPTEST=true -t oduwsdl/ipwb .HelpUsage of sub-commands in ipwb can be accessed through providing the-hor--helpflag, like any of the below.$ ipwb -h
usage: ipwb [-h] [-d DAEMON_ADDRESS] [-v] [-u] {index,replay} ...

InterPlanetary Wayback (ipwb)

optional arguments:
  -h, --help            show this help message and exit
  -d DAEMON_ADDRESS, --daemon DAEMON_ADDRESS
                        Multi-address of IPFS daemon (default
                        /dns/localhost/tcp/5001/http)
  -v, --version         Report the version of ipwb
  -u, --update-check    Check whether an updated version of ipwb is available

ipwb commands:
  Invoke using "ipwb <command>", e.g., ipwb replay <cdxjFile>

  {index,replay}
    index               Index a WARC file for replay in ipwb
    replay              Start the ipwb replay system$ ipwb index -h
usage: ipwb [-h] [-e] [-c] [--compressFirst] [-o OUTFILE] [--debug]
            index <warc_path> [index <warc_path> ...]

Index a WARC file for replay in ipwb

positional arguments:
  index <warc_path>      Path to a WARC[.gz] file

optional arguments:
  -h, --help            show this help message and exit
  -e                    Encrypt WARC content prior to adding to IPFS
  -c                    Compress WARC content prior to adding to IPFS
  --compressFirst       Compress data before encryption, where applicable
  -o OUTFILE, --outfile OUTFILE
                        Path to an output CDXJ file, defaults to STDOUT
  --debug               Convenience flag to help with testing and debugging$ ipwb replay -h
usage: ipwb replay [-h] [-P [<host:port>]] [index]

Start the ipwb relay system

positional arguments:
  index                 path, URI, or multihash of file to use for replay

optional arguments:
  -h, --help            show this help message and exit
  -P [<host:port>], --proxy [<host:port>]
                        Proxy URLProject HistoryThis repo contains the code for integratingWARCs andIPFSas developed at theArchives Unleashed: Web Archive Hackathonin Toronto, Canada in March 2016. The project was also presented at:TheJoint Conference on Digital Libraries 2016in Newark, NJ in June 2016.TheWeb Archiving and Digital Libraries (WADL) 2016 workshopin Newark, NJ in June 2016.TheTheory and Practice on Digital Libraries (TPDL) 2016in Hannover, Germany in September 2016.TheArchives Unleashed 4.0: Web Archive Datathonin London, England in June 2017.TheInternational Internet Preservation Consortium (IIPC) Web Archiving Conference (WAC) 2017in London, England in June 2017.TheDecentralized Web Summit 2018'sIPFS Lab Day in San Francisco, CA in August 2018.Citing ProjectWe have numerous publications related to this project, but the most significant and primary one was published in TPDL 2016. (Read the PDF)Mat Kelly, Sawood Alam, Michael L. Nelson, and Michele C. Weigle.InterPlanetary Wayback: Peer-To-Peer Permanence of Web Archives. InProceedings of the 20th International Conference on Theory and Practice of Digital Libraries, pages 411–416, Hamburg, Germany, June 2016.@INPROCEEDINGS{ipwb-tpdl2016,AUTHOR={Mat Kelly andSawood Alam andMichael L. Nelson andMichele C. Weigle},TITLE={{InterPlanetary Wayback}: Peer-To-Peer Permanence of Web Archives},BOOKTITLE={Proceedings of the 20th International Conference on Theory and Practice of Digital Libraries},PAGES={411--416},MONTH={June},YEAR={2016},ADDRESS={Hamburg, Germany},DOI={10.1007/978-3-319-43997-6_35}}LicenseMIT