Package: warc2zim
warc2zimwarc2zim provides a way to convert WARC files to ZIM, storing the WARC payload and WARC+HTTP headers separately.Additionally, theReplayWeb.pageis also added to the ZIM, creating a self-contained ZIM
that can render its content in a modern browser.UsageExample:warc2zim ./path/to/myarchive.warc --output /output --name myarchive.zim -u https://example.com/The above will create a ZIM file/output/myarchive.zimwithhttps://example.com/set as the main page.Installationpython3-mvenv./env# creates a virtual python environment in ./env folder./env/bin/pipinstall-Upip# upgrade pip (package manager). recommended./env/bin/pipinstall-Uwarc2zim# install/upgrade warc2zim inside virtualenv# direct access to in-virtualenv warc2zim binary, without shell-attachment./env/bin/warc2zim--help# alternatively, attach virtualenv to shellsourceenv/bin/activate
warc2zim--help
deactivate# unloads virtualenv from shellURL FilteringBy default, all URLs found in the WARC files are included unless the--include-domains/-iflag is set.To filter URLs that may be out of scope (eg. ads, social media trackers), use the--include-domains/-iflag to specify each domain you want to include.Other URLs will be filtered and not pushed to the ZIM.Note that the domain passedand all its subdomainsare included.Eg. if main page is on a subdomainhttps://subdomain.example.com/but all URLs from*.example.comshould be included, use:warc2zim myarchive.warc --name myarchive -i example.com -u https://subdomain.example.com/starting/page.htmlIf main page is on a subdomain,https://subdomain.example.com/and only URLs fromsubdomain.example.comshould be included, use:warc2zim myarchive.warc --name myarchive -i subdomain.example.com -u https://subdomain.example.com/starting/page.htmlIf main page is on a subdomain,https://subdomain1.example.com/and only URLs fromsubdomain1.example.comandsubdomain2.example.comshould be included, use:warc2zim myarchive.warc --name myarchive -i subdomain1.example.com -i subdomain2.example.com -u https://subdomain1.example.com/starting/page.htmlCustom CSS--custom-cssallows passing an URL or a path to a CSS file that gets added to the ZIM and gets included onevery HTML articleat the very end of</head>(if it exists).Seewarc2zim -hfor other options.ZIM Entry LayoutThe WARC to ZIM conversion is performed by splitting the WARC (and HTTP) headers from the payload.Forresponserecords, the WARC + HTTP headers are stored underH/<url>while the payload is stored underA/<url>Forresourcerecords, the WARC headers are stored underH/<url>while the payload is stored underA/<url>. (Three are no HTTP headers for resource records).Forrevisitrecords, the WARC + optional HTTP headers are stored underH/<url>, while no payload record is created.If the payloadA/<url>is zero-length, the record is omitted to conform to ZIM specifications of not storing empty records.Duplicate URIsWARCs allow multiple records for the same URL, while ZIM does not. As a result, only the first encountered response or resource record is stored in the ZIM,
and subsequent records are ignored.For revisit records, they are only added if pointing to a different URL, and are processed after response/revisit records. A revisit record to the same URL
will always be ignored.All other WARC records are skipped.i18nwarc2zimhas very minimal non-content text but still uses gettext throughbabelto internationalize.To add a new locale (frin this example, use only ISO-639-1):init for your locale:python setup.py init_catalog -l frmake sure the POT is up to datepython setup.py extract_messagesupdate your locale's catalogpython setup.py update_catalogtranslate the PO file (poeditis your friend)compile updated translationpython setup.py compile_catalogLicenseGPLv3or later, seeLICENSEfor more details.