Package: warcit
warcitis a command-line tool to convert on-disk directories of web documents (commonly HTML, web assets and any other data files) into an ISO standard web archive (WARC) files.Conversion to WARC file allows for improved durability in a standardized format, and allows for any web files stored on disk to be uploaded intoWebrecorder, or replayed locally withWebrecorder Playerorpywb(Many other tools also operate on WARC files, see:Awesome Web Archiving – Tools and Software)WARCIT supports converting individual files, directories (including any nested directories) as well as ZIP files into WARCs.Basic Usagewarcit <prefix> <dir or file> ...Seewarcit-hfor a complete list of flags and options.For example, the following example will download a simple web site viawget(for simplicity, this retrieves one level deep only), then usewarcitto convert towww.iana.org.warc.gz:wget -l 1 -r www.iana.org/
warcit http://www.iana.org/ ./www.iana.org/The WARCwww.iana.org.warc.gzshould now have been created!Mime Type Detection and OverridesBy default,warcitsupports the the default Pythonmimetypeslibrary to determine a mime type based on a file extension.However, it also supports usingpython-magic(libmagic) if available and custom mime overrides configured via the command line.The mime detection is as follows:If the filename matches an override specified via--mime-overrides, use that as the mime type.Ifmimetypes.guess_type()returns a valid mime type, use that as the mime type.If--use-magicflag is specified, use themagicapi to determine mime type (warcitwill error ifmagicis not available when using this flag).Default totext/htmlif all previous attempts did not yield a mime type.The--mime-overridesflag can be used to specify wildcard query (applied to the full url) and corresponding mime types as a comma-delimeted property list:warcit '--mime-overrides=*.html=text/html; charset="utf-8",image.ico=image/png' http://www.iana.org/ ./www.iana.org/When a url ending in*.htmlor*.icois encountered, the specified mime type will be used for theContent-Typeheader, by passing any auto-detection.Charset DetectionCharset detection is disabled by default, but can be enabled with the--charsetautoflag.Detection is done using thecchardetnative chardet library.A specific charset can also be specified, eg.--charsetutf-8will add;charset=utf-8to alltext/*resources.If detection does not produce a result, or if the result isascii, no charset is added to theContent-Type.ZIP Fileswarcitalso supports converting ZIP files to WARCs, including portions of zip files.For example, if a zip file contains:my_zip_file.zip
|
+-- www.example.com/
|
+-- another.example.com/
|
+-- some_other_data/It is possible to specify the two paths in the zip file to be converted to a WARC separately:warcit --name my-warc.gz http:// my_zip_file.zip/www.example.com/ my_zip_file.zip/another.example.com/This should result in a new WARCmy-warc.gzconverting the specified zip file paths. Thesome_other_datapath is not processed.WARC Structure and FormatThe tool produces ISO standard WARC 1.0 files.Awarcinforecord is added at the beginning of the WARC, unless the--no-warcinfoflag is specified.The warcinfo record contains the full command line and warcit version:WARC/1.0
WARC-Type: warcinfo
WARC-Record-ID: ...
WARC-Filename: example.com.warc.gz
WARC-Date: 2017-12-05T18:30:58Z
Content-Type: application/warc-fields
Content-Length: ...

software: warcit 0.2.0
format: WARC File Format 1.0
cmdline: warcit --fixed-dt 2011-02 http://example.com/ ./path/to/somefile.htmlEach file specified or found in the directory is stored as a WARCresourcerecord.By default, warcit uses the file-modified date as theWARC-Dateof each url.
This setting can be overriden with a fixed date time by specifying the--fixed-dtflag.
The datetime can be specified as--fixed-dtYYYY-MM-DDTHH:mm:ssor--fixed-dtYYYYMMDDHHmmssor partial date,
eg.--fixed-dtYYYY-MMThe actual WARC creation time and path to the source file on disk are also stored, using theWARC-Creation-DateandWARC-Source-URIextension headers, respectively.For example, if when runningwarcit--fixed-dt2011-02http://example.com/./path/to/somefile.html, the resulting WARC Record might look as follows:WARC/1.0
WARC-Date: 2011-02-01T00:00:00Z
WARC-Creation-Date: 2017-12-05T18:30:58Z
WARC-Source-URI: file://./path/to/somefile.html
WARC-Type: resource
WARC-Record-ID: ...
WARC-Target-URI: http://www.example.com/to/somefile.html
Content-Type: text/html
Content-Length ...

...Additionally, warcit addsrevisitrecords for top-level directories if index files are present.
Index files can be specified via the--index-filesflag, the default being--index-files=index.html,index.htmFor example, when running:warcithttp://example.com/./path/and there exists a file:./path/subdir/index.html, warcit will create:aresourcerecord forhttp://example.com/path/subdir/index.htmlarevisitrecord forhttp://example.com/path/subdir/pointing tohttp://example.com/path/subdir/index.htmlWARC Video Conersions and Embeds ManifestWith warcit 0.4.0, warcit also includeswarcit-converterand the ability to
useffmpegto generate video/audio conversions, store them as conversion records and generate a manifest.SeeWARCIT Media Conversions and Transclusions<conversions-and-transclusions.md>`_ for more details on how to convert video/audio, create WARC
records and metadata to support replay of converted media.