Package: metawarc
metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)metawarc (pronouncedme-ta-warc) is a command line WARC files processing tools.Its goal is to make CLI interaction with files inside WARC archives so easy as possible.It provides a simplemetawarccommand that allows to extract metadata from images, documents and other files insideWARC archives.Main featuresBuilt-in WARC supportMetadata extraction for a lot of file formatsLow memory footprintDocumentationTest coverageFile formats supportedMS Office OLE: .doc, .xls, .pptMS Office XML: .docx, .xlsx, .pptxAdobe PDF: .pdfImages: .png, .jpg, .tiff, .jpeg, .jp2InstallationAny OSA universal installation method (that works on Windows, Mac OS X, Linux, …,and always provides the latest version) is to use pip:# Make sure we have an up-to-date version of pip and setuptools:$pipinstall--upgradepipsetuptools$pipinstall--upgrademetawarc(Ifpipinstallation fails for some reason, you can tryeasy_install metawarcas a fallback.)Python versionPython version 3.6 or greater is required.UsageSynopsis:$metawarc[command][flags]inputfileSee alsometawarc--helpandmetawarc [command]--helpfor help for each command.ExamplesExtract metadata of all supported file types from ‘digital.gov.ru.warc.gz’ and output results to default filename ‘metadata.jsonl’:$metawarcmetadatadigital.gov.ru.warc.gzExtract metadata for .doc and .docx file types from ‘digital.gov.ru.warc.gz’ and output results to default filename ‘metadata.jsonl’:$metawarcmetadata--filetypesdoc,docxdigital.gov.ru.warc.gzExtract metadata for .doc and .docx file types from ‘digital.gov.ru.warc.gz’ and output results to filename ‘digital_meta.jsonl’:$metawarcmetadata--filetypesdoc,docx--outputdigital_meta.jsonldigital.gov.ru.warc.gzCommandsMetadata commandExtracts metadata from files inside .warc files. Returns JSON lines output for each file found.Extract metadata for .doc and .docx file types from ‘digital.gov.ru.warc.gz’ and output results to filename ‘digital_meta.jsonl’:$metawarcmetadata--filetypesdoc,docx--outputdigital_meta.jsonldigital.gov.ru.warc.gzAnalyze commandReturns list of mime mimetypes with stats as number of files and total files size for each mime type.Will be merged or replaced by ‘stats’ command that uses sqlite db to speed up data processingAnalyzes ‘digital.gov.ru.warc.gz’ and output results of list of mime types as table to console$metawarcanalyzedigital.gov.ru.warc.gzIndex commandGenerates ‘metawarc.db’ SQLite database with records HTTP metadata. Requred for ‘stats’ command to calculate stats quicklyAnalyzes ‘digital.gov.ru.warc.gz’ and writes ‘metawarc.db’ with HTTP metadata.$metawarcindexdigital.gov.ru.warc.gzIndex commandSame as ‘analyze’ command but uses ‘metawarc.db’ to speed up data processing. Returns total length and count of records by each mime or file extension.Processes data in ‘metawarc.db’ and prints total length and count for each mime$metawarcstats-mmimesProcesses data in ‘metawarc.db’ and prints total length and count for each file extension$metawarcstats-mextsExport commandExtracts HTTP headers, WARC headers or text content from WARC file and saves as NDJSON (JSON lines) data file.Exports http headers from ‘digital.gov.ru.warc.gz’ and writes as ‘headers.jsonl’$metawarcexport-theaders-oheaders.jsonldigital.gov.ru.warc.gzExports WarcIO index from ‘digital.gov.ru.warc.gz’ and writes as ‘data.jsonl’ with fields listed in ‘-f’ option.$metawarcexport-twarcio-foffset,length,filename,http:status,http:content-type,warc-type,warc-target-uri-odata.jsonldigital.gov.ru.warc.gzExports text (HTML) content from ‘digital.gov.ru.warc.gz’ and writes as ‘content.jsonl’$metawarcexport-tcontent-ocontent.jsonldigital.gov.ru.warc.gz