Package: undatum
undatum (pronouncedun-da-tum) is a command line data processing tool.
Its goal is to make CLI interaction with huge datasets so easy as possible.
It provides a simpleundatumcommand that allows to convert, split, calculate frequency, statistics and to validate
data in CSV, JSON lines, BSON files.Contents1Main features2Installation2.1macOS2.2Linux2.3Windows, etc.2.4Python version3Usage3.1Examples4Commands4.1Frequency command4.2Uniq command4.3Convert command4.4Validate command4.5Headers command4.6Stats command4.7Analyze command4.8Split command4.9Select command4.10Flatten command5Advanced5.1Filtering5.2Data containers5.3Date detection6Data types1Main featuresCommon data operations against CSV, JSON lines and BSON filesBuilt-in data filteringSupport data compressed with ZIP, XZ, GZ, BZ2Conversion between CSV, JSONl, BSON, XML, XLS, XLSX, Parquet file typesLow memory footprintSupport for compressed datasetsAdvanced statistics calculationsDate/datetime fields automatic recognitionData validationDocumentationTest coverage2Installation2.1macOSOn macOS, undatum can be installed viaHomebrew(recommended):$brewinstallundatumA MacPortsportis also available:$portinstallundatum2.2LinuxMost Linux distributions provide a package that can be installed using the
system package manager, for example:# Debian, Ubuntu, etc.$aptinstallundatum# Fedora$dnfinstallundatum# CentOS, RHEL, ...$yuminstallundatum# Arch Linux$pacman-Sundatum2.3Windows, etc.A universal installation method (that works on Windows, Mac OS X, Linux, вЂ¦,
and always provides the latest version) is to use pip:# Make sure we have an up-to-date version of pip and setuptools:$pipinstall--upgradepipsetuptools$pipinstall--upgradeundatum(Ifpipinstallation fails for some reason, you can tryeasy_install undatumas a fallback.)2.4Python versionPython version 3.6 or greater is required.3UsageSynopsis:$undatum[flags][command]inputfileSee alsoundatum--help.3.1ExamplesGet headers from file asheaders command,JSONldata:$undatumheadersexamples/ausgovdir.jsonlAnalyze file and generate statisticsstats command:$undatumstatsexamples/ausgovdir.jsonlGetfrequency commandof values for field GovSystem in the list of Russian federal government domains fromgovdomains repository$undatumfrequencyexamples/feddomains.csv--fieldsGovSystemGet all unique values usinguniq commandof theitem.typefield$undatumuniq--fieldsitem.typeexamples/ausgovdir.jsonlconvert commandfrom XML to JSON lines file on tagitem:$undatumconvert--tagnameitemexamples/ausgovdir.xmlexamples/ausgovdir.jsonlValidate data withvalidate commandagainst validation ruleru.org.innand fieldVendorINNin  data file. Output is statistcs only :$undatumvalidate-rru.org.inn--modestats--fieldsVendorINNexamples/roszdravvendors_final.jsonl>inn_stats.jsonValidate data withvalidate commandagainst validation ruleru.org.innand fieldVendorINNin  data file. Output all invalid records :$undatumvalidate-rru.org.inn--modeinvalid--fieldsVendorINNexamples/roszdravvendors_final.jsonl>inn_invalid.json4Commands4.1Frequency commandField value frequency calculator. Returns frequency table for certain field.
This command autodetects delimiter and encoding of CSV files and encoding of JSON lines files by default. You may override it providng “-d” delimiter and “-e” encoding parameters.Get frequencies of values for fieldGovSystemin the list of Russian federal government domains$undatumfrequencyexamples/feddomains.csv--fieldsGovSystem4.2Uniq commandReturns all unique files of certain field(s). Accepts parameterfieldswith comma separated fields to gets it unique values.
Provide single field name to get unique values of this field or provide list of fields to get combined unique values.
This command autodetects delimiter and encoding of CSV files and encoding of JSON lines files by default. You may override it providng “-d” delimiter and “-e” encoding parametersReturns all unique values of fieldregionsin selected JSONl file$undatumuniq--fieldsregionexamples/reestrgp_final.jsonlReturns all unique combinations of fieldsstatusandregionsin selected JSONl file$undatumuniq--fieldsstatus,regionexamples/reestrgp_final.jsonl4.3Convert commandConverts data from one format to another. Supports most common data files
Supports conversions:XML to JSON linesCSV to JSON linesXLS to JSON linesXLSX to JSON linesXLS to CSVCSV to BSONXLS to BSONJSON lines to CSVCSV to ParquetJSON lines to ParquetConversion between XML and JSON lines require flagtagnamewith name of tag which should be converted into single JSON record.Converts XML ausgovdir.xml with tag nameditemto ausgovdir.jsonl$undatumconvert--tagnameitemexamples/ausgovdir.xmlexamples/ausgovdir.jsonlConverts JSON lines file roszdravvendors_final.jsonl to CSV file roszdravvendors_final.csv$undatumconvertexamples/roszdravvendors_final.jsonlexamples/roszdravvendors_final.csvConverts CSV file feddomains.csv to Parquet file feddomains.parquet$undatumconvertexamples/feddomains.csvexamples/feddomains.parquet4.4Validate commandValidatecommand used to check every value of of field against validation rules like rule to validate email or url.Current supported rules:common.email- checks if value is emailcommon.url- checks if value is urlru.org.inn- checks if value is russian organization INN identifierru.org.ogrn- checks if value if russian organization OGRN identifierValidate data withvalidate commandagainst validation ruleru.org.innand fieldVendorINNin  data file. Output all invalid records :$undatumvalidate-rru.org.inn--modeinvalid--fieldsVendorINNexamples/roszdravvendors_final.jsonl>inn_invalid.json4.5Headers commandReturns fieldnames of the file. Supports CSV, JSON, BSON file types.
For CSV file it takes first line of the file and for JSON lines and BSON files it processes number of records provided aslimitparameter with default value 10000.
This command autodetects delimiter and encoding of CSV files and encoding of JSON lines files by default. You may override it providng “-d” delimiter and “-e” encoding parametersReturns headers of JSON lines file with top 10 000 records (default value)$undatumheadersexamples/ausgovdir.jsonlReturns headers of JSON lines file using top 50 000 records$undatumheaders--limit50000examples/ausgovdir.jsonl4.6Stats commandCollects statistics about data in dataset. Supports BSON, CSV an JSON lines file types.Returns table with following data:key- name of the keyftype- data type of the values with this keyis_dictkey- if True, than this key is identified as dictionary valueis_uniq- if True, identified as unique fieldn_uniq- number of unique valuesshare_uniq- share of unique values among all valuesminlen- minimal length of the fieldmaxlen- maximum length of the fieldavglen- average length of the fieldReturns stats for JSON lines file$undatumstatsexamples/ausgovdir.jsonlAnalysis of JSON lines file and verifies each field that it’s date field, detects date format:$undatumstats--checkdatesexamples/ausgovdir.jsonl4.7Analyze commandAnalyzes data format and provides human-readable information.$undatumanalyzeexamples/ausgovdir.jsonlReturned values will include:Filename - name of the fileFile type - type of the file, could be: jsonl, xml, csv, json, bsonEncoding - file encodingDelimiter - file delimiter if CSV fileFile size - size of the file, bytesObjects count - number of objects in fileFields - list of file fieldsAlso for XML AND JSON files:Miltiple tables exists - True or False, if multiple data tables in XML filesFull data key - full path to data key (field with list of objects) in XML fileShort data key - final name of field with objects in XML fileFor JSON files: JSON type - could be “objects list”, “objects list with key” and “single object”
For XML, JSON lines and JSON files: Is flat table? - True if table is flat and could be converted to CSV, False if not convertable
For CSV and JSON lines: Number of lines - number of lines in file4.8Split commandSplits dataset into number of datasets based on number of records or field value.
Chunksize parameter-cused to set size of chunk if dataset should be splitted by chunk size rule.
If dataset should be splitted by field value than–fieldsparameter used.Split dataset as 10000 records chunks, procuces files like filename_1.jsonl, filename_2.jsonl wherefilenameis name of original file except extension.$undatumsplit-c10000examples/ausgovdir.jsonlSplit dataset as number of files based of fielditem.type”, generates files filename_value1.jsonl, filename_value2.jsonl and e.t.c.
There are *[filename]- ausgovdir and[value1]- certain unique value fromitem.typefield$undatumsplit--fieldsitem.typeexamples/ausgovdir.jsonl4.9Select commandSelect or re-order columns from file. Supports CSV, JSON lines, BSONReturns columnsitem.titleanditem.typefrom ausgovdir.jsonl$undatumselect--fieldsitem.title,item.typeexamples/ausgovdir.jsonlReturns columnsitem.titleanditem.typefrom ausgovdir.jsonl and stores result as selected.jsonl$undatumselect--fieldsitem.title,item.type-oselected.jsonlexamples/ausgovdir.jsonl4.10Flatten commandFlatten data records. Write them as one value per rowReturns all columns as flattened key,value$undatumflattenexamples/ausgovdir.jsonl5Advanced5.1FilteringYou could filter values of any file record by usingfilterattr for any command where it’s suported.Returns columns item.title and item.type filtered withitem.typevalue asrole. Note: keys should be surrounded by “`” and text values by “’”.$undatumselect--fieldsitem.title,item.type--filter"`item.type` == 'role'"examples/ausgovdir.jsonl5.2Data containersSometimes, to keep keep memory usage as low as possible to process huge data files.
These files are inside compressed containers like .zip, .gz, .bz2 or .tar.gz files.undatumcould process compressed files with little memory footprint, but it could slow down file processing.Returns headers from subs_dump_1.jsonl file inside subs_dump_1.zip file. Require parameter–format-into force input file type.$undatumheaders--format-injsonlsubs_dump_1.zipExtracts unique values of the field countryCode from XZ compressed file data.jsonl.xz. Require parameter–format-into force input file type.$undatumuniq-fcountryCode--format-injsonldata.jsonl.xz5.3Date detectionJSON, JSON lines and CSV files do not support date and datetime data types.
If you manually prepare your data, than you could define datetime in JSON schema for example.B
But if data is external, you need to identify these fields.undatum supports date identification viaqddatepython library with automatic date detection abilities.$undatumstats--checkdatesexamples/ausgovdir.jsonl6Data typesJSONlJSON lines is a replacement to CSV and JSON files, with JSON flexibility and ability to process data line by line, without loading everything into memory.