Package: tsv2py
Parse and generate tab-separated values (TSV) dataTab-separated values(TSV) is a simple and popular format for data storage, data transfer, exporting data from and importing data to relational databases. For example, PostgreSQLCOPYmoves data between PostgreSQL tables and standard file-system files or in-memory stores, and itstextformat (a text file with one line per table row) is a generic version of TSV. Meanwhile, packages likeasyncpghelp efficiently insert, update or query data in bulk with binary data transfer between Python and PostgreSQL.This package offers a high-performance alternative to convert data between a TSV text file and Python objects. The parser can read a TSV record into a Python tuple consisting of built-in Python types, one for each field. The generator can produce a TSV record from a tuple.Quick startfromtsv.helperimportParser# specify the column structureparser=Parser(fields=(bytes,date,datetime,float,int,str,UUID,bool))# read and parse an entire filewithopen(tsv_path,"rb")asf:py_records=parser.parse_file(f)# read and parse a file line by linewithopen(tsv_path,"rb")asf:forlineinf:py_record=parser.parse_line(line)TSV formatText format is a simple tabular format in which each record (table row) occupies a single line.Output always begins with a header row, which lists data field names.Fields (table columns) are delimited bytabcharacters.Non-printable characters and special values are escaped withbackslash(\), as shown below:EscapeInterpretation\NNULL value\0NUL character (ASCII 0)\bBackspace (ASCII 8)\fForm feed (ASCII 12)\nNewline (ASCII 10)\rCarriage return (ASCII 13)\tTab (ASCII 9)\vVertical tab (ASCII 11)\\Backslash (single character)This format allows data to be easily imported into a database engine, e.g. with PostgreSQLCOPY.Output in this format is transmitted as media typetext/plainortext/tab-separated-valuesin UTF-8 encoding.ParserThe parser understands the following Python types:None. This special value is returned for the TSV escape sequence\N.bool. A literaltrueorfalseis converted into a boolean value.bytes. TSV escape sequences are reversed before the data is passed to Python as abytesobject. NUL bytes are permitted.datetime. The input has to comply with RFC 3339 and ISO 8601. The timezone must be UTC (a.k.a. suffixZ).date. The input has to conform to the formatYYYY-MM-DD.time. The input has to conform to the formathh:mm:ssZwith no fractional seconds, orhh:mm:ss.ffffffZwith fractional seconds. Fractional seconds allow up to 6 digits of precision.float. Interpreted as double precision floating point numbers.int. Arbitrary-length integers are allowed.str. TSV escape sequences are reversed before the data is passed to Python as astr. NUL bytes are not allowed.uuid.UUID. The input has to comply with RFC 4122, or be a string of 32 hexadecimal digits.decimal.Decimal. Interpreted as arbitrary precision decimal numbers.ipaddress.IPv4Address.ipaddress.IPv6Address.listanddict, which are understood as JSON, and invoke the equivalent ofjson.loadsto parse a serialized JSON string.The backslash character\is both a TSV and a JSON escape sequence initiator. When JSON data is written to TSV, several backslash characters may be needed, e.g.\\nin a quoted JSON string translates to a single newline character. First,\\in\\nis understood as an escape sequence by the TSV parser to produce a single\character followed by anncharacter, and in turn\nis understood as a single newline embedded in a JSON string by the JSON parser. Specifically, you need four consecutive backslash characters in TSV to represent a single backslash in a JSON quoted string.Internally, the implementation uses AVX2 instructions toparse RFC 3339 date-time strings into Pythondatetimeobjects,parse RFC 4122 UUID strings or 32-digit hexadecimal strings into PythonUUIDobjects,and find\tdelimiters between fields in a line.For parsing integers up to the range of thelongtype, the parser calls the C standard library functionstrtol.For parsing IPv4 and IPv6 addresses, the parser calls the C functioninet_ptonin libc or Windows Sockets (WinSock2).If installed, the parser employsorjsonto improve parsing speed of nested JSON structures. If not available, the library falls back to thebuilt-in JSON decoder.Date-time formatYYYY-MM-DDThh:mm:ssZ
YYYY-MM-DDThh:mm:ss.fZ
YYYY-MM-DDThh:mm:ss.ffZ
YYYY-MM-DDThh:mm:ss.fffZ
YYYY-MM-DDThh:mm:ss.ffffZ
YYYY-MM-DDThh:mm:ss.fffffZ
YYYY-MM-DDThh:mm:ss.ffffffZDate formatYYYY-MM-DDTime formathh:mm:ssZ
hh:mm:ss.fZ
hh:mm:ss.ffZ
hh:mm:ss.fffZ
hh:mm:ss.ffffZ
hh:mm:ss.fffffZ
hh:mm:ss.ffffffZPerformanceDepending on the field types,tsv2pyis up to 7 times faster to parse TSV records than a functionally equivalent Python implementation based on the Python standard library. Savings in execution time are more substantial for dates, UUIDs and longer strings with special characters (up to 90% savings), and they are more moderate for simple types like small integers (approx. 60% savings).