Package: typeddfs
Typed DataFramesPandas DataFrame subclasses that self-organize and serialize robustly.Film=TypedDfs.typed("Film").require("name","studio","year").build()df=Film.read_csv("file.csv")assertdf.columns.tolist()==["name","studio","year"]type(df)# FilmYour types remember how to be read,
including columns, dtypes, indices, and custom requirements.
No index_cols=, header=, set_index, or astype needed.Read and write any format:path=input("input file? [.csv/.tsv/.tab/.json/.xml.bz2/.feather/.snappy.h5/...]")df=Film.read_file(path)df.write_file("output.snappy")Need dataclasses?instances=df.to_dataclass_instances()Film.from_dataclass_instances(instances)Save metadata?df=df.set_attrs(dataset="piano")df.write_file("df.csv",attrs=True)df=Film.read_file("df.csv",attrs=True)print(df.attrs)# e.g. {"dataset": "piano")Make dirs? Don’t overwrite?df.write_file("df.csv",mkdirs=True,overwrite=False)Write / verify checksums?df.write_file("df.csv",file_hash=True)df=Film.read_file("df.csv",file_hash=True)# fails if wrongGet example datasets?print(ExampleDfs.penguins().df)#    species     island  bill_length_mm  ...  flipper_length_mm  body_mass_g     sex# 0    Adelie  Torgersen            39.1  ...              181.0       3750.0    MALEPretty-print the obvious way?df.pretty_print(to="all_data.md.zip")wiki_txt=df.pretty_print(fmt="mediawiki")All standard DataFrame methods remain available.
Use.of(df)to convert to your type, or.vanilla()for a plain DataFrame.Read the docs 📚for more info and examples.🐛 Pandas serialization bugs fixedPandas has several issues with serialization.See: Fixed issuesDepending on the format and columns, these issues occur:columns being silently added or dropped,errors on either read or write of empty DataFrames,the inability to use DataFrames with indices in Feather,writing to Parquet failing with half-precision,lingering partially written files on error,the buggy xlrd being preferred by read_excel,the buggy odfpy also being preferred,writing a file and reading it back results in a different DataFrame,you can’t write fixed-width format,and the platform text encoding being used rather than utf-8.invalid JSON is written via the built-in json library🎁 Other featuresSee more in theguided walkthrough ✏️See: Short feature listDtype-aware natural sortingUTF-8 by defaultNear-atomicity of read/writeMatrix-like typed dataframes and methods (e.g.matrix.is_symmetric())DataFrame-compatible frozen, hashable, ordered collections (dict, list, and set)Serialize JSON robustly, preserving NaN, inf, −inf, enums, timezones, complex numbers, etc.Serialize more formats like TOML and INIInterpreting paths and formats (e.g.FileFormat.split("dir/myfile.csv.gz").compression # gz)Generate good CLI help text for input DataFramesParse/verify/add/update/delete files in a .shasum-like file💔 LimitationsSee: List of limitationsMulti-level columns are not yet supported.Columns and index levels cannot share names.Duplicate column names are not supported. (These are strange anyway.)A typed DF cannot have columns "level_0", "index", or "Unnamed: 0".inplaceis forbidden in some functions; avoid it or use.vanilla().🔌 Serialization supportTypedDfs provides the methodsread_fileandwrite_file, which guess the format from the
filename extension. For example, this will convert a gzipped, tab-delimited file to Feather:TastyDf=typeddfs.typed("TastyDf").build()TastyDf.read_file("myfile.tab.gz").write_file("myfile.feather")Pandas does most of the serialization, but some formats require extra packages.
Typed-dfs specifiesextrasto help you get required packages and with compatible versions.Here are the extras:feather:Feather(uses: pyarrow)parquet:Parquet (e.g. .snappy)(uses: pyarrow)xml(uses: lxml)excel: Excel and LibreOffice .xlsx/.ods/.xls, etc. (uses: openpyxl, defusedxml)toml:TOML(uses: tomlkit)html(uses: html5lib, beautifulsoup4)xlsb: rare binary Excel file (uses: pyxlsb)HDF5{no extra provided}(use:tables)For example, for Feather and TOML support use:typeddfs[feather,toml]As a shorthand for all formats, usetypeddfs[all].📊 Serialization in-depthSee: Full tableformatpackagesextrasanityspeedfile sizesFeatherpyarrowfeather++++++++++Parquetpyarroworfastparquet†parquet+++++++++csv/tsvnonenone++−−−−flexwf ‡nonenone++−−−−.fwfnonenone+−−−−jsonnonenone−−−−−−−−xmllxmlxml−−−−−−−.propertiesnonenone−−−−−−tomltomlkittoml−−−−−−INInonenone−−−−−−−.linesnonenone++−−−−.npynonenone−++++.npznonenone−++++.htmlhtml5lib,beautifulsoup4html−−−−−−−−picklenonenone−−−−−−−−XLSXopenpyxl,defusedxmlexcel+−−+ODSopenpyxl,defusedxmlexcel+−−+XLSopenpyxl,defusedxmlexcel−−−−+XLSBpyxlsbxlsb−−−−++HDF5tableshdf5−−−++⚠ Note:Thehdf5extra is currently disabled.See: serialization notes†fastparquetcan be used instead. It is slower but much smaller.Parquet only supports str, float64, float32, int64, int32, and bool.
Other numeric types are automatically converted during write.‡.flexwfis fixed-width with optional delimiters.JSON has inconsistent handling ofNone. (orjsonis more consistent).XML requires Pandas 1.3+.Not all JSON, XML, TOML, and HDF5 files can be read..ini and .properties can only be written with exactly 2 columns + index levels:
a key and a value. INI keys are in the formsection.name..lines can only be written with exactly 1 column or index level..npy and .npz only serialize numpy objects.
They are not supported inread_fileandwrite_file..html is not supported inread_fileandwrite_file.Pickle is insecure and not recommended.Pandas supports odfpy for ODS and xlrd for XLS. In fact, it prefers those.
However, they are very buggy; openpyxl is much better.XLSM, XLTX, XLTM, XLS, and XLSB files can contain macros, which Microsoft Excel will ingest.XLS is a deprecated format.XLSB is not fully supported in Pandas.HDF may not work on all platforms yet due to atables issue.Feather offers massively better performance over CSV, gzipped CSV, and HDF5
in read speed, write speed, memory overhead, and compression ratios.
Parquet typically results in smaller file sizes than Feather at some cost in speed.
Feather is the preferred format for most cases.🔒 SecurityRefer to thesecurity policy.📝 Extra notesSee: Pinned versionsDependencies in the extras only have version minimums, not maximums.
For example, typed-dfs requires pyarrow >= 4.natsortis also only assigned a minimum version number.
This means that the result of typed-df’ssort_naturalcould change.
To fix this, pin natsort to a specific major version;
e.g.natsort = "^8"withPoetryornatsort>=8,<9with pip.🍁 ContributingTyped-Dfs is licensed under theApache License, version 2.0.New issuesand pull requests are welcome.
Please refer to thecontributing guide.
Generated withTyrannosaurus.