Package: tablite
TabliteContentsintroductioninstallationfeature overviewtutoriallatest updatescreditsIntroductionTabliteseeks to be the go-to library for manipulating tabular data with an api that is as close in syntax to pure python as possible.Even smaller memory footprintTablite usesnumpys fileformatas a backend with strong abstraction, so that copy, append & repetition of data is handled in pages. This is imperative forincremental data processing.Tablite testsfor memory footprint. One test compares the memory footprint of 10,000,000 integers wheretablitewill use < 1 Mb RAM in contrast to python which will require around 133.7 Mb of RAM (1M lists with 10 integers). Tablite also tests to assure that working with1Tb of datais tolerable.Tablite achieves this minimal memory footprint by using a temporary storage set inconfig.Config.workdirastempfile.gettempdir()/tablite-tmp.
If your OS (windows/linux/mac) sits on a SSD this will benefit from high IOPS and permit slices of9,000,000,000 rows in less than a second.Multiprocessing enabled by defaultTablite uses numpy whereever possible and applies multiprocessing for bypassing the GIL on all major operations.
CSV import is performed in C through usingnims compiler and is as fast the hardware allows.All algorithms have been reworked to respect memory limitsTablite respects the limits of free memory by tagging the free memory and defining task size before each memory intensive task is initiated (join, groupby, data import, etc).
If you still run out of memory you may try to reduce theconfig.Config.PAGE_SIZEand rerun your program.100% support for all python datatypesTablite wants to make it easy for you to work with data.tablite.Table'sbehave like a dict with lists:my_table[column name] = [... data ...].Tablite uses datatype mapping to native numpy types where possible and uses type mapping for non-native types such as timedelta, None, date, time… e.g. what you put in, is what you get out. This is inspired bybank python.Light weightTablite is ~200 kB.HelpfulTablite wants you to be productive, so a number of helpers are available.Table.import_fileto import csv*, tsv, txt, xls, xlsx, xlsm, ods, zip and logs. There is automatic type detection (seetutorial.ipynb)To peek into any supported file useget_headerswhich shows the first 10 rows.Usemytable.rowsandmytable.columnsto iterate over rows or columns.Create multi-key.indexfor quick lookups.Perform multi-key.sort,Filter using.anyand.allto select specific rows.use multi-key.lookupand.jointo find data across tables.Perform.groupbyand reorganise data as a.pivottable with max, min, sum, first, last, count, unique, average, st.deviation, median and modeAppend / concatenate tables with+=which automatically sorts out the columns - even if they're not in perfect order.Should you tables be similar but not the identical you can use.stackto "stack" tables on top of each otherIf you're still missing something add it to thewishlistInstallationGet it from pypi:Install:pip install tabliteUsage:>>> from tablite import TableBuild & testinstall nim >= 2.0.0run:chmod +x ./build_nim.shrun:./build_nim.shShould the default nim not be your desired taste, please usenimsenvironment manager (atlas) and runsource nim-2.0.0/activate.shon UNIX ornim-2.0.0/activate.baton windows.install python >= 3.8
python -m venv /your/venv/dir
activate /your/venv/dir
pip install -r requirements.txt
pip install -r requirements_for_testing.py
pytest ./testsFeature overviewwant to...this way...loop over rows[ row for row in table.rows ]loop over columns[ table[col_name] for col_name in table.columns ]slicemyslice = table['A', 'B', slice(0,None,15)]get column by namemy_table['A']get row by indexmy_table[9_000_000_001]value updatemytable['A'][2] = new valueupdate w. list comprehensionmytable['A'] = [ x*x for x in mytable['A'] if x % 2 != 0 ]joina_join = numbers.join(letters, left_keys=['colour'], right_keys=['color'], left_columns=['number'], right_columns=['letter'], kind='left')lookuptravel_plan = friends.lookup(bustable, (DataTypes.time(21, 10), "<=", 'time'), ('stop', "==", 'stop'))groupbygroup_by = table.groupby(keys=['C', 'B'], functions=[('A', gb.count)])pivot tablemy_pivot = t.pivot(rows=['C'], columns=['A'], functions=[('B', gb.sum), ('B', gb.count)], values_as_rows=False)indexindices = old_table.index(*old_table.columns)sortlookup1_sorted = lookup_1.sort(**{'time': True, 'name':False, "sort_mode":'unix'})filtertrue, false = unfiltered.filter( [{"column1": 'a', "criteria":">=", 'value2':3}, ... more criteria ... ], filter_type='all' )find anyany_even_rows = mytable.any('A': lambda x : x%2==0, 'B': lambda x > 0)find allall_even_rows = mytable.all('A': lambda x : x%2==0, 'B': lambda x > 0)to jsonjson_str = my_table.to_json()from jsonTable.from_json(json_str)TutorialTo learn more see thetutorial.ipynb(Jupyter notebook)Latest updatesSeechangelog.mdCreditsMartynas Kaunas - GroupBy functionality.Audrius Kulikajevas - Edge case testing / various bugs, Jupyter notebook integration.Sergej Sinkarenko - various bugs.Ovidijus Grigas - various bugs, documentation.Lori Cooper - spell checking.