Package: pyfsdb
ObjectiveTheFSDB"flat-file streaming database" is a structured data file
that includes column names, formatting specifications (e.g. tab vs
space vs comma), and a command history that generated each file.
PyFSDB is a a python implementation of the original functionality that
was implemented in perl.  Both the perl and python version come with a
long list ofcommand line toolsthat can be used to quickly process
datasets using traditional unix pipeline processing.  There is also aC implementationand a Go implementation (ref needed) of FSDB.Getting started documentation is below, but also see thefull
documentationover on readthedocs.InstallationUsing pip (or pipx):pip3 install pyfsdbOr manually:git clone git@github.com:gawseed/pyfsdb.git
cd pyfsdb
python3 setup.py build
python3 setup.py installExample UsageThe FSDB file format contains headers and footers that supplement the
data within a file.  The most common separator is tab-separated, but
can wrap CSVs and other datatypes (see the FSDB documentation for full
details).  The file also contains footers that trace all the piped
commands that were used to create a file, thus documenting the history
of its creation within the metadata in the file.Example pyfsdb code for reading a fileReading in row by row:import pyfsdb
db = pyfsdb.Fsdb("myfile.fsdb")
print(db.column_names)
for row in db:
    print(row)Example FSDB file#fsdb -F t col1 two andthree
1	key1	42.0
2	key2	123.0Example writing to an FSDB formatted file.import pyfsdb
db = pyfsdb.Fsdb(out_file="myfile.fsdb")
db.out_column_names=('one', 'two')
db.append([4, 'hello world'])
db.close()Read below for further usage details.Installationpip3 install pyfsdbAdditional Usage DetailsThe real power of the FSDB comes from the build up of tool-suites that
all interchange FSDB formatted files.  This allows chaining multiple
commands together to achieve a goal.  Though the original base set of
tools are in perl, you don't need to know perl for most of them.Let's create a ./mydemo.py script:importsys,pyfsdbdb=pyfsdb.Fsdb(file_handle=sys.stdin,out_file_handle=sys.stdout)value_column=db.get_column_number('value')forrowindb:# reads a row from the input streamrow[value_column]=float(row[value_column])*2db.append(row)# sends the row to the output streamdb.close()And then feed it this file:#fsdb -F t col1 value
1	42.0
2	123.0We can run it thus'ly:# cat test.fsdb | ./mydemo.py#fsdb -F t col1 value184.02246.0#   | ./mydemo.pyOr chain it together with multiple FSDB commands:# cat test.fsdb | ./mydemo | dbcolstats value | dbcol mean stddev sum min max | dbfilealter -R C
#fsdb -R C mean stddev sum min max
mean: 165
stddev: 114.55
sum: 330
min: 84
max: 246
#   | ./mydemo.py
#   | dbcolstats value
#   | dbcol mean stddev sum min max
#   | dbfilealter -R CCommand line tools includedAll the command line utilities that come withpyfsdbstart withpby convention so as not to conflict with the utilities from perl
package.  The leadingpalso serves to distinguish the CLI argument
differences as well (e.g. the python versions allow file names to be
specified on the command line, and most keys must be passed with a-kflag).Data processing toolspdbrow: select rows based on logic criteriapdbroweval: modify rows based on python codepdbtopn: given a key and a value column, print the top N rows with
unique keys and the highest values.pdbaugment: a fast way to merge two fsdb files, where one is stored
entirely in memory for speed.  Unlike other tools, this does not
sort the data for speed purposes.pdbcoluniq: find all unique values of a key column, optionally with
counting.  Requires no sorting (unlike dbrowuniq) at the cost of
greater memory usage.pdbzerofill: fills a column with zeros if the value is otherwise blankpdbkeyedsort: sorts a potentially large file that is already
"mostly" sorted by performing a double-pass on reading it.  This
will be less and less efficient the more random the rows are in
order.pdbfullpivot: description TBDpdbreescape: converts a column full of data to regex quoted for
safetypdbensure:pdbcdf: performs cdf analysis on a columnConversion toolsbro2fsdb: converts azeek/brolog into an fsdbjson2fsdb: converts a json file to fsdbfsdb2json: converts an fsdb file to jsonpdb2tex: converts a fsdb file to a latex tablepdbformat: generically formats each row according to a python column
specifierpdbsplitter: splits a FSDB file into multiple sub-files based on a
column setpdbdatetoepoch: converts columns from a date string to an integer
epoch columnpdbepochtodate: formats a unix epoch seconds date to human readablepdbnormalize: normalizes a column to a limited rangepdbsum: tbdpdbj2: formats results based on a jinja2 templatepdb2sql: converts a fsdb file into an sqlite3 databasegraphical utilitiespdbheatmap: creates a heat map based on incoming data columnspdbroc: creates a ROC graph for incoming fsdb dataAuthorWes Hardaker @ USC/ISISee alsoThe FSDB website and manual page for the original perl module:https://www.isi.edu/~johnh/SOFTWARE/FSDB/