Package: hicsv-python
hicsv-pythonPython implementation of the I/O interface for the Header-Included Comma-Separated Values (hicsv) file format.DocumentationThe API documentation is available here.What ishicsv?Header-included comma-separated values (hicsvorhi-csvin short)
is a text file format consisting of a header (metadata)
and a tabulated values, both in a human-readable and editable form.CSV is a great universal format that can store a table of values
in a human-readable way.
Many instruments and softwares can export data in CSV.
We can then avoid dealing with the proprietary binary formats
whose specifications are often not disclosed to consumers.However, there is no commonly accepted method to store metadata
along with a table of values.
Metadata such as experimental conditions, author, etc.
are often essential in science.
People can easily lose track of these metadata when the CSV file
only contains the table part.There are some formats or conventions that (try to) treat a table
and metadata nicely (e.g. Tabular Data Packagehttps://specs.frictionlessdata.io/tabular-data-package/#introduction).
But, to the best of the author's knowledge, there is no text-based
generic format thatcombinesa table and metadata in a single file.It is very important to combine the metadata and the table
because it would be difficult to force everyone to
always keep two separate files together.Binary formats such as HDF5 can store metadata in addition to
image or array data. But again, there is no text-based,
human-readable format with embedded metadata.hicsvformat has been developed as a solution to the above problem.
It is basically just a CSV table with a JSON header part attached
at the top. The rough, tentative specification ofhicsvis as follows.It should be a Unicode text file.It starts with consequtive lines starting with#.The last one of the#-capped lines represents the "keys" of each column
in the table.The rest of the#-capped lines are an indented JSON which represents
the dictionary of metadata.The lines not starting with#are the CSV table.
The CSV part follows the generally accepted conventions
(mostly RFC 4180 but may not be strict)
except for following additional restrictions.The columns in the CSV table should always be of the same length.Each column must consist of only one type.Example ofhicsvformat#{
#    "some metadata entry": "some metadata",
#    "it can be anything": {
#        "that": "can be made into a JSON file",
#        "for example": "a dictionary"
#    },
#    "もちろん": "ユニコード文字もOK",
#    "hicsv-python version": "0.0.0",
#    "hicsv version": "20220812"
#}
#"first column"     ,"2nd column","column 3","四番目のカラム"
0.391006442374058   ,0           ,"string"  ,"いち"     
0.6943031206081407  ,1           ,"is"      ,"に"      
0.03433735164688634 ,2           ,"also"    ,"さん"     
0.015028457129899642,3           ,"OK"      ,"よん"     
0.5900201005940872  ,4           ,"!"       ,"ご"Scope of this packageThehicsvpackage is a minimal implementation of the I/O interface
to thehicsv-formatted files.
It, therefore, does not have (or aim for) any advanced functionalities.
Please usenumpyand/or other useful packages for anything beyond I/O.ExampleTo read a hicsv-formatted text file,importhicsvd=hicsv.hicsv("foo.txt")Or you can just create an empty object to which
you can add columns and header info.e=hicsv.hicsv()You can get one or more columns.c1,c2=d.ga("column 1","column 2")The header info is stored as a dictionary.
You can set and get information at any time::d.h["new header entry"]="some value"print(d.h["new header entry"])If you need to create a hicsv-formatted file from scratch::out=hicsv()To add a column::out.append_column("column 1",arr)Herearrshould be a 1-d numpy array.
Note that all columns must be of the same length,
so adding an array with different length will cause error.
Then you can write into a hicsv-formatted text file like::out.save("new hicsv file.txt")NoteSome users may find the following behavior of this package unexpected.The parser deletes the spaces at the beginning and at the end of each cell.
This is true even if the cell content is double-quoted.
For example, the two cells"spam", "  spam  "results in identical strings.