Package: xcsv
xcsvxcsv is a package for reading and writing extended CSV files.Extended CSV formatExtended header section of parseable atttributes, introduced by '#'.Header row of variable and units for each column.Data rows.ExampleExtended header sectionNo leading/trailing whitespace.Each line introduced by a comment ('#') character.Each line contains a single header item.Key/value separator ': '.Multi-line values naturally continued over to the next lines following the line introducing the key.Continuation lines that contain the delimiter character in the value must be escaped by a leading delimiter.Preferably use a common vocabulary for attribute name, such asCF conventions.Preferably include recommended attributes fromAttribute Convention for Data Discovery (ACDD).Preferably use units fromUnified Code for Units of Measureand/orUdunits.Units in parentheses.Certain special keys are used tofurther process the data, for example themissing_valuekey.# id: 1
# title: The title
# summary: This dataset...
# The second summary paragraph.
# : The third summary paragraph.  Escaped because it contains the delimiter in a URL https://dummy.domain
# authors: A B, C D
# latitude: -73.86 (degree_north)
# longitude: -65.46 (degree_east)
# elevation: 1897 (m a.s.l.)
# [a]: 2012 not a complete yearHeader rowNo leading/trailing whitespace.Preferably use a common vocabulary for variable name, such asCF conventions.Units in parentheses.Optional notes in square brackets, that reference an item in the extended header section.time (year) [a],depth (m)Data rowNo leading/trailing whitespace.2012,0.575Automated post-processing of the dataDepending on the presence of special keys in the extended header section, these will be used to automatically post-process the data.  To turn off this automatic behaviour, either remove or rename these keys, or setparse_metadata=Falsewhen reading in the data.missing_value:  This is used to define those values in the data that are to be considered as missing values.  This is typically a value that is outside the domain of the data such as-999.99, or can be a symbolic value such asNA.  All such values appearing in the data will be masked, appearing as anNAvalue to pandas (i.e.pd.isna(value)returnsTrue).  Note that pandas itself will automatically do this for certain values regardless of this key, such as for the stringsNaNorNA, or the constantNone.InstallThe package can be installed from PyPI:$pipinstallxcsvUsing the packageThe package has a generalXCSVclass, that has ametadataattribute that holds the parsed contents of the extended file header section and the parsed column headers from the data table, and adataattribute that holds the data table (including the column headers as-is).Themetadataattribute is adict, with the following general structure:{'header':{},'column_headers':{}}and thedataattribute is apandas.DataFrame, and so has all the features of thepandaspackage.The package also has aReaderclass for reading an extended CSV file into anXCSVobject, and similarly aWriterclass for writing anXCSVobject to a file in the extended CSV format.  In addition there is aFileclass that provides a convenient context manager for reading and writing these files.ExamplesSimple read and printRead in a file and print the contents tostdout.  This shows how the contents of the extended CSV file are stored in theXCSVobject.  Note how multi-line values, such assummaryhere, are stored in a list.  Given the following script called, say,simple_read.py:importargparseimportxcsvparser=argparse.ArgumentParser()parser.add_argument('filename',help='filename.csv')args=parser.parse_args()withxcsv.File(args.filename)asf:content=f.read()print(content.metadata)print(content.data)Running it would produce:$python3simple_read.pyexample.csv{'header':{'id':'1','title':'The title','summary':['This dataset...','The second summary paragraph.','The third summary paragraph.  Escaped because it contains the delimiter in a URL https://dummy.domain'],'authors':'A B, C D','latitude':{'value':'-73.86','units':'degree_north'},'longitude':{'value':'-65.46','units':'degree_east'},'elevation':{'value':'1897','units':'m a.s.l.'},'[a]':'2012 not a complete year'},'column_headers':{'time (year) [a]':{'name':'time','units':'year','notes':'a'},'depth (m)':{'name':'depth','units':'m','notes':None}}}time(year)[a]depth(m)020120.575120111.125220102.225Simple read and print with missing valuesIf the above example header section included the following:# missing_value: -999.99and the data section looked like:time (year) [a],depth (m)
2012,0.575
2011,1.125
2010,2.225
2009,-999
2008,999
2007,-999.99
2006,999.99
2005,NA
2004,NaNRunning it would produce:$python3simple_read.pymissing_example.csv{'header':{'id':'1','title':'The title','summary':['This dataset...','The second summary paragraph.','The third summary paragraph.  Escaped because it contains the delimiter in a URL https://dummy.domain'],'authors':'A B, C D','latitude':{'value':'-73.86','units':'degree_north'},'longitude':{'value':'-65.46','units':'degree_east'},'elevation':{'value':'1897','units':'m a.s.l.'},'missing_value':'-999.99','[a]':'2012 not a complete year'},'column_headers':{'time (year) [a]':{'name':'time','units':'year','notes':'a'},'depth (m)':{'name':'depth','units':'m','notes':None}}}time(year)[a]depth(m)020120.575120111.125220102.22532009-999.00042008999.00052007NaN62006999.99072005NaN82004NaNNote that the-999.99value has been automatically masked as a missing value (shown asNaNin the printed pandasDataFrame), as well as theNAandNaNstrings in the original data, which pandas automatically masks itself, irrespective of themissing_valueheader item.Simple read and plotRead a file and plot the data:importargparseimportmatplotlib.pyplotaspltimportxcsvparser=argparse.ArgumentParser()parser.add_argument('filename',help='filename.csv')args=parser.parse_args()withxcsv.File(args.filename)asf:content=f.read()content.data.plot(x='depth (m)',y='time (year) [a]')plt.show()Simple read and writeRead a file in, manipulate the data in some way, and write this modifiedXCSVobject out to a new file:importargparseimportxcsvparser=argparse.ArgumentParser()parser.add_argument('in_filename',help='in_filename.csv')parser.add_argument('out_filename',help='out_filename.csv')args=parser.parse_args()withxcsv.File(args.in_filename)asf:content=f.read()# Manipulate the data...withxcsv.File(args.out_filename,mode='w')asf:f.write(xcsv=content)