Package: cassava-csv
cassavaCassava is a package for reading, plotting and quality-checking CSV files.  It's primary purpose is for giving a quick, first assessment of a CSV file, highlighting common quality issues such as wholly empty columns or rows, differing column counts and basic outlier detection.  The package can be integrated as part of a larger workflow, or used directly from the command line with a simple but functional command line interface (CLI).InstallThe package can be installed from PyPI (note the package distribution name):$pipinstallcassava-csvFrom the command lineThe cassava CLI runs in a number of modes.  The main commands areplot, to visually inspect a file, andprint, to print its findings tostdout.  For each of these commands, there are two subcommands;qcfor producing a QC plot or report, andstatsfor producing a summary statistics plot or report.  There are then many options to specify how to read and process the CSV file, e.g., whether it has a header row, which column to use for the x-axis in the plot, which columns to use for the y-axis in the plot etc.SynopsisThe general usage is to call the cassava package main, followed by any options, then the command (one ofplot,print), a subcommand (one ofqc,stats) and finally the input CSV file.  Note that the subcommand is optional and if not supplied, will default toqc.$python-mcassava[opts]command[subcommand]input.csvSpecifying the--helpoption, will print the CLI usage and quit.  To get help on a given command, specify the--helpoption after that command.  For example:$python-mcassavaplot--helpNote that the options are global to all modes (commands and subcommands), even when some only really make sense for a given mode.  This is by design, and to make for an uncomplicated CLI, which lends itself well to just pressing "up arrow" in the shell and using the same options but in a different mode.Options-h, --help            show this help message and exit
  -H HEADER_ROW, --header-row HEADER_ROW
                        row containing the header
  -i FIRST_DATA_ROW, --first-data-row FIRST_DATA_ROW
                        first row containing data to plot
  -C, --common-header-row
                        shorthand for -H 0 -i 1, as this is such a commonplace
                        configuration
  -x XCOL, --x-column XCOL
                        column containing values for the x-axis
  -y YCOL, --y-column YCOL
                        column containing values for the y-axis (specify
                        multiple columns separated by commas and/or as ranges
                        to plot multiple curves on y-axis)
  -d, --x-as-datetime   treat the x-axis values as datetimes
  -f DATETIME_FORMAT, --datetime-format DATETIME_FORMAT
                        datetime format specification
  -m MISSING_VALUE, --missing-value MISSING_VALUE
                        value to be treated as missing data
  -l DELIMITER, --delimiter DELIMITER
                        alternative delimiter
  -s, --skip-initial-space
                        ignore whitespace immediately following the delimiter
  -F, --forgive         be forgiving when parsing numeric data
  -N NCOLS, --plot-in-n-columns NCOLS
                        number of columns for a multi-plot grid
  -k K, --tukey-fence-factor K
                        factor to multiply IQR by in Tukey's rule
  -O, --hide-outliers   don't show outliers on stats plots
  -P PLOT_OPTS, --plot-options PLOT_OPTS
                        options for the plot, specified as a simple JSON
                        object
  -S, --scatter-plot    set plot options (see -P) to produce a scatter plot
  -v, --verbose         emit verbose messages
  -V, --version         show program's version number and exitExamplesGiven the following CSV file:Datetime,Temperature,Relative_Humidity,Sea_Level_Pressure,Wind_Speed,,,
1999-12-20T00:00:00,-10,40,990,,,,
1999-12-21T00:00:00,-11,51,971,,,,
1999-12-22T00:00:00,-10,62,952,17,,,
1999-12-23T00:00:00,-10,56,956,,,,
1999-12-24T00:00:00,-12,78,,,,,
1999-12-25T00:00:00,-11,77,995,,,,
1999-12-26T00:00:00,-11,70,986,,,,
1999-12-27T00:00:00,-12,47,966,22,,,
1999-12-28T00:00:00,-11,48,990,230,,,
1999-12-29T00:00:00,-11,57,967,25,,,
,,,,,,,
,,,,,,,
,,,,,,,
,,,,,
,,,,,,,,,,,we can see a number of issues.  It's likely that this was exported from a spreadsheet, as it has a number of trailing empty columns, and trailing empty rows.  In addition, there's been some subsequent "finger trouble", as the last two empty rows have differing column counts.  There are wholly empty columns, as well as some sparse, but not wholly empty columns, and at least one highly probable outlier.Often a quick plot will tell us a lot about our data, so let's try that.  If our CSV file contained only numeric columns, with no header, then we could simply do:$python-mcassavaplotdata.csv
Failedtoconvertcolumn0atrow0withfloat:['Datetime','Temperature','Relative_Humidity','Sea_Level_Pressure','Wind_Speed','','',''].Cause:couldnotconvertstringtofloat:'Datetime'however in this case, the default y-axis column (0) contains ISO datetime strings, and the first row is a header row, so cassava will raise the above exception.Note that by default, cassava will return focused, colour-coded exception messages, with the data context where appropriate.  To see the full, chained exceptions including the tracebacks (as per normal Python exception reporting), run cassava in verbose mode (-v).At this point, we could replace theplotcommand withprint, and get some useful QC information about the file, as theprintcommand doesn't need to interpret the data as much as theplotcommand does.A key feature of cassava is that you can run it in "forgive" mode (-F), where it will simply replace any invalid numeric values with a placeholder value (the default is NaN).  By default the forgive mode isFalse, otherwise it would mask quality issues with the data, and so defeat the central point of cassava!  However, in cases where we just want to get a first look at the data, the forgive mode is invaluable.  So let's do that:$python-mcassava-Fplotdata.csvThis produces a plot, but it's empty!  This is because the default y-axis column contains datetime strings, and so the forgive mode has skipped over all of them.  Let's specify the y-axis data (-y) as column 1:$python-mcassava-F-y1plotdata.csv
Nohandleswithlabelsfoundtoputinlegend.That works, and produces a line plot of theTemperaturedata.  This is a minimal working command line for plotting this file, but we can do better!Let's provide a few more options for processing.  First we can address the notification thatmatplotlibemitted about no labels for the legend.  We can tell cassava that the first row (row 0 - all cassava coordinates have origin zero) is a header row (-H 0) and the first data row is the next row (-i 1).  Having these as two separate options gives us flexibility for cases where the CSV file may have a complex structured header section.  However, as this is such a commonplace header configuration, there is a shorthand option (-C), which is identical to-H 0 -i 1.  Furthermore, we note that this is a timeseries, so we can provide options to cassava so that it can treat it as such.  We specify the x-axis data as column 0 (-x 0), and tell cassava to treat the x-axis as a datetime column (-d).  The datetime format is ISO 8601, which is the cassava default, so we don't need to specify the datetime format.  Trying this also raises an exception:$python-mcassava-H0-i1-x0-d-y1plotqcdata.csv
Failedtoconvertcolumn0atrow11withstrptime:['','','','','','','',''].Cause:timedata''doesnotmatchformat'%Y-%m-%dT%H:%M:%S'This is valuable QC information, as it tells us exactly the row that failed to parse as a datetime (again, we can run in verbose mode to see the chained tracebacks).  Particularly useful if the file is large.  At this point we could address the issue directly, by editing the CSV file accordingly (in this example, by removing the empty trailing rows), or just run cassava without specifying the x-axis data (cassava then defaults to integer indices).  Let's do the latter:$python-mcassava-H0-i1-y1plotqcdata.csv
Failedtoconvertcolumn1atrow11withfloat:['','','','','','','',''].Cause:couldnotconvertstringtofloat:''More valuable QC information!  The empty rows at the bottom of the file cause this exception.  Let's reintroduce the forgive option.  This allows us to get on with evaluating the data, but of course eventually, we will remove those empty rows.$python-mcassava-H0-i1-y1-Fplotqcdata.csvThis gives us a nice working command line.  Now let's plot all the numeric columns.  We can specify multiple columns for the y-axis by giving a comma-separated list and/or an inclusive range - the following are all equivalent:$python-mcassava-H0-i1-y1,2,3,4-Fplotqcdata.csv
$python-mcassava-H0-i1-y1,2,3-4-Fplotqcdata.csv
$python-mcassava-H0-i1-y1,2-4-Fplotqcdata.csv
$python-mcassava-H0-i1-y1-4-Fplotqcdata.csvThis works, but as theSea_Level_Pressurevalues are far greater than the other columns, it's not easy to pick out the detail.  We could drop theSea_Level_Pressurecolumn from the y-axis list (-y 1,2,4).  This is an improvement, but the outlier inWind_Speedis now causing problems.  In cases where your data are of greatly differing scales, it's better to plot multiple curves on separate plots.  This can be achieved using the-N NCOLSoption, which tells cassava to plot a grid of NCOLS-wide plots:$python-mcassava-H0-i1-y1,2,3,4-F-N2plotqcdata.csvThis plots a 2x2 grid of plots, with each variable in its own plot, with a suitably-scaled y-axis.Another case where you may find large values are obscuring the detail, is where missing values in the data have been specified by a value that is outside of the data domain - for example-999.  In such a case, we can tell cassava to treat these values as missing data and replace them with NaN.  This will then mean that they are not shown when plotting the data:$python-mcassava-H0-i1-y1,2,3,4-F-m-999plotqcdata.csvIf we have isolated points that are separated by NaN values (either direct NaN values or converted missing values), then the default plot type (a line plot) may not show these points, as matplotlib cannot draw a line segment between them.  In such a case, we can ask cassava to produce a scatter plot (-S) instead of a line plot:$python-mcassava-H0-i1-y1,2,3,4-F-m-999-Splotqcdata.csvThis scatter-plot option is actually a convenience option for the more general purpose plot-options option (-P), which takes a valid set of matplotlib plot options as a simple JSON object, and then converts it into a Pythondict.  Note that as it's a JSON object, you must use double-quotes for the keys and any string values.  For example, the scatter-plot option above can equivalently be specified as:$python-mcassava-H0-i1-y1,2,3,4-F-m-999-P'{"marker": ".", "ls": ""}'plotqcdata.csvFor any of the working command lines above, we could replace theqcsubcommand with thestatssubcommand, to get summary statistics plots for the specified y-columns.  Let's do that for the 2x2 grid command line.  The first thing to note, is that the-N 2option is not required for stats plots.  However, as noted above, it doesn't hurt to leave it there and thus allows for rapid tweak/repeat cycles:$python-mcassava-H0-i1-y1,2,3,4-F-N2plotstatsdata.csvThis produces three plots for each specified y-column: a density plot of the distribution of the data, a line plot of the data including bounds to highlight potential outliers, and a boxplot of the data.  In this example, the outlier in theWind_Speedis clearly identified.  If an outlier is so large that it dominates the remaining data, then we can instruct cassava to not show outliers (-O), thereby revealing the detail:$python-mcassava-H0-i1-y1,2,3,4-F-N2-Oplotstatsdata.csvSimilarly, we can replace theplotcommand with theprintcommand.  Let's do that (again, no need to remove extraneous options):$python-mcassava-H0-i1-y1,2,3,4-F-N2-Oprintqcdata.csvColumncounts:firstrow1:ncols=8row14:ncols=6row15:ncols=12Rowcounts:totalrows=16,datarows=15Emptycolumns:column5isemptycolumn6isemptycolumn7isempty
Emptyrows:row11isemptyrow12isemptyrow13isemptyrow14isemptyrow15isemptywhich produces the above QC report.  The output is colour-coded using a traffic light system, thereby highlighting quality issues.  For theColumn countssection, only those rows which have differing column counts to the first data row are listed, so ideally in good data, you would only see the column count of the first data row.  Running the above in verbose mode (-v) would list all column counts, irrespective of whether they agree with the first data row or not.We can also print summary statistics for the specified columns, and list any cells that contain suspected outlier values:$python-mcassava-H0-i1-y1,2,3,4-F-N2-Oprintstatsdata.csvColumnstats:columnminmeanmaxq1medianq3std1-12-11-10-11-11-100.724059784956681239.5e+029.7e+021e+039.7e+029.7e+029.9e+0215417742.3e+0221247690Columnoutliers(1.5*IQR):column,rowvalue4,92.3e+02Using the packageTo use cassava in your own code, setup a configurationdict, and then call the required methods from within theCassavacontext manager:fromcassavaimportCassavafilename='data.csv'conf=Cassava.DEFAULTS.copy()opts={'header_row':0,'first_data_row':1,'ycol':[1,2,3,4],'forgive':True}conf.update(opts)withCassava(path=filename,conf=conf)asf:f.read()f.plot()Configuration dictAs can be seen above, cassava requires a fully-specified configurationdict, so the easiest way to ensure this is to take a copy of theCassavaclassDEFAULTS dict, and then override any specific configuration items.  The default configurationdictis:DEFAULTS={'header_row':None,'first_data_row':0,'xcol':None,'ycol':[0],'x_as_datetime':False,'datetime_format':'%Y-%m-%dT%H:%M:%S','missing_value':None,'delimiter':',','skip_initial_space':False,'forgive':False,'verbose':False}header_row: Integer row index of the input file's headerfirst_data_row: Integer row index of the input file's first data rowxcol: Integer column index from the input file for the plot x-axisycol: Integer column indexlistfrom the input file for the plot y-axisx_as_datetime: Consider the x-axis data as datetime stringsdatetime_format:datetime.datetime.strptime()format specificationmissing_value: Value in the data that indicates a missing datum (e.g. -999)delimiter: Column delimiter characterskip_initial_space: Skip any spaces following the delimiter characterforgive: Forgive mode. Replace invalid numeric values with placeholder (NaN)verbose: Print extra messages inprintmode methodsNote that all cassava column/row coordinates have origin zero.Persist custom default configuration for a sessionA feature of theDEFAULTS dictis that, because it's a class variable (rather than an instance variable), you can set perimportsession defaults for all instances by directly updatingDEFAULTS:fromcassavaimportCassavafilenames=['data1.csv','data2.csv']opts={'header_row':0,'first_data_row':1}Cassava.DEFAULTS.update(opts)forfilenameinfilenames:withCassava(path=filename)asf:f.read()f.plot()Note that if no configurationdictis passed via theconfkwarg, then cassava usesDEFAULTSinstead.The cassava objectTheCassavaobject has the following instance variables:self.path=pathself.conf=conforself.DEFAULTSself.fp=Noneself.header_row=[]self.rows=[]pathandconfare the input file pathstrand the configurationdictthat are (optionally) passed to the constructor, as shown in the above examples.fpis the file pointer for the input file.header_row(alist) holds the (optional) header row, parsed from the input data, androws(alistoflists) contains the data rows parsed from the input data.Reading input dataThe easiest way to get data into cassava, is by using it as a context manager:withCassava(path=filename,conf=conf)asf:f.read()f.plot_stats()however, you can use theopen/closemethods directly, should you wish:c=Cassava(conf=conf)c.open(path=filename)c.read()c.print_qc()c.close()As noted in the previous section, theheader_rowandrowsattributes hold the header row and data rows parsed from the input data.  Hence you could also directly setrows(and optionallyheader_row), rather than reading them from a file using thereadmethod, and still be able to make use of the methods that cassava provides.Reading input data with different delimitersAlthough by default, cassava is setup to read CSV data, it can actually read any similarly-delimited tabular data.  This is controlled by thedelimiterconfiguration item.  For instance a space (conf['delimiter'] = ' ') or a tab (conf['delimiter'] = '\t').  Note that if the columns are separated by multiple spaces (e.g. a fixed width format), then settingconf['skip_initial_space'] = Truewill consume all spaces between the columns.Analysing the dataOnce we have the data in cassava, we can produce quick-look plots, generate QC reports and compute summary statistics.Plot the dataTo give an initial exploratory look at the data and assess any QC issues, we can simply plot our dependent variables (ourycolcolumns) by callingplot():f.conf['ycol']=[1,2,3,4]f.plot()Our input file may contain an independent variable (e.g. datetime) which we can use for ourxcolcolumn:f.conf['xcol']=0f.conf['ycol']=[1,2,3,4]f.plot()If our dependent variables contain data with differing scales, we can choose to plot them as separate subplots, rather than all on a single plot with a shared y-axis.  We do this by providing alayouttuple specifying anmrows byncolumns plot grid (wherem * n > 1):f.conf['ycol']=[1,2,3,4]f.plot(layout=(2,2))Often we don't care how many rows the plot grid requires, rather just how many columns wide the grid should be, so we can use thecompute_multi_plot_layoutmethod to compute ourlayoutfor us, given our desired number of columns:f.conf['ycol']=[1,2,3,4]f.plot(layout=f.compute_multi_plot_layout(2))which will produce a 2x2 plot grid.To allow some customisation of the plot, we can use theoptskwarg.  This takes adict, and can contain any plot kwargs accepted bymatplotlib:f.plot(opts={'lw':4,'c':'green','ls':'--'})In addition, we can defer the showing of the plot via theshowkwarg.  This allows us to do further customisation of the plot before showing it.  Note that to do this, we have toimport matplotliband callshow()ourselves:importmatplotlib.pyplotaspltfig,axs=f.plot(show=False)fig.suptitle(f'Meteorological Data -{filename}')plt.show()Plot summary statistics for the dataPlotting summary statistics is straightforward:f.conf['ycol']=[1,2,3,4]f.plot_stats()This produces three plots for each specified y-column: a density plot of the distribution of the data, a line plot of the data including bounds to highlight potential outliers, and a boxplot of the data.Note that the bounds in the line plot, and the whiskers in the boxplot, are scaled according to Tukey's rule and are 1.5 * IQR above/below the IQR.  We can change this scale factor via thekkwarg:f.plot_stats(k=2.5)If we have outliers that are so large that they dominate the remaining data, then we can choose to omit them, thereby revealing the detail:f.plot_stats(showfliers=False)By default, the number of bins to use for the density plot is automatically calculated bymatplotlib.  We can specify these explicitly via thebinskwarg, which takes any form supported bymatplotlib.pyplot.hist:f.plot_stats(bins=10)Print a QC report of the dataTo print a QC report of the data tostdout, we can do:filename='data.csv'withCassava(path=filename)asf:f.read()f.print_qc()which outputs the following (using our example data from above):Columncounts:firstrow0:ncols=8row14:ncols=6row15:ncols=12Rowcounts:totalrows=16,datarows=16Emptycolumns:column5isemptycolumn6isemptycolumn7isempty
Emptyrows:row11isemptyrow12isemptyrow13isemptyrow14isemptyrow15isemptyNote that this really requires no configuration, but assumes that our input file has no header row (and sototal rows==data rows), or more generally, that the data begin on the first row (row 0).  If the file does contain a header row, or an extended header section, then we should specify the first data row index for completeness, and so generate a more accurate report:f.conf['first_data_row']=1f.print_qc()Theprint_qcmethod is a wrapper around the methods that produce the different output sections in the above output.  Of course, we can call these directly to produce a custom report.  For example:f.print_column_counts()f.print_empty_columns()Print a summary statistics report of the dataHere we require a little more configuration than for a QC report.  We need to specify those columns that we wish to compute summary statistics for, and, taking our example data above, we'd need to run cassava in forgive mode to skip over our trailing empty rows:f.conf['first_data_row']=1f.conf['ycol']=[1,4]f.conf['forgive']=Truef.print_stats()which outputs the following (again using our example data from above):Columnstats:columnminmeanmaxq1medianq3std1-12-11-10-11-11-100.7417742.3e+0221247690Columnoutliers(1.5*IQR):column,rowvalue4,92.3e+02Here we can see that cassava has identified the value in column 4 (Wind_Speed) and row 9 as an outlier, according to Tukey's rule.Access the underlying QC and summary statistics dataProducing the plots and printing the reports is fine, but for tighter integration, we can access the underlyingmessage dictthat encapsulates the QC and statistics information.Message dictThe general form of themessage dictis:msg={'x':x_coordinate,'y':y_coordinate,'data':data_dict,'status':status_code}x: Integer column index of the input data (Noneindicates all columns)y: Integer row index of the input data (Noneindicates all rows)data: Adictof the output, specific to each methodstatus: One of the possible status constants fromclass CassavaStatusNote that all cassava column/row coordinates have origin zero.For example, thecheck_column_countsmethod calculates the column count of each row in the input data.  In particular, it calculates and holds the column count of the configured first data row, and then compares the column counts of all subsequent rows to that of the first, and flags up if they differ.  As this is a per-row check, the x-coordinate isNonebecause it refers to all columns for that row.  The column count for the first data row will always havestatus = CassavaStatus.ok:msg={'x':None,'y':y,'data':{'is_first_row':True,'ncols':len(row)},'status':CassavaStatus.ok}Subsequent rows will haveis_first_row = False, andstatus = CassavaStatus.okif their column counts agree with that of the first data row, otherwise they will havestatus = CassavaStatus.error.  (It is thestatusattribute that is used for colour-coding the output in the print functions.)The following describe themessage['data'] dictfor each method:check_column_counts: {'is_first_row': Boolean, 'ncols': Integer column count}check_empty_columns: {'is_empty': Boolean}check_empty_rows: {'is_empty': Boolean}compute_column_stats: {'min': Minimum, 'mean': Mean, 'max': Maximum, 'q1': Quartile1, 'median': Quartile2, 'q3': Quartile3, 'std': Standard deviation}check_column_outliers_iqr: {'value': Cell value}ExamplesAs an example, say we wanted to compute the percentage of empty columns and rows in our example file.  We could do:fromcassavaimportCassavafilename='data.csv'withCassava(path=filename)asf:f.read()ncols=len(f.rows[f.conf['first_data_row']])# Uncomment to only consider data rows, rather than total rows# f.conf['first_data_row'] = 1# nrows = len(f.rows) - f.conf['first_data_row']nrows=len(f.rows)empty_ncols=0empty_nrows=0formsginf.check_empty_columns():ifmsg['data']['is_empty']:empty_ncols+=1formsginf.check_empty_rows():ifmsg['data']['is_empty']:empty_nrows+=1print(f'Empty cols ={empty_ncols/ncols*100:0.2f}%')print(f'Empty rows ={empty_nrows/nrows*100:0.2f}%')To print the mean of each configured column, we could do:filename='data.csv'conf=Cassava.DEFAULTS.copy()opts={'header_row':0,'first_data_row':1,'ycol':[1,2,3,4],'forgive':True}conf.update(opts)withCassava(path=filename,conf=conf)asf:f.read()labels=f.get_column_labels_from_header(f.conf['ycol'])formsg,labelinzip(f.compute_column_stats(),labels):print(f'{label}mean ={msg["data"]["mean"]:0.2f}')