Package: csvinsight
csvinsightFast & simple summary for large CSV filesFree software: MIT licenseDocumentation:https://csvinsight.readthedocs.io.FeaturesCalculates basic stats for each column: max, min, mean length; number of non-empty valuesCalculates exact number of unique values and the top 20 most frequent valuesSupports non-orthogonal data (list fields)Works with very large files: does not load the entire CSV into memoryFast splitting of CSVs into columns, one file per columnMultiprocessing-enabledExample UsageGiven a CSV file:bash-3.2$ cat tests/sampledata.csv
name|age|fave_color
Alexey|33|red;yellow
Boris|31|blue
Valentina|0|you can obtain a CsvInsight report with:bash-3.2$ csvi tests/sampledata.csv --list-fields fave_color
CSV Insight Report
Total # Rows: 3
Column counts:
        3  columns ->  3 rows

Report Format:
Column Number. Column Header -> Uniques: # ; Fills: # ; Fill Rate:
Field Length: min #, max #, average:
 Top n field values -> Dupe Counts


1. name -> Uniques: 3 ; Fills: 3 ; Fill Rate: 100.0%
    Field Length:  min 5, max 9, avg 6.67
        Counts      Percent  Field Value
        1           33.33 %  Valentina
        1           33.33 %  Boris
        1           33.33 %  Alexey

2. age -> Uniques: 3 ; Fills: 3 ; Fill Rate: 100.0%
    Field Length:  min 1, max 2, avg 1.67
        Counts      Percent  Field Value
        1           33.33 %  33
        1           33.33 %  31
        1           33.33 %  0

3. fave_color -> Uniques: 4 ; Fills: 3 ; Fill Rate: 75.0%
    Field Length:  min 0, max 6, avg 3.25
        Counts      Percent  Field Value
        1           25.00 %  yellow
        1           25.00 %  red
        1           25.00 %  blue
        1           25.00 %  NULLSince CSV comes in different flavors, you may need to tweak the underlying CSV parser’s parameters to read your file successfully.
CSVInsight handles this via CSV dialects.
For example, to read a comma-separated file, you would use the following command:bash-3.2$ csvi your/file.csv --dialect delimiter=,You may combine as many dialect parameters as needed:bash-3.2$ csvi your/file.csv --dialect delimiter=, quoting=QUOTE_NONEFor a full list of dialect parameters, see the documentation for Python’scsv module.
Constant values like QUOTE_NONE are resolved automagically.Once you’ve discovered the winning parameter combination for your file, save it to a YAML file:list_fields:
  - fave_color
  - another_field_name
list_separator: ;
dialect:
  - "delimiter=|"
  - "quoting=QUOTE_NONE"You can then invoke CSVI as follows:bash-3.2$ csvi your/file.csv --config your/config.yamlCreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.HistoryUnreleased0.3.3 (2020-12-02)Handle numeric quoting parameter, e.g. “–dialect quoting=3”0.3.2 (2019-07-01)Set the field size limit to sys.maxsize0.3.1 (2019-06-26)Make Jupyter notebook an optional dependency0.3.0 (2018-07-11)Added –most-common parameter (resolved Issue #14)Added –no-tiny parameterRefactored temporary file namingImprove error message when handling empty CSV filesFixed “Argument list too long” error (Issue #15)Added –json parameterAdded –ipynb parameter to generate IPython notebook0.2.3 (2017-12-09)Fix bug: Unicode column names now work under Py20.2.2 (2017-12-04)Fix bug: Unicode characters no longer break CsvInsight on Py20.2.1 (2017-11-27)Fix bug: opening gzipped files with Py3 now works0.2.0 (2017-11-25)Split files using gsplit and process them in parallel for faster processingNo longer work with streams; works exclusively with filesGet rid of csvi_summarize and csvi_split entry pointsIntegrated plumbum for cleaner pipelinesFixed issue #11: added support for more CSV parameters via the –dialect optionFixed issue #10: reading from empty files no longer raises StopIterationFixed issue #8: use the correct link to the GitHub project in the documentationFixed issue #2: implemented in-memory mode for smaller files0.1.0 (2017-10-29)First release on PyPI.