Package: csvprofiler
CSV ProfilerAn extensible CSV column profiling and validation utility written in Python 3.PurposeThis open-source project was developed to provide a feature-rich column validation and profiling utility for CSV files without the need for writing any code.  It was also designed to allow quite a bit of extensibility for those understanding Python and regular expressions, and with the expectation that the main testing module be used in the future to validate other file types.  It should serve as a helpful tool for those working with extremely large CSV files or who work with relatively large CSV files with the same format on a routine basis where validation would be useful.Python Scriptscsvpcg.py- the CSV modeling and configuration generator scriptcsvprofiler.py- the CSV profiling and validation utilityprofmod.py- the field testing engine used bycsvpcg.pyandcsvprofiler.pywrapdemo.py- a script demonstrating wrapper and imported script approaches to adding user functionsFeaturesAutomatic generation of configuration and parameter template files with column test recommendations.Profile any column, producing a report of unique column values and occurrence counts.New: "unique" profile option reports duplicate values and the record sequence numbers where they were found.New: "statistical" profile option uses a pandas series to generate a statistical column profile (i.e. mean, median, etc.)Column test options:Over 70 built-in tests including custom developed regular expressions and hard coded tests utilizing Python functions.User defined regular expression tests.User defined lookup test lists.User defined range tests using integers, floating point numbers and dates.User defined column cross-reference tests allowing extensive testing of interrelated columns.Column length validation.Maximum column length validation.Output error CSV and log file options.Extensive control over error output limits.Ability to use external files for regular expressions, lookup lists, and column cross-reference test lists.Simple integration of database and user function hooks without changes to modules:Using a Python wrapper script and a couple lines of code.Using a configuration file entry to import a user function script.Easily integrate the field testing engine into existing scripts.Detailed report of column testing statistics, record level statistics and detailed column cross-reference test statistics.Parameter File Examplecsvp_optionsIDPrefixNameGenderLink_IDDeptTerritoryT_StateColumn TestdigitAbbrevNameAlpharegex_LIDlookup_Deptxcheck_Txcheck_TColumn Length106Max Length1050101022Profile (y/n/p/u/s)yyyyyBlank is Error (y/n)yyyStrip Surrounding Spaces (y/n)yyyyyyError Output Limit5050505050Error Output Limit - Length Tests505050505050Error Output Limit - Blank Test50User Data(A\d{5}|B\d{5})AdminnothingnothingFinanceE(PA|NY)HRSEFLTrainingMW(IL|WI)SalesSTXOpsNW(WA|WY)WCAGrange(1:99)DocumentationComplete user documentation is availablehere.InstallationSoftware is available above and can be downloaded into your local Python scripts directory or installed using pip from PyPI.org (sudo pip3 install csvprofiler or sudo pip3 install --user csvprofiler).  If you are installing on linux, you may need to chmod the scripts to allow execution.RequirementsVersion 1.2.0 now requires pandas to be installed (used in the new statistical profiling option).Revision HistoryVersion 1.2.0, March 24, 2021New "unique" profile optionNew "statistical" profile option9 new latitude/longitude built-in testsother minor changes; full change list and details included in documentationPrior versions are available on PyPI.org.Supplemental ScriptsThe"AWS Lambda"folder contains scripts and documentation on how to implement this software as an AWS Lambda function, using S3 or EFS as an input/output resource.LicensingCopyright (c) 2021 Larry Kuhn (larrykuhn at outlook.com)This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
any later version.This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.You should have received a copy of the GNU General Public License
along with this program.  If not, seehttps://www.gnu.org/licenses/.
A copy can be found with the project at:https://github.com/LarryKuhn/CSV-Profiler/blob/master/LICENSE