Package: vladiate
DescriptionVladiate helps you write explicit assertions for every field of your CSV
file.FeaturesWrite validation schemas in plain-old PythonNo UI, no XML, no JSON, just code.Write your own validatorsVladiate comes with a few by default, but there’s no reason you can’t write
your own.Validate multiple files at onceEither with the same schema, or different ones.DocumentationInstallationInstalling:$ pip install vladiateQuickstartBelow is an example of avladfile.pyfromvladiateimportVladfromvladiate.validatorsimportUniqueValidator,SetValidatorfromvladiate.inputsimportLocalFileclassYourFirstValidator(Vlad):source=LocalFile('vampires.csv')validators={'Column A':[UniqueValidator()],'Column B':[SetValidator(['Vampire','Not A Vampire'])]}Here we define a number of validators for a local filevampires.csv,
which would look like this:Column A,Column B
Vlad the Impaler,Not A Vampire
Dracula,Vampire
Count Chocula,VampireWe then runvladiatein the same directory as your.csvfile:$ vladiateAnd get the following output:Validating YourFirstValidator(source=LocalFile('vampires.csv'))
Passed! :)Handling ChangesLet’s imagine that you’ve gotten a new CSV file,potential_vampires.csv, that looks like this:Column A,Column B
Vlad the Impaler,Not A Vampire
Dracula,Vampire
Count Chocula,Vampire
Ronald Reagan,Maybe A VampireIf we were to update our first validator to use this file as follows:- class YourFirstValidator(Vlad):
-     source = LocalFile('vampires.csv')
+ class YourFirstFailingValidator(Vlad):
+     source = LocalFile('potential_vampires.csv')we would get the following error:Validating YourFirstFailingValidator(source=LocalFile('potential_vampires.csv'))
Failed :(
  SetValidator failed 1 time(s) (25.0%) on field: 'Column B'
    Invalid fields: ['Maybe A Vampire']And we would know that we’d either need to sanitize this field, or add
it to theSetValidator.Starting from scratchTo make writing a newvladfile.pyeasy, Vladiate will give
meaningful error messages.Given the following asreal_vampires.csv:Column A,Column B,Column C
Vlad the Impaler,Not A Vampire
Dracula,Vampire
Count Chocula,Vampire
Ronald Reagan,Maybe A VampireWe could write a bare-bones validator as follows:classYourFirstEmptyValidator(Vlad):source=LocalFile('real_vampires.csv')validators={}Running this withvladiatewould give the following error:Validating YourFirstEmptyValidator(source=LocalFile('real_vampires.csv'))
Missing...
  Missing validators for:
    'Column A': [],
    'Column B': [],
    'Column C': [],Vladiate expects something to be specified for every column,even if it
is an empty list(more on this later). We can easily copy and paste
from the error into ourvladfile.pyto make it:classYourFirstEmptyValidator(Vlad):source=LocalFile('real_vampires.csv')validators={'Column A':[],'Column B':[],'Column C':[],}When we runthiswithvladiate, we get:Validating YourSecondEmptyValidator(source=LocalFile('real_vampires.csv'))
Failed :(
  EmptyValidator failed 4 time(s) (100.0%) on field: 'Column A'
    Invalid fields: ['Dracula', 'Vlad the Impaler', 'Count Chocula', 'Ronald Reagan']
  EmptyValidator failed 4 time(s) (100.0%) on field: 'Column B'
    Invalid fields: ['Maybe A Vampire', 'Not A Vampire', 'Vampire']
  EmptyValidator failed 4 time(s) (100.0%) on field: 'Column C'
    Invalid fields: ['Real', 'Not Real']This is because Vladiate interprets an empty list of validators for a
field as anEmptyValidator, which expects an empty string in every
field. This helps us make meaningful decisions when adding validators to
ourvladfile.py. It also ensures that we are not forgetting about a
column or field which is not empty.Built-in ValidatorsVladiate comes with a few common validators built-in:classValidatorGeneric validator. Should be subclassed by any custom validators. Not to
be used directly.classCastValidatorGeneric “can-be-cast-to-x” validator. Should be subclassed by any
cast-test validator. Not to be used directly.classIntValidatorValidates whether a field can be cast to aninttype or not.empty_ok=False:Specify whether a field which is an empty string should be ignored.classFloatValidatorValidates whether a field can be cast to anfloattype or not.empty_ok=False:Specify whether a field which is an empty string should be ignored.classSetValidatorValidates whether a field is in the specified set of possible fields.valid_set=[]:List of valid possible fieldsempty_ok=False:Implicity adds the empty string to the specified set.ignore_case=False:Ignore the case between values in the column and valid setclassUniqueValidatorEnsures that a given field is not repeated in any other column. Can
optionally determine “uniqueness” with other fields in the row as well viaunique_with.unique_with=[]:List of field names to make the primary field unique with.empty_ok=False:Specify whether a field which is an empty string should be ignored.classRegexValidatorValidates whether a field matches the given regex usingre.match().pattern=r'di^':The regex pattern. Fails for all fields by default.full=False:Specify whether we should use a fullmatch() or match().empty_ok=False:Specify whether a field which is an empty string should be ignored.classRangeValidatorValidates whether a field falls within a given range (inclusive). Can handle
integers or floats.low:The low value of the range.high:The high value of the range.empty_ok=False:Specify whether a field which is an empty string should be ignored.classEmptyValidatorEnsure that a field is always empty. Essentially the same as an emptySetValidator. This is used by default when a field has no
validators.classNotEmptyValidatorThe opposite of anEmptyValidator. Ensure that a field is never empty.classIgnoreAlways passes validation. Used to explicity ignore a given column.classRowValidatorGeneric row validator. Should be subclassed by any custom validators. Not
to be used directly.classRowLengthValidatorValidates that each row has the expected number of fields. The expected
number of fields is inferred from the CSV header row read bycsv.DictReader.Built-in Input TypesVladiate comes with the following input types:classVladInputGeneric input. Should be subclassed by any custom inputs. Not to be used
directly.classLocalFileRead from a file local to the filesystem.filename:Path to a local CSV file.classS3FileRead from a file in S3. Optionally can specify either a full path, or a
bucket/key pair.Requires thebotolibrary, which should be
installed viapip install vladiate[s3].path=None:A full S3 filepath (e.g.,s3://foo.bar/path/to/file.csv)bucket=None:S3 bucket. Must be specified with akey.key=None:S3 key. Must be specified with abucket.classStringRead CSV from a string. Can take either anstror aStringIO.:string_input=NoneRegular Python string input.:string_io=NoneStringIOinput.Running Vlads ProgramaticallyclassVladInitialize a Vlad programaticallysource:Required. AnyVladInput.validators={}:List of validators. Optional, defaults to the class variablevalidatorsif set, otherwise usesEmptyValidatorfor all fields.delimiter=',':The delimiter used within your csv source. Optional, defaults to,.ignore_missing_validators=False:Whether to fail validation if there are fields in the file for which theVladdoes not have validators. Optional, defaults toFalse.quiet=False:Whether to disable log output generated by validations.
Optional, defaults toFalse.file_validation_failure_threshold=None:Stops validating the file after this failure threshold is reached.
Input a value between0.0and1.0.1.0`(100%) validates the entire file.
Optional, defaults to `None.For example:fromvladiateimportVladfromvladiate.inputsimportLocalFileVlad(source=LocalFile('path/to/local/file.csv')).validate()TestingTo run the tests:make testTo run the linter:make lintCommand Line ArgumentsUsage:vladiate[options][VladClass[VladClass2...]]Options:-h,--helpshowthishelpmessageandexit-fVLADFILE,--vladfile=VLADFILEPythonmodulefiletoimport,e.g.'../other.py'.Default:vladfile-l,--listShowlistofpossiblevladiateclassesandexit-V,--versionshowversionnumberandexit-pPROCESSES,--processes=PROCESSESattempttousethisnumberofprocesses,Default:1-q,--quietdisableconsolelogoutputgeneratedbyvalidationsContributorsDustin IngramClara BennettAditya NatrajSterling PetersenAleixBob LannonSantiDavid ParkJon Bonafatoharitha-raviLicenseOpen source MIT license.