Package: acryl-datahub-classify
datahub-classifyPredict InfoTypes forDataHub.Installationpython3 -m pip install --upgrade acryl-datahub-classifyAPIpredict_infotypesThis API populates infotype proposal(s) for each input column by using metadata, values & confidence level threshold. Following are the input and output contractAPI InputAPI expects following parameters in the outputcolumn_infos- This is a list of ColumnInfo objects. Each ColumnInfo object contains metadata (col_name, description, datatype, etc) and values of a column.confidence_level_threshold- If the infotype prediction confidence is greater than the confidence threshold then the prediction is considered as a proposal. This is the common threshold for all infotypes.global_config- This dictionary contains configuration details about all supported infotypes. Refer sectionInfotype Configurationfor more information.infotypes- This is a list of infotypes that is to be processed. This is an optional argument, if specified then it will override the default list of all supported infotypes. If user is interested in only few infotypes then this list can be specified with correct infotype names. Infotype names are case sensitive.minimum_values_threshold- Minimum number of column values required for processing. This is an optional argument, default is 50.API OutputAPI returns a list of ColumnInfo objects of length same as input ColumnInfo objects list. A populated list of Infotype proposal(s), if any, is added in the ColumnInfo object itself with a variable name asinfotype_proposals. The infotype_proposals list contains InfotypeProposal objects which has following informationinfotype- A proposed infotype name.confidence_level- Overall confidence of the infotype proposal.debug_info- confidence score of each prediction factor involved in the overall confidence score calculation. Refer sectionDebug Informationfor more information.Convention:Ifinfotype_proposalslist is non-empty then it indicates that there is at least one infotype proposal with confidence greater thanconfidence_level_threshold.Infotype ConfigurationInfotype configuration is a dictionary with all infotypes at root level key. Each infotype has following configurable parameters (value of each parameter is a dictionary)Prediction_Factors_and_Weights- This is a dictionary that specifies the weight of each prediction factor which will be used in the final confidence calculation. Following are the prediction factorsNameDescriptionDatatypeValuesExcludeName- optional exact match list for column names to exclude from classification for this info_typeName- regex list which is to be matched against column nameDescription- regex list which is to be matched against column descriptionDatatype- list of datatypes to be matched against column datatypeValues- this dictionary contains following informationprediction_type- values evaluation model (regex/library)regex- regex list which is to be matched against column valueslibrary- library name which is to be used to evaluate column valuesSample Infotype Configuration Dictionary{'<Infotype1>':{'Prediction_Factors_and_Weights':{'Name':0.4,'Description':0,'Datatype':0,'Values':0.6},'Name':{'regex':[<regexpatterns>]},'Description':{'regex':[<regexpatterns>]},'Datatype':{'type':[<listofdatatypes>]},'Values':{'prediction_type':'regex/library','regex':[<regexpatterns>],'library':[<libraryname>]}},'<Infotype2>':{......}}Debug InformationA debug information is associated with each infotype proposal, it provides details about confidence score from each prediction factor involved in overall confidence score calculation. This is a dictionary with following four prediction factors as keyNameDescriptionDatatypeValues{'Name':0.4,'Description':0.2,'Values':0.6,'Datatype':0.3}Supported InfotypesBelow Infotypes are supported out of the box.AgeGenderPerson Name / Full NameEmail AddressPhone NumberStreet AddressCredit-Debit Card NumberInternational Bank Account NumberVehicle Identification NumberUS Social Security NumberIpv4 AddressIpv6 AddressSwift CodeUS Driving License NumberRegex based custom infotypes are supported. Specify custom infotype configuration in format mentionedhere.AssumptionsIf value prediction factor weight is non-zero (indicating values should be used for infotype inspection) then a minimum 50 non-null column values should be present.DevelopmentSet up your Python environmentcddatahub-classify
../gradlew:datahub-classify:installDev# OR pip install -e ".[dev]"sourcevenv/bin/activateRunnning testspytesttests/--capture=no--log-cli-level=DEBUGSanity check code before committing# Assumes: pip install -e ".[dev]" and venv is activatedblacksrc/tests/
isortsrc/tests/
flake8src/tests/
mypysrc/tests/Build and Test../gradlew:datahub-classify:buildYou can also run these steps via the gradle build:../gradlew:datahub-classify:lint
../gradlew:datahub-classify:lintFix
../gradlew:datahub-classify:testQuick