Package: openclean-pattern
AboutThis package identifies patterns and creates Openclean Patterns from data. It is part of the openclean-core library to create profiled results as well as to detect anomalies.
Currently, Openclean Patterns support the following data types, but are fairly extensible to any other basic / nonbasic implementations:BasicStringIntegersPunctuationsSpacesNon-Basic/AdvancedDatesdays of the week and monthsBusiness Entitiesusing corporation suffixesGeospatial Entitiesusing datamart-geo for administrative levels (in progress)AddressUSPS street abreviations and secondary unit designators for addressesThe package has been extended to identify anomalous patterns inside the data as well.InstallationInstallopenclean-patternfrom thePython Package Index (PyPI)usingpipwith:pipinstallopenclean-patternUsageThe library comes with many predefined classes to support the pattern detection process. One could use theOpencleanPatternFinderclass or otherwise the general process should look similar to the following:Sample the columnIn case of very large dataset two Samplers have been added for the user’s convenience to help extract the distribution of the column:RandomSampler: considers each item in the iterable equally probable to get selectedWeightedRandomSampler: takes a Counter of type {value:frequency} and creates a sample using the Counter distribution.Distinct: selects only distinct rowsTokenize it to remove punctutationAt this point TypeResolvers can also be injected to tokenize and encode in the same run instead of running it as a separate step 3:RegexTokenizer: tokenizes using the default regex that breaks the row values into a list of tokens keeping the delimiters intact (unless a user provides a custom regex). It also changes the tokens to lower case letters. The user also has the option to define if they want to consider e.g. the string ‘a.b.c’ as delimited by the ‘.’ character or consider it as an abbreviation character and keep ‘abc’ intact.DefaultTokenizerFollows the Regex Tokenizer process and the uses the DefaultTypeResolver to resolve token types.Resolve TypesThis stage converts the tokens to theirBasic and Non-Basic representations:BasicTypeResolver: converts the row into the above mentioned BasicTypes.AdvancedTypeResolver: has numerous implementations and can be easily extended to add new AdvancedTypeResolver classes.DateResolverBusinessEntityResolverAddressDesignatorResolverGeoSpatialResolverDefaultTypeResolver: does both Basic and Non-Basic type resolution by letting a user add Non-Basic interceptors before the Basic type resolution operation.Collect and/or AlignCreate groups of similar rows and align them:Cluster: Collect similar tokenized rows by either clustering them using DBSCAN choosing a precomputeddistance.Group: Grouping tokenized rows with similar lengthsCombAlign[1]: looks at all the possible combinations of each token in each row with other all other rows, calculates the distance, clusters the closest alignments together using DBSCAN and returns the clustered groups.Compile a patternGenerate a regex pattern from the aligned groups:DefaultRegexCompilerAnalyzes each token position and the different datatypes that appear at that position iterating through each row. Then selects the majority type as the pattern at that position. Combining positional regex’s compiles a full expression for the column.method=col: Compiles the pattern based on the positions of different tokens at in each row. It flags values that don’t match the specific position’s majority types as anomalies.method=row: Compiles the pattern using each full row as a possible pattern.[1]Not recommended for large datasets or cases where the number of combinations between rows is too large (e.g. one row has 16 tokens and other has 6, the total no. of distance computation just for this combination would be 16P6 =  5765760) - to be updated to using majority pooling alignment / minimum set cover.Upcoming Modulesserializer / deserializermultiple sequence alignmentExamplesWe include severalnotebooksin this repository that demonstrateopenclean-pattern’s usage.See also:openclean-coreopenclean-notebookDatamart-Geo