Package: datafilter
Data FilterQuickly find tokens (words, phrases, etc) within your data.Data Filter is a lightweightdata cleansingframework that can be easily extended to support different data types, structures or processing requirements. It natively supports the following data types:CSV filesText filesText stringsTable of ContentsRequirementsInstallationBasic UsageFeaturesBaseFiltersCSVTextTextFileRequirementsPython 3.6+InstallationTo install, simply usepipenv(or pip):>>>pipenvinstalldatafilterBasic UsageCSVfromdatafilterimportCSVtokens=["Lorem","ipsum","Volutpat est","mi sit amet"]data=CSV("test.csv",tokens=tokens)data.save("filtered.csv")In this example, we open a CSV file, search all rows for normalized tokens and flag them. Thesavemethod creates a new CSV file with all rows that weren't flagged.TextfromdatafilterimportTexttext="Lorem ipsum dolor sit amet, consectetur adipiscing elit"data=Text(text,tokens=["Lorem"])print(next(data.results()))In this example, we search a text string for normalized tokens. We can then iterator over the results using the.results()method, which returns a generator that yieldsformatted results.Text FilefromdatafilterimportTextFiledata=TextFile("test.txt",tokens=["Lorem","ipsum"],re_split=r"(?<=\.)")print(next(data.results()))In this example, we open a text file and split the data based on a regular expression defined byre_split.FeaturesData Filter was designed to be highly extensible. Common or useful filters can be easily reused and shared. A few example use cases include:Filters that can handle different data types such as Microsoft Word, Google Docs, etc.Filters that can handle incoming data from external APIs.BaseAbstract base class that's subclassed by every filter.Baseincludes several methods to ensure data is properly normalized, formatted and returned. The.results()method is an@abstractmethodto enforce its use in subclasses.Parameterstokenstype <list>A list of strings that will be searched for within a set of data.translationstype <list>A list of strings that will be removed during normalization.Default['0123456789','(){}[]<>!?.:;,`\'"@#$%^&*+-|=~–—/\\_','\t\n\r\x0c\x0b']bidirectionaltype <bool>WhenTrue, token matching will be bidirectional.DefaultTrueNote:A common obfuscation method is to reverse the offending string or phrase. This helps detect those instances.caseinsensitivetype <bool>WhenTrue, tokens and data are converted to lowercase during normalization.DefaultTrueMethods.results()Abstract method used to return results within a filter. This is defined by aBasesubclass.maketrans()Returns a translation table used during normalization.Returnstype <dict>.normalize(data)Returns normalized data. Normalization includes converting data tolowercaseandremoving strings.Accepts parameterdata.Returnstype <tuple>Note:Normalized data is returned as a tuple. The first element is the original data. The second element is the normalized data..parse(data)Returns parsed and formatted data.Accepts parameterdata.Returnstype <dict>ExampleAssume we're searching for the token "Lorem" in a very short text string.data=Text("Lorem ipsum dolor sit amet",tokens=["Lorem"])print(next(data.results()))The returned result would be formatted as:{"data":"Lorem ipsum dolor sit amet","flagged":True,"describe":{"tokens":{"detected":["Lorem"],"count":1,"frequency":{"Lorem":1,},}},}Note:.parse()should never be called directly. Use.results()instead.FiltersFilters subclass and extend theBaseclass to support various data types and structure. This extensibility allows for the creation of powerful custom filters specifically tailored to a given task, data type or structure.CSVParametersCSVis a subclass ofBaseand inherits all parameters.pathtype <str>Path to a CSV file.MethodsCSVis a subclass ofBaseand inherits all methods..save(path)Saves results to a file.Accepts parameterpath.pathis the absolute path and filename of the new file.TextParametersTextis a subclass ofBaseand inherits all parameters.texttype <str>A text string.re_splittype <str>A regular expression pattern or string that will be applied totextwithre.splitbefore normalization.MethodsTextis a subclass ofBaseand inherits all methods..save(path, endofline=" ")Saves results to a file.Accepts parameterpathandendofline.pathis the absolute path and filename of the new file.endoflineis a line delimiter that will be added to the end of every row.TextFileParametersTextFileis a subclass ofBaseand inherits all parameters.pathtype <str>Path to a text file.re_splittype <str>A regular expression pattern or string that will be applied totextwithre.splitbefore normalization.MethodsTextFileis a subclass ofBaseand inherits all methods..save(path, endofline=" ")Saves results to a file.Accepts parameterpathandendofline.pathis the absolute path and filename of the new file.endoflineis a line delimiter that will be added to the end of every row.