Package: ai-data-preprocessing-queue
ai-data-preprocessing-queueWhat it doesThis tool is intended for preparing data for further processing.
It contains different text processing steps that can be enabled or disabled dynamically.Installationpip install ai-data-preprocessing-queueHow to usefromai_data_preprocessing_queueimportPipelinestate={}pre_processor_dict={'to_lower':None,'spellcheck':'test\r\ntesting'}pipeline=Pipeline(pre_processor_dict)value=pipeline.consume('Input text',state)stateis optional here and can be used to cache preprocessing data between pipeline calls.The preprocessors that the pipeline should use have to be transmitted as keys within a dictionary.
Some preprocessors also require additional data to function.
The data must be converted into string form and assigned to its preprocessor within the dictionary.This dictionary then needs to be transmitted to the pipeline through its constructor.Note: Pipeline has to be instantiated only once and can be reused.Existing preprocessorsTo Lower CaseName: to_lowerRequired additional data: -Converts the text to lower case characters.Remove NumbersName: remove_numbersRequired additional data: -Removes all numbers from the text.Remove PunctuationName: remove_punctuationRequired additional data: -Removes all special characters from the text.Text onlyName: text_onlyRequired additional data: -Removes all special characters and numbers from the text.Spellcheck (Levenshtein)Name: spellcheckRequired additional data: A string containing words, separated by newline, i.e. "word1\r\nword2"Takes a list of words representing the correct spelling. Words within the given text that are close to a word from this list will be replaced with the listed word.Regex replacementName: regex_replacementRequired additional data: CSV data in string form with the following line format: <pattern>,<replacement>,<order>pattern: a regex pattern that is to be found within the textreplacement: the word/text by which any match should be replacedorder: the order in which the regex entries are supposed to be applied (lowest number will be applied first!)This preprocessor will search for occurrences of specific entities in your text and replace them by a specified pattern.Token ReplacementName: token_replacementRequired additional data: CSV data in string form with the following line format: <text>,<replacement>,<order>text: one or multiple words to search within the textreplacement: the word/text by which any match should be replacedorder: the order in which the entries are supposed to be applied (largest number will be applied first!)With this preprocessor you can replace specific words and abbreviations within the text with specified tokens. It is also possible to replace abbreviations ending with a dot. Other special characters are not supported, though.How to start developingWith VS CodeJust install VS Code with the Dev Containers extension. All required extensions and configurations are prepared automatically.With PyCharmInstall the latest PyCharm versionInstall PyCharm plugin BlackConnectInstall PyCharm plugin MypyConfigure the Python interpreter/venvpip install requirements-dev.txtpip install black[d]Ctl+Alt+S => Check Tools => BlackConnect => Trigger when saving changed filesCtl+Alt+S => Check Tools => BlackConnect => Trigger on code reformatCtl+Alt+S => Click Tools => BlackConnect => "Load from pyproject.yaml" (ensure line length is 120)Ctl+Alt+S => Click Tools => BlackConnect => Configure path to the blackd.exe at the "local instance" config (e.g. C:\Python310\Scripts\blackd.exe)Ctl+Alt+S => Click Tools => Actions on save => Reformat codeRestart PyCharmHow to publishUpdate the version in setup.py and commit your changeCreate a tag with the same version numberLet GitHub do the rest