Package: data-preprocessors
Data PreprocessorsAn easy-to-use tool for Data Preprocessing especially for Text PreprocessingTable of ContentsInstallationQuick StartFeaturesSplit TextfileBuild Parallel CorpusSeparate Parallel CorpusDecontruct Words of SentenceRemove PunctuationSpace PunctuationText File to ListText File to DataframeList to Text FileRemove FileCount Characters of a SentenceCount Words of SentenceCount No of Lines in a Text FileConvert Excel to Multiple Text FilesApply Any Function in a Full Text FileInstallationInstall the latest stable releaseFor windowspip install -U data-preprocessorsFor Linux/WSL2pip3 install -U data-preprocessorsQuick Startfromdata_preprocessorsimporttext_preprocessorastpsentence="bla! bla- ?bla ?bla."sentence=tp.remove_punc(sentence)print(sentence)>>blablablablaFeaturesSplit TextfileThis function will split your textfile into train, test and validate. Three separate text files. By changingshuffleandseedvalue, you can randomly shuffle the lines of your text files.fromdata_preprocessorsimporttext_preprocessorastptp.split_textfile(main_file_path="example.txt",train_file_path="splitted/train.txt",val_file_path="splitted/val.txt",test_file_path="splitted/test.txt",train_size=0.6,val_size=0.2,test_size=0.2,shuffle=True,seed=42)# Total lines:  500# Train set size:  300# Validation set size:  100# Test set size:  100Separate Parallel CorpusBy using this function, you will be able to easily separatesrc_tgt_fileinto separatedsrc_fileandtgt_file.fromdata_preprocessorsimporttext_preprocessorastptp.separate_parallel_corpus(src_tgt_file="",separator="|||",src_file="",tgt_file="")Decontracting Words from Sentencetp.decontracting_words(sentence)Remove PunctuationBy using this function, you will be able to remove the punction of a single line of a text file.fromdata_preprocessorsimporttext_preprocessorastpsentence="bla! bla- ?bla ?bla."sentence=tp.remove_punc(sentence)print(sentence)# bla bla bla blaSpace PunctuationBy using this function, you will be able to add one space to the both side of the punction so that it will easier to tokenize the sentence. This will apply on a single line of a text file. But if we want, we can use it in a full twxt file.fromdata_preprocessorsimporttext_preprocessorastpsentence="bla! bla- ?bla ?bla."sentence=tp.space_punc(sentence)print(sentence)# bla bla bla blaText File to ListConvert any text file into list.mylist=tp.text2list(myfile_path="myfile.txt")List to Text FileConvert any list into a text file (filename.txt)tp.list2text(mylist=mylist,myfile_path="myfile.txt")Count Characters of a SentenceThis function will help to count the total characters of a sentence.tp.count_chars(myfile="file.txt")Convert Excel to Multiple Text FilesThis function will help to Convert an Excel file's columns into multiple text files.tp.excel2multitext(excel_file_path="",column_names=None,src_file="",tgt_file="",aligns_file="",separator="|||",src_tgt_file="",)Apply a function in whole text fileIn the place offunction_nameyou can use any function and that function will be applied in the full/whole text file.fromdata_preprocessorsimporttext_preprocessorastptp.apply_whole(function_name,myfile_path="myfile.txt",modified_file_path="modified_file.txt")