Package: purewords
# purewordsPurewords is a package used to clean raw texts for all languages.## Install`pip install purewords`## Usage### Module usage:```pythonimport purewords# raw sentenceinputs = "ha hi!!hello I\'m at http:www.google.com.tw\n\n"+ "you know yahoo? my_computer is great. My phone number"+ "is 02-3366-5678. <br>的啦<br> my password: 123-abc$99&^%Y)\'_\'(Y "```#### Treat inputs as a sentence and clean it.Word tokens are splitted with whitespace```python# result: stringpurewords.clean_sentence(inputs)'ha hi hello i am at _url_ you know yahoo my computer is great my phone number is _phone_ 的 my password _num_ abc _num_ y y'```#### Treat inputs as a document and clean it.Split document with some confident splitting token such as '.' or '?'.```python# result: list of cleaned stringpurewords.clean_document(inputs)['ha hi', 'hello i am at _url_', 'you know yahoo', 'my computer is great', 'my phone number is _phone_', '的 my password _num_ abc _num_ y y']```### Customed your purewordsYou can use different setting in purewords.```pythonimport purewordsfrom purewords.tokenizer import YoctolTokenizerfrom purewords.filter_collection import document_filtersfrom purewords.filter_collection import token_filterstokenizer = YoctolTokenizer()pw = purewords.PureWords(tokenizer=tokenizer, # select your tokenizerdocument_filters=document_filters, # select your document filterstoken_filters=token_filters, # select your token filtersmax_len=200, # cut long sentence whose length exceed max_lenmin_len=1 # ignore short sentence)inputs = 'This is a sentence.'pw.clean_sentence(inputs)pw.clean_document(inputs)```#### Tokenizer##### Select your tokenizer in purewordsYou can select `WhitespaceTokenizer` tokenizer if you prefer tokenizesentences with whitespace or `JiebaTokenizer` for default jieba setting.Otherwise, we use yoctol jeiba tokenizer as our default setting.```pythonfrom purewords.tokenizer import WhitespaceTokenizertokenizer = WhitespaceTokenizer()pw = purewords.PureWords(tokenizer=tokenizer)```##### Add new words in JiebaTokenizerYou can add new word in JiebaTokenizer to customize your tokenizer.```pythonfrom purewords.tokenizer import JiebaTokenizertokenizer = JiebaTokenizer()tokenizer.add_word(new_word, freq, tag) # The setting is same with jieba.add_wordtokenizer.add_words(new_word_list, freq, tag)pw = purewords.PureWords(tokenizer=tokenizer)```#### Filter collectionYou can customize your preprocesing ways in purewords.* document_filters: preprocess the raw sentence before sentence splitting* token_filters: preprocess tokens after tokenization of each sentence##### Organize your filtersYou can create your customized filters by adding your filters in our filter collection class.Filter means a callable object which receives a raw sentence and returns the processed one.The preprocessing order is consistent with the adding order of filters.```pythonfrom purewords.filter_collection import BaseFilterCollectioncustom_filters = BaseFilterCollection()custom_filters.add(filter_1)custom_filters.add(filter_2)...custom_filters.add(filter_n)pw = purewords.PureWords(tokenizer=tokenizer,document_filters=custom_filters,)```#### StopwordsYou can add stopwords in `purewords/config/stopwords.txt`.### Command line usage:Preprocess text files into a single cleaned document from command line.Usage:#### Clean single txt files```python -m purewords input_file_path```#### Clean text files in a directoryOr, you can use following command to clean all the txt files in your directory.```python -m purewords -d your_raw_text_dir```#### Ignore short sentencesIf you prefer long sentences and want to ignore short sentences less than 5 words, you can try this.```python -m purewords -min 5 your_text_file```#### Cut long sentencesOr you prefer short sentences less than 30 words and want to cut long sentences into short sentences.You can set up the maximun sentence length like this.```python -m purewords -max 30 your_text_file```#### Use multi-thread to speed upYou can also use multi-trhead to speed up the cleaning process.In the follwoing example, you clean all the text files with 4 threads```python -m purewords -j 4 -d your_raw_text_dir```