Natural Language Processing in Production: 27 Fast Text Pre-Processing Methods
By: Bruce H. Cottman
Tags: Text Preprocessing, Machine Learning, Spacy, Data Science, NLP

Natural Language Processing in Production: 27 Fast Text Pre-Processing Methods

We show Python code and benchmarks for 27 different NLP text pre-processing actions.

A natural pipeline: Source: Unsplash

Outline

Estimates state that 70%–85% of the world’s data is text (unstructured data) [1]. New deep learning language models (transformers) have caused explosive growth in industry applications [5,6.11].

This blog is not an article introducing you to Natural Language Processing. Instead, it assumes you are familiar with noise reduction and normalization of text. It covers text preprocessing up to producing tokens and lemmas from the text.

We stop at feeding the sequence of tokens into a Natural Language model.

The feeding of that sequence of tokens into a Natural Language model to accomplish a specific model task is not covered here.

In production-grade Natural Language Processing (NLP), what is covered in this blog is that fast text pre-processing (noise cleaning and normalization) is critical.

I discuss packages we use for production-level NLP; I detail the production-level NLP preprocessing text tasks with python code and packages; Finally. I report benchmarks for NLP text pre-processing tasks;

Dividing NLP processing into two Steps

We segment NLP into two major steps (for the convenience of this article):

Text pre-processing into tokens. We clean (noise removal) and then normalize the text. The goal is to transform the text into a corpus that any NLP model can use. A goal is rarely achieved until the introduction of the transformer [2]. A corpus is an input (text preprocessed into a sequence of tokens) into NLP models for training or prediction.

The rest of this article is devoted to noise removal text and normalization of text into tokens/lemmas (Step 1: text pre-processing).

Noise removal deletes or transforms things in the text that degrade the NLP task model.

Noise removal is usually an NLP task-dependent. For example, e-mail may or may not be removed if it is a text classification task or a text redaction task.

I show replacement or removal of the noise.

Normalization of the corpus is transforming the text into a common form. The most frequent example is normalization by transforming all characters to lowercase.

In follow-on blogs, we will cover different deep learning language models and Transformers (Steps 2-n) fed by the corpus token/lemma stream.

NLP Text Pre-Processing Package Factoids

There are many NLP packages available. We use spaCy [2], textacy [4], Hugging Face transformers [5], and regex [7] in most of our NLP production applications. The following are some of the “factoids” we used in our decision process.

Note: The following “factoids” may be biased. That is why we refer to them as “factoids.”

NLTK [3]

NLTK is a string processing library. All the tools take strings as input and return strings or lists of strings as output [3] .

is a string processing library. All the tools take strings as input and return strings or lists of strings as output . NLTK is a good choice if you want to explore different NLP with a corpus whose length is less than a million words.

is a choice if you want to explore different with a corpus whose length is less than a million words. NLTK is a bad choice if you want to go into production with your NLP application [3].

Regex

The use of regex is pervasive throughout our text-preprocessing code. Regex is a fast string processor. Regex, in various forms, has been around for over 50 years. Regex support is part of the standard library of Java and Python, and is built into the syntax of others, including Perl and ECMAScript (JavaScript);

spaCy [2]

spaCy is a moderate choice if you want to research different NLP models with a corpus whose length is greater than a million words.

is a choice if you want to different models with a corpus whose length is greater than a million words. If you use a selection from spaCy [3] , Hugging Face [5] , fast.ai [13] , and GPT-3 [6] , then you are performing SOTA (state-of-the-art) research of different NLP models (my opinion at the time of writing this blog).

, , , and , then you are performing (state-of-the-art) of different models (my opinion at the time of writing this blog). spaCy is a good choice if you want to go into production with your NLP application .

is a choice if you want to go into production with your application spaCy is an NLP library implemented both in Python and Cython . Because of the Cython, parts of spaCy are faster than if implemented in Python [3];

is an library implemented both in and . Because of the parts of are faster than if implemented in spacy is the fastest package, we know of, for NLP operations;

is the fastest package, we know of, for operations; spacy is available for operating systems MS Windows, macOS, and Ubuntu [3] ;

is available for operating systems and ; spaCy runs natively on Nvidia GPUs [3] ;

runs natively on GPUs ; explosion /spaCy has 16,900 stars on Github (7/22/2020);

has 16,900 stars on Github (7/22/2020); spaCy has 138 public repository implementations on GitHub;

has 138 public repository implementations on spaCy comes with pre-trained statistical models and word vectors;

comes with pre-trained statistical models and word vectors; spaCy transforms text into document objects, vocabulary objects, word- token objects, and other useful objects resulting from parsing the text ;

transforms text into document objects, vocabulary objects, word- token objects, and other useful objects resulting from parsing the text ; Doc class has several useful attributes and methods. Significantly, you can create new operations on these objects as well as extend a class with new attributes (adding to the spaCy pipeline);

class has several useful attributes and methods. Significantly, you can create new operations on these objects as well as extend a class with new attributes (adding to the pipeline); spaCy features tokenization for 50+ languages;

spacy pipeline Source: https://github.com/explosion/spaCy/blob/master/website/docs/images/pipeline.svg

Creating long_s Practice Text String

We create long_ , a long string that has extra whitespace, emoji, email addresses, $ symbols, HTML tags, punctuation, and other text that may or may not be noise for the downstream NLP task and/or model.

MULPIPIER = int(3.8e3)

text_l = 300

long_s += ' 888 eihtg DoD Fee

long_s += ':( cat-

nip'

long_s += ' immed-

natedly <html><h2>2nd levelheading</h2></html> . , '

long_s += '#

long_s +=' $Shine $$beighty?$ '

long_s *= MULPIPIER

print('size: {:g} {}'.format(len(long_s),long_s[:text_l])) %time long_s = ':( 😻 😈 #google +1 608-444-0000 08-444-0004 608-444-00003 ext. 508 'long_s += ' 888 eihtg DoD Fee https://medium.com/ #hash ## Document Title 'long_s += ':( cat-

nip'long_s += ' immed-

natedly 2nd levelheading . , 'long_s += '# bhc@gmail.com f@z.yx can\'t Be a ckunk. $4 $123,456 won\'t seven 'long_s +=' $Shine $$beighty?$ 'long_s *= MULPIPIERprint('size: {:g} {}'.format(len(long_s),long_s[:text_l]))

output =>



Wall time: 8.11 µs

size: 1.159e+06

:( 😻 😈 #google +1 608-444-0000 08-444-0004 608-444-00003 ext. 508 888 eihtg DoD Fee

nip immed-

natedly <html><h2>2nd levelheading</h2></html> . , # bhc@gmail.com f@z.yx can't Be a ckunk. $4 $123,456 won't seven $Shine $$beigh CPU times: user 3 µs, sys: 1 µs, total: 4 µsWall time: 8.11 µssize: 1.159e+06:( 😻 😈 #google +1 608-444-0000 08-444-0004 608-444-00003 ext. 508 888 eihtg DoD Fee https://medium.com/ #hash ## Document Title :( cat-nip immed-natedly 2nd levelheading . , # bhc@gmail.com f@z.yx can't Be a ckunk. $4 $123,456 won't seven $Shine $$beigh

A string, long_s of 1.159 million characters is created in 8.11 µs.

Python String Corpus Pre-processing Step and Benchmarks

All benchmarks are run within a Docker container on MacOS Version 14.0 (14.0).

Model Name: Mac Pro

Processor Name: 12-Core Intel Xeon E5

Processor Speed: 2.7 GHz

Total Number of Cores: 24

L2 Cache (per Core): 256 KB

L3 Cache: 30 MB

Hyper-Threading Technology: Enabled Memory: 64 GB

Note: Corpus/text pre-processing is dependent on the end-point NLP analysis task. Sentiment Analysis requires different corpus/text pre-processing steps than document redaction. The corpus/text pre-processing steps given here are for a range of NLP analysis tasks. Usually. a subset of the given corpus/text pre-processing steps is needed for each NLP task. Also, some of required corpus/text pre-processing steps may not be given here.

1. NLP text preprocessing: Replace Twitter Hash Tags

from textacy.preprocessing.replace import replace_hashtags

%time text = replace_hashtags(long_s,replace_with= '_HASH_')

print('size: {:g} {}'.format(len(text),text[:text_l])))

output =>



Wall time: 223 ms

size: 1.159e+06 :

( 😻 😈 _HASH_ +1 608-444-0000 08-444-0004 608-444-00003 ext. 508 888 eihtg DoD Fee

nip immed-

natedly <html><h2>2nd levelheading</h2></html> . , # bhc@gmail.com f@z.yx can't Be a ckunk. $4 $123,456 won't seven $Shine $$beigh CPU times: user 223 ms, sys: 66 µs, total: 223 msWall time: 223 mssize: 1.159e+06 :( 😻 😈 _HASH_ +1 608-444-0000 08-444-0004 608-444-00003 ext. 508 888 eihtg DoD Fee https://medium.com/ _HASH_ ## Document Title :( cat-nip immed-natedly 2nd levelheading . , # bhc@gmail.com f@z.yx can't Be a ckunk. $4 $123,456 won't seven $Shine $$beigh

Notice that #google and #hash are swapped with _HASH_, and ## and _# are untouched. A million characters were processed in 200 ms. Fast enough for a big corpus of a billion characters (example: web server log).

2. NLP text preprocessing: Remove Twitter Hash Tags

from textacy.preprocessing.replace import replace_hashtags

%time text = replace_hashtags(long_s,replace_with= '')

print('size: {:g} {}'.format(len(text),text[:text_l]))

output =>



Wall time: 220 ms

size: 1.1134e+06 :( 😻 😈 +1 608-444-0000 08-444-0004 608-444-00003 ext. 508 888 eihtg DoD Fee

nip immed-

natedly <html><h2>2nd levelheading</h2></html> . , # bhc@gmail.com f@z.yx can't Be a ckunk. $4 $123,456 won't seven $Shine $$beighty?$ CPU times: user 219 ms, sys: 0 ns, total: 219 msWall time: 220 mssize: 1.1134e+06 :( 😻 😈 +1 608-444-0000 08-444-0004 608-444-00003 ext. 508 888 eihtg DoD Fee https://medium.com/ ## Document Title :( cat-nip immed-natedly 2nd levelheading . , # bhc@gmail.com f@z.yx can't Be a ckunk. $4 $123,456 won't seven $Shine $$beighty?$

Notice that #google and #hash are removed and ##, and _# are untouched. A million characters were processed in 200 ms.

3. NLP text preprocessing: Replace Phone Numbers

from textacy.preprocessing.replace import replace_phone_numbers

%time text = replace_phone_numbers(long_s,replace_with= '_PHONE_')

print('size: {:g} {}'.format(len(text),text[:text_l]))

output =>

CPU times: user 384 ms, sys: 1.59 ms, total: 386 ms

Wall time: 383 ms

size: 1.0792e+06

:( 😻 😈 _PHONE_ 08-_PHONE_ 608-444-00003 ext. 508 888 eihtg

Notice phone number 08-444-0004 and 608-444-00003 ext. 508 were not transformed.

4. NLP text preprocessing: Replace Phone Numbers - better

RE_PHONE_NUMBER: Pattern = re.compile(

# core components of a phone number

r"(?:^|(?<=[^\w)]))(\+?1[ .-]?)?(\(?\d{2,3}\)?[ .-]?)?(\d{2,3}[ .-]?\d{2,5})"

# extensions, etc.

r"(\s?(?:ext\.?|[#x-])\s?\d{2,6})?(?:$|(?=\W))",

flags=re.UNICODE | re.IGNORECASE) text = RE_PHONE_NUMBER.sub('_PHoNE_', long_s)

print('size: {:g} {}'.format(len(text),text[:text_l]))

output =>



Wall time: 350 ms

size: 1.0108e+06

:( 😻 😈 _PHoNE_ _PHoNE_ _PHoNE_ 888 eihtg DoD Fee

nip immed-

natedly <html><h2>2nd levelheading</h2></html> . , # bhc@gmail.com f@z.yx can't Be a ckunk. $4 $123,456 won't seven $Shine $$beighty?$ CPU times: user 353 ms, sys: 0 ns, total: 353 msWall time: 350 mssize: 1.0108e+06:( 😻 😈 _PHoNE_ _PHoNE_ _PHoNE_ 888 eihtg DoD Fee https://medium.com/ ## Document Title :( cat-nip immed-natedly 2nd levelheading . , # bhc@gmail.com f@z.yx can't Be a ckunk. $4 $123,456 won't seven $Shine $$beighty?$

Notice phone number 08-444-0004 and 608-444-00003 ext. 508 were transformed. A million characters were processed in 350 ms.

5. NLP text preprocessing: Remove Phone Numbers

Using the improved RE_PHONE_NUMBER pattern, we put '' in for ' PHoNE' to remove phone numbers from the corpus.

text = RE_PHONE_NUMBER.sub('', long_s)

print('size: {:g} {}'.format(len(text),text[:text_l]))

output =>



Wall time: 351 ms

size: 931000

:( 😻 😈 888 eihtg DoD Fee

nip immed-

natedly <html><h2>2nd levelheading</h2></html> . , # bhc@gmail.com f@z.yx can't Be a ckunk. $4 $123,456 won't seven $Shine $$beighty?$ CPU times: user 353 ms, sys: 459 µs, total: 353 msWall time: 351 mssize: 931000:( 😻 😈 888 eihtg DoD Fee https://medium.com/ ## Document Title :( cat-nip immed-natedly 2nd levelheading . , # bhc@gmail.com f@z.yx can't Be a ckunk. $4 $123,456 won't seven $Shine $$beighty?$

A million characters were processed in 375 ms.

6. NLP text preprocessing: Removing HTML metadata

I admit removing HTML metadata is my favorite. Not because I like the task, but because I screen-scrape frequently. There is a lot of useful data that resides on an IBM mainframe, VAX-780 (huh?), or whatever terminal-emulation that results in an HTML-based report.

These techniques of web scraping of reports generate text that has HTML tags. HTML tags are considered noise typically as they are parts of the text with little or no value in the follow-on NLP task.

Remember, we created a test string ( long_s ) a little over million characters long with some HTML tags. We remove the HTML tags using BeautifulSoup .

from bs4 import BeautifulSoup

%time long_s = BeautifulSoup(long_s,'html.parser').get_text()

print('size: {:g} {}'.format(len(long_s),long_s[:text_l])))

output =>



Wall time: 971 ms

size: 817000

:( 😻 😈 888 eihtg DoD Fee

nip immed-

natedly 2nd levelheading CPU times: user 954 ms, sys: 17.7 ms, total: 971 msWall time: 971 mssize: 817000:( 😻 😈 888 eihtg DoD Fee https://medium.com/ ## Document Title :( cat-nip immed-natedly 2nd levelheading

The result is that BeautifulSoup is able to remove over 7,000 HTML tags in a million character corpus in one second. Scaling linearly, a billion character corpus, about 200 million word, or approximately 2000 books, would require about 200 seconds.

The rate for HTML tag removal by BeautifulSoup is about 0. 1 second per book. An acceptable rate for our production requirements.

I only benchmark BeautifulSoup . If you know of a competitive alternative method, please let me know.

Not: The compute times you get may be multiples of time longer or shorter if you are using the cloud or Spark.

7. NLP text preprocessing: Replace currency symbol

The currency symbols “[$¢£¤¥ƒ֏؋৲৳૱௹฿៛ℳ元円圆圓﷼\u20A0-\u20C0] “ are replaced with _CUR_ using the textacy package:

%time textr = textacy.preprocessing.replace.replace_currency_symbols(long_s)

print('size: {:g} {}'.format(len(textr),textr[:text_l]))

output =>



Wall time: 33.7 ms

size: 908200

:( 😻 😈 888 eihtg DoD Fee

nip immed-

natedly 2nd levelheading . , # bhc@gmail.com f@z.yx can't Be a ckunk. _CUR_4 _CUR_123,456 won't seven _CUR_Shine _CUR__CUR_beighty?_CUR_ CPU times: user 31.2 ms, sys: 1.67 ms, total: 32.9 msWall time: 33.7 mssize: 908200:( 😻 😈 888 eihtg DoD Fee https://medium.com/ ## Document Title :( cat-nip immed-natedly 2nd levelheading . , # bhc@gmail.com f@z.yx can't Be a ckunk. _CUR_4 _CUR_123,456 won't seven _CUR_Shine _CUR__CUR_beighty?_CUR_

Note: The option textacy replace_<something> enables you to specify the replacement text. _CUR_ is the default substitution text for replace_currency_symbols.

You may have the currency symbol $ in your text. In this case you can use a regex:

%time text = re.sub('\$', '_DOL_', long_s)

print('size: {:g} {}'.format(len(text),text[:250]))

output =>



Wall time: 8.25 ms

size: 1.3262e+06

:( 😻 😈 #google +1 608-444-0000 08-444-0004 608-444-00003 ext. 508 888 eihtg DoD Fee

nip immed-

natedly <html><h2>2nd levelheading</h2></html> . , # bhc@gmail.com f@z.yx can't Be a ckunk. _DOL_4 _DOL_123,456 won't seven _DOL_Shine _DOL__DOL_beighty?_DOL_ : CPU times: user 8.06 ms, sys: 0 ns, total: 8.06 msWall time: 8.25 mssize: 1.3262e+06:( 😻 😈 #google +1 608-444-0000 08-444-0004 608-444-00003 ext. 508 888 eihtg DoD Fee https://medium.com/ #hash ## Document Title :( cat-nip immed-natedly 2nd levelheading . , # bhc@gmail.com f@z.yx can't Be a ckunk. _DOL_4 _DOL_123,456 won't seven _DOL_Shine _DOL__DOL_beighty?_DOL_ :

Note: All symbol $ in your text will be removed. Don't use if you have LaTex or any text where multiple symbol $ are used.

8. NLP text preprocessing: Replace URL String

from textacy.preprocessing.replace import replace_urls

%time text = replace_urls(long_s,replace_with= '_URL_')

print('size: {:g} {}'.format(len(text),text[:text_l]))

output =>

CPU times: user 649 ms, sys: 112 µs, total: 649 ms

Wall time: 646 ms

size: 763800

:( 😻 😈 888 eihtg DoD Fee _URL_ ## Document Title :(

9. NLP text preprocessing: Remove URL String

from textacy.preprocessing.replace import replace_urls

%time text = replace_urls(long_s,replace_with= '')

print('size: {:g} {}'.format(len(text),text[:text_l]))

output =>

CPU times: user 633 ms, sys: 1.35 ms, total: 635 ms

Wall time: 630 ms

size: 744800

:( 😻 😈 888 eihtg DoD Fee ## Document Title :(

The rate for URL replace or removal is about 4,000 URLs per 1 million characters per second. Fast enough for 10 books in a corpus.

10. NLP text preprocessing: Replace E-mail string

%time text = textacy.preprocessing.replace.replace_emails(long_s)

print('size: {:g} {}'.format(len(text),text[:text_l]))

output =>

CPU times: user 406 ms, sys: 125 µs, total: 406 ms

Wall time: 402 ms

size: 725800

:( 😻 😈 888 eihtg DoD Fee ## Document Title :( cat-

nip immed-

natedly 2nd levelheading . , # _EMAIL_ _EMAIL_ can't Be a ckunk. $4 $123,456 won't seven $Shine $$beighty?$

The rate for email reference replace is about 8,000 emails per 1.7 million characters per second. Fast enough for 17 books in a corpus.

11. NLP text pre-processing: Remove E-mail string

from textacy.preprocessing.replace import replace_emails



%time text = textacy.preprocessing.replace.replace_emails(long_s,replace_with= '')

print('size: {:g} {}'.format(len(text),text[:text_l]))

output =>

CPU times: user 413 ms, sys: 1.68 ms, total: 415 ms

Wall time: 412 ms

size: 672600 :( 😻 😈 888 eihtg DoD Fee ## Document Title :( cat-

nip immed-

natedly 2nd levelheading . , # can't Be a ckunk. $4 $123,456 won't seven $Shine $$beighty?$

The rate for email reference removal is about 8,000 emails per 1.1 million characters per second. Fast enough for 11 books in a corpus.

12. NLP text preprocessing: normalize_hyphenated_words

from textacy.preprocessing.normalize import normalize_hyphenated_words

%time long_s = normalize_hyphenated_words(long_s)

print('size: {:g} {}'.format(len(long_s),long_s[:text_l])))

output =>

CPU times: user 186 ms, sys: 4.58 ms, total: 191 ms

Wall time: 190 ms

size: 642200 :

( 😻 😈 888 eihtg DoD Fee ## Document Title :( catnip immednatedly

Approximately 8,000 hyphenated-words, cat — nip and immed- iately (mispelled) were corrected in a corpus of 640,000 characters in 190 ms or abouut 3 million per second.

13. NLP text preprocessing: Convert all characters to lower case

### - **all characters to lower case;**

%time long_s = long_s.lower()

print('size: {:g} {}'.format(len(long_s),long_s[:text_l]))

output =>

CPU times: user 4.82 ms, sys: 953 µs, total: 5.77 ms

Wall time: 5.97 ms

size: 642200

:( 😻 😈 888 eihtg dod fee ## document title :( catnip immednatedly 2nd levelheading . , # can't be a ckunk. $4 $123,456 won't seven $shine $$beighty?$

I only benchmark the .lower Python function. The rate for lower case transformation by .lower() of a Python string of a million characters is about 6 ms. A rate that far exceeds our production rate requirements.

14. NLP text preprocessing: Whitespace Removal

%time text = re.sub(' +', ' ', long_s)

print('size: {:g} {}'.format(len(text),text[:text_l]))

output =>

CPU times: user 44.9 ms, sys: 2.89 ms, total: 47.8 ms

Wall time: 47.8 ms

size: 570000

:( 😻 😈 888 eihtg dod fee ## document title :( catnip immednatedly 2nd levelheading . , # can't be a ckunk. $4 $123,456 won't seven $shine $$beighty?$

The rate is about 0.1 seconds for 1 million characters.

15. NLP text preprocessing: Whitespace Removal (slower)

from textacy.preprocessing.normalize import normalize_whitespace



%time text= normalize_whitespace(long_s)

print('size: {:g} {}'.format(len(text),text[:text_l]))

output =>

CPU times: user 199 ms, sys: 3.06 ms, total: 203 ms

Wall time: 201 ms

size: 569999

:( 😻 😈 888 eihtg dod fee ## document title :( catnip immednatedly 2nd levelheading . , # can't be a ckunk. $4 $123,456 won't seven $shine $$beighty?$

normalize_whitespce is 5x slower but more general. For safety in production, we use normalize_whitespce. To date, we do not think we had any problems with faster regex.

16. NLP text preprocessing: Remove Punctuation

from textacy.preprocessing.remove import remove_punctuation



%time text = remove_punctuation(long_s, marks=',.#$?')

print('size: {:g} {}'.format(len(text),text[:text_l]))

output =>