Tokenize Text Columns Into Sentences in Pandas
By: Baris Sari
Tags: Spacy, Python, Pandas, Regex, Tokenization

READ CSV

I’ll keep this very short. Put scripts.csv to the same directory with your Python script or Jupyter notebook and then run the following commands:

give the path of CSV file to FILE_PATH variable

You can directly give scripts.csv to Pandas’s read_csv() but it is always more robust to use os or glob to give file path. It’s also a good practice for production. You might forget to change all of magic filenames if you use strings.

Let’s create a DataFrame from our CSV file and assign the first row’s Dialogue column to first_dialogue variable.

we will use “first_dialogue” in sentence tokenization section

So, we are ready to try different sentence tokenizers. This might be a great example because it has many punctuations such as three dots, exclamation mark, question mark; long sentences; sentences without punctuation between them; etc.

Sentence Tokenization

1. Tokenize an example text using Python’s split()

This will be a naive method, which you should never use for sentence tokenization! I prefer using built-in functions as much as possible in my projects. But if it doesn’t fit, then don’t use it, check for the alternatives (you will probably find a better library because Python has a vast community and lots of great open-source libraries!). So, you might use the following gist, which will only work if you are very lucky and all of your sentences end with “.”:

Not so good, right? I see that split() is used in many articles for word tokenization. Which might be acceptable because it splits texts by taking care of extra spaces, etc. For sentence tokenization, it just doesn’t work. You might use replace() and then split() to replace all end-of-line characters with one character and split the text into sentences using that character. It would give a better result, but the performance of your code would decrease.

Another problem is we’re losing the character we used for splitting. If we aggregated the transformed data, we wouldn’t have the original punctuations any more.

2. Tokenize an example text using regex

Regular expression is always useful if you’re working on texts. It’s a quite old and robust approach. Many programming languages offer it natively. So, you can use your regex across different languages with small or no changes.

Let’s give it a try to one of the accepted answers on Stackoverflow (with a small change, by adding |! to the third group):

Let me explain the regex pattern: (?<!\w\.\w.)(?<![A-Z][a-z]\.)(?<=\.|\?|!)\s step by step:

(?<!X)Y is called negative lookbehind. It tries to capture Y (any whitespace character \s in our case) where preceded characters of Y do not match with X. For example, let’s say you have this regex: (?<!TEST)MATCH . If your text is “TESTMATCH”, it will not match, if it’s “RANDOM_MATCH” (preceded characters are not “TEST”), it will match.

As you may see, all “MATCH” values are matched but the first row because it has “TEST” as preceded characters

(?<=X)Y is called positive lookbehind. It tries to capture Y (any whitespace character \s in our case) where preceded characters of Y match with X. Let’s use the same example as above and check the outcomes:

This time only the first row is captured because the rest of the rows don’t have “TEST” as preceded characters

In our regex, we have three capturing groups: the first two are looking for negative lookbehind and the last one is looking for a positive lookbehind. Now, let’s look at the inside of these capturing groups to understand what kind of strings we’re looking for:

Inside of the first capturing group is as follows: \w\.\w. . \w matches any word character (letter, number, or underscore). \. matches literal dot character (backslash is the escape character in regex). . matches any character. An example match: i.e. MATCH

. matches any word character (letter, number, or underscore). matches (backslash is the escape character in regex). matches any character. An example match: Inside of the second capturing group is [A-Z][a-z]\. . It matches any lowercase or uppercase letter following with dot character. An example match: Mr. MATCH .

. It matches any lowercase or uppercase letter following with dot character. An example match: . Inside of the third capturing group is \.|\?|! . We already know that \ is used for literal characters. | means to match either the first part or the latter part. In other words, (A|B|C) means either match A, B or C. In our case, it will match either dot, question, or exclamation mark.

We may finally know all we need to know about our regex. To sum up, we’re searching for whitespace characters by checking its preceded characters. If it passes from our three validations, we match it. So, we can split the text from these matches.

The disadvantage of using regex is:

It might be quite painful to explain what is going on! It’s not intuitive . You should spend some time to learn it, otherwise, it will look scary.

. You should spend some time to learn it, otherwise, it will look scary. You might create infinite loops or low-performance regular expressions if you’re not much familiar with it.

with it. It’s not easy to cover edge cases before you face with them . When you start to write a regex, you might easily miss some of these cases.

. When you start to write a regex, you might easily miss some of these cases. Our regex relies on whitespace characters. If there is no whitespace between two sentences, then it doesn’t match anything. So, it might fail if your text is not properly formatted. For example, if you give this text to our regex, “This is an example sentence.This is another one without space after the dot.”, it will think it’s one sentence.

The advantage of using regex is:

You don’t need to rely on third-party libraries .

to rely on . It’s lightning fast because just as many of Python’s built-in functions, re module is converted and executed on C.

because just as many of Python’s built-in functions, module is converted and executed on C. It exists since the 1950s, and it’s a quite known thing for string search, replace, etc. You can write one regex in Python and use it in Javascript with minor changes. You only need to know the programming language specific behaviours, the rest will be easy. But, you can’t use a Python library in Javascript. If you can, you will need a workaround, or you will use a Python API etc.

3. Tokenize an example text using spaCy

spaCy is capable of preprocessing texts in many languages. It offers tokenization, lemmatization, linguistic features, creating pipelines, training, running on GPU, etc. So, it’s a resourceful and powerful library. If you have unstructured text data such as scraped texts from the web, I’d suggest using it without any hesitation. If you think of only doing basic preprocessing, it might add extra complexity which you don’t actually need.

Before you try spaCy’s tokenization, you should download the library and one of its models. We will use the small English model which will be sufficient for our task.

Download library and language model, then tokenize first_dialogue

This approach takes a long time. However, this is a bit misleading because spaCy might work much faster. It took a long time because we need to convert each value to an nlp object and we used the default sentence segmentation component. It is called DependencyParser. It’s possible to modify -to remove dependency parser and use custom segmentation- nlp() object and define it without using any model. Let’s use sentencizer pipeline component this time:

As you can see, there is a trade-off between the speed and the result. sentencizer struggled just as nltk and regex because it uses a rule-based strategy.

The advantage of using spaCy is:

It is a well-documented library which is maintained actively by a large community.

library which is by a large community. It offers tons of NLP functionalities that you might use.

that you might use. It supports many languages such as German, Dutch, French, and Chinese.

The disadvantage of using spaCy is:

It’s slower than re module in normal usage.

in normal usage. It might be overkill to include it to your project and use it only for tokenization.

4. Tokenize an example text using nltk

nltk is another NLP library which you may use for text processing. It is natively supporting sentence tokenization as spaCy. To use its sent_tokenize function, you should download punkt (default sentence tokenizer).

nltk tokenizer gave almost the same result with regex. It struggled and couldn’t split many sentences.