NLP | How tokenizing text, sentence, words works

05 Nov, 2021
Natural Language Processing (NLP) is a subfield of computer science, artificial intelligence, information engineering, and human-computer interaction. This field focuses on how to program computers to process and analyze large amounts of natural language data. It is difficult to perform as the process of reading and understanding languages is far more complex than it seems at first glance.
Tokenization is the process of tokenizing or splitting a string, text into a list of tokens. One can think of token as parts like a word is a token in a sentence, and a sentence is a token in a paragraph.
Key points of the article –
Text into sentences tokenization
Sentences into words tokenization
Sentences using regular expressions tokenization
Code #1: Sentence Tokenization – Splitting sentences in the paragraph
from nltk.tokenize import sent_tokenize  text = "Hello everyone. Welcome to GeeksforGeeks. You are studying NLP article"sent_tokenize(text)
Output :
['Hello everyone.',
 'Welcome to GeeksforGeeks.',
 'You are studying NLP article']

How sent_tokenize works ?The sent_tokenize function uses an instance of PunktSentenceTokenizer from the nltk.tokenize.punkt module, which is already been trained and thus very well knows to mark the end and beginning of sentence at what characters and punctuation. Code #2: PunktSentenceTokenizer – When we have huge chunks of data then it is efficient to use it.
import nltk.data  # Loading PunktSentenceTokenizer using English pickle filetokenizer = nltk.data.load('tokenizers/punkt/PY3/english.pickle')  tokenizer.tokenize(text)
Output :
['Hello everyone.',
 'Welcome to GeeksforGeeks.',
 'You are studying NLP article']

 Code #3: Tokenize sentence of different language – One can also tokenize sentence from different languages using different pickle file other than English.
import nltk.data  spanish_tokenizer = nltk.data.load('tokenizers/punkt/PY3/spanish.pickle')  text = 'Hola amigo. Estoy bien.'spanish_tokenizer.tokenize(text)
Output :
['Hola amigo.', 
 'Estoy bien.']

 Code #4: Word Tokenization – Splitting words in a sentence.
from nltk.tokenize import word_tokenize  text = "Hello everyone. Welcome to GeeksforGeeks."word_tokenize(text)
Output :
['Hello', 'everyone', '.', 'Welcome', 'to', 'GeeksforGeeks', '.']

How word_tokenize works?word_tokenize() function is a wrapper function that calls tokenize() on an instance of the TreebankWordTokenizer class. Code #5: Using TreebankWordTokenizer
from nltk.tokenize import TreebankWordTokenizer  tokenizer = TreebankWordTokenizer()tokenizer.tokenize(text)
Output :
['Hello', 'everyone.', 'Welcome', 'to', 'GeeksforGeeks', '.']

These tokenizers work by separating the words using punctuation and spaces. And as mentioned in the code outputs above, it doesn’t discard the punctuation, allowing a user to decide what to do with the punctuations at the time of pre-processing. Code #6: PunktWordTokenizer – It doen’t separates the punctuation from the words.
from nltk.tokenize import PunktWordTokenizer  tokenizer = PunktWordTokenizer()tokenizer.tokenize("Let's see how it's working.")
Output :
['Let', "'s", 'see', 'how', 'it', "'s", 'working', '.']

 Code #6: WordPunctTokenizer – It separates the punctuation from the words.
from nltk.tokenize import WordPunctTokenizer  tokenizer = WordPunctTokenizer()tokenizer.tokenize("Let's see how it's working.")
Output :
['Let', "'", 's', 'see', 'how', 'it', "'", 's', 'working', '.']

 Code #7: Using Regular Expression
from nltk.tokenize import RegexpTokenizer  tokenizer = RegexpTokenizer("[\w']+")text = "Let's see how it's working."tokenizer.tokenize(text)
Output :
["Let's", 'see', 'how', "it's", 'working']

 Code #7: Using Regular Expression
from nltk.tokenize import regexp_tokenize  text = "Let's see how it's working."regexp_tokenize(text, "[\w']+")
Output :
["Let's", 'see', 'how', "it's", 'working']

Akanksha_Rai
marcosarcticseal
Natural-language-processing
Python-nltk
Machine Learning
Python
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Getting started with Machine Learning
Introduction to Recurrent Neural Network
Support Vector Machine Algorithm
Random Forest Regression in Python
Running Python script on GPU.
Read JSON file using Python
Python map() function
Adding new column to existing DataFrame in Pandas
Python Dictionary
How to get column names in Pandas dataframe