Normalizing Textual Data with Python

07 Jul, 2021
In this article, we will learn How to Normalizing Textual Data with Python. Let’s discuss some concepts :
Textual data ask systematically collected material consisting of written, printed, or electronically published words, typically either purposefully written or transcribed from speech.
Text normalization is that the method of transforming text into one canonical form that it’d not have had before. Normalizing text before storing or processing it allows for separation of concerns since the input is sure to be consistent before operations are performed thereon. Text normalization requires being conscious of what sort of text is to be normalized and the way it’s to be processed afterwards; there’s no all-purpose normalization procedure.
Steps Required
Here, we will discuss some basic steps need for Text normalization.
Input text String,
Convert all letters of the string to one case(either lower or upper case),
If numbers are essential to convert to words else remove all numbers,
Remove punctuations, other formalities of grammar,
Remove white spaces,
Remove stop words,
And any other computations.
We are doing Text normalization with above-mentioned steps, every step can be done in some ways. So we will discuss each and everything in this whole process.
Text String
Python3
# input stringstring = "       Python 3.0, released in 2008, was a major revision of the language that is not completely backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g. Windows 7 (and old installers not restricted to 64-bit Windows)."print(string)
Output:
”       Python 3.0, released in 2008, was a major revision of the language that is not completely backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2’s end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g. Windows 7 (and old installers not restricted to 64-bit Windows).”
Case Conversion (Lower Case)
In Python, lower() is a built-in method used for string handling. The lower() methods returns the lowercased string from the given string. It converts all uppercase characters to lowercase. If no uppercase characters exist, it returns the original string.
Python3
# input stringstring = "       Python 3.0, released in 2008, was a major revision of the language that is not completely backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g. Windows 7 (and old installers not restricted to 64-bit Windows)." # convert to lower caselower_string = string.lower()print(lower_string)
Output:
”       python 3.0, released in 2008, was a major revision of the language that is not completely backward compatible and much python 2 code does not run unmodified on python 3. with python 2’s end-of-life, only python 3.6.x[30] and later are supported, with older versions still supporting e.g. windows 7 (and old installers not restricted to 64-bit windows).”
Removing Numbers
Remove numbers if they’re not relevant to your analyses. Usually, regular expressions are used to remove numbers.
Python3
# import regeximport re # input stringstring = "       Python 3.0, released in 2008, was a major revision of the language that is not completely backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g. Windows 7 (and old installers not restricted to 64-bit Windows)." # convert to lower caselower_string = string.lower() # remove numbersno_number_string = re.sub(r'\d+','',lower_string)print(no_number_string)
Output:
”       python ., released in , was a major revision of the language that is not completely backward compatible and much python  code does not run unmodified on python . with python ‘s end-of-life, only python ..x[] and later are supported, with older versions still supporting e.g. windows  (and old installers not restricted to -bit windows).”
Removing punctuation
The part of replacing with punctuation can also be performed using regex. In this, we replace all punctuation by empty string using certain regex.
Python3
# import regeximport re # input stringstring = "       Python 3.0, released in 2008, was a major revision of the language that is not completely backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g. Windows 7 (and old installers not restricted to 64-bit Windows)." # convert to lower caselower_string = string.lower() # remove numbersno_number_string = re.sub(r'\d+','',lower_string) # remove all punctuation except words and spaceno_punc_string = re.sub(r'[^\w\s]','', no_number_string)print(no_punc_string)
Output:
‘       python  released in  was a major revision of the language that is not completely backward compatible and much python  code does not run unmodified on python  with python s endoflife only python x and later are supported with older versions still supporting eg windows  and old installers not restricted to bit windows’
Removing White space
The strip() function is an inbuilt function in Python programming language that returns a copy of the string with both leading and trailing characters removed (based on the string argument passed).
Python3
# import regeximport re # input stringstring = "       Python 3.0, released in 2008, was a major revision of the language that is not completely backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g. Windows 7 (and old installers not restricted to 64-bit Windows)." # convert to lower caselower_string = string.lower() # remove numbersno_number_string = re.sub(r'\d+','',lower_string) # remove all punctuation except words and spaceno_punc_string = re.sub(r'[^\w\s]','', no_number_string) # remove white spacesno_wspace_string = no_punc_string.strip()print(no_wspace_string)
Output:
‘python  released in  was a major revision of the language that is not completely backward compatible and much python  code does not run unmodified on python  with python s endoflife only python x and later are supported with older versions still supporting eg windows  and old installers not restricted to bit windows’
Removing Stop Words
Stop words” are the foremost common words during a language like “the”, “a”, “on”, “is”, “all”. These words don’t carry important meaning and are usually faraway from texts. It is possible to get rid of stop words using tongue Toolkit (NLTK), a set of libraries and programs for symbolic and statistical tongue processing.
Python3
# download stpwordsimport nltknltk.download('stopwords') # import nltk for stopwordsfrom nltk.corpus import stopwordsstop_words = set(stopwords.words('english'))print(stop_words) # assign stringno_wspace_string='python  released in  was a major revision of the language that is not completely backward compatible and much python  code does not run unmodified on python  with python s endoflife only python x and later are supported with older versions still supporting eg windows  and old installers not restricted to bit windows' # convert string to list of wordslst_string = [no_wspace_string][0].split()print(lst_string) # remove stopwordsno_stpwords_string=""for i in lst_string:    if not i in stop_words:        no_stpwords_string += i+' '         # removing last spaceno_stpwords_string = no_stpwords_string[:-1]print(no_stpwords_string)
Output: 
In this, we can normalize the textual data using Python. Below is the complete python program:
Python3
# import regeximport re # download stpwordsimport nltknltk.download('stopwords') # import nltk for stopwordsfrom nltk.corpus import stopwordsstop_words = set(stopwords.words('english'))  # input stringstring = "       Python 3.0, released in 2008, was a major revision of the language that is not completely backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g. Windows 7 (and old installers not restricted to 64-bit Windows)." # convert to lower caselower_string = string.lower() # remove numbersno_number_string = re.sub(r'\d+','',lower_string) # remove all punctuation except words and spaceno_punc_string = re.sub(r'[^\w\s]','', no_number_string) # remove white spacesno_wspace_string = no_punc_string.strip()no_wspace_string # convert string to list of wordslst_string = [no_wspace_string][0].split()print(lst_string) # remove stopwordsno_stpwords_string=""for i in lst_string:    if not i in stop_words:        no_stpwords_string += i+' '         # removing last spaceno_stpwords_string = no_stpwords_string[:-1] # outputprint(no_stpwords_string)
Output:
 
adnanirshad158
Picked
python-string
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.