# Systematic Comparison of Vectorization Methods in Classification Context
## 3 Methodology
### Text Preparation  
Performing the learning process on text data requires the appropriate prior preparation. Each document is transformed in the same way so that it can be represented in vector form, then placed in the appropriate analysis space. We refer to the basic operations that a text document can undergo shallow text analysis and the following transformations, performed in the following order:  
1. _Removal of special characters_--most electronic texts contain characters related to text formatting, whose removal is important for optimal processing.
2. _Uniform size of letters_--since all text processing algorithms are case-sensitive, it is necessary to force all letters to be the same size.
3. _Remove punctuation and quotation marks_--punctuation and quotation marks in sentences play a communicative role. In automatic natural language analysis, it is necessary to decide whether the value added by punctuation is large enough. Leaving punctuation unmodified increases the dataset size to be processed and further complicates the computation. In the case of classifying the topic of an utterance, it is not so important. It is different when we analyze the emotional coloring of an utterance (whether it is positive or negative). Then, the punctuation, especially emotional signs (exclamation marks, question marks, ellipsis), may prove crucial for the interpretation of the text.
4. _Remove possessive pronouns_--they indicate belonging to someone or something. In the framework of automatic text analysis, especially the classification of the subject of the text, the information about affiliation is not necessary, but as in the case of punctuation marks, each case of automatic text analysis should be considered individually in terms of the usefulness of this information.
5. _Lemmatization_--is the process by which we replace words with their base form or lemma. The use of lemmatization is important because the same word can take different forms, each of which carries the same information, and unfortunately, the machine treats them as completely different words. In the case of English, we lemmatize only the verbs. In order to determine what part of speech a given word is and what its lemma is, dictionary methods are most often used. They consist in storing in memory the entire dictionary for a given language. In practice, dictionary methods are often imperfect and may not contain a complete dictionary due to the high dynamics of language change; this may be especially important when analyzing informal or youth texts.
6. _Removal of \"stop words\"_--stop words are words that occur relatively often in the studied set of documents and, at the same time, are irrelevant because they do not carry information. For each language, there are commonly available sets of such words. Removing them from the text allows the statistical models needed for classification to work correctly but may completely change the meaning of the text. The step of removing the \"stop words\" in preprocessing the text can be as follows: stop_words = list(stopwords.words('english')) df['text_6'] = df['text_5'] for stop_word in stop_words: regex_stopword = r^\b\" + stop_word + r^\b\" df['text_6'] = df['text_6'].str.replace(regex_stopword, '') The last step is to re-save the result of all performed operations to \"_csv_ files--the first file contains the texts together with all data preprocessing steps, while the second contains only category information, original text, and the result of all processing.