## The Investigation Framework
### Framework Overview
#### 4.2.2 Module M1-Pre-Processor in Details  
When users write messages, they write punctuation, single characters and stop-words that are not useful for topic classification (and even decrease the accuracy of classification). So, before features are extracted, messages must be pre-processed in order to be cleaned from noise. Specifically, module _Pre-processor_ performs text tokenization, special symbol removal, stop-words filtering and stemming. Hereafter, we describe these activities in more details.  
Let us denote the input data set as \(T=\{lp_{1},lp_{2},\dots,\}\), where each \(lp_{i}\) is a labeled message, such that \(lp_{i}=\langle mt_{i},at_{i}\rangle\), i.e., \(mt_{i}\) is the message text and \(at_{i}\) is the label class or topic associated to the message.  
For each message \(lp\in T\), on its message text \(lp.mt\) the module performs the following processing steps, in order to generate the set \(T_{p}\) of pre-processed messages.  
1. The \(lp.mt\) message is tokenized, in order to represent it as a vector \(d_{1}\) of terms, where a term is a token found within the message text.  
Figure 3: The Investigation Framework.  
From vector \(d_{1}\), we obtain vector \(d_{2}\) by removing special symbols, punctuation marks, numbers and special characters.
3. Stop-word removal is performed, by comparing each term in \(d_{2}\) with \(NLTK\)[46], a static list of stop-words. We formalize this process by defining the recursive function \(RemoveSW\) hereafter. \[RemoveSW(d,pos)=\] \[=\left\{\begin{array}{ll}RemoveSW(remove(d,pos),pos)&\text{if }(1 \leq pos\leq|d|)\wedge(|d[pos]|\leq 2\lor d[pos]\in S_{L}))\\ RemoveSW(d,pos+1)&\text{if }(1\leq pos\leq|d|)\wedge-(|d[pos]|\leq 2\lor d[pos] \in S_{L}))\\ d&\text{if }\
eg(1\leq pos\leq|d|)\end{array}\right.\] where \(d\) is the message represented as a vector of terms, \(|d|\) is the size of the vector, \(pos\) denotes a position index, \(d[pos]\) denotes the term (string within vector \(d\) in position \(pos\)), \(|d[pos]|\) denotes the length of the term (string) in position \(pos\) in vector \(d\). Furthermore, \(S_{L}\) is the list of stop-words, while function \(remove(d,pos)\) removes the item in position \(pos\) from vector \(d\). The function is defined by the following formula. \[remove(d,pos)=\left\{\begin{array}{ll}d[1,(pos-1)]\bullet d[(pos+1),|d|]& \text{if }1<pos<\ |d|\\ d[2,|d|]&\text{if }pos=1\\ d[1,(pos-1)]&\text{if }pos=|d|\end{array}\right.\] where with \(d[i,j]\) we denote the sub-vector with items from position \(i\) to position \(j\) and the \(\bullet\) denotes an operator that concatenates two vectors. The \(d_{3}\) vector representing the message without stop-words is obtained by calling the \(RemoveSW\) function as \(d_{3}=RemoveSW(d_{2},1)\).
4. After stop-word removal, stemming is performed on vector \(d_{3}\) by applying the _Porter stemming algorithm_[47]; we obtain the final \(d_{4}\) vector, i.e., the vector of terms that represent the \(lp.mt\) message text after pre-processing. The \(d_{4}\) vector is paired with the class label \(lp.at\), obtaining the pair \(lm_{p}=\langle d_{4},lp.at\rangle\) that is inserted into \(T_{p}\), the set of pre-processed messages. \(T_{p}\) is the final output of this module: the source data set \(T\) has been transformed into \(T_{p}\), where instead of strings, message texts are represented by vectors of terms.