# Classification of Retail Products: From Probabilistic Ranking to Neural Networks
## 4 Proposed Model
#### Preprocessing  
As previously mentioned, MDD-DS consists of 20,888 products, which is the result of cleaning the original dataset (31,193 products) before applying the classification algorithm. This first cleaning procedure consisted of removing all the products with no name, no ingredients, and no legal name. After that, MDD-DS is pre-processed by extracting all the meaningful words for the attributes 'Name', 'Legal Name' and 'Ingredients'. For each product \(p\), the three attributes are merged into a single text which aims to describe the product \(des(p)\). This description (\(des(p)\)) is obtained after a cleansing process described [35] step by step in Algorithm 1: (i) parenthesis are transformed into blank spaces, (ii) numbers, stop-words, punctuation, and extra spaces are removed; (iii) all letters are converted to lowercase, and (iv) repeated strings are removed. ```
1:procedurePreprocess(MDD-DS)
2: cleaning(\(MDD\_DS\))
3:\(product\_words[]\leftarrow\text{new\_list}(m)\)
4:\(all\_products\_words\leftarrow\text{new\_vector}\) (0)
5:for\(i\gets 1:m\)do
6:\(p\leftarrow\text{MDD-DS}[i,\;]\)
7:\(des(p)\leftarrow\text{concatenate}(\)
8:\(p\)[Name], \(p\)[Legal Name], \(p\)[Ingredients]\()\)
9:\(des(p)\leftarrow\text{transform\_brackets\_into\_space}(des(p))\)
10:\(des(p)\leftarrow\text{transform\_into\_lowercase}(des(p))\)
11:\(des(p)\leftarrow\text{remove\_numbers}(des(p))\)
12:\(des(p)\leftarrow\text{remove\_stopwords}(des(p))\)
13:\(des(p)\leftarrow\text{remove\_punctuation}(des(p))\)
14:\(des(p)\leftarrow\text{remove\_extra\_spaces}(des(p))\)
15:\(des(p)\leftarrow\text{split\_by\_spaces}(des(p))\)
16:\(des(p)\leftarrow\text{remove\_duplicates}(des(p))\)
17:\(des(p)\leftarrow\text{remove\_empty\_words}(des(p))\)
18:\(product\_words[i]\gets des(p)\)
19:\(all\_products\_words\)
20:\(\leftarrow\)\(all\_products\_words\cup des(p)\)
21:endfor
22:endprocedure
```  
**Algorithm 1** Preprocessing pseudocode  
Then, words are split, and a vector of words \(product\_words(p)\) is generated, an example is shown in Table 3. Additionally, we also create the \(all\_products\_words\) (for all the products in MDD-DS). This is another vector of words that contains as many elements as different words in the products' vectors (from the fields Name, Legal Name, and Ingredients). After preprocessing, \(all\_products\_words\) contains a total of 11,359 unique words, an example is shown in Table 4. \begin{table}
\begin{tabular}{l l} \hline \hline
**Product Id** & **Product-Vector** \\ \hline
1 & [’oil’, ’olive’, ’virgin’,...] \\... &... \\
218 & [’pepper’, ’sweet’, ‘glass’,...] \\... &... \\
20,888 & [’biochips’, ‘cheese’, ‘spelt’, ‘oil’,...] \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Examples of \(product\_words\) for every \(p\). With the information provided by the _all_products_words_, we obtain the product matrix \(X[m,n]\), which is mathematically defined as follows. Let \(\vec{w}\) be the \(n\)-dimensional vector obtained from _all_products_words_ such that \(\vec{w}=(w_{1},\ldots,w_{n})\) and \(\forall\ k\in[1,\ldots,n],\ w_{k}\) is a string \(\in\)_all_products_words_ and \(N=\) dim(_all_products_words_) is the total number of different meaningful words in the dataset (after cleaning). * The product matrix \(X[m,n]\) is an \(MxN\) matrix, where each column represents a word \(w_{i}\in\vec{w}\), and each row represents a product \(p\) of the dataset so that \(M\) is the total number of products
* Let \(\vec{p}\) be the \(product\_words[]\) list such that each element contains a vector \(\vec{p}_{i}\), hence \(\vec{p}=(\vec{p}_{1},\ldots,\vec{p}_{m})\). Likewise, \(\vec{p}_{i}=(p_{i}[1],\ldots,p_{i}[l])\), where \(l=\) dim(\(product\_words[i]\)), length of the vector in the \(i\) element of the list \(\vec{p}\). Note that \(\forall\ k\in[1,\ldots,l],\ p_{i}[k]\) is a string. Therefore,  
\[X[i,j]=\left\{ \begin{array}{l}1,\text{ if }\vec{w}[j]\in\vec{p}[i]\\ 0,\text{ if }\vec{w}[j]\
ot\in\vec{p}[i]\end{array} \right.