# Automated Design of the Deep Neural Network Pipeline
## Appendix A.1.2 Text Processing
### Feature Engineering  
Feature engineering is specific to text processing. Techniques generally used include:  
1. One hot encoding creates a vector representation for each word within the text. The vector representation will have elements that are either zero or one only, and each position in the vector corresponds to a position within the text. If the word for which we are creating the vector appears in a specific position in the text, the same position in the vector will have a one. Conversely, if the word for which we are creating the vector does not appear in a specific position within the text, the same position in the vector will have a 0. A vector representation is created for each and every word within the text, including punctuation, if it was not removed during preprocessing. The resulting collection of vectors will be the feature representation.
2. Term frequency--inverse document frequency (TF-IDF) [66] calculates how frequently a word occurs in a single textual instance (term frequency) and then multiplies that value by the inverse of how frequently that word occurs across multiple texts (inverse document frequency).
3. Word2 Vec (CBOW model) [67] uses cosine similarity to group similar words together. A word vector (the context) will represent the words that are most similar to the word. Each unique word in the text as a whole will have a word vector generated for it. Word2 Vec makes use of a two-layer neural network to generate the word vectors (CBOW) architecture and the second the Skip-gram architecture. In the CBOW architecture, the neural network is given a specific context (collection of words) from the text as a whole and attempts to predict the single word that would fit that context. If the prediction is incorrect, the vector for the single word is adjusted accordingly.
4. Word2 Vec (Skip-gram model) [67] uses cosine similarity to group similar words together. A word vector (the context) will represent the words that are most similar to the word. Each unique word in the text as a whole will have a word vector generated for it. Word2 Vec makes use of a two-layer neural network to generate the word vectors, and has two distinct variations in architecture, the first being the continuous bag of words (CBOW) architecture and the second the Skip-gram architecture. In the Skip-gram architecture, the neural network is given a single word and attempts to predict the context (collection of words) within the text as a whole that the single word would occur in. If the prediction is incorrect, the vector for the single word is adjusted accordingly.
5. GloVe [68] makes use of unsupervised learning and, like TF-IDF, makes use of word frequency vectors for each word. The GloVe technique will then calculate the dot product of two word vectors and compare it to the log of the amount of times the two words occur near one another. The two word vectors are then adjusted based on how different the dot product is from the log result. The concept of how \"near\" two words are to one another is controlled by a hyper-parameter.  
FastText [69] extends the Word2 Vec model by encoding the individual words into their n-gram representation before training the neural network. An n-gram is created by sampling a number of contiguous characters from a word. The number of characters to be sampled is greater than one and smaller than or equal to n. The use of n-grams for the representation allows for longer words to be broken up into their prefixes and suffixes and shorter words to be better explicated.