## 4 Experimental Validation
### Implementation  
We further preprocessed all three datasets in order to clean them of any noisy data to further reduce the complexity of our datasets. We decided to remove irrelevant features, such as common stop words (such as I or and); we tokenized the datasets to split up their words into terms of tokens; and we stemmed the latter to reduce the tokens into a single type, normally a root word. As such, the stemming process reduced redundant words in our datasets. Our aim is to create a vocabulary of approximately 40,000 different word stems. From these, approximately 2000 of the most popular ones were kept in the training datasets. As a result, each posting is represented as a vector containing a 2000 word count. Even though word embedding is currently the-state-of-art of the NLP field in resolving text-related problems, due to a number of limitations that presented, we decided to use bag-of-words (BoW). Particularly, one limitation of individual word embeddings is their inability to represent phrases where the combination of two or more words does not represent the combination of the meanings of the individual words [60]. Another limitation comes from learning embeddings based only a small window of surrounding words: sometimes words, such as _good_ and _bad_, share almost the same embedding [61], which is problematic if used in tasks such as SA [62]. Moreover, a general caveat for word embeddings is that they are highly dependent on the applications in which they are used [63]; as such, BoW works better than the current word-embedding models (Word2 Vec, GloVe) for our examined scenario. We trained our networks using Keras [64] with Tensorflow as the back-end [65]. Following [66], to make a direct comparison, we set our representation size \(h_{d}\) (the size of the denoising autoencoder (DAE) hidden state) to 50. The generator input noise vector \(h_{g}\) was also set to be the same size. The generator is a three-layer feedforward network, with rectified linear unit (ReLU) activations in the first two layers, and a sigmoid nonlinearity in the output layer. Layers 1 and 2 are both of size 300, with the final layer being the same size as the vocabulary. Layers 1 and 2 use batch normalization [15]. We decided to use the latter, as it has been recommended for use in both networks in order to stabilize training in deeper models. The discriminator encoder consists of a single linear layer followed by a leaky ReLU nonlinearity function (with a leak of 0.02). We decided to do so, as it has been shown that using a leaky ReLU activation functions between the intermediate layers of the discriminator, giving a superior performance to that when using regular ReLUs [5]. The decoder is a linear transformation back to the vocabulary size. We optimized both \(G\) and \(D\) using the first-order gradient-based Adam optimizer [67] with an initial learning rate of 0.0001. Our denoising autoencoder (DAE) corruption process was to randomly set to zero 40% of the input values, and we used a margin size \(m\) of 5% of the vocabulary size. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Datasets** & **Training** & **Validation** & **Test** \\ \hline
20 Newsgroup ([[http://qwone.com/](http://qwone.com/)]([http://qwone.com/](http://qwone.com/))\textasci{-}jason/20 Newsgroups/) & 10.163 & 1130 & 7528 \\ Movie Reviews [58] & 7424 & 76 & 2500 \\ Finegrained Dataset (FSD) [59] & 2582 & 287 & 956 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Test collections for experiments on the use of generative adversarial networks to learn distributed representations of natural language at the document, sentence, and aspect level. We followed the same validation procedure as [PERSON] and [PERSON] (2016), and we set the validation set to perform model selection of other hyperparameters, such as the learning rate and the number of learning passes over the training set (based on early stopping).