## 3 Proposed Approach
### Adversarial Document-Level Neural Network Architecture  
The original GAN formulation [4], presented in Figure 1, consists of a min-max adversarial game between a generative model \(G\) and a discriminative model \(D\). \(G(z)\) is a neural network that is trained to map samples \(z\) from a prior noise distribution \(p(z)\) to the data space. \(D(x)\) is another neural network that takes a data sample \(x\) as input and outputs a single scalar value representing the probability that \(x\) came from the data distribution instead of \(G(z)\). \(D\) is trained to maximize the probability of assigning the correct label to the input \(x\), while \(G\) is trained to maximally confuse \(D\), using the gradient of \(D(x)\) with respect to \(x\) to update its parameters. This min-max game can be optimized by the following risk, given by Equation (1), and is typically implemented with neural network models; however,  
Figure 1: The architecture of a generative adversarial network (GAN) model, consisting of a discriminator \(D\) and a generator \(G\) model which are learned during the training process of a GAN. these models could be implemented by any form of differentiable system that maps data from one space to another. \[\phi=min_{G}max_{D}E_{x\sim p(data)}[logD(x)]+E_{z\sim p(z)}[log(1-D(G(z)))] \tag{1}\]  
One shortcoming of this model is that there are no explicit means for inference, and so it is unclear how GANs could be applied to unsupervised representation learning. In [4], two possible solutions were suggested and have been explored by the research community in subsequent works. The first approach is to train another network to do inference, learning a mapping from \(x\) back to \(z\)[55], with a variation of this method being to instead use the adversarial training process to regularize an autoencoder's representation layer [56]. The second idea is to use internal components of the discriminator network as a representation [5]. Nevertheless, both approaches fail to result in an architecture with stable training across a range of datasets and model hyperparameters when using a probabilistic discriminator network. Thus, our approach to improving GAN training is to assess the empirical symptoms that are experienced during training [5] by switching to use the energy-based GAN architecture, where the discriminator is an autoencoder [10]. Document representations can then be formed from the encoded representation of the discriminator. Particularly, our proposed model's architecture is formulated as follows: Let \(x_{i}\in\{0,1\}^{V}\) be the binary bag-of-words representations of a document, where \(V\) is the vocabulary size and \(x_{i}\) is a binary value indicating whether the \(i\)th word is present in the document or not. We define a feedforward generator network \(G(z)\) that takes a vector \(z\in R^{h_{g}}\) as input and produces a vector \(\hat{x}\in R^{V}\), with \(h_{g}\) being the number of dimensions in the input noise vector (sampled from \(N(0,1)\)). We also define a discriminator network \(D(x)\), seen as an energy function, that takes vectors \(x\in R^{V}\) and produces an energy estimate \(E\in R\). One main difference compared to the work of [10] is that we used a denoising autoencoder (DAE) as our energy function, as the DAE has been found to produce superior representations to the standard autoencoder [57]. In this work, we used single encoding and decoding layers, so the encoding process is given in Equation (2):  
\[h=f(W^{e}x^{c}+b_{e}) \tag{2}\]  
where \(W^{e}\) is a set of learned parameters referring to a deterministic mapping from a data space into the latent/representation space, \(b_{e}\) is a learned bias term, \(f\) is a nonlinearity, \(x^{c}\) is a corrupted version of \(x\), and \(h\in R^{h_{d}}\) is the hidden representation of size \(h_{d}\), with \(h_{d}\) being the size of the denoising autoencoder (DAE) hidden state. The decoding process is then given by:  
\[y=W^{d}h+b_{d} \tag{3}\]  
where \(W^{d}\) and \(b_{d}\) are another learned set of weights and bias terms.