## 3 EssayGAN
### Generator and Reinforcement Learning  
Figure 3 depicts the architecture of the \(i\)-th generator assigned to generate essays scored as \(c_{i}\). We trained \(r\) generators to generate an essay with a specified score. The value of \(r\) is determined by the range of scores, which are specified in a scoring rubric. Figure 2: The architecture of the discriminator used in EssayGAN. We utilized LSTM networks as a basic architecture of the generators. The LSTM networks were initially pre-trained with a sentence-level language model using a training dataset and using a conventional maximum likelihood estimation (MLE) method. Therefore, the pre-trained LSTM networks can predict the most likely next sentence based on previously selected sentences. After the pre-training phase, adversarial training was employed to train the generators and the discriminator in turn. The output layer of each LSTM cell has the same dimensions as a sentence-level one-hot vector that can identify a specific sentence. Each LSTM cell, \(h_{t}\), can be recursively defined according to Equation (1), and a predicted sentence of the LSTM cell can be defined as in Equation (2). \[h_{t}=LSTM(h_{t-1},E_{s_{t}}), \tag{1}\]  
\[p(\hat{s}_{t+1}S_{0},s_{1},\ldots,s_{t})=softmax(Vh_{t}+b), \tag{2}\]  
where \(E_{s_{t}}\) is the embedding vector of the \(t\)-th sentence \(s_{t}\) and \(S_{0}\) is a start state. \(V\) is a weight matrix and \(b\) is a bias. The next sentence is chosen through random sampling based on the expectation probability. A new essay is organized according to the sequence of sentences generated by LSTMs. There is an obstacle involved in applying adversarial training to essay generation. The discriminator can only provide a reward value for a completed essay, whereas the generators require a reward value for incomplete essays at every sampling step. Therefore, to reward the generators at every sampling step, we applied a Monte Carlo search [17] to estimate the next unknown sentence to complete an essay. The parameter \(\theta_{i}\) of the \(i\)-th generator \(G_{i}\) was updated by means of the REINFORCE algorithm [18] using rewards provided by the discriminator \(D_{\phi}\). The objective of the generator \(G_{i}\) is to maximize the expected reward of Equation (3), in which \(R_{n}\) is the reward for a completed essay with the length \(n\) and \(Q_{D_{\phi}}^{G_{\theta_{i}}}(s,a)\) is the action-value function of a sequence, i.e., the expected accumulative reward starting from state \(s\), taking action \(a\) and following the policy \(G_{\theta_{i}}\). \[J(\theta_{i})=\mathbb{E}[R_{n}S_{0}\cdot\theta_{i}]=\sum_{t=1}^{t=n}G_{\theta_ {i}}(s_{t}S_{1:t-1})\cdot Q_{D_{\phi}}^{G_{\theta_{i}}}(S_{1:t-1},s_{t}), \tag{3}\]  
where \(S_{0}\) is the start state, \(S_{1:t-1}\) is the previously generated sentence (current state), and \(s_{t}\) is the selected current sentence. Figure 3: The architecture of the \(i\)-th generator in EssayGAN. The action-value function \(Q^{G_{\beta_{i}}}_{D_{\phi}^{\phi}}(s,a)\) is estimated as the output value of the discriminator, as defined in Equation (4). \(D_{\phi}(S_{1:n})\) is the probability that an essay consisting of \(n\) sentences \((s_{1},s_{2},\ldots,s_{n-1},s_{n})\) is real. \[Q^{G_{\beta_{i}}}_{D_{\phi}^{\phi}}(S_{1:n-1},s_{n})=D_{\phi}(S_{1:n}), \tag{4}\]  
where \(S_{1:n-1}\) is a state which specifies a generated sentence \((s_{1},s_{2},\ldots,s_{n-1})\), whereas \(s_{n}\) is an action which selects the \(n\)-th sentence. To evaluate the action-value for an intermediate state, we applied a Monte Carlo search with a roll-out policy \(G_{\beta}\) to sample the last unknown \(n-t\) sentences. In this work, \(G_{\beta_{i}}\) has the same parameters as those of the generator \(G_{\theta_{i}}\). Hence, the action-value function can be calculated using Equation (5). \[Q^{G_{\beta_{i}}}_{D_{\phi}}(S_{1:t-1},s_{t})=