Question: What training settings did they try?
Evidence: Training and testing are done in alternating steps: In each epoch, for training, we first present to an LSTM network 1000 samples in a given language, which are generated according to a certain discrete probability distribution supported on a closed finite interval. We then freeze all the weights in our model, exhaustively enumerate all the sequences in the language by their lengths, and determine the first $k$ shortest sequences whose outputs the model produces inaccurately. We remark, for the sake of clarity, that our test design is slightly different from the traditional testing approaches used by BIBREF10 , BIBREF9 , BIBREF12 , since we do not consider the shortest sequence in a language whose output was incorrectly predicted by the model, or the largest accepted test set, or the accuracy of the model on a fixed test set.
It has been shown by BIBREF9 that LSTMs can learn $a^n b^n$ and $a^n b^n c^n$ with 1 and 2 hidden units, respectively. Similarly, BIBREF24 demonstrated that a simple RNN architecture containing a single hidden unit with carefully tuned parameters can develop a canonical linear counting mechanism to recognize the simple context-free language $a^n b^n$ , for $n \le 250$ . We wanted to explore whether the stability of the networks would improve with an increase in capacity of the LSTM model. We, therefore, varied the number of hidden units in our LSTM models as follows. We experimented with 1, 2, 3, and 36 hidden units for $a^n b^n$ ; 2, 3, 4, and 36 hidden units for $a^n b^n c^n$ ; and 3, 4, 5, and 36 hidden units for $a^n b^n c^n d^n$ . The 36 hidden unit case represents an over-parameterized network with more than enough theoretical capacity to recognize all these languages.
Following the traditional approach adopted by BIBREF7 , BIBREF12 , BIBREF9 and many other studies, we train our neural network as follows. At each time step, we present one input character to our model and then ask it to predict the set of next possible characters, based on the current character and the prior hidden states. Given a vocabulary $\mathcal {V}^{(i)}$ of size $d$ , we use a one-hot representation to encode the input values; therefore, all the input vectors are $d$ -dimensional binary vectors. The output values are $(d+1)$ -dimensional though, since they may further contain the termination symbol $\dashv $ , in addition to the symbols in $\mathcal {V}^{(i)}$ . The output values are not always one-hot encoded, because there can be multiple possibilities for the next character in the sequence, therefore we instead use a $k$ -hot representation to encode the output values. Our objective is to minimize the mean-squared error (MSE) of the sequence predictions. During testing, we use an output threshold criterion of $0.5$ for the sigmoid output layer to indicate which characters were predicted by the model. We then turn this prediction task into a classification task by accepting a sample if our model predicts all of its output values correctly and rejecting it otherwise.
Training and testing are done in alternating steps: In each epoch, for training, we first present to an LSTM network 1000 samples in a given language, which are generated according to a certain discrete probability distribution supported on a closed finite interval. We then freeze all the weights in our model, exhaustively enumerate all the sequences in the language by their lengths, and determine the first $k$ shortest sequences whose outputs the model produces inaccurately. 
experimented with 1, 2, 3, and 36 hidden units for $a^n b^n$ ; 2, 3, 4, and 36 hidden units for $a^n b^n c^n$ ; and 3, 4, 5, and 36 hidden units for $a^n b^n c^n d^n$ . 
Following the traditional approach adopted by BIBREF7 , BIBREF12 , BIBREF9 and many other studies, we train our neural network as follows. At each time step, we present one input character to our model and then ask it to predict the set of next possible characters, based on the current character and the prior hidden states. Given a vocabulary $\mathcal {V}^{(i)}$ of size $d$ , we use a one-hot representation to encode the input values; therefore, all the input vectors are $d$ -dimensional binary vectors. The output values are $(d+1)$ -dimensional though, since they may further contain the termination symbol $\dashv $ , in addition to the symbols in $\mathcal {V}^{(i)}$ . The output values are not always one-hot encoded, because there can be multiple possibilities for the next character in the sequence, therefore we instead use a $k$ -hot representation to encode the output values. Our objective is to minimize the mean-squared error (MSE) of the sequence predictions.
Answer: The answers are shown as follows:
* Training and testing are done in alternating steps: In each epoch, for training, we first present to an LSTM network 1000 samples in a given language, which are generated according to a certain discrete probability distribution supported on a closed finite interval. We then freeze all the weights in our model, exhaustively enumerate all the sequences in the language by their lengths, and determine the first $k$ shortest sequences whose outputs the model produces inaccurately. 
* experimented with 1, 2, 3, and 36 hidden units for $a^n b^n$ ; 2, 3, 4, and 36 hidden units for $a^n b^n c^n$ ; and 3, 4, 5, and 36 hidden units for $a^n b^n c^n d^n$ . 
* Following the traditional approach adopted by BIBREF7 , BIBREF12 , BIBREF9 and many other studies, we train our neural network as follows. At each time step, we present one input character to our model and then ask it to predict the set of next possible characters, based on the current character and the prior hidden states. Given a vocabulary $\mathcal {V}^{(i)}$ of size $d$ , we use a one-hot representation to encode the input values; therefore, all the input vectors are $d$ -dimensional binary vectors. The output values are $(d+1)$ -dimensional though, since they may further contain the termination symbol $\dashv $ , in addition to the symbols in $\mathcal {V}^{(i)}$ . The output values are not always one-hot encoded, because there can be multiple possibilities for the next character in the sequence, therefore we instead use a $k$ -hot representation to encode the output values. Our objective is to minimize the mean-squared error (MSE) of the sequence predictions.