Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to add an extra residual connection between a transformed word embedding and the final output layer, which easily generalizes over different recurrent architectures. The language model experiments show that their proposed model performs better than the same recurrent model without the residual connection, but the performance gain becomes marginal after optimal hyperparameter setups were used.
main_review: Strengths:
+ The technique is very simple and is applicable to any embedding-recurrent-softmax (ERS) architecture.
+ The presentation of the method is very clear. Sufficient details were given on the experiment setup and hyperparameter search.

Weaknesses:
+ The proposed technique, while simple, is not very novel. A very similar technique has been examined in a previous paper (https://ieeexplore.ieee.org/document/9207238), and similar experiments on PTB have also been conducted.
+ It's hard to justify whether the performance gain in the Table 1 actually due to the residual connections or because of the extra input projections introduced. The authors could have examined this by simply doing a version of the residual connection by directly adding the input embedding $x_t$ to the hidden state $h_t$, which will incur no extra parameters at all.
+ After hyperparameter tuning, the performance gain brought by the architectural change becomes very marginal.

I also have a question for the author: why is the parameter number for Dual mdLSTM smaller than Mogrifier-LSTM in Table 3?

=== UPDATE ===

I have read the authors response and the other reviewers' reviews and do not think any change in my overall evaluation is justified. See my detailed comment below.
summary_of_the_review: The idea is simple and generally applicable, but the lack of novelty and the marginal result improvement greatly limit the impact of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple improvement to recurrent network architectures for language modeling. The idea is to insert a single additional layer into a network with one or more recurrent layers, just before the output layer (Eq. 6). This is termed a "dual connection" and it combines the output of the last recurrent layer directly with the input to the network at the current step. The paper presents the results of adding this modification to both LSTM and Mogrifier-LSTM networks on the Penn Treebank and WikiText-2 datasets.

main_review: I found the paper to be clearly written and easy to follow. This is an experimental paper, and the authors have taken care to describe the setup in sufficient detail. I also appreciated the use of two separate sets of experiments, one with hyperparameters for the baselines transferred over to the "dual" models, and another with hyperparameters tuned for the "dual" models.

There are a few concerns that make it difficult for me to accept this paper in its current form:

1. It is unclear where the improvement is coming from. Is the added dual connection improving ease of optimization, model capacity, or regularization (providing a useful inductive bias)? It's possible that this will be much clearer if the training and validation loss curves are compared for models in Table 2. Results for additional baseline models with slightly larger sizes might also help: those models should have slightly more capacity in principle, so how do their training and validation loss curves compare to the "dual" models.
2. Related to the above question: it appears that Melis et al. [A] use skip connections in their LSTMs to improve optimization, but this paper doesn't appear to use them. This appears to be very relevant here if the improvement from the proposed dual layer is coming from improved optimization. Would simpler skip connections provide a similar benefit? This can be checked by modifying the baseline to use additive skip connections.
3. Overall the experimental results are not strongly in favor of the proposed model. In Table 2, we find that dual layer leads to worse results without dynamic evaluation when added to "mdLSTM", but better results with dynamic evaluation. We don't know why this should be the case, and it casts doubt on a universal benefit of this design. We see a similar situation in Table 3 too, where the results are worse without dynamic eval, as well as with dynamic eval on the val set. Only on the test do we see a small improvement. It is very difficult to say from these results that the proposed design can be expected to improve performance in general. On this point, I don't have concrete suggestions to offer, but perhaps the authors can address this by adding experiments on more datasets, and/or clarify the reason for these mixed results.

If 1 & 2 are satisfactorily addressed, I'm willing to raise my score up to 6. If 3 can also be addressed, I will increase the score further.

[A] Melis, G., Dyer, C., & Blunsom, P. (2017). On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589.

————
Update:
I am thankful to the authors for providing additional results and analysis. My concerns are partially addressed so I am increasing my score. However, I believe more work is needed to make sure that the increased capacity provided by the _dual_ connection can actually be useful in practice. What I mean by this is that many results in Table 2, while useful from a comparative perspective, feel less relevant from a practical perspective because they show that the _dual_ connection improves performance of models _underpowered for the task/dataset_ at hand. The most relevant results are when we get close to the “best” test set performance for the task. It is here the evidence gets fuzzy, and the use of only 2 datasets makes the issue difficult to ignore. 
Can I take a setup that’s otherwise well tuned for a task, add this connection to improve capacity, and expect improvement? If not, why not, and under which conditions should I expect it? I don’t have sufficient confidence to answer these questions based on the results so far. The experiments suggested by the authors in the responses and in the paper’s Discussion section may be necessary to provide that confidence.
summary_of_the_review: The paper proposes a simple modification to recurrent language models that can potentially improve performance, but additional analysis and results are needed to establish the utility of the approach.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work revisits the LSTM architecture. They propose to modify a recurrent architecture by adding a direct connection between the input and the output of the recurrent module, called "dual". They also consider a double-layer LSTM, where the output of the recurrent module is obtained by the concatenated application of two LSTM layers.
In experiments, the dual modification can get consistent improvement.
main_review: Strength:

It's good to revisit the LSTM recurrent structure.

Hyper-parameter tuning is reported in detail.

I got a bit surprised by the improvement from the dual structure.

Weakness:

The used datasets like PTB or WT2 are small-scale. I'd like to see something like wiki-103. 

Why there's no comparison with transformers?

Not much analysis or explanation about why the dual structure is better is given.

Below are some minor comments:

I sometimes got confused by "dual" and "double" in the writing.

Some possible typos:

Sec1: "for the Penn Treebank problem "-> PTB "dataset"?

Sec2.2: "achieving the highest perplexity scores" -> lowest PPL?
summary_of_the_review: While I think this revisit to LSTM is interesting, the scope of the experiments are still limited. Comparison with transformers is not conducted, and the datasets are small scale.
review_title: null

Meta-review: This paper considers augmenting LSTM language models with a form of residual connection that adds and additional feed forward layer before the softmax that integrates the output of the recurrent cell with the input embedding. This architectural variation is evaluated on the standard Penn Treebank and Wikitext-2 language modelling tasks and shown to lead to lower perplexities on the test sets, particularly when dynamic evaluation is used. 

The reviewers agree that the proposed addition is well motivated, however they also observe that there has been substantial work in language modelling on various forms of residual and skip connections and it is not clear how this work relates to that body of work. The authors have provided some additional comparisons during the discussion, however the reviewers feel that further evaluation and analysis is needed. There was also some additional confusion about the varying hyperparamter tuning protocols employed in the different evaluations. The author’s have clarified this in their response so that it is clearer how the different results were obtained.

Overall this paper presents an promising initial result, but it would benefit from more complete evaluation, analysis, and hyperparameter tuning. This could include ablation studies and analysis to shed more light on what the proposed architectural addition is contributing, how this relates to other varieties of residual connection, and it’s positive interaction with dynamic evaluation. It would also be useful to include a tuned model with a comparison to previously reported Wikitext-2 results.
Decision: Reject