Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes to improve the learnability of the gating mechanism in RNN by two modifications on the standard RNN structure, uniform gate initialization and refine gate. The authors give some propositions to show that the refine gate can maintain an effective forget effect within a larger range of timescale. The authors conduct experiments on four different tasks and compare the proposed modification with baseline methods.

Strong points:
1. The authors propose a new refine structure that seems to have a longer "memory".
2. The authors designed a good synthetic experiment to demonstrate whether the proposed refine structure can help to remember information in longer sequence.

Weak points:
1. There are several parts in the experiment that are not very convincing.
     a. In Figure 3(a), where are the other baselines? Are they performing too badly so that they can not show up in the figure? It needs more explanation.
     b.  In Figure 3(b), actually a lot of methods are performing similar, while some methods converge similarly. What is the reason?
2. It is not defined why the uniform gate initialization works.
3. The proposed results actually not always perform the best. For instance, in Table 3, purely using the UR-LSTM only achieve good results on sMNIST. What is the reason? The proposed method seems not very general.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 
This paper introduces and studies modifications to gating mechanisms in RNNs.
The first modification is uniform gate initialization. Here the biases in the forget and input gate bias are sampled such that after the application of the sigmoid the values are in the range (1/d, 1-1/d) where d is the dimensionality of the hidden space for the bias. The second modification is the introduction of a refine gating mechanism with a view to allow for gradients to flow when the forget gates f_t in the LSTM updates are near saturation. The idea is to create an effective gate g = f_t +/- phi(f_t, x_t). The paper proposes using phi (f_t, x_t) = f_t(1-f_t) * (2*r_t-1) where (r_t is between 0 or 1). The effect of such a change is that g_t can reach values of 0.99 when the value of f_t is 0.9 allowing gradients to flow more freely through the parameters that constitute the forget gate. Overall the change corresponds to improving gradient flow for the forget gate by interpolating between f_t^2 and (1-f_t)^2. i.e. the authors note that the result of these changes is that it corresponds to sampling biases from a heavier tailed distribution while the refine gate (by allowing the forget gate to reach values close to 0 and 1), allows for capturing information on a much longer time scale.


The paper studies various combinations of the two changes proposed to gating architectures. Other baselines include a vanilla LSTM, a Chrono initialized LSTM, and an ordered Neuron LSTM. The models are trained on several synthetic and real world tasks. On the copy and add tasks, the LSTMs that contain the refine gate converge the fastest. A similar story is observed on the task of pixel by pixel image classification. The refine gate was also adapted to memory architectures such as the DNC and RMA where it was found to improve performance on two different tasks.

Overall, the paper is written well, I like the (second) idea of the refine gate and the contributions are explained in an accessible manner. While I'm not entirely convinced about the proposed initialization scheme but across the many different tasks tried, the use of the refine gate does appear to give performance improvements that lead me to conclude that this aspect of the work is a solid contribution to the literature.

Questions and comments:
* This manuscript already quite long and has several formatting issues. Several of the figures are unreadable when printed. For example, every piece of text on Figure 2(d) is unreadable on paper. Figure 3 and 5 are difficult to read; they contain too many alternatives with a colour scheme that makes it difficult to distinguish between them -- consider displaying a subset of the options via a plot and using a table to display (# steps to convergence) as a metric instead. It also appears as if the caption for Table 6 is deleted?
* I think that for this approach to work, two conditions need to be satisfied (a) there must be foreseeable improvements in the use of a forget gate that can reach values close to 0/1 for the task at hand and (b) r_t needs to function well despite not being too close to 0 or 1 (lest its parameters suffer from gradient flow issues).
  * Was there any visualizations done on whether (a) happened? i.e. for the URLSTMs that performed well, were the values of the forget gate closer to 0/1 than the baselines?
  * What were typical values of r_t, did the models need the refine gate to reach values close to 0 or 1 for the overall approach to work?
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper introduces two novel techniques to help long term signal propagation in RNNs. One is an initialization strategy which uses inverse sigmoid function to avoid the decay of the contribution of the input earlier in time and another is a new design of a refine gate which pushes the value of the gate closer to 0 or 1.  The authors conduct exhaustive ablation and empirical studies on copy task, sequential MNIST, language modeling and reinforcement learning. 

Though the experiment section is solid, I still vote for rejection for the following reasons:

1. The writing in the description of the UGI and refine fate is not clear.
a. The authors compares UGI to standard initialization but where is the standard initialization? I do not see "standard initialization" clearly defined in the paper.
b. I am not convinced how the UGI gate help avoid decaying of the input. There is a proposition 2 trying to explain some part of the mechanism of UGI. But the proposition is never proved anywhere and I am not sure why this proposition is important. More explanations are needed. Also this proposition is far away from the place the authors introduce the UGI. The authors may want to refer it in the place introducing UGI.
c. Similar to proposition 2, proposition 3 is not explained and proved in the paper. It is hard for me to analyze the importance of these two propositions. Overall, propositions 2 and 3 look isolated in the section.
d. Proposition 1 looks like a definition. Not sure why the authors name it as a proposition.

2. Even though the title of the paper is "improving the gating mechanism of recurrent neural networks", the authors try to solve signal propagation problems. It is unclear why "gate" is important. Maybe other designs of the recurrent neural network can satisfy better the desiderata the authors want. Based on my limited knowledge, the initialization the authors mention (saturation) is exactly from the need of using a sigmoid gate. The importance of using "gate" should be discussed.

3. The authors shrink the space before and after headings in the paper. I think this is not allowed in ICLR. It would be better that the authors correct the spacing in the revised version.


Minors:
1. page 1 second paragraph: repeated “proven”
2. page 1 second last paragraph: “due” -> “due to” 
3. page 2 second last paragraph: repeated “be”
4. page 2 Equation (4) and (5): using some symbols like \odot for element wise multiplication will be good for the readers.



review_title: Official Blind Review #2

Meta-review: This submission proposes a new gating mechanism to improve gradient information propagation during back-propagation when training recurrent neural networks.

Strengths:
-The problem is interesting and important.
-The proposed method is novel.

Weaknesses:
-The justification and motivation of the UGI mechanism was not clear and/or convincing.
-The experimental validation is sometimes hard to interpret and the proposed improvements of the gating mechanism are not well-reflected in the quantitative results.
-The submission was hard to read and some images were initially illegible.

The authors improved several of the weaknesses but not to the desired level.

AC agrees with the majority recommendation to reject.
Decision: Reject