Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary
----------

This paper presents an approach to uncertainty modeling in recurrent neural networks through a discrete hidden state. The training of this discrete model is done using a reparameterizable approximation (in particular, using the Gumbel-Softmax trick). The authors show the utility of this method on a variety of problems, including showing effective out of distribution detection and improved calibration in classification tasks.

Comments
----------

This paper presents a relatively simple idea that builds relatively directly from previous work, and uses the now common Softmax-Gumbel trick to enable differentiability. The main strength of this paper is the thorough experimental evaluation, on a wide variety of problems. 

The main weakness of this paper is the very unclear presentation of the method. In section 2.1, the authors do not define all quantities, the mathematics of the method is interspersed with discussions of the approaches of others, and the writing is unclear. The authors must clarify the presentation of their method, and have this presentation be distinct from discussion of previous work. 

Overall, the experimental results seem compelling and interesting. The authors should clarify their discussion of the partially observed RL task. In the partially observed task, is the agent only provided lagged measurements of the state? The presentation if quite confusing and the authors should state what this task is as clearly as possible. 

Post-Rebuttal
----------
I thank the authors for their response. Both of the sections are now more clear, although the authors should make an effort to polish the narrative of the paper and the clarity of exposition throughout. The discussion of epistemic versus aleatoric uncertainty in the appendix is also interesting. I have increased my score from 6 to 7. 
review_title: Review 3

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a novel approach for uncertainty estimation with RNNs. More precisely, the task is to both fit a model on the data and to learn the uncertainty of the fitted model at the same time.

The proposed approach fits a random model, with its randomness adjusted to the level of uncertainty. The probability of the potential outputs on a given input is then estimated by sampling the model (i.e., re-evaluating it multiple times on the same input). This, in turn, can also be used to estimate the uncertainty of the model.

One important detail that the paper does not discuss but would be important to understand is how S_t is trained/updated? (Actually, the same question goes for \tau.) In fact, referring to S_t as states is quite confusing; from the formulas it seems that they are used as weights. The authors should discuss these questions in detail.
 
Apart of these issues, the paper is relatively well written and the considered problem is important to various applications. The proposed model also makes sense on the high level (although the missing details make it hard to claim the same in general). Finally, empirical evaluations show the effectiveness of the method, and also that its performance is comparable - and in many cases superior - to vanilla LSTM, Bayesian RNN. RNN with variational dropout, and a deep ensemble of LSTM based model.



REMARKS

Section 2.2.
Setting \varphi to be a dot-product does not seem right: as its two attributes are \theta_t \in R^d and S_t \in S^{d x k}, the dimensions do not match. Simple matrix-vector product does work though.

In fact, Section 2 could be somewhat polished; it is not always easy to understand what is part of the proposed method, and what is explained in relation to other models only. Additionally, it would be helpful to have a brief recap at the end of the section about how the uncertainty estimation is done for the model.

In (1), t_i does not seem to be defined. Actually, should it not be {t,i}? Additionally, \alpha_i two lines below (2) should be \alpha_{t,i}, presumably.
review_title: Nice way to augment RNNs with internal randomness - but important details are missing from the paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper proposes a method to quantify the uncertainty for RNN. Different from  the traditional Bayesian RNN, the proposed method is more efficient. At each  time, based on the current hidden state and memory, it generates a probability  distribution over the state transition paths on the transition probability by  using the Gumbel softmax function. The next state is computed based on the weighted average of the sampled states and its uncertainty can be  qualified by the sample variance. The hyper-parameter tau of the Gumbel function  is learnt from data to better capture the inherent uncertainty in the data.

To demonstrate their method, they perform several experiments. First, they show that their model can  capture the stochastics in language better than other methods  Second, they demonstrate their  method performs better in classification on benchmark datastes than baseline methods  such as the ensemble and BBB methods in terms of both prediction accuracy and efficiency.   Third, they evaluated their method for out-of-distribution detection and their experiments again show their method performs better than the baseline methods on benchmark datasets.  Finally, they show that when applied to reinforcement  learning, their method is better than existing  methods in sample complexity. 

Strengths:
The proposed method for uncertainty quantification is efficient, compared with other methods such as Bayesian RNN.  The performances of their methods have been evaluated for different tasks on benchmark datasets and show competitive performance versus the baseline methods.

Weaknesses: 
First, technical novelty is minor; it is largely based on the exiting work on Gumbel function.  More importantly, is unclear why the Gumbel softmax function, even with the learnt tau parameter, can capture the data uncertainty and better theoretical justification  is needed.   Second, it is unclear how to compute the aleraeroic and epistemic uncertainties separately from their method as the latter is needed for OOD detection.  Third, it is unclear how to quantify the accuracy with the estimated uncertainty and how the improved uncertainty quantification can translate into improved performance in classification /regressions.  Fifth, the experimental comparisons are only done for baseline methods for each task.  The authors should also compare their methods to SOTA methods for each task.  Finally, they need do an ablation study on their method to figure out what contributes to their method’s improved performance for certain tasks. 

review_title: A good paper with extensive experimental validations and minor novelty

final_score: rating: 7: Good paper, accept
review_content: This work proposes a novel method to estimate uncertainties in recurrent neural networks. The proposed model explicitly computes a probability distribution over a set of discrete hidden states given the current hidden state in an RNN. Leveraging the Gumbel softmax trick, the proposed method performs MC gradient estimation. A temperature parameter is also learned to control the concentration of state transition distribution. To estimate uncertainty of a given input, the proposed model is run multiple times to draw samples for estimating the mean and variance. Experiments are conducted in a variety of sequential prediction problems, including a reinforcement learning task, demonstrating the effectiveness of the proposed uncertainty estimation method.


Pros:
Estimating uncertainty of predictions is important for data-driven machine learning models, especially for detecting out-of-distribution data;
The proposed method directly quantifies and calibrates uncertainty, and therefore does not use much more parameters (compared to BNNs) and requires less parameter tuning;
The paper selects a good range of task domains and strong baseline methods, demonstrating comparable performance.

Cons:
While the proposed method demonstrates good performance on both modeling stochastic processes and estimating out-of-distribution data, it is unclear whether the method itself can separate epistemic uncertainty from aleatoric  uncertainty if both exists; meanwhile, most of the selected baseline methods focuses exclusively on estimating the epistemic uncertainty; if possible, it is desired to see a comparison of the proposed method with baseline methods that are designed to exclusively model aleatoric uncertainties for RNNs;
It is mentioned that a large number of states improves performance in the experiments for predicting OOD data; a plot for the relationship between performance and the number of states used would be useful to understand how sensitive the performance is to the number of states used;
If possible, the authors should also discuss the proposed work’s relationship with the sampling-free method of Hwang et al. [1] and how the choice of using discrete state distribution would outperform a parametric distribution.


[1] Hwang, S. J., Mehta, R. R., Kim, H. J., Johnson, S. C., & Singh, V. (2020, August). Sampling-free uncertainty estimation in gated recurrent units with applications to normative modeling in neuroimaging. In Uncertainty in Artificial Intelligence (pp. 809-819). PMLR.


------------------------------------
Update: the major concerns above have been addressed in the appendix of the updated manuscript. I'm moving my initial rating of 6 to 7.
review_title: a novel uncertainty estimation method for RNNs

Meta-review: This paper proposes a method to quantify the uncertainty for RNN, which is an important problem in various applications. It provides results in a variety of domains demonstrating that the proposed method outperforms baselines. However, these experiments would benefit greatly from a comparison with SOTA methods for the specific tasks in addition to the considered baselines (e.g. covariance propagation, prior network, and orthonormal certificates). The paper could also be improved by adding a theoretical justification to explain how the Gumbel softmax function is able to capture the underlying data and model uncertainty.
Decision: Accept (Poster)