Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an interesting regularization method for encouraging function values to be discrete. The main idea is very interesting: with two correlated random vectors, the function is encouraged to maximize the correlation of outputs from the two vectors. The function that maximizes the correlation is the indicator function of a half-space. Combining this objective and original training objectives, a learning model can output random values that are near 0 or 1. The model plays the same role as the Gumbel-softmax trick. 

The submission has done extensive experiments comparing the proposed technique against Gumbel-softmax. The results indicate that the new technique improves the performance in several learning tasks, though the improvement is not consistent. 
main_review: Overall, I think the proposed new method is interesting and worth further study, but I also feel that the contribution of the submission is somewhat limited. 

In particular, I have several concerns about the submission. 

1. I think the study is not thorough enough. Though we see some performance improvement, can we compare the proposed method against Gumbel-softmax at a more detailed level. For example, Gumbel-softmax uses temperature to control the extremeness of function values. I think the corresponding hyper-parameter in the proposed method is the strength of regularization. Can we do a side-by-side comparison between the two hyper-parameters. For example, checking the performance vs temperature, and the performance vs regularization strength. 

My hypothesis is that the temperature value is hard to tune while the regularization strength is easier to tune. The performance difference might be from this reason. 

In general, I think the optimization involving discrete variables is hard, and the backpropagation through these variables does not behave. I don't feel the proposed method is changing the property. 

2. The performance improvement does not seem to be consistent. Sometimes we see performance drops. Is the experiment in 5.3 a good comparison? The proposed method uses 20 clusters while the first two baselines use 30 clusters. More clusters should leads to higher performance, right? The authors may want to increase the number of clusters and check the performance. How do you get the performance of SB-VAE? I don't find it in the original paper. 

3. Which experiments use mean-centering? Do you see a performance improvement by using mean-centering? I don't see a reason for mean centering in experiments. I feel that the model will learn the function and decide an underlying mean even without mean-centering. 





summary_of_the_review: Overall I think the technique is interesting, but I also feel the overall contribution of the submission is somewhat limited.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for regularizing neural networks with boolean and categorical stochastic variables. The proposed stability regularization is based on the Gaussian noise stability, which is maximized by the indicator functions of half spaces. Experiments with various benchmarks show promising results comparing to existing methods.
main_review: This work introduces stability regularization for training neural network with discrete stochastic variables. Unlike most of existing methods for handling discrete stochastic variables, the proposed method promotes the output of continuous functions of Gaussian random variables to discrete, which leverages the nice property of the Gaussian noise stability. The authors have conducted various experiments on different benchmarks to showcase its superiority over existing methods.

It has been known that halfspaces maximize noise stability subject to a measure constraint since Borell's work in 1985. But the theorem only states about the relationship between noise stability and partitions with two parts. This can be directly related to the binary variables, but it is not obvious how this could be extended to the categorical variables (that is, partitions with more than two parts). The way that the authors handle this is to sum across dimension, which is more like treating each dimension as independent binary variables rather than a single categorical variable. This does not prevent the output from generating all zeros or multiple ones, which is no longer a categorical variable. It has been shown that the standard simplex partition fails to maximize multi-part noise stability unless all of the parts have equal measure [1, 2]. Therefore, the analogue of Borell's theorem in the multi-part partitions case seems still an open problem. The authors might want to discuss more regarding the way that they extend the two-parts result to the categorical case and how the proposed regularization (4) promotes a single one for the categorical variable.

The authors present two versions of stability regularization: one with mean centering and the other without centering. It seems natural to compare these two versions quantitatively and give recommendations for situations where one is better than the other. However, in all 6 experiments, the authors only choose to provide results for either one of the two versions without any direct comparisons between them and the choice of which one to apply in each experiment appears a bit arbitrary. On the other hand, there are a few hyper-parameters introduced for the regularization: the correlation variable, the maximum stability, and the regularization strength. The authors claim that these either do not impact the performance much or are easier to tune without any quantitative results. The most ideal case is that one set of hyperparameters can be applied to multiple problems and gives reasonably good performance for all of them. But this does not seem to be the case here. The maximum stabilities are different across applications, and the regularization strength is only mentioned for the neural relational inference experiments. Since the "hassle-free" is one of the most important features of the proposed method, it is critical to provide sensitivity analysis of the introduced parameters and demonstrate that competitive performance can be obtained without careful tuning.

The authors also discuss how the proposed regularization can be combined with the Gumbel-Softmax method, where the output is from the Gumbel-Softmax computation but the stability is computed with the vanilla softmax without adding the Gumbel noise. This creates a mismatch here: the regularization is not applied on the "real" output. The authors might want to provide the rationale behind this and why the Gumbel noise is necessary to produce the output as input for the downstream network. One benefit of combining the stability regularization and the continuous relaxations, as the authors argue, is to provide "a form of implicit temperature control" and mitigate the need to manually tune the temperature. There are quantitative results supporting the combination is better than using Gumbel-Softmax alone. But it would be more interesting to see either the less sensitive to the temperature with the combination, or the universal improvement of the combination over the Gumbel-Softmax alone for different temperatures.

It is really nice that the authors conduct various experiments across different problem domains, but they are mostly in the same flavor without providing complementary support. I would suggest move some of the experiments to the supplementary material. The saved space can be used for the sensitivity analysis mentioned above, and the detailed comparisons with existing methods handling stochastic neural networks similar to the experiments done in [3] (the same toy experiment) and the Section B.2 in the supplementary material. After all, this is a work of training stochastic neural network. It would be nicer to give a thorough comparison with methods discussed in the Related Work.

Some typos:
- In Algorithm 2, $v_1$ and $v_2$ are used in Line 2, but $y_1, y_2$ are used in Line 3.
- The error of the mean-centered stability approximation in Proposition 1 is $O(\rho_2(\rho_2 - \rho_1))$, but $O(\rho_1 (\rho_1 - \rho_2))$ in Section A in the supplementary material.

[1] S. Heilman, E. Mossel, and J. Neeman. Standard simplices and pluralities are not the most noise stable. ITCS 2015.

[2] A. De, E. Mossel, and J. Neeman. Noise stability is computatble and approximately low-dimensional. Theory of Computing, 2019.

[3] A. Pervez, T. Cohen, and E. Gavves. Low bias low variance gradient estimates for boolean stochastic networks. ICML 2020.
summary_of_the_review: The proposed stability regularization has nice theoretical foundation and shows good performance across different problems involving discrete stochastic variables. But I am lean to reject mainly due to the following reasons: 1) the lack of theoretical foundation for the extension to the categorical variable, 2) the insufficient quantitative results supporting the "hassle-free" characteristic of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes a computational approach to discrete optimisation, based on a probabilistic regularization procedure that enforces the output of a continuous function to be quasi-discrete. The idea is to use a property of Gaussian noise described by Borell's Isoperimetric Theorem.

main_review: Strengths:
The method implemented in the paper is innovative and exploits a nice and not-so-popular property of the Gaussian distributions. Connecting real and discrete outputs through the expectation of the indicator function is a promising idea and may have a good impact on various discrete optimization applications. The paper has an extensive set of experiments.

weaknesses:
I have two main concerns about this work. One is the computational cost of the proposed method. Probabilistic approaches to discrete optimization often suffer from high computational complexity, e.g. the likelihood is intractable. The proposed procedure is based on a different idea but it looks like the resulting optimization requires Monte Carlo sampling or ELBO approximations. The second concern is about a possible comparison with other methods for discrete optimization, e.g. STE or more recent versions of it. If the authors have some available results, these could have been put in more evidence in the experiments section.

questions:
- Would it be possible to use the method for learning a binary NN where only the weights and not the layer representations are discrete?
- Is this the first time Gaussian stability is used for learning approximations of discrete-valued functions?
- Could be the proposed method compared with other approaches for discrete optimization, e.g. STE, on both the quality of the output and the computational cost?
- How many artificial Gaussian variables are needed for a d-dimensional optimization? Is the procedure expected to be scalable for networks with millions of parameters?
summary_of_the_review: A very nice idea, probably a bit expansive on the computational side.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new method to train neural networks with stochastic discrete variables called stability regularisation. The method pushes 
the outputs of functions of Gaussian random variables to be close to discrete, and unlike other methods used with discrete variables, 
such as Gumbel Softmax which requires temperature annealing, it is easy to tune. The method is demonstrated on a very rich variety of
tasks and models and shows state of the art performance. 

main_review: The paper builds on the concept of noise stability of functions of Gaussian variables.  Noise stability measures the resilience of such functions 
to noise by estimating how their outputs correlate when they receive as input correlated Gaussian variables; the higher the correlation the more 
stable the function. For a family of functions, bounded and fix volume, the stability is maximized for these functions that are indicator functions 
of half spaces which is a natural way to construct binary variables. The paper proposes to maximize stability in order to obtain categorical 
variables.

The implementation of the method is rather simple and it comes with a small computational overhead due to the fact that it requires a double evaluation
of these computational units whose output we would want to push to be discrete. 

The paper explores the performance of the method in a diverse set of experiments learning discrete latent spaces, gating computational units using binary
variables, and generating discrete outputs. I appreciate the breadth of scenarios, I only have a small comment here on the fact that the comparisons are 
only done with the Gumbel-softmax, why this choice and not compare also with some of the other approaches that the authors discuss, e.g. straigh-through
and/or some variants of the score based estimators?


summary_of_the_review: Overall this is a nice contribution to what is an important problem, learning discrete (stochastic) variables. The idea is simple and the authors demonstrate it in a variety of datasets. The only thing that I would have liked to see is a comparison with other methods that are meant to also work in such discrete settings, e.g. straight-through, score-based estimators.
review_title: null

Meta-review: The paper introduces a method to train neural networks based on so-called stability regularisation. The method encourages the outputs of functions of Gaussian random variables to be close to discrete and does not require temperature annealing like the Gumbel Softmax. All reviewers agreed that the proposed method was novel and of interest. The authors conducted extensive experiments. They also adequately addressed the concerns raised by the reviewers (e.g., theoretical foundation and computational cost).
Decision: Accept (Poster)