Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This work is an application of interval bound propagation on evaluating the robustness of NLI model. This work is well-motivated, assuming that the confidence of a neural model should be lower when part of the sentence is missed. However, the application of vanilla IBP is quite limited in certain model architectures. In this work, the author considers specifically the decomposable attention model, which is a very shallow network, and not a state-of-the-art model anymore. It is non-trivial to adapt the proposed method to other more advanced models, such as the ones based on the Transformer model. Hence, this work does not make enough contribution to be accepted.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: -- Overall --
This submission tackles to verify the “under-sensitivity” problem of neural network models in the natural language inference by ensuring modes do not become more confident in the predictions when arbitrary subsets of words from the input text are deleted. The authors developed new verification approaches based on decomposable attention mechanism with interval bound propagation (IBP), which can prove the under-sensitivity issue given a model and a particular sample. The experimental results on SNLI and MNLI show that the proposed approach leads to a much improved verified accuracy.

-- In general, “under-sensitivity” is a very critical problem for applying neural models in natural language understanding where powerful neural networks tend to capture spurious correlations from the biased datasets. This submission formulates “under-sensitivity” as a mathematical specification and then try to verify it with IBP verification. Although the used technique IBP is not new, it would interesting to have the verification in NLI models.

-- Section 5 is a bit unclear how to compute the IBP for deleting several words, and what is the output. It would be better to have a clear example for how this was computed.

-- As the author mentioned, the verification of under-sensitivity can also be done by using beam-search, although it is costly and not accurate. IBP is another more efficient option, but not the optimal neigher. Maybe consider to change the title as “efficient verification”?

-- Specific Questions -- 
The entire paper builds on decomposable attention. Is the same approach also applicable to other model types, or only single layer attention-based models? 
Also, how this methods work for other NLI or NLU tasks?
In experiments, how the data augumentation penalize the model with a loss for specification violation? What does the equation look like?
Can you explain a bit more for IBP-training? How that hinge loss applies to the objective function? Is the IBP training differentiable?

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This works considers the task of Natural Language Inference (NLI).
The question addressed is that SOTA NLI models tend to lead to
higher confidence when some parts are deleted from the "premise".
It is a problem known as under-sensitivity.
A method based on IBP is proposed to address this issue.
The idea of Interval Bound Propagation (IBP) is to use interval arithmetic to propagate
intervals and bound the variation of the target based
on variation of the input. In other words, one propagates
upper and lower interval bounds through the network.
The DAM model from (Parikh et al., 2016)
is studied in particular.

The paper is well written and easy to follow.

My only concern is about the relevance of approach based on DAM when
there are now more accurate models for this task. The paper is however
interesting and addressed a relevant topic.

Misc:
- transpose should be written with $^\top$ (not $^T$).


review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a model to verify the robustness of NLP models (change in the original probability), more specifically DAM, in the case of word removals in the input. The idea is given the lower and upper bound on the hidden state at previous layer, compute the new bound by propagating the bounding box around the hidden state at previous layer. The upper bound at the final layer is then compared with the label probability of the original input to assess if the probability increases or not. By training model with a hinge loss based on this verification method, they show that the model becomes more robust to word removals.

Overall, the paper is well written and the idea of using IBP with an attentive model seems to work empirically for SNLI datasets. But, the technical contribution feels incremental over previous approaches, especially Huang (2019). I have several questions related to some parts of the paper:

- Since upper and lower bounds are also propagated, do you backpropagate the gradients via these bounds or only via the original inputs?
- How sensitive is the label in SNLI dataset to word removal? For some label types, such as entailment, it might have less of an effect that for the others.
- How is the accuracy distributed wrt different label types?
- Since the accuracy of the proposed model drops the most, I am wondering how the verfied accuracy and accuracy are related during training? For example, can you show what is the verified accuracy with accuracy being close to the standard training?
review_title: Official Blind Review #3

Meta-review: This paper deals with the under-sensitivity problem in natural language inference tasks.  An interval bound propagation (IBP) approach is applied to predict the confidence of the model when a subsets of words from the input text are deleted.  The paper is well written and easy to follow.  The authors give detailed rebuttal and 3 of the 4 reviewers lean to accept the paper.
Decision: Accept (Poster)