Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use Lipschitz constraints to train certifiable robust transformers. With some modifications to the Transformers models, they bound the Lipschitz constant for each layer, which is related to the robustness of the model. Compared to previous work, their proposed model, named one-Lipschitz self-attention mechanism (OLSA) can achieve tighter bound (therefore larger robustness radius) as well as being more efficient, which is empirically verified on both training-from-scratch and fine-tuning scenarios.
strength_and_weaknesses: Strengths:

1. OLSA has advantages over its baseline, as it has a tighter Lipschitz bound (compared to Kim et al. (2021)) and requires only the forward pass (compared to Shi et al. (2020)). With those improvements, it achieves a higher robustness radius and gets faster than Shi et al. (2020), which is verified on two datasets in both training-from-scratch and fine-tuning scenarios.
2. The modification OLSA makes to the Transformer model makes it easier to bound their 1-Lipschitz constant during training, which is well-designed and works well both theoretically and empirically.

Weaknesses:

1. While the changes on Transformers are necessary for the OLSA to achieve robustness, certain features in Transformers must be abandoned. E.g. only additive attention can be used, while most of the pre-trained transformers rely on the dot-product attention mechanism. LayerNorm is removed, which is important when many layers of transformers are stacked. Those changes might be acceptable for 3-layer transformers, but can be dangerous for larger models.
2. Certain steps to bound the Lipschitz constant have dependencies on the sequence length, such as Theorem 4.1 and 4.2, which means the actual Lipschitz bound would differ for different lengths of the input sequences. 

Typo: In the last paragraph before the "technical contributions", the paper says OLSA Transformer achieves a radius of 0.071 ... previous work ... 0.368. Should it be 0.368 and 0.071?

The paper seems to use both n and N symbols to represent the sequence length. E.g. in Theorem 4.1 it uses "n" and the remark below 4.1 uses "N". Can you make them consistent? As a reference, N in Shi et al. represents the number of layers.

Questions

1. I have a general question for this line of work: How is it practically useful to model defense? Lipschitz bound guarantees that the model is robust to slight changes in the embedding space, which is not naturally common in attacks. If one or two words are changed/added/deleted, the embeddings will be significantly changed, then how is Lipschitz helpful?
2. Why do you put the comparison with Kim et al. (2021) in the appendix, and not show their efficiency?
clarity,_quality,_novelty_and_reproducibility: The paper is clear about its methods, theorems, and corresponding proofs. The paper is well-written, and the statements about the comparison between OLSA and previous work are helpful to distinguish their contributions. However, certain details in the model training might make this work not easy to reproduce. If the authors can release the source codes and training scripts, it would be most helpful.
summary_of_the_review: The model proposes a new method that guarantees the Lipschitz bound of the Transformer with a smaller robustness radius and efficient certifications. However, certain modifications to the Transformers are necessary under their settings, which could be harmful to its usefulness.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes OLSA, an algorithm to train transformers for 1-Lipschitzness in the l2 norm, which is directly linked to robustness to l2 perturbations. The authors show that OLSA attains stronger verified robustness than relevant previous work, at a minimal cost in standard accuracy.
strength_and_weaknesses: OLTA achieves 1-Lipschitzness via a number of modifications to the standard training process. The authors enforce orthogonality in the self-attention layers (in their additive form), remove dropout and LayerNorm, and employ the GroupSort activation function. While each of the ideas was previously presented in the 1-Lipschitzness literature, this is the first application to training 1-Lipschitz transformers.
In spite of the modifications, the authors show that their method achieves larger certified robustness than the relevant previous work from Shi et al. 2020, with a minor cost in terms of standard accuracy. This is definitely a big strength of the work.

Nevertheless, I have the following comments/questions on the experimental section:
- the authors should compare against the bounds from (Bonaert et al. 2021)[https://files.sri.inf.ethz.ch/website/papers/pldi21-transformers.pdf], which were shown to be more effective than those from Shi et al. 2020;
- the speedup against previous certification methods is repeatedly stated. However, bounding algorithms do not necessarily require an ad-hoc training process, which might be more costly. Could the authors provide information on the training runtimes? What is the overhead of running the Newton's iteration?
- it would be nice to present an ablation study of the non-standard components of the training process on standard accuracy. In other words, how does the use of GroupSort, the lack of dropout or LayerNorm, and the use of additive attention?



clarity,_quality,_novelty_and_reproducibility: The paper is mostly well-written and easy to follow, except a couple of typos. Some bibliographic entries miss the publication year.
While many of the paper's building blocks already come from the relevant literature (e.g., orthogonality for 1-Lipschitzness), the work is original as it is the first example of 1-Lipschitz transformers (to the best of my knowledge).
The authors provide code in the supplementary material for reproducibility (though I have not run it).
summary_of_the_review: OLTA achieves state-of-the-art deterministic certified accuracy on transformers at a minimal cost in terms of standard accuracy. Nevertheless, the paper could benefit from a more recent baseline (see above), reporting and commenting on training runtimes, and an ablation study.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new multi-head self-attention variant -- OLSA. The proposed method can satisfy the 1-Lipschitz constraint under certain conditions. Thus, the author further proposes a Lipschitz constrained Transformer model. The model achieves a strong certified radius and robust accuracy while maintaining a reasonable vanilla accuracy.
strength_and_weaknesses: Strength:
1. The proposed method can achieve a strong certified radius and it is straightforward and easy to implement.
2. The paper is well-written and easy to follow. The comparison with related works is comprehensive.
3. Experiments show the good robustness of the proposed model.

Weakness:
1. The so-called fine-tuning experiment is not a standard fine-tuning setting used by modern works.
2. The OLSA-based transformer model doesn't have layer norm and dropout. I suspect that's why the author is using a transformer with only 3 layers. In order to make this work has a broader impact, it's would be great the find an adequate substitute for these two modules and train a transformer model with more layers.
3. The paper didn't explain how the method could be applied to natural language generation tasks.
clarity,_quality,_novelty_and_reproducibility: Although limited by several constraints, the proposed method seems sufficiently novel to me. The paper is easy to follow and the experiment results are enough to show that the model can achieve better robustness under these constraints.
summary_of_the_review: Overall the paper proposed an interesting new variant of transformer that can easily satisfy the 1-Lipschitz constraint. The idea of building a robust transformer model could potentially have a big impact on many downstream tasks. However, the current method still has many limitations, especially since it's not compatible with some important components of SOTA models and its scalability is still unclear.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces Lipschitz constraints into the transformer for higher adversarial robustness. The author proposes the first One-Lipschitz Self-Attention (OLSA) mechanism for Transformer models. The author orthogonalizes all linear operations in the self-attention mechanism and introduces Lipschitz constraints. The proposed method also provides certified guarantees. Experiments demonstrate the effectiveness of this method.
strength_and_weaknesses: Strength:
1. This paper proposes the first One-Lipschitz Self-Attention mechanism (OLSA) and proves the Lipschitz bound.
2. Because the prediction gap is only calculated on a forward pass in the OLSA model, the robustness radius can be certified in much less time.

Weaknesses:
1. Certified defense is usually divided into bound propagation and randomized smoothing (RS). The paper lacks a comparison with the RS method.
2. The method is not evaluated on TectFooler, BERT-Attack and TextBugger.

clarity,_quality,_novelty_and_reproducibility: The presentation of this paper is clear and has certain theoretical contributions.
summary_of_the_review: We tend to accept this paper because the method is effective and the presentation is clear.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes One-Lipschitz Self-Attention (OLSA) mechanism for Transformer models that proposes to orthogonalize all linear operations. 

I am recommending rejection for the following reasons:
1. reviewers are generally not excited about the paper based on scores.
2. orthogonalizing transformers have already been tried in https://aclanthology.org/2021.acl-short.48/ although not really for robustness.
3. this paper does not consider the cost both in terms of engineering and efficiency of these approaches.
4. the experiments look toy-ish and its quite hard to really imagine the impact of such an approach. most likely large scale transformer models (or even small scale transformers) will likely not adopt this. 


justification_for_why_not_higher_score: I doubt this paper would be impactful given that similar approaches have been tried and none of them have really been folded into the main transformer. 

robustness in NLP is a tricky topic and i don't think certified robustness is important at all for language. paper also fails to explain why it's important. if certified robustness is so important than chatGPT would already be using it. 
justification_for_why_not_lower_score: Already a reject. Can't be lower.
Decision: Reject