Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper develops transformer-RKDE by leveraging robust kernel density estimation as a replacement of dot-product attention. The authors show that the optimal estimation of density functions via robust KDE requires computing a set of weights by solving an iterative re-weighted least-square problem. Multiple experiments including NLP and vision demonstrate the proposed method on performance and efficiency improvement. 
strength_and_weaknesses: Strength 
1. The topic is interesting and important 
2. The paper is clear to understand and easy to follow 
3. Related works and backgrounds are well discussed and explained.
4. Strong experiments are shown the benefits of the proposed method

Weakness
1. The novelty is weak and lacks of a major contribution
2. The comparison is not strong and lacks of other robust transformer models instead of KDE-based models 
clarity,_quality,_novelty_and_reproducibility: clarity, quality and reproducibility are good
novelty is weak 

summary_of_the_review: The paper simply employs the robust KDE (20212 JMRL) for transformer models. The idea is great and the connection is natural and straightforward but the major contribution seems marginal. 

I also expect to see the performance benefits compared with other robust transformer models not only KDE-based vanilla models. That's important to demonstrate the necessity of the proposed robust KDE.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a robustified transformer model motivated by the robust kernel density estimation. The proposed model alleviates the influence of bad data (e.g., outliers). The empirical studies on NLP and CV tasks demonstrate the effectiveness of the proposed methods.
strength_and_weaknesses: Strength:
The proposed method enjoys promising numerical performance when the noise level is high.

Weakness:

1.  The writing could be significantly improved. For example, equations (5-11) in subsection 2.2 on page 3 are almost analogous to equations (3-8) in Nguyen et al. (2022). It may not necessary to use a whole page to restate the existing result in the literature. 
Moreover, in Aglrotihm 1 on page 6, a) the $\hat{p}_{\mathrm{robust}}^{(k)}(k)$ is used without declaration, in the input
line the initial weights of $\omega\_{\mathrm{marginal}}^{(0)}$ and $\omega\_{\mathrm{joint}}^{(0)}$  are not used, and I don't find the definition of $\omega_j^{\mathrm{joint}}$.

2. The bond between theoretical motivation and real implementation is kind of weak. For example, the theoretical analysis requires the solve for $\hat{p}_{\mathrm{robust}}^{(k)}$ via an iterative method (e.g., KIRWLS). However, when checking the code, the RBFMultiHeadAttn class in Line 391 of src/mem_transformer.py, I don't find the for loop in the forward function and it seems the attention 
being computed via only a single path...  If it is the case, the $\hat{p}\_{\mathrm{robust}}^{(k)}$ can be far away from its underlying *optimal* value. If it is not the case, I suggest authors report the computing cost of the proposed method since the multi-round of the iterative method can increase the computation time by a significant amount. 


clarity,_quality,_novelty_and_reproducibility: Clarity: The current presentation could be improved.

Quality: Based on the current presentation, the overall quality doesn't meet the bar of the top ML conference like ICLR.

Novelty: This paper considered an interesting attention variant motivated by robust kernel estimation.

Reproducibility: The code is provided. 
summary_of_the_review: This paper proposes a variant of Transformer motivated by robust kernel estimation. In particular, a decayed weight $\omega$ is computed to minimize the influence of the outliers, and in high noise settings, the proposed method reaches promising numerical performance.

My major concern is the writing of the paper which could be significantly improved. I also suggest authors to further highlight the computation cost of the iterative approach or show the single path can yield an accurate enough estimation of $\hat{p}_{\mathrm{robust}}^{(k)}$.


Minor Issue: I suggest authors use \eqref instead of \ref when referring to the equation number.


At the current stage, I tend to reject this paper but I'm willing to change my evaluation after rebuttal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authoor leverage the robust kernel density estimation (RKDE) in the self-attention mechanism, to alleviate the issue of the contamination of data by down-weighting the weight of bad samples in the estimation process.  Empirical results on language modeling and image classification tasks have demonstrated the effectiveness of this approach. The motivation is clear, but still have few concerns.
strength_and_weaknesses: strengths:
1. propose a novel robust transformer framework which replace the dotproduct attention by an attention arising from the robust kernel density estimators (RKDE) associated with the robust kernel regression problem.
2. the proposed method only requires computing an extra set of weights.

weaknesses:
1. My most critical concern is regarding the review of the related works. Several robust transformer methods have been proposed, I suggest that the author should review those works in the related works, please see an incomplete list here, Mao X, Qi G, Chen Y, et al. Towards robust vision transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 12042-12051. Li N, Liu Y, Wu Y, et al. Robutrans: A robust transformer-based text-to-speech model[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(05): 8228-8235. Liu J, Singhal T, Blessing L T M, et al. Crisisbert: a robust transformer for crisis classification and contextual crisis embedding[C]//Proceedings of the 32nd ACM Conference on Hypertext and Social Media. 2021: 133-141. Yang J, Gupta A, Upadhyay S, et al. TableFormer: Robust Transformer Modeling for Table-Text Encoding[J]. arXiv preprint arXiv:2203.00274, 2022.
2. The second concern is the baselines. The current baseline methods seem insufficient. Why does the author choose these methods as the baseline? Why does comparison with these methods can prove the effectiveness of the proposed method? I think the authors should select some similar methods (see weaknesses 1) as a baseline to prove the effectiveness of the proposed method.


clarity,_quality,_novelty_and_reproducibility: The quality of this paper is good, the problem is clearly stated, but the novelty seems limited.
summary_of_the_review: Motivation of this paper is clear, but the reviwe of the related works and baselines are insufficient.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a robust variant of the standard Transformer named Transformer-RKDE to improve the robustness of data with contaminated samples. The idea is based on the interpretation that the self-attention in transformer can be viewed as a non-parametric estimator based on the kernel density estimation (KDE). The authors re-interpret KDE as a regression in the Reproducing Kernel Hilbert Space (RKHS) and indicate that the vanilla KDE is sensitive to data corruption. Therefore, the authors introduce a robust version of the KDE and use the robust KDEs to construct the robust self-attention mechanism for the Transformer.
strength_and_weaknesses: Strengths:
+ The idea seems interesting. Starting from the inherent problem of self-attention (the KDE is not robust to the outliers), the authors introduce RKDE, the robust version of the KDE, to construct robust attention in transformer.
+ The experiments verify the effectiveness of the proposed model in both language modeling and image classification tasks.
 
Weaknesses:
+ Lack of quantitative analysis on the efficiency of the proposed model. Since additional weight calculation requires iteration, I wonder whether it will affect the efficiency of the algorithm.
+ Because the experimental verification is mainly designed for different adversarial attacks, I wonder how the final effect of the proposed method is compared with other models designed for adversarial attacks. This can better show the significance of the proposed method.

clarity,_quality,_novelty_and_reproducibility: The presentation is clear and well-organized. The contributions are significant and somewhat new. The authors provide the code to reproduce their results. 
summary_of_the_review: Overall, the innovation is somewhat interesting to me. However, I have some concerns about its efficiency and practical application value.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper advocates to use robust kernel density estimation in the self-attention mechanism to mitigate the issue of data contamination. In particular, the idea is to down-weight the weight of bad samples in the estimation process. The authors conduct empirical experiments to demonstrate the effectiveness of the proposal.
strength_and_weaknesses: Strength:
1. The problem and issues to address are well motivated.

Weakness:
1. The paper appears to be largely overlapped with the existing literature of Kim and Scott (2012), which is intensively cited in the manuscript.
2. The robust self-attention mechanism does not have strong theoretical support 
clarity,_quality,_novelty_and_reproducibility: I fail to identify the original of this manuscript over existing literature.
summary_of_the_review: I fail to identify the technical contribution of this manuscript over existing literature.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a robust variant of the standard Transformer named Transformer-RKDE to improve the robustness of data with contaminated samples. The idea is based on the interpretation that the self-attention in transformer can be viewed as a non-parametric estimator based on the kernel density estimation (KDE). 

Strengths: The problem and issues to address are well motivated. The idea is great and the connection is natural and straightforward.
Weakness: All the reviewers agree that the critical part is the limited novelty(simply employs the robust KDE for transformer model), concerns about its efficiency and practical application value. The reviewers are not satisfactory on the rebuttal since majority concerns are not well addressed. 
justification_for_why_not_higher_score: The experiment is insufficient and novelty is limited. 
justification_for_why_not_lower_score: NA
Decision: Reject