# RobustQuote: Using Reference Images for Adversarial Robustness
## 3 Method
### Rectification Module  
The rectification module (right module in Figure 3) aims to correct the adversarial features in \(X_{\texttt{p}}\), referring to the unattacked information in \(R_{\omega}\)'s as follows:  
\[\bar{X}=X+G(X,\{R_{\omega}\}), \tag{5}\]  
where \(G\) calculates vectors to mitigate the effect of adversarial perturbation. Here, we propose to use the self-attention mechanism [33] and two derived strategies involving \(R_{\omega}\) to a greater and greater degree to generate the term \(G\).  
**Self-attention block**: We use the traditional self-attention module without intervention of \(R_{\omega}\) as a baseline, formulated as follows:  
\[G_{\texttt{sa}}(X,\{R_{\omega}\})=P_{\texttt{sa}}(\text{Att}(Q_{\texttt{sa}}( X),K_{\texttt{sa}}(X))V_{\texttt{sa}}(X)), \tag{6}\]  
where \(Q_{\texttt{sa}}(\cdot)\), \(K_{\texttt{sa}}(\cdot)\), \(V_{\texttt{sa}}(\cdot)\), and \(P_{\texttt{sa}}(\cdot)\) are all linear projections. These projections allow us to find robust features in the original image's patch and combine them to create an efficient rectification, guided by a secondary loss described in Section 3.4.  
**Reference-aware self-attention**: \(R_{\omega}\) patches are used as a comparison ground to decide the natural/corrupted state of \(X\). This approach selects patch features in \(X\) that are similar to those in \(R_{\omega}\) to correct patch features in \(X\) that most diverge from \(R_{\omega}\). Specifically,we use a slightly modified version of the scaled dot-product attention to find the attention weight for the correcting patches from \(X\):  
\[A_{\omega}^{\text{tr}}=\max(0,Q_{\text{rr}}(X)K_{\text{rr}}(R_{\omega})^{\top}), \tag{7}\]  
where \(Q_{\text{rr}}(\cdot)\) and \(K_{\text{rr}}(\cdot)\) are linear projections. We use the element-wise max function instead of softmax as we wish to use only positively relevant patch features in \(X\), expecting that the similarity to a natural feature works as a proxy to the relevance for robust correction. Likewise, the attention for the correction's targets in \(X\) is provided by the following:  
\[A_{\omega}^{\text{ri}}=\max(0,-Q_{\text{ri}}(R_{\omega})K_{\text{ri}}(X)^{\top}), \tag{8}\]  
where \(Q_{\text{ri}}(\cdot)\) and \(K_{\text{ri}}(\cdot)\) are linear projections. This time, we negate the similarity as we wish to target only patch features in the original \(X\) that are most different from natural features (this negation is used to emphasize our expectation and is unnecessary as it can be learned in \(Q_{\text{ri}}\) and \(K_{\text{ri}}\)).  
The final attention map is the weighted average of these attention maps, provided by the following:  
\[A_{\text{rsa}}=\sum_{\omega\in\Omega}a_{\texttt{cls},\omega}A_{\omega}^{ \text{ri}}A_{\omega}^{\text{rr}}, \tag{9}\]  
where \(a_{\texttt{cls},\omega}\) is the attention weight in \(A_{\texttt{cls}}\) in Equation (2), corresponding to the category \(\omega\), which provides the semantic relevance between the input and the reference for \(\omega\). Eventually, we compute \(G\) with this attention map as follows:  
\[G_{\text{rsa}}(X,\{R_{\omega}\})=P_{\text{rsa}}(A_{\text{rsa}}V_{\text{rsa}}(X)), \tag{10}\]  
where \(P_{\text{rsa}}(\cdot)\) and \(V_{\text{rsa}}(\cdot)\) are linear projections.  
**Reconstruction**: Similarly to the quotation module, we can reconstruct \(X\) directly from \(R_{\omega}\)'s. The attention weight can be provided by the following:  
\[A_{\text{r}}=\sum_{\omega\in\Omega}a_{\texttt{cls},\omega}\text{Att}(Q_{\text {r}}(R_{\omega}),K_{\text{r}}(X)), \tag{11}\]  
and \(G\) is calculated as follows:  
\[G_{\text{r}}(X,\{R_{\omega}\})=P_{\text{r}}(A_{\text{r}}V_{\text{r}}(R_{ \omega})), \tag{12}\]  
where \(Q_{\text{r}}(\cdot)\), \(K_{\text{r}}(\cdot)\), \(V_{\text{r}}(\cdot)\), and \(P_{\text{r}}(\cdot)\) are linear projections, respectively.