Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A. Paper summary:

- This paper proposes a trigger inversion framework that can be generalized to different types of triggers. Specifically, this paper introduces a new optimization problem on top of the previous trigger detection formulation with an input transformation and inversion function. Using two neural networks to approximate the transformation and inversion functions, the method can be generalized to all types of triggers. The result shows strong performance in reverse engineering trojan patterns. 
strength_and_weaknesses: B. Strength
- The visualization is good. 
- The novelties and motivations are strong.

C. Weakness
- Some of the design details are missing.
- The paper presentation can be improved.
- The results are not easy to interpret. 
clarity,_quality,_novelty_and_reproducibility: - Clarity can be improved. (-)
- The quality of this paper is good. (+)
- Novelty is strong. (++)
- Code is provided (reproducibility).  (++)
summary_of_the_review: D. Questions:
- ASR-Inj is applied in both Table 4 and Table 5 and it is introduced at the end of the evaluation function. Maybe you should reorganize the evaluation section a little bit. 

- You use ASR-Inv as the key metric throughout the paper. So I am confused by the goal of this paper. Aren't you trying to detect if a model is poisoned or not? What is the prediction accuracy of testing trojan models using your proposed method?

- Also, you never mentioned how you implement P and Q. How you designed the neural networks to fulfill the inversion and transformation? 

- A lot of ablation studies can be added. For example, the portion of the training dataset that is required to reverse-engineer the trojan. 


Overall, the novelty of this paper is strong.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new framework for inverting backdoor triggers from backdoor classifiers. Previous approaches is only applicable to patch-based backdoor. The authors proposed an unified approaches. Experiments on various different attacks demonstrate that the proposed approach is able to generate triggers with high attack success rate.
strength_and_weaknesses: Strengths:
1. This is well-written. It tries to propose a uniform framework for backdoor attack and it is novel in that respect.
2. The proposed optimization framework is general and can be applied to various forms of backdoor.
3. The authors have done extensive experiments to validate the effectiveness of the proposed approach.


Weaknesses:
1. I don’t quite follow section 3.3. Many notations, e.g. u and A is not well defined. For now there are only text descriptions. Please formally define these vectors in mathematic equation and write out the dimensions of these vectors. Also many equations are put inline so it becomes even harder to read.
On another note, you were saying the backdoor behaviors in intermediate representations in this part. Is there any empirical evidence to back this claim or is it proposed and established in previous work (if so, please cite the source)? I don’t see why this is obvious without any supporting evidence. 
2. The optimization problem in equation 3 involves many parameters to be optimized. Is it stable? I don’t feel a small paragraph in A.3 would convince me that this optimization problem is easy to implement and run in practice. Some plot on the training loss should help readers understand this optimization procedure better. Also how long does this procedure takes compared to previous approaches given that you are optimizing so many parameters at the same time?
3. Why would the formulation in definition 1 generalizes all those previous backdoors you listed? For example, you mentioned wrapping based backdoors, i don’t see what function of phi, m and t would recover such backdoor.
4. How do you decide this target label in equation 1? If you are running equation 1 for every class in the classifier, then how do you decide if a class is poisoned or not?
5. In Figure 3, the inverse trigger does not look similar with the original trigger in some cases, e.g. SIG/Filter/WaNet/BppAttack. Some discussions on this is needed.
6. There are some typos in equation 1. For the first equation, I think M is not needed.


clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is written clearly. 
Quality: The quality is good.
Novelty: The problem and the approach are both novel.
Reproducibility: The authors have provided the code in an anonymous link.

summary_of_the_review: I think this paper is interesting and the problem it tries to tackle is an important one. I am leaning towards accept. But i hope the authors can address the concerns i listed above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a backdoor inversion method called UNICORN. This approach is motivated by the fact that some backdoor attacks would apply their triggers in a space other than the pixel space. To alleviate this issue, UNICORN employs an objective function that uses an invertible transformation to map the image to a different space and apply the trigger. The goal is to generalize the previous trigger inversion methods that would only consider triggers applied in the pixel space. Experimental results indicate the success of the proposed approach in creating triggers that have similar characteristics to the original triggers and successfully fool the backdoored model.
strength_and_weaknesses: ### Strengths:
- The general idea of the paper is novel and interesting. Specifically, since the previous methods would only use a fixed objective function in the pixel space, they lack the universality to encompass all the existing backdoor triggers. UNICORN seems to be a step in the right direction.

 - A broad range of backdoor attacks and neural network architectures are used in the experimental results.


### Weaknesses:

The weaknesses can be divided into three broad categories:
#### **1. Motivation and Methodology**:
- The motivation behind the proposed approach during the introduction seems clear. However, as the reader goes through Section 3, some aspects start to be introduced without proper motivation. For instance, while the material presented in Section 3.3 makes sense, a reader might wonder why this information is presented there.

- The issue with the material presented in Section 3.3 shows itself more profoundly in Eqs. (1) and (2). A reader might say: "Okay, based on the material, I can understand that we need to map the image to a different space, but how did we get to a point where we need the *disentanglement constraint*? What was the authors' motivation behind adding this regularizer? More broadly, among all the regularizers in the world that one could have picked about backdoor attacks, why this particular one? Because similar types of properties about backdoor attacks have already been observed in the literature (e.g., backdoored models have a shortcut path when inferring backdoored examples [1, 2]). What is so specific about the proposed *disentanglement constraint* that makes it vital to the success of UNICORN?"

- In the leadup to Section 3, the paper mentions several times that "...[existing methods] assume that the trigger is a static pattern with a *small size* in the pixel space." Nevertheless, for Eq. (1) in Section 4, the paper says that: "similar to Neural Cleanse, we also constrain the size of the input space mask." These seem to contradict each other, as the paper argues that the assumption of having a small-size trigger is problematic, yet the same constraint is used! 

- The motivation behind trigger inversion needs to be more solid. The reader needs to understand what is the advantage of reverse-engineering the trigger. Otherwise, they could argue that a simple detection-based strategy can tell us if the model has backdoors or not, and we might not need the hassle of finding the underlying trigger.

- In the abstract, introduction, and background, it is mentioned that: "...existing work does not formally define the trigger and the inversion problem," which can create misinterpretations as those methods define the trigger inversion problem in their own mathematical ways. It would be better to replace these with something like: "...existing work does not consider the trigger's design space in their formulation of the inversion problem."

#### **2. Presentation and Writing**:
- Besides the motivational issues about the origins of the *disentanglement constraint* mentioned above, presentation of the paper, especially Section 4, needs lots of work. The reader suddenly faces an objective function in Eq. (1), where most of the terms have not been introduced already, and their definitions are deferred to the bottom of the page. Perhaps reasoning about different properties of the sought trigger and then presenting the mathematical expressions would be a better idea and makes the paper a lot easier to read and follow.

#### **3. Experimental Results**:
- The number of baselines used in the paper is limited. More recent trigger inversion baselines, such as [3], need to be added.

- While the experimental results show that the reverse-engineered triggers can have a high attack success rate, they do not tell us the other side of the story: false positives. It is common practice to show whether the trigger inversion helps detect backdoor models and report the detection accuracy among models. In other words, given a set of clean and poisoned models, how many instances could the proposed method identify the trigger correctly? How many clean cases still were identified as malicious?

- Since the model uses a trainable neural network for trigger inversion, it would be beneficial to see how it compares to existing methods (such as Neural Cleanse) in terms of the detection time and computational complexity.

- Ablation studies are limited. First, it is unclear what benchmark and dataset were used for the "effects of the mask size constraint." Secondly, the results in Table 3 need to be repeated for different triggers, especially those with large sizes. Finally, given the importance of the *disentanglement constraint*, what would happen if one adds this regularizer to the existing methods such as Neural Cleanse? Can this be used to boost their performance? If so, what is the role of the invertible transformation given in Eq. (1)?

- In Table 5, how is the reverse-engineered trigger's success rate better than the original one?

- The effects of different hyper-parameters and how they are set seems missing.

[1] Zheng et al. "Topological detection of Trojaned neural networks," *NeurIPS*, 2021.

[2] Li et al. "Deep learning backdoors," *Security and Artificial Intelligence*, *Springer*, 2022.

[3] Hu et al. "Trigger hunting with a topological prior for trojan detection," *ICLR*, 2022.
clarity,_quality,_novelty_and_reproducibility: Detailed comments are given in the **Strength And Weaknesses**. Below is a summary:

#### **Clarity**: as mentioned in the weaknesses above, several parts, such as the proposed method, need to be rewritten to help with the clarity.

#### **Quality**: overall, the paper is of moderate quality. Although the work has some positive aspects (such as the motivation for looking into the inversion problem in a different space than the pixel space), the motivations behind different parts of the methodology are obscure. Moreover, the experimental results require a deeper investigation and a broader set of baselines.

#### **Novelty And Reproducibility**: while the proposed method seems novel, certain aspects of the work need proper motivation. In terms of reproducibility, the authors provide the codebase. However, a detailed account of the hyper-parameter selection is missing. For instance, while $\gamma$ and $\delta$ values were given on page 5, it is unclear how those specific values were found.
summary_of_the_review: Based on the grounds given above, the paper seems to require a better explanation of the method and better conduction of experiments. Thus, this reviewer would recommend rejection at this stage but would be happy to increase the rating if the authors can address the abovementioned concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper provides a formal definition and analysis of triggers in the context of a backdoor attack on Deep Neural Network (DNN) models. It then proposes a unified framework to invert backdoor triggers based on the formalization of triggers and the identified inner behaviors of backdoor models. Finally, the authors provide a prototype called UNICORN which is general and effective in inverting backdoor triggers in DNNs.

Strengths:
- Well-written paper proposing a uniform framework for backdoor attack, Visualization is good 
- General optimization framework can be applied to various forms of backdoor
- Extensive experiments to validate effectiveness of proposed approach
- Code is provided (reproducibility) 
- A broad range of backdoor attacks and neural network architectures are used in the experimental results.

Weaknesses:
- Paper presentation and clarity can be improved (eg Motivations behind the proposed approach during the introduction/Section 3.3 not well defined and notations not formally defined/Section 4)
- Related to point above: Some of the design details are missing and sometimes the results were not easy to interpret 
- Questions still remain on implementation of P and Q, prediction accuracy, and ablation studies
- No empirical evidence to back claim of backdoor behaviors in intermediate representations
- Optimization problem involves many parameters (and is not well explained, related to point 1 above)
- Not clear how target label in equation 1 is decided
- Inverse trigger does not look similar to original trigger in some cases
- The number of baselines used in the paper is limited, and experimental results do not highlight cases of false positives.
-The computational complexity of the proposed methods need to be compared against baselines methods.


Although the reviewers noted many points of concern the rebuttal was very thorough from the authors perspective, and addressed many of them. The authors were quite extensive in providing many points of clarification (which was in fact the biggest concern of all reviewers). Some reviewer engagement was present post rebuttal, and all converged (including myself) on a positive assessment of the novelty and intellectual contribution of this work. 


justification_for_why_not_lower_score: The work presents significant and novel insights on a very relevant and technical challenging problem.
Decision: Accept: notable-top-25%