Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a diversity loss and a topological prior to not only increase the chances of finding the appropriate triggers but also improve the quality of the found triggers. These loss terms significantly improve the efficiency in finding trojaned triggers. The experiments results show that the proposed method performs substantially better than the baselines on the Trojaned-MNIST/CIFAR10 and TrojAI
datasets, respectively.
main_review: This paper proposed a novel target-label-agnostic reverse engineering method for reverse engineering in recovering trojaned triggers on clean images. The proposed diversity loss and topological prior significantly improved the performance in finding trojaned triggers and the quality of the found triggers compared to other existing methods. The proposed method is novel, the experiments were well conducted, and analysis was well performed. 
summary_of_the_review: This paper proposed a novel target-label-agnostic reverse engineering method for reverse engineering in recovering trojaned triggers on clean images. he proposed diversity loss and topological prior significantly improved the performance in finding trojaned triggers and the quality of the found triggers compared to other existing methods. The proposed method is novel, the experiments were well conducted, and analysis was well performed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a trojan detection method using reverse engineering techniques. It improves existing approaches by incorporating a trigger diversity loss and a topological loss during trigger generation. The diversity loss aims to generate a set of triggers with different patterns and locations. The topological loss is to make triggers more connected with fewer perturbations spread out on the input. The paper builds a neural network to detect trojaned models by using the features of generated triggers such as the number of trigger pixels, the mean and standard deviation of triggers in horizontal and vertical directions, etc. This paper evaluates the proposed approach on two synthetic datasets and TrojAI benchmarks.
main_review: The introduction of the diversity loss and topological loss is interesting and seems to improve the quality of generated triggers. The evaluation is conducted on a large benchmark. I have the following concerns with the paper.

1. The current approach seems to heavily rely on a large set of models with the same or closely related task. A reasonable detection result (>0.8 AUC) requires at least 50 models as shown in Table 3. This does not seem practical. How can a defender acquire such a large set of clean models? Even if the defender can obtain a set of clean models, the task for these model also need to be similar, e.g., traffic sign detection in TrojAI datasets. What if these models have different tasks? Can the proposed approach still work? The paper needs to be better motivated why having a large set of clean models with similar task is practical and realistic.

2. Following up the first point, the comparison with several baselines does not seem fair. NC, ABS, and TABOR do not require any clean models for training. The detection is directly carried out on a given model (either benign or trojaned). The proposed approach relies on the assumption of having a set of clean models, which has a different threat model than these baselines. A fair comparison would be some modified version of NC, ABS or TABOR that also takes a set of clean models into consideration during detection. For instance, the defender could first run these baselines on the clean set and get an estimation of how clean models would behave. She then uses this information to guide the detection.

3. A recent state-of-the-art approach (Shen et al.) is not compared. The paper only briefly mentions it in the related work but does not empirically compare with it. The results from the original paper (Shen et al.) are very similar with marginally differences. The experiment of the proposed approach is only evaluated on 10% of the TrojAI datasets as it requires training and validating on the remaining 90%, whereas Shen et al. evaluated on the whole set. The proposed approach can be better compared with the baseline by presenting the results from TrojAI test set that is evaluated remotely by the TrojAI organizer. This can rule out the possibility that the approach does not overfit on the local training set as this is a training based method.

4. There is no discussion and evaluation on adaptive attacks. When knowing the existence of the proposed detection, the attacker can trojan a model by maximizing the mask using the losses in Equation (2). She can also add those reverse engineered triggers in the training set during poisoning to counter the effect of generating a set of small triggers. The trojan detection network can also be considered by incorporating in the loss function.

5. The paper does not clearly separate the contribution of existing works and this paper. For instance, the trigger reverse engineering technique by using a mask and a pattern in Section 3.1 is proposed by NC. Utilizing continuous values instead of binary values for the mask is also proposed by NC. This should either be discussed in a section regarding existing works or explicitly credits the contribution to existing works.

6. From the ablation results in Table 3, without the topological loss or the diversity loss, the results are already better than evaluated baselines. Why is this the case? What are other factors contributing to the final good detection results? Without the above two losses, it is more or less NC, which should have a very low detection result. Do the author care to explain in detail which component of the approach leads to the good results.

7. The presentation needs improvement. Equation (2) introduces three losses, namely, the label-flipping loss, diversity loss, and topological loss. The first two losses are discussed in Section 3.1, while the topological loss is presented in a separate section, Section 3.2. These three losses should be at the same section level. The caption of Figure 5 is hard to be distinguished from the main text. There are no images with the injected triggers in Figure 6.
summary_of_the_review: 1. The setup does not seem practical.
2. The comparison with existing works is unfair.
3. There is no comparison with a state-of-the-art approach.
4. The paper is not evaluated on adaptive attacks.
5. It is not clear about the contribution of different components.
6. The presentation of the paper needs improvements.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a method for the detection of backdoored neural networks.
It is assumed that the user only will have access to the trained model and a few clean validation samples, not the training data.
As such, a reverse engineering approach is taken to solve the task: i.e. using the validation samples, the method aims to recover possible backdoor triggers.
To this end, a loss function with three terms is minimized:

1. a _label flipping_ loss to extract triggers that can subvert the classifier's decision,
2. a _diversity_ loss that aims to generate various triggers with different patterns, and
3. a _topological_ loss that tries to encourage the compactness of the recovered triggers.

Based on these loss terms, triggers are reverse engineered for a dataset containing malicious and benign models.
Then, using these triggers a set of features are built for each model (Trojaned vs. benign).
Using these features, a binary classifier is trained to predict if the models are backdoored or not.
The performance of the proposed approach is tested under various experimental settings and ablation studies.
main_review: ### Strengths and Weaknesses:

+ The paper is very well-written.
It flows well and provides extensive explanations on abstract topics such as persistent homology to help the reader understand them easily.

+ The proposed approach seems novel and promising.
The experimental results are comprehensive, and are done on synthetic (MNIST and CIFAR-10) as well as real-world (TrojAI) backdoored models.
It is shown that using the proposed approach one can get significant improvements over state-of-the-art baselines such as Neural Cleanse [[1](https://ieeexplore.ieee.org/abstract/document/8835365)] and DL-TND [[2](https://arxiv.org/abs/2007.15802)].

+ Although the use of topological loss seems to be enhancing the detection of backdoor models, an explanation as to why this choice is valid is somehow missing.
It is pointed out that this way the topological loss would limit the search space of all possible triggers, which as a result would increase the chances of finding the true one.
While this makes total sense, it also brings a natural question to mind: are all the possible triggers compact?
In other words, how does the introduced topological loss behave in case the adversary poisons the model with a scattered trigger?
I think this question needs to be answered to help the readers understand why a topological loss needs to be employed.

### Minor Comments/Questions:

+ To minimize Eq. (2), a sum over all the validation samples is taken, or they are minimized separately? If it is the latter, please add a summation to indicate this.

+ In Table 3, the number of samples means the number of models in the training data or the number of validation samples used for reverse engineering?

+ In the assumptions of Section 3.1 $\mathbf{x}$ needs to be in $\mathbb{R}^{3\times M \times N}$ to accommodate the three RGB channels.
summary_of_the_review: This work introduces a novel approach for the detection of backdoor neural networks.
The performance of the approach seems to be promising.
However, there are some speculations around the use of topological loss for detection as the compactness of the triggers is not a necessity in backdoor attacks.
If the authors can elaborate on this assumption, I would be happy to increase my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel method for trigger detection against trojan attacks (especially the TrojAI benchmark). The key idea is to introduce the diversity penalty and the topological simplicity to help find more different high-quality triggers in the image domain.
main_review: Strengths:
1. The experiments show fantastic Trojan detection accuracy under the TrojAI benchmark settings.
2. The idea of the diversity penalty is simple and straightforward. I like the consideration of unknown target labels and multiple triggers, which may be the closest to the real scenes.
3. A very detailed ablation study for the contribution of each item.

Weaknesses:

1. It took me a while to understand the idea of topological simplicity and why such prior exists in the trojan attack settings. The motivation needs to be further clarified. For example, can the embedded trigger be a set of scattering points of pixels? If can, will the topological simplicity exist? This also leads to another question about the quality evaluation of recovered triggers. In fig 6, It is doubtful to consider the “more compact” as a good metric for triggers.
2. The second limitation of the proposed method is the training of the detection network, which needs annotation of the trojan model. To my knowledge, a very unique characteristic of a trojan attack is its stealthiness. Once the type and method of attack are exposed publicly, the threat does not exist. Are there any possible ways to apply the proposed method in a more realistic scene without the annotations of the trojaned model? 
3. It will be better if the authors provide a more detailed implementation of baselines since the performance increase is very astonishing. Also, a clear analysis of the impact of trigger types is better to be exhibited.
summary_of_the_review: This paper is well-written and proposes a pretty novel trigger hunting method. However, the limitation of this method is also very obvious, not good enough for real threats. I will increase my score if the authors can provide more insights under more general settings of trojan attacks except for only TrojAI benchmarks.
review_title: null

Meta-review: This paper proposes a diversity loss and a topological prior to not only increase the chances of finding the appropriate triggers but also improve the quality of the found triggers. These loss terms significantly improve the efficiency in finding trojaned triggers. The experiments results show that the proposed method performs substantially better than the baselines on the Trojaned-MNIST/CIFAR10 and TrojAI datasets, respectively. This paper shows detailed ablation study with great empirical performance. Some reviewers have doubts about the experimental comparisons and some of the assumptions made in the algorithm. Overall, the thorough experimental investigation fo the proposed method makes this paper worthy of publication and widely being shared.
Decision: Accept (Poster)