Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This submission considers how to detect Trojaned Neural Networks from a particular feature extraction method - which the authors call topological descriptor. Overall, my impression is that the paper neglects the duty of defining the problem in a concise, mathematical language - I still don't understand what the proposed algorithm does to extract features, descriptions are too handwavy. 
If assuming some black-box method is given to extract (topological) features of fully-trained neural network, main claim is that it is easy to classify whether a (fully-trained) Neural-Net is Trojaned or not, a reduced-task which I believe also should be formally defined. Empirically, extracted features are shown to be well-separated between Trojaned and Non-Trojaned neural networks. 
main_review: Writing

There are too many places where descriptions should be more concise and precise. Here are a few examples

1. Line 116-122: what is "m" and "x"? Are they vectors or matrices? How are (1-m) and multiplication defined?

2. Line 146-153: what is, and how do you compute $\rho$? What are neurons here? Please define it mathematically.

3. Line 168-177: on birth/death time: where the notion of "time" comes from? In which procedure?

There are also too many jargons which are not very standard in ML community, like "persistence diagrams" or "cycles of high persistence" in Line 184-185. They are not defined anywhere in the paper. Overall the paper is very hard to read and follow.




- Theory

Theories do not back-up the idea anyway either. For instance,

Theorem 4.1: a cherry-picked good example (Gaussian ones here) does not justify your method. It may be used to give a good intuition, but it doesn't go the other way - your method should be general enough to handle broad class of examples.

Theorem 4.2: This is not much more than a "uniform concentration bound" for a finite function-class. And what is $\psi$? is it ever defined before? 


- Experiment

I don't understand why your experiment should suddenly be in data-limited setting, while your theory is based on the law of large numbers. Numbers in Table 1 seems impressive, probably this is the strength of the paper. 



Overall, I think the paper is poorly written and should be significantly revised, but there seems to be potential as an empirical work to detect Trojaned Neural Networks. At this time, I recommend rejection. 

limitations_and_societal_impact: See above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors investigate an approach to detecting Trojan Networks, using topological features based on the correlation matrices of the network activations.  They provide primarily empirical evidence that such features are discriminative for Trojan networks, including a comparison of the method against previous approaches on a synthetically designed task using MNIST and CIFAR10 datasets.  The further provide a theoretical analysis on a toy problems (Th. 1) and provide an analysis of the estimation error of the persistence diagram necessary for their approach (Th. 2).
main_review: The results are interesting, and provide a compelling case for the usefulness of topological features in detecting Trojan networks.  This is the primary contribution of the paper, which appears to be a robust phenomenon in the contexts they test.  The theoretical analysis of the toy problem in Def. 1 and Th. 1 is less convincing, since this appears to establish only that in this setting, topologically distinct models can be found for the clean/Trojan settings, and not that this must or is more likely to be the case (indeed, the same may be true for any distinct conditions, say with the output labels grouped into meta-classes).  Also, the definition of what counts as a trigger (Sec 2) appears subjective, although this is not specific to the paper.  However, the paper is well written and clearly presented, appears to be theoretically and empirically sound in terms of the claims made, and the phenomenon observed is likely to be of interest, and may lead to further work on theoretical understanding.

Minor points:

The paper should be checked for typos before the final version.
limitations_and_societal_impact: Not relevant.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents an approach for the detection of malicious backdoored neural networks using persistent homology from topological data analysis.
First, a neuron connectivity graph is built.
This is a complete weighted graph whose nodes correspond to different neurons of a neural network.
Each edge connects two different neurons, and its weight is set to be negatively proportional to the correlation of the activation vectors of the two neurons evaluated for the input data.
Having this graph, its topological features are extracted using the Vietoris-Rips filtration.
Using this framework, it is then argued (both theoretically and empirically) that there exist topological dissimilarities between clean and poisoned models.
Based on this topological analysis, it is shown that in Trojaned models: (1) the neurons exhibit a larger correlation with each other, and (2) there exists a short-cut path connecting shallow and deep layers.
Further, a theoretical guarantee is provided to show the convergence of the estimated topological features using sufficient data samples to the true one.
Finally, a practical algorithm for backdoor model detection is proposed.
The experiments indicate the better performance of the proposed approach compared to well-known baselines such as neural cleanse [1], universal litmus patterns [2], and data-limited Trojaned network detection [3].
main_review: ### Strengths and Weaknesses:
+ First, the paper is very well-written.
Considering the novelty of persistent homology to the machine and deep learning audience, this paper does an extremely good job of describing such concepts in a straightforward, yet informative, way.
Nonetheless, it would benefit from careful proofreading. See the additional feedback for a few minor suggestions.

+ More importantly, the contributions of this paper are sound and promising.
The proposed approach sets up a novel direction in the analysis of Trojaned neural networks using topological data analysis.
These findings are analyzed meticulously, shedding light on the behavior of neural networks under backdoor attacks.
Although that some of these behaviors, such as the existence of a short-cut in backdoored models, have been previously discussed (see [4] for e.g.), this paper uses a different, novel route to reach the same conclusions.

+ The empirical results indicate the potential of the proposed approach in the detection of backdoored neural networks in contrast to the state-of-the-art.

+ The main concerns with the current version are three-fold:

	1. The relevance of the theoretical study in Section 4.1 to Trojan attacks is obscure.
	Specifically, the data that is used in this study do not resemble the general settings in backdoor attacks:
	
		- number of poisoned and clean data are equal, and
		- the final poisoned dataset $\mathcal{D}_3$ is completely different from the clean data distribution, and one cannot expect that training a model on it can be equal to training on the clean data.
		
	Perhaps a point is missing here, and this should be made clear in Section 4.1.
	
	2. Throughout the paper, especially for the empirical studies in Section 4.2 which form the foundation of the proposed approach, it is assumed that the data poisonings are done in a dirty-label fashion, meaning that the attacker flips the labels besides injecting triggers.
	In the past couple of years, there has been a good progress in stealthy backdoor attacks that do not change the labels, e.g., see [5].
	To make more conclusive judgements about Trojaned models, such attacks should perhaps be considered.
	
	3. Possible shortcomings of the proposed approach are not discussed.
	For instance, does the current approach only work for the detection of networks with the same architecture?
	In other words, can one train the detector on say ResNet-18, and detect malicious DenseNets?
	Furthermore, does the poisoned models used for training and testing use different triggers?
	Or the training and test models should be poisoned with the same triggers?
	These are key questions when it comes to backdoor model detection and should be shown through experiments.

### Additional Feedback:
+ It would be beneficial if the Gebhart and Schrater [6] paper regarding the detection of adversarial attacks in neural networks using persistent homology is added to the list of related work.

+ Line 108 says "These Trojaned samples are assigned specific target class labels – different from the labels of the original training samples."
As stated before, this is not always the case.
Consider adding a few sentences to state that this is the assumption in the paper.

+ In the Vietoris-Rips filtration, why $t$ needs to be varied from $-\infty$ to $\infty$?
Because the edge weights are always between -1 and 1.

+ In Definition 1, datasets $ \mathcal{D}_2 $ and $ \mathcal{D}_3 $ should use $\mu_j$ instead of $\mu_i$.

+ Consider adding different hatches and line-styles to Figures 2 and 4 to help with their readability for the color-blind.

+ In Figure 4, apparently some of the sub-figures are missing since the caption has 4 parts but there are only two sub-figures.

+ In Algorithm 1, does one have to use a different testing for each poisoned model?
Or the same one can also be used multiple times?

+ Minor suggestions, typos, and grammatical errors:
	1. Line 3 and 4: do not split the paragraphs in the abstract.
	2. Line 15: "attach" -> "attack"
	3. Line 75: add a few references for the sentence "Few methods investigate..."
	4. Line 116: "Let clean dataset $\mathcal{D}$..." -> "Let the clean dataset be $\mathcal{D}$..."
	5. Theorem 2: "iid" -> "i.i.d."
	6. Line 286: "can we" -> "we can"
	7. Line 297: "is provide" -> "is provided"
	8. Line 298 and 299: "(or a small patch) modify its value." -> "(or a small patch) and modify its value."
	9. Line 349 and 350: "These topological methodology lead to..." -> "This topological methodology leads to..."
	10. Line 155: "is a collections of..." -> "is a collection of..."

#### References

[1] Wang, Bolun, et al. "Neural Cleanse: Identifying and mitigating backdoor attacks in neural networks." _IEEE Symposium on Security and Privacy (SP)_, 2019.

[2] Kolouri, Soheil, et al. "Universal litmus patterns: Revealing backdoor attacks in CNNs." _CVPR_, 2020.

[3] Wang, Ren, et al. "Practical detection of Trojan neural networks: Data-limited and data-free cases." _ECCV_, 2020.

[4] Li, Shaofeng, et al. "Invisible backdoor attacks on deep neural networks via steganography and regularization." _IEEE Transactions on Dependable and Secure Computing_ (2020).

[5] Turner, Alexander, Dimitris Tsipras, and Aleksander Madry. "Label-consistent backdoor attacks." _arXiv preprint arXiv:1912.02771_ (2019).

[6] Gebhart, Thomas, and Paul Schrater. "Adversary detection in neural networks via persistent homology." _arXiv preprint arXiv:1711.10056_ (2017).
limitations_and_societal_impact: The potential negative societal impacts are not discussed.
Perhaps you should discuss how the method developed here can be exploited by an adversary.
review_title: null

Meta-review: This is an interesting paper with a creative, deep and promising approach.
Moreover, the thorough rebuttal and subsequent discussion seems to have addressed successfully the majority of the main comments the reviewers have raised. 
Decision: Accept (Poster)