Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposed a method of attacking deep neural networks that were trained using transfer learning. The primary claim is that the proposed technique only requires knowledge of the base model (i.e., if the frozen parameters taken from pretrained VGG, ResNET models are known, then that is sufficient) and doesn’t require any samples of target classes of the post-transfer application to successfully attack the DNN.

Core idea is that Softmax layers used in classification can be exploited to craft a perturbation that will fool the network. The algorithm uses a brute force approach to iterate through each neuron on the final layer before the softmax. For each activated neuron, all other neurons are zeroed out and a regularized loss is computed between the activated feature and the neuron. The gradient of the loss function is then used to craft a perturbation using typical k-step gradient based method (as used in FGSM, PGD attacks). In other words, they are trying to craft a perturbation that will zero out all other class probability and put maximum weight on the target adversary class when applied to the network.

Experiments were conducted on VGG-Face and Pannous Speech dataset.

1. Paper is easy to follow and different empirical results show a few intuitive observations for a few different settings. 

2. The main issue is that the algorithmic is rather simplistic and seems impractical. Most security applications that motivate the paper can easily defend themselves by simply limiting the number of attempts. Even if this core motivation issue is discarded, the algorithmic contribution is very minimal, no analytical understanding either. 

3. Some comparison with Black Box models should be added. Although authors claim that Black Box models would require many queries, a specific comparison and contrasting with number of attempts and queries would help understand the critical advantages of the proposed technique. 

4. Table 1 provides a good overview of the results but it should include standard deviations for experiments that were averaged.

5. Is there a notion of imperceptibility/attack budget here? Some of the attacks are clearly not imperceptible. This should be discussed. 

6. Section 6 seems a little brief and can use more details. Authors claims that using EVM, “the model successfully defeat all our crafted images” but “there is 7.38% chance that the EVM-base model classifies the input as one of the target classes”. This statement is a little confusing.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors proposed an attack scheme to any model that pretrained from a general model. The merit comes from that the attacker, by taking advantage of the vulnerability of softmax, has no access to examples from the target task to finetune the model.  

Pros:
-	The setting where the authors focused on is much more practical, i.e., the attacker is blind to examples in a target task.  
-	The blackdoor for attack in this work, namely the softmax layer, is novel and interesting to me, at least to my knowledge.
-	The paper is well written and easy to follow.

Cons:
-	In the experiments, the authors should still compare one or two black-box attacks which also use model outputs only, to demonstrate the effectiveness/efficiency of the proposed attack scheme. 
-	Considering the design of \gamma and \beta in Eqn. (1), it is expected to investigate the influence of both on the performance of the attack, as well as the vanilla version of Eqn. (1) by only considering MSE.
-	Currently, it seems that the authors did not study the influence of different target datasets to finetune. It is highly expected that the same adversarial input crafted using Alg. 1 could fool all networks finetuned using different datasets. What if a wildly different dataset is used to finetune the pretrained model?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes an interesting new adversarial attack concern: any machine learning that use a linear classifier on an off-the-shelf feature extractor, suffers the risk that the feature extractor has adversarial examples that output arbitrary feature vectors. The paper proposes a concrete attack where a set of samples is crafted to activate each feature vector. 
 

Pro:

The setup is interesting and seems novel. To my knowledge, attack on the features used in transfer learning has not been considered in the literature. Several adversarial attack papers consider attacking the network layer by layer, or attack an intermediate layer, but they are different from the proposed setup in this paper. 

The proposed attack is simple but effective (though unsurprisingly). 

The experiments show reasonable performance even beyond the motivating setup, for example, with fine-tuning or with non-linear classifiers on the feature space. 

Con:

I think the range of the input feature (to the linear classifier) should be reported. A simple check on the range of the input feature should be a good defense. Therefore, the input feature should not be unreasonably large. 

I think the experiments need comparison with the naive baseline by simply trying random faces. It seems that without any attack, it should be possible to try random faces and find one that is incorrectly classified. Assuming the model it well calibrated, among the time it says it is 99% confident, it should still make a mistake with approximately 1% probability. In addition, deep networks are known to be not calibrated and over confident. 

I do not think the attack is task-agnostic, because the attacker still needs to query the final linear layer as a black box to find which feature unit to activate. This is a special hybrid setup, where the feature layer is white-box while the final layers are black box. I think task-agnostic is somewhat an oversell. 


Overall I believe this paper is okay. It explores a novel task with a somewhat interesting approach, with sufficient empirical support. 


review_title: Official Blind Review #3

Meta-review: The reviewers were generally in agreement that the paper presents a valuable contribution and should be accepted for publication. However, I would strongly encourage the authors to carefully read over the reviews and address the suggestions and concerns insofar as possible for the final.
Decision: Accept (Poster)