Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method for generating adversarial examples for satisfying multiple goals based on multi-objective optimisation (MOO). The paper can be classified into the family of adversarial attack algorithms. A multi-gradient decent algorithm is utilised to solve the optimisation problem by integrating all gradients w.r.t. different objectives into a single gradient direction, but the optimisation process of MOO always tends to treat all tasks equally. Therefore, a regularisation term is proposed to emphasise unachieved tasks while putting less weight on already satisfying tasks. The paper then gives four experiments, including adversarial attack and training over multiple models, universal perturbation, and adversarial attack over data transformations, to validate the utility of the proposed approach.
strength_and_weaknesses: Strength:

1. The proposed method is basically considering a generating attacks scenario where multiple domain losses are needed to be optimised together simultaneously. The insight behind the technique is to balance the weights of all tasks, and a regularizer is added to push greater weights on unsatisfied tasks.

2. The paper is well-written, and the main idea is easy to follow.

3. Comprehensive experiments show that the proposed method is better than the uniform case and minimax case.


Main concerns:

1. The novelty of such formulations is not clear to the reviewer. The main contribution claimed by the authors is that this is the first work that regards adversarial attacks as a multi-objective optimisation problem. But, there are some works in the literature which takes this formulation into consideration in adversarial example generation scenario (especially, e.g., [1], [2]).

2. This problem and considered scenarios are similar in spirit to [2]. The proposed regularizer added to the optimisation objective function is a fairly basic and straightforward trick to weigh different domain losses. It cannot be considered a sufficient contribution for an ICLR paper.

3. From the perspective of multi-objective optimisation theory, any analysis in terms of convergence and Pareto-optimal frontier is essential.
Reference 

[1] Qiu, H., Du, Y. and Lu, T., 2022. The Framework of Cross-Domain and Model Adversarial Attack against Deepfake. Future Internet, 14(2), p.46.

[2] Wang, J., Zhang, T., Liu, S., Chen, P.Y., Xu, J., Fardad, M. and Li, B., 2021. Adversarial attack generation empowered by min-max optimization. Advances in Neural Information Processing Systems, 34, pp.16020-16033.

clarity,_quality,_novelty_and_reproducibility: See Section Strength And Weaknesses

summary_of_the_review: See Section Strength And Weaknesses
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use multi-objective optimization to generate adversarial examples for model ensembles, adversarial examples against several transformations, and universal adversarial perturbation. Specifically, by setting the different models in the ensemble as different objectives, it turns the problem into multi-objective optimization. To further improve the performance, a regularizer is then proposed to suppress the successful attack and encourage the unsuccessful attack. Extensive experiments have been done in several cases including generating adversarial examples for model ensembles, adversarial examples against several transformations, and universal adversarial perturbation to show the effectiveness of the proposed method. The experiment results show the proposed method could achieve a better performance compared to several baselines.
strength_and_weaknesses: Pros:
1. The paper is well-written and easy to follow.
2. The experiment results show that the proposed method could achieve a good result in generating adversarial examples in several settings.


Cons:
1. To my understanding, the proposed framework could only be applied into the ensemble that uses the max, min, and majority vote as its aggregation rule. It is not clear if the proposed method still works if the ensemble is taking the average of every model's output.
2. The proposed settings are kind of similar to each other. A better setting will be generating adversarial examples with the different norms and proposing a defense to defend against several norms attacking at the same time. 
3. I am not quite sure if the novelty is strong or not since the proposed method just applies multi-objective optimization. Also, I am not sure whether the proposed regularizer is already proposed in the multi-objective optimization before or not.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written and easy to follow.

Quality: The experiment is extensive and detailed. 

Novelty: The novelty is limited if the proposed regularizer is already proposed in the multi-objective optimization before because the proposed method just applies multi-objective optimization to generating adversarial examples.

Reproducibility: Although the code is not provided, all hyperparameters are listed and I believe the proposed method should be reproducible.
summary_of_the_review: The paper proposes to use multi-objective optimization to generate adversarial examples in several settings including model ensemble, universal perturbation, etc.  The experiment shows the proposed method could achieve good results in several settings. However,  as all the settings use the max as its ensemble, it is not clear whether the proposed method could be used in other ensemble cases.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The other suggest to extend the multi-objective optimization (MOO) method. Their new proposed method is called Task Oriented MOO. They claim that naive MOO invest useless effort in trying to maximize already achieved goals, their method let the optimizer spend more effort on improving the goal-unachieved tasks. They formalize the adversarial generation task as a multi-objective optimization problem and apply their TA-MOO approach to the problem. The authors show comprehensive experiments in which their method is better than the current baseline.
strength_and_weaknesses: ### Strength  
The paper is written in a clear and easy to follow fashion. The results seems to improve the baseline and the MOO method by a nice margin most of the time. The method and formulation, therefore, make sense to the adversarial generation task. The authors are the first to view the adversarial generation task as a MOO problem. The discussions and comprehensive experiment are clear and supporting the understanding of the method.

### Weaknesses
The paper did not fully explained the evaluation metrics. Why does the A-All is the most important metric for this task?
clarity,_quality,_novelty_and_reproducibility: The paper is well written and very informative to the reader and community. The paper show originality by casting the adversarial generation task as a MOO problem and applying MOO algorithms to the problem. The paper present promising results improving the baseline with a large margin most of the time.
summary_of_the_review: The paper is original and improves the baselines. The paper is written in high quality fashion.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a multi-objective optimization approach capable of dealing with objectives that converge at different rates. The authors then claim that three problems within the realm of generating adversarial examples can be cast as multi-objective optimization and apply the proposed Task Aware Multi-objective optimization (TA-MOO) algorithm for these problems. The main rationale behind TA-MOO is to augment the optimization problem to find a common descent direction within the standard multi gradient descent (MGD) with a regularization term that enforces the weights of the common descent direction to be larger for objectives with smaller losses (in the case of a problem of maximization of objectives). Experiments on two datasets, namely CIFAR-10 and CIFAR-100, considering the tasks of generating i- adversarial examples for an ensemble model, ii- universal perturbations, and iii- adversarial examples against transformations showed that TA-MOO outperformed the compared approaches in most of the cases.   
strength_and_weaknesses: Strengths:
- The authors proposed a general multi-objective approach that showed promising results on three different (but related) problems related to adversarial examples generation;

- The proposed algorithm, TA-MOO, has a mechanism to enforce the solutions of the multi-objective problem to focus on the central regions of the Pareto front, which is a desired feature in the case of the studied applications;

- The experiments on generation of adversarial perturbations are broad in the sense that they account for different aspects of the considered problems. 


Weaknesses:

- Lack of motivation for the contribution: one of the main claims of this work is “multi-objective optimization is a natural tool for adversarial example generation”, however, other than the fact that there are multiple objectives to be simultaneously satisfied, I am not able see why this is the case since there is no clear evidence that there is a tradeoff between the objectives when generating adversarial examples;

- Limited evaluation: the main contribution of this work is proposing a variation of a gradient-based multi-objective optimization algorithm capable of dealing with objectives that converge at different rates. However, it is not clear to me whether the choice of problems for evaluation is in fact suffering from the issue TA-MOO is proposed to solve. Moreover, the experiments in the main paper do not analyze important aspects of the proposed algorithm, such as if it is able to reach Pareto stationary solutions in practical applications;  
 
- Lack of strong baselines: I believe using unconstrained MGD (referred to in the manuscript as MOO) is not a fair choice of baseline in a case where it is known a priori that the desired solutions are in the central region of the Pareto front since MGD has no mechanism whatsoever to incorporate user preferences to guide the optimization. In the following section of the review I suggest stronger baselines to be considered. 

clarity,_quality,_novelty_and_reproducibility: - The authors claim that in some cases multi-objective optimization does not work. However, it is not clear to me if this is indeed the case. In more details, the purpose of (unconstrained) multi-objective optimization is to find a point within the Pareto front, and, perhaps in the case of the reported results MOO was able to find point in a region of the Pareto front that does not yield equally good performance across all objectives, which means it is actually “working”. I would like to learn more about the authors’ opinion on this aspect and suggest a more detailed discussion is added to the manuscript.

- Importantly, it is still unclear to me whether there are indeed conflicts between the objectives in the considered test cases within the paper. Notably, previous work showed evidence that adversarial attacks are transferable across different models and architectures [1, 2]. Therefore, I believe that in such cases the conflict between gradients should be negligible. Moreover, since there are no theoretical guarantees that TA-MOO finds Pareto stationary solutions, empirical evidence needs to be provided to illustrate why the proposed algorithm works. As an example, the authors can report quantities such as the norm of the common descent direction, since for Pareto stationary points this quantity should be close to zero. Finally, as I mentioned in my review, I strongly believe the authors should include in the empirical evaluation further gradient-based multi-objective approaches that aim at finding solutions in the central region of the Pareto front.

- Since in the case of MOO / MGD there is no constraint to enforce the solutions to reach the desired preferred region of the Pareto front, could it be the case that a particular initialization of MOO would suffice to make it find the desired solutions? Also, could a previously proposed constrained version of MGD such as [3] be able to do so and potentially achieve a performance as good as TA-MOO?

- Recent work [4, 5, 6] on multi-task learning has shown that well-tuned, simple baselines such as scalarizing the objectives with fixed weights equal to 1 random weights, yields solutions as good as the ones found by some multi-objective approaches, including MGDA, the base method for TA-MOO. Given that, and the fact that it seems that the tasks considered in this submission are similar to the considered multi-task settings mentioned in [4, 5, 6], I wonder if similar conclusions would be drawn in case extensive tuning of baselines were to be performed.

- As mentioned in the previous section of the review, the manuscript lacks in the choice of multi-objective optimization baselines. For example, I believe multi-objective approaches that aim at finding solutions lying in the central region of the Pareto front [7, 8] should be included, as well as methods that have been shown to prevent the bias issue in MGD [9].

[1] Liu et al. "Delving into Transferable Adversarial Examples and Black-box Attacks", 2016. \
[2] Che et al. "A New Ensemble Adversarial Attack Powered by Long-term Gradient Memories", 2019. \
[3] Lin et al. "Pareto multi-task learning", 2019. \
[4] Kurin et al. “In defense of the unitary scalarization for deep multi-task learning”, 2022. \
[5] Xin et al. "Do Current Multi-Task Optimization Methods in Deep Learning Even Help?”, 2022. \
[6] Lin et al. “Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning”, 2022. \
[7] Albuquerque et al. "Multi-objective training of generative adversarial networks with multiple discriminators", 2019. \
[8] Miranda et al. ”Single-solution hypervolume maximization and its use for improving generalization of neural networks”, 2016. \
[9] Navon et al. “Multi-Task Learning as a Bargaining Game”, 2022.

summary_of_the_review: The main contributions of this work are casting problems related to generating adversarial perturbations as multi-objective optimization and proposing a gradient-based multi-objective algorithm to solve the aforementioned problems. In the current version of the manuscript, the the major limitations of the work are lack of motivation for the use of multi-objective optimization (i.e. why using multi-objective optimization for this problem?), and limited scope of experiments both in terms of baselines and aspects considered in the analysis. In my review, I raised questions and provided suggestions that could address some of the limitations. At this point, I believe this contribution is not yet ready to be considered for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper has proposed a method for generating adversarial examples which satisfy multiple objectives based on multi-objective optimization (MOO). To solve the optimization problem, authors has leveraged a multi-gradient decent algorithm which combines all gradients with regards to different objectives into a single gradient direction. In addition, it has a regularization term so that it can focus on (or give higher weights to) unachieved tasks. In experimental part, the paper conducted four experiments to validate the utility of the proposed approach.

The main drawback is that the novelty of the problem formulations is not so clear to the reviewer, as some work also take this formulation into consideration in adversarial example generation scenario.
justification_for_why_not_higher_score: See the reviewers' discussion
Decision: Reject