Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new loss function for end-to-end training of deep networks for AUC maximization.  The proposed compositional loss combines the standard cross-entropy (CE) loss within the AUC loss such that the AUC is maximized for good classification while the CE portion of the loss helps to learn robust features. An optimization method is presented and is shown to have the same convergence rate as standard SGD. The proposed method is tested on 4 benchmark and 4 medical image datasets and compared against related losses, showing consistent improvement in AUC. 

main_review: While I cannot speak to the proposed optimization approach, I feel the following are the major strengths and weaknesses of the paper:

Strengths:

- The authors present the first end-to-end training framework for AUC maximization (previous approaches required pretraining e.g. using standard CE loss first for good results)

- The authors present an optimization method for the new loss that has similar rate of convergence as standard SGD

- Experiments are presented using a large number of natural and medical image public datasets.

- Generally the results show consistent improvement over AUC / CE - only loss, a standard linear combination of AUC + CE losses, and 2 other methods for maximizing AUC loss.

Weaknesses:

- My primary concern is some missing details in the experiments, which makes it difficult to accept the results at face value. While some of the datasets do mention a separation of training/validation/test, several datasets only discuss having either only validation or test set (eg. 3 out of the 4 medical datasets). Thus, it is unclear how the parameters for the different methods are tuned - even though all the methods are tuned on the same dataset partitions, if tuned on the set used for reporting results, this will not capture the generalization capability of the methods and thus it is difficult to assess which is really best. Please clarify the parameter tuning process.

- I am confused about the DDSM+ dataset, where the authors state they have combined the DDSM with CBIS-DDSM datasets. This doesn't make sense to me, as my understanding is that CBIS-DDSM is an updated/standardized version of the DDSM, i.e. CBIS-DDSM is derived from the DDSM and contains the same patient data (also, the authors do not have the correct citation for CBIS-DDSM, which should be: Lee et al., A curated mammography data set for use in computer-aided detection and diagnosis research, 2017). Thus depending on how the data is partitioned, the same patient data will be in both training/testing sets, which may partially explain the extremely high AUC results. Furthermore, I am not sure how there are so many images (55K training and ~14K testing), as each dataset has approximately 2600 patients (again, the same patients), where each should have ~4 associated mammogram images (left/right and MLO/CC views). Please clarify the use of the mammography datasets.

- The authors test on all the same datasets as those presented in the paper by Yuan et al., 2020, which introduced the AUCM loss. However, there are no comparisons made to the results from this paper, which seem to report higher AUC results for many of the datasets (eg. CheXpert, Melonoma, PatchCam). While the authors compared to AUCM loss, it was for training from scratch, not using the same training framework as proposed in Yuan et al. It would make sense that the authors should be comparing to the results in this paper if directly comparable, or running their own comparison to this method.

- Regarding the ablation studies on algorithmic design choices for beta and k in Sec. 4.1, I feel like we do not learn much new from this, as it simply verifies that it is better to tune hyperparameters than to not tune them. 

Minor comments:

- Although understandable, would be good to define what "imratio" means 
- The convergence curve results don't really seem to fall under "ablation" study, as nothing is being ablated...consider moving to other section or renaming 4.1 for clarity of presentation
- p.2: "presentations" --> representations

summary_of_the_review: My recommendation is based on the introduction of the novel end-to-end learning approach for AUC maximization and the presented empirical results on 8 different datasets that show general improvement compared to other AUC maximization / CE approaches. However, my enthusiasm was dampened by the unclear experimental setup for parameter tuning/use of validation sets and lack of comparison to highly related work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a new compositional loss function that aims to help deep AUC maximization. Specifically, the outer function of the newly proposed loss function is a surrogate loss for AUC, i.e. AUC square loss; while the inner component can be understood as to facilitate the convergence of CE loss. The authors explain their compositional loss through Taylor expansion theoretically and the lens of feature purification and classifier robustification intuitively. Furthermore, they demonstrate the difference between their proposal and the linear combination of AUC and CE loss from both mathematical perspective and empirical perspective. Experimental wise, this paper has done extensive studies on different dataset ranging from popular natural image benchmarks to several medical image datasets. Though the improvement is rather limited on some cases, CT(AUC) outperforms all the other methods in all cases. 
main_review: Strengths:
1. The paper provides good explanation on their newly proposed compositional loss and training method. 
2. The paper performs extensive experiments to demonstrate the effectiveness of their method.
3. This paper is well written and organized. 

Weaknesses:
Overall, this paper is a good work. But I do have several questions and concerns:
1. The author hypotheses that using CE loss, different examples roughly have equal weights regardless which classes they belong to. And it is one of the key reasons that using CE loss has advantage over raw AUC loss. It would be great to show that the compositional loss can help achieve the equal weights property empirically to validate this hypothesis in the paper.
2.  In the experiments section, the authors list AUC for multi-class classification tasks (e.g. CIFAR10 and CIFAR100), I wonder how the AUC is computed for these multi-class tasks. The details are missing in the paper.
3. For medical image datasets, since the authors only report the results from a single run, it may not be very convincing to show the improvement. I would suggest the authors either report results from multiple runs or report the evaluation noise level (i.e train same model several times and perform evaluation on the test data, AA test to understand the eval noise), so that we can make sure the improvement comes from the proposed method.
4. In figure 4, there are several bump ups in the convergence curves. What are the reasons of these bumps? Are they because a learning rate scheduling or something else? The detailed explanations are missing.
5. In table 2, the authors mention that they tune the inner gradient steps k \in {1,2,3} for the left table while keep k = 1 for the right table. However, the detailed tuning strategy is missing.  Does the algorithm tries out 1,2,3 and pick up the optimal one automatically during each iteration?
6. In the runtime analysis, it would also be useful to report what hardware is used for testing, since different hardware may results different running time.

summary_of_the_review: Overall, this paper is well written and explained. I have listed several questions and concerns in the Main Review.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses the problem of how to perform end-to-end training for deep AUC ("area under ROC curve") maximisation. This is a challenging problem as this yields a harder optimisation problem than the standard CE minimisation. Existing approaches use two stage optimisation approaches where the first stage optimises a traditional loss function (e.g. the CE loss) and the second stage fine-tunes the network by optimising the AUC loss. The proposed approach is termed compositional deep AUC maximisation which can be trained end-to-end. The paper furthermore proposes an efficient stochastic optimisation algorithm solving compositional deep AUC maximisation problem. It has been evaluated on standard benchmark computer vision and medical imaging classification datasets.
main_review: The paper includes a good summary of the related work and the contributions. The proposed compositional approach to the deep AUC maximisation problem seems interesting and novel. It uses a reformulation of the optimised loss function that contains three different terms which jointly optimize the feature representation and the classification performance. A primal-dual stochastic algorithm is proposed that optimises the proposed cost function and an informal argument for its convergence is given. The reported experimental results are quite convincing, both for the computer vision tasks and for the medical imaging tasks, however a proper comparison of the experimental results with other state-of-art methods is missing.

summary_of_the_review: This is an interesting paper that proposed a number of technical advances (formulation of the compositional loss function and its optimization) with reasonable results, albeit there are weaknesses in the experimental evaluation.
review_title: null

Meta-review: I recommend this paper to be accepted. All reviewers are in agreement that this paper is above the bar.
Decision: Accept (Spotlight)