Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors propose two modifications to an algorithm from [Rafique et al 2018] for optimizing AUC under a min-max formulation, prove bounds for the two modifications, and experimentally compare the modifications against SGD and the original algorithm by varying class ratios of four datasets.

The proposal builds on [Rafique et al 2018], so it may be considered incremental. However, the algorithm is carefully analyzed and resulting bounds are stronger. The experimental analysis is fairly minimal, with the proposed modifications performing similarly to the original algorithm from [Rafique et al 2018].
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary: 
The authors propose stochastic algorithms for AUC maximization using a deep neural network. Under the assumption that the underlying function satisfies the PL condition, they prove convergence rates of the proposed algorithms. The key insight is to use the equivalence between AUC maximization and some min-max function. Experiments results show the proposed algorithms works better than some baselines. 

Comments: 
The technical contribution is to show stochastic optimization algorithms for some kind of min-max functions converge to the optimum under the PL condition. The proposed algorithms have better convergence rates than a naïve application of Rafique et al. The technical results rely on previous work on the PL condition and stochastic optimization of min-max functions. The techniques are not straightforward but not seem to be highly innovative, either. 

As a summary, non-trivial algorithms for AUC maximization with neural networks are presented, which could be useful in practice.

Minor Comments:

-How the validation data for tuning parameter are chosen in the experiments? This is absent in the descriptions for experiments. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes two algorithms for the non-convex concave AUC maximization problem, along with theoretical analysis. Experiments show the proposed methods are effective, especially in data imbalanced scenarios.

Strengths:

This paper might be useful and interesting to related research, which overcomes some limitations in previous works such as: 1. the convex assumptions; 2. only considering simple models like linear models; 3. the need of extra memory to store/maintain samples. The proposed method extends existing works to a non-convex setting, which can be applied to deep neural networks, and is applicable for batch-learning and online learning.

The proposed methods achieve better experimental results, especially in the data imbalanced scenarios, which is a real problem that may arise in many scenarios. The paper provides theoretical analysis on the proposed methods, based on Assumption 1, and inspired by the PL condition.

Weaknesses:

I think some comparisons with AdaGrad and related methods should be performed in experiments. Since PPD-AdaGrad is “AdaGrad style”.

The assumptions seem a bit unclear. What does the first assumption in Assumption 1 imply?

Minor Comments:
1. Since the experiments label the first 5/50 classes as negative, and the last 5/50 classes as positive for CIFAR10/CIFAR100, is it possible to provide results on experiments that label in the opposite way (or randomly label 5/50 classes) and add these results in the paper/appendix? Just to make results more convincing and reduce some potential dataset influences.

2. Is it possible to provide some results on more imbalanced positive-negative ratio like 20:1?

3. Is it possible to provide some comparison in terms of actual time, like learning curves with time as x-axis?

4. In the multi-class problems, why are the lower layers shared while last layer separated? 

5. Since the extension to multi-classes problems are mentioned in the paper. I like to see some experimental results on this setting.

6. How do the proposed methods perform on models other than NN?

7. I think there is a typo on Page 4, the definition of AUC definition: the latter y should be -1.

review_title: Official Blind Review #1

Meta-review: The paper proposed using stochastic AUC for dealing with imbalanced data. This paper provides useful insights and experiments on this important problem. I recommend acceptance.
Decision: Accept (Poster)