Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors propose to learn surrogate loss functions for non-differentiable and non-decomposable loss. An alternative minimization method is used for training the surrogate network and prediction model. Learning surrogate loss functions for different tasks is somewhat novel, although there are some prior works on learning the loss, e.g., [1]. 

Pros.
1.	The paper is well written and easy to follow.
2.	Learning a unified loss for different tasks is interesting. It has the potential to reduce human efforts to design losses.

Cons.
1.	Some important details of learning the surrogate loss are missing. The function g for extracting the latent error component is not clear. The composition function h is not provided either.
2.	The details of the experiments are not clear. Are the results on the test set? How many independent runs are performed?
3.	Learning surrogate loss incurs additional approximation error, time complexity, and model complexity. The benefit of the trade-off is not systematically evaluated.

Questions
1.	The function g for extracting the latent error component is not clear. Is it required to design for different tasks by an expert specifically? Or, does it have the same differentiable form for different tasks? Please provide the details of the function g for the different losses in the experiments.

2.	Can the proposed loss work well for multi-class classification tasks? In the experiments, only binary classification is evaluated. Multi-class classification will increase the number of classes, thus increasing the difficulty of approximation. It is better to provide MCR compared with CE on the CIFAR100 dataset for evaluation. Also, please provide the running time of CE and SL-R in the same running environment.

3.	Is the results in Table 3 on the test set? CE has fewer parameters compared with the proposed loss, why does SL-R have better generalization performance compared with CE? How many independent runs performed in experiments?

4.	 CE does not need training the loss compared with SL-R. Please provide the running time of CE in the same environment for a fair comparison. 

5.	The time complexity analysis treats extracting function g as a black-box function. However, the complexity of function g depends on the tasks. Please provide a detailed discussion about time complexity for different tasks (e.g., AUC, F1, MCR for multi-class classification, and ranking tasks). 


[1] Learning Loss Functions for Semi-supervised Learning via Discriminative Adversarial Network, 2017


review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: In this paper authors propose to jointly optimize a critic, that estimates some non-differentiable objective (or an objective with intractable derivatives and/or derivatives that are 0 almost everywhere). Authors conduct numerous experiments, and show improvements over some sota methods, and maybe more importantly - provide a unified way of achieving these across multiple metrics. Paper is well written, very easy to understand and in reviewers opinion, a nice, simple story worth sharing.


Major concerns:

The theoretical result provided is just a citation of an existing proof, rather than a new contribution. More importantly however, the setup considered strongly violates the assumptions of the theorem, for example assumption (b) does not hold for the neural networks, as there are continuum many global minima (imagine having a unit in a neural net, that has a huge negative bias, which causes it to always be "turned off" by relu, then any weight on top of it will have the same final loss value; or simply notice that relu(a*x)=a*relu(x) if x>0, and so you can always "push" the norm of weights from one layer to the next one without affecting the outcome etc.), and so argmin is not a singleton, but the set of exactly same power as the whole parameter space (since for each k there exists a bijection from R^k to R, and R is of continuum power). This property is not a mild assumption, but rather a critical element guaranteeing convergence of such systems, with powerful universal approximators, one cannot hope for such strong convergence results without actually analysing the approximators family. Reviewer strongly believes this should be either removed completely, or just briefly mentioned, rather than made a strong statement in the paper. It is an empirical work, and stands strong on its own rights, there is no need to add theorems with assumptions that are never satisfied in the proposed scenario.

Results provided use somewhat non-standard datasets for deep learning, and as such it is hard to asses statistical significance of the differences reported; while the test datasets sizes are big enough to trust error to 1e-3 level, they are heavily imbalanced and relatively low dimensional problems, which have proven to be hard for neural network many times in the past, consequently I would expect to see confidence intervals or stds of each result, at least for Table 3. The problem becomes even more severe for metrics such as F1 which do not decompose additively and so can be very sensitive to the (false) positive rates values - reported improvements might disappear once these are introduced, but even if one does not outperform hand-crafted proxies for specific objectives, having a unified method that is on-part with those in black-box scenario is a good result (I would argue that even if all the results are slightly worse, it is still worth publishing).

I find it a bit disappointing, that authors did not try to analyse trained models, it would be invaluable to see what kind of aggregation g and h came up with, that is well aligned with losses such as F1 or JAC.


Minor concerns:

The work resembles closely methods of error critic learning - where one uses an update direction coming from a model, that regresses towards the loss itself, or the gradient of the loss (is available), see:
- Neural Network Design for J Function Approximation in Dynamic Programming (Pang and Werbos paper from '98)
- Sobolev Training of Neural Networks (from NIPS; more precisely "Critic" baseline, which has the same functional form as the loss presented in the paper, if applied at the very top of the network only)
It might be worth discussing in the paper.


Overall I recommend weak acceptance, and encourage authors to address some of the concerns raised above.

**Update**

Based on discussion, corrections and new additions, I recommend acceptance of the above work.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method of learning loss functions in addition to the learning of predictors. Since it's not easy to optimize loss functions that evaluate the accuracy, surrogate loss functions have been widely employed. The design of the surrogate loss is problem-dependent, and handcraft is required. This paper tries to tackle this problem from the viewpoint of meta-learning, i.e., the surrogate loss learning. Typically, deep neural networks (DNN) are used to design a surrogate loss that approximates the original loss while maintaining the tractability of the optimization. Some convergence properties of the proposed method are analyzed. Some empirical studies showed the efficiency of the proposed method to the state-of-the-art baselines. 

The design of the surrogate loss is important for machine learning problems. However, the proposed method in this paper seems an ad-hoc approach rather. For example, the 0-1 loss is often replaced with convex loss functions such as the hinge loss or logistic loss. Using these surrogate loss functions, the statistical properties of the predictors obtained from 0-1 loss are maintained. See the following paper for details.
P. L. Bartlett, et al., (2006), Convexity, Classification, and Risk Bounds, Journal of the American Statistical Association March , Vol. 101, No. 473. 

On the other hand, the current approach does not have such a theoretical guarantee for each learning problems. Though certainly, the proposed method is widely applicable to many problems, there is no theoretical guarantee. Theorem 1 in page 5 shows the convergence property. However, the number of iterations, K_beta, should tend to infinity. This is not a practical operation in the learning algorithm
review_title: Official Blind Review #2

Meta-review: Unfortunately, this was a borderline paper that generated disagreement among the reviewers.  After high level round of additional deliberation it was decided that this paper does not yet meet the standard for acceptance.  The paper proposes a potentially interesting approach to learning surrogates for non-differentiable and non-decomposable loss functions.  However, the work is a bit shallow technically, as any supporting theoretical justification is supplied by pointing to other work.  The paper would be stronger with a more serious and comprehensive analysis.  The reviewers criticized the lack of clarity in the technical exposition, which the authors attempted to mitigate in the rebuttal/revision process.  The paper would benefit from additional clarity and systematic presentation of complete details to allow reproduction.
Decision: Reject