Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper proposes a framework for incorporating and optimising meta-information as additional parameters when optimising the validation loss for a specific network on a given task. More precisely, they instantiate a teacher network that supplies the meta-information (could be anything from attention maps, to importance weights over sample points, to parameters of a data-augmentation scheme) that is then used by a student network, and propose a formulation whereby the two are trained in tandem so as to optimise the output loss of the student network. 

General remarks: The paper is well-written and well-presented. The idea is clear, language is effective and visualisations/tabulations are accessible. 

Pros (Originality/Significance)
(a) I believe the biggest contribution of the paper is in the unifying of different existing approaches that explicate and optimise meta-information as part of the network pipeline in a bid to improve performance (accuracy or robustness). Ideas such as example reweighting, example blending (e.g. mixup), describing attention masks, or transforming input data (rotation, etc) are treated as special cases of a particular student-teacher formulation. To the best of my knowledge, this unified treatment is unique and comes at a crucial time when same techniques seem to reappear under different names and in different forms and prevent a wholecome understanding of the approach and its wider implications.

Cons (Originality/Significance)
(a) The maths of the reunification is rather simple, teacher outputs values that a student uses and optimises, and could lead to a time-consuming training.
(b) The instantiations of meta-information studied in this paper have been explored before and don't seem to offer any significant advantages over existing methods (the benchmarks in some tables are not up-to-date, for example, consider referring to [1] for sample reweighting benchmark and [2] wherein an attention based classification network for CUBS has been proposed.
(c) Some use cases are shown to work only for small datasets, such as CIFARs for example reweighting. Newer techniques that are able to scale example reweighting to ImageNet exist [3]. Their method seems to need curriculum learning on top of example reweighting for full benefit, please comment.


[1] Learning to Reweight Examples for Robust Deep Learning, Mengye Ren et.al.
[2] Learn to pay attention, Jetley et.al. 
[3] Sample Balancing for Deep Learning-Based Visual Recognition, Chen et.al.
review_title: Interesting concept that ties together disparate ways of using meta-information for network training, concerns around scalability

final_score: rating: 7: Good paper, accept
review_content: The authors propose a framework for learning a commentary or teacher model
that provides helpful information during training. Under this framework, a
teacher model learns to provide meta-information, referred to as a commentary,
on a given training example, with the goal of improving the student network
(e.g., improving convergence speed or robustness). The learning of the teacher
model is done by minimizing the validation set loss of the student model under
parameterization from a training loop that used the commentaries.

The optimization of the teacher parameters involves backpropagating
through gradient descent steps. The authors propose doing this directly
in small toy examples, or using an approximation using the Implicit Function
Theorem for more realistic problems.

The authors then demonstrate that the commentary learning framework can
be used in a variety of ways:
learning to provide example weights, learning a blending policy for data augmentation, and learning to provide an attention mask for image classification.
In the latter case, they show that using the attention masks leads to a more
robust student when the backgrounds are modified to spuriously correlate
particular class labels on the training/validation sets but not on the
test set.

This paper has many strengths. It introduces an interesting meta-learning
framework and demonstrate its flexibility to implement three different
kinds of commentary for improving a student network. Additionally, it supports
its statements with a variety of interesting experiments. While I would
like to see this paper accepted, it could be improved 
in its organization and explication.

For example, it is not immediately clear that the commentary network is learned with equations 1 & 2, but the student network in this process is ultimately discarded and only after the finished teacher network is obtained, is a
student network actually trained for evaluation purposes. An "illustrative example" of commentaries is put in the appendix instead of the main paper which
feels like a lost opportunity to holistically explain the algorithm and
clear up these misconceptions.

In the example weighting curricula it is not clear what is meant by curricula.
In an ablation study, example weights produced by a teacher network with and
without curricula are compared, but what constitutes the curricula here is
never defined. I'm inferring that in the curricula case, the teacher network
produces weights using both the example x and the current training iteration
i, while the non-curricula case only uses x. This should be stated explicitly
somewhere. 

In equations 1 and 2 the authors write

optimize(phi/theta)[loss func]

when

argmin_phi/theta loss func

would convey what the authors intend while using standard notation.

On the masking experiments, the masks are used both at training and test time
which seems somewhat odd. The commentary formalism proposed is supposed to
be limited to learning a commentary model to facilitate training. In this case,
the commentary model seems less like a separate teaching model and more like
another component of the classification model, similar to "Recurrent Models of Visual Attention" where a component of the classifier determines where to
focus in an image. How well does the student network trained with masks provided by the teacher do on the validation/test sets without masks?

review_title: An interesting Meta-Learning Approach

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a general framework for boosting CNNs performance on different tasks by using'commentary' to learn meta-information. The obtained meta-information can also be used for other purposes such as the mask of objects within spurious background and the similarities among classes. The commentary module would be incorporated into standard networks and be iteratively optimized with the host via the proposed objective. To effectively optimize both the commentary and the standard network, this paper adopts the techniques including implicit function theorem and efficient inverse Hessian approximations. 

This paper studies three kinds of commentary named weight curriculum commentary that learns individual weights for data samples, augmentation commentary that learns data augmentation strategy similar to mix-up, and attention mask commentary that learns to focus on objects of images. All three commentaries have been examined via extensive experiments on small-scale benchmarks and shown improvements. 

The studied direction is very interesting that how to obtain meta-information with standard data annotations. Although, the reviewer thinks all three commentaries' tasks are studied in the literature. This paper integrates them into a general framework. Also, the weight curriculum commentary can be naturally compatible with MAML. There may be possible to derive more different auxiliary tasks via properly designing the commentary structure. The experimental details are described in detail and make the experiments are easy to replicate. Overall the paper is easy to follow.

-------------------------

However, the proposed framework needs human efforts to specifically design the formula of the commentary structure to involve proper inductive bias for learning the targeted meta-information. There're works that also only use the standard annotations to study adjusting the weights for data samples, data augmentation (like the mix-up), and focusing the object.  Compared to them, the reviewer does not see the clear advantages of the proposed method. Also, the author does not provide many comparisons. 

The reviewer also concerns about scalability. Currently, the method is only tested on several small-scale benchmarks including MNIST, CIFAR, TINY-ImageNet, and CelebA, X Ray Data. However, the objective would need a lot of computation resources, though approximately computing the inverse Hessian. The reviewer doubts that it is hard to apply in real-world tasks and the method can not bring benefits when the data is big. But, this concern may be beyond the scope of this paper and can be considered in the following works.

In conclusion, the reviewer would rate 7 due to it nicely integrates several meta-information tasks into a unified framework and hope this work can bring some new understandings to the communities. 

---------------after rebuttal----------

I've read all reviews and the rebuttal. I think overall this is a good paper and would like to keep my score.
review_title: A general framework to learn meta-information via properly designing 'commentary'

final_score: rating: 5: Marginally below acceptance threshold
review_content: [Summary] This paper introduces a gradient-based meta-learning method to learn weighting for each training sample, called "commentary" in the paper's description, as a form of auxiliary learning, updated on training loss to accelerate training speed, and improve generalisation.

[Strength] The motivation is interesting, as to balance training by automatically computing training sample weighting. The toy example on rotated MNIST clearly shows that the teacher network would put a higher weighting on samples closing to the decision boundary, which is expected and confirms the authors' assumption.

[Weakness] The paper has several weaknesses which I will outline below.
-- Curriculum v.s. Non-curriculum learning. The authors present two versions of "commentary", one is with a curriculum and the other is without a curriculum. The curriculum version, which I assume is to update training sample weighting iteratively with the updated teacher network. However, the authors did not mention any details on the non-curriculum learning version, on how they compute the training sample weighting in a fixed way.

-- Inconsistent results on ResNet18. In Fig 3., ResNet 18 seems to perform unreasonably low on CIFAR-10/100 with roughly ~70 and ~45 of accuracy, respectively. However, with the same network of ResNet18, they perform in a reasonable performance in Table 2.

-- Marginal performance. Using commentary to generate mix-up style sample weighting seems reasonable and interesting. But the results are quite marginal, or do not consistently improve over the standard mix-up baseline as shown in Table 2.

Further questions: What is the distribution of training and validation data in CIFAR10/100 results? Is it possible that the teacher network simply put higher weights on training samples that look most similar to validation data, as a form of a trivial/degenerated solution? I assume this might happen particularly in the rotated MNIST example? And how authors generate the labels for those overlapping samples in rotated MNIST? 

I hope the authors could elaborate on these questions and further refine the paper. I think in general the idea is interesting and worth exploring, but this paper is not ready yet.
review_title: Missing important details, inconsistent baselines

Meta-review: This paper proposes an interesting unified framework for meta-learning with commentaries, which contains information helpful for learning about new tasks or new data points. The authors present three kinds of different instantiations, i.e., example weighting, example blending, and attention mask, and show the effectiveness with the extensive experiments. The proposed method has a potential to be used for a wide variety of tasks.
Decision: Accept (Poster)