Teaching with Commentaries
Authors: Aniruddh Raghu
Maithra Raghu
Simon Kornblith
David Duvenaud
Geoffrey Hinton
Venue: ICLR 2021 Poster (2021)
Keywords: learning to teach
metalearning
hypergradients

Abstract: Effective training of deep neural networks can be challenging, and there remain many open questions on how to best learn these models. Recently developed methods to improve neural network training examine teaching: providing learned information during the training process to improve downstream model performance. In this paper, we take steps towards extending the scope of teaching. We propose a flexible teaching framework using commentaries,  learned meta-information helpful for training on a particular task. We present gradient-based methods to learn commentaries, leveraging recent work on implicit differentiation for scalability. We explore diverse applications of commentaries, from weighting training examples, to parameterising label-dependent data augmentation policies, to representing attention masks that highlight salient image regions. We find that commentaries can improve training speed and/or performance, and provide insights about the dataset and training process. We also observe that commentaries generalise: they can be reused when training new models to obtain performance benefits, suggesting a use-case where commentaries are stored with a dataset and leveraged in future for improved model training. 

Published as a conference paper at ICLR 2021
TEACHING WITH COMMENTARIES
Aniruddh Raghu
Maithra Raghu
Simon Kornblith Google Research
MIT araghu@mit.ecm edu
Google Research
David Duvenaud
Geoffrey Hinton
Google Research University of Toronto
Google Research University of Toronto
ABSTRACT
Effective training of deep neural networks can be challenging, and there remain many open questions on how best learn these models. Recently developed meth- ods to improve neural network training examine teaching: providing learned in- formation during the training process improve downstream model performance. In this paper, we take steps towards extending the scope of teaching. We propose a flexible teaching framework using commentaries, learned meta-information help- ful for training commentaries, leveraging recent work on implicit differentiation for scalability. We explore diverse applications of commentaries, from weighting training exam- ples, to parameterising dabe-ddepndeett data augmentation policies, to represent- ing attention masks that highlight salient image regions. We find that commen- taries can improve training speed and/or performance, and provide insights about the dataset and training process. We also observe that commentaries generalise: they can be reused when training new models to obtain performance benefits, sug- gesting a use-case where commentaries are stored with dataset and leveraged future for improved model training.
particular task. We present gradient-based methods to learn
INTRODUCTION
Training, regularising, and understanding complex neural network models is challenging. There remain central open questions on making training faster and more data-efficient (Kornblith al., 2019; Raghu et al., 2019a;b), ensuring better generalisation (Zhang et al., 2016) and improving transparency and robustness (Bau al., 2017: Madry al., 2017). promising approach for ad- dressing these questions is learning to teach (Zhu, 2015), in which learned auxiliary information about task provided objectives. Examples include providing auxiliary training targets (Liu et al., 2019; Navon et al., 2020; Pham et al., 2020) and reweighting training examples to emphasise important datapoints (Fan et al., 2020; Jiang et al., 2018; Ren et al., 2018: Shu et al., 2019).
neural network to inform the training process and help downstream
Learning to teach approaches have achieved promising results vision and language applications (Jiang et al., 2018; Ren al., 2018; Shu et al., 2019; Hu et al., 2019) using a handful of specific mod- ifications to the training process. this paper, we take steps towards generalising these approaches, introducing flexible and effective learning to teach framework using commentaries. Commentaries represent learned meta-information helpful for training a model on a task, and once learned, such commentaries be reused is to improve the training of new models. We demonstrate that com- mentaries can be used for applications ranging from speeding up training to gaining insights into the neural network model. Specifically, our contributions are:
1. We formalise the notion of commentaries, providing a unified framework for learning meta- information that can be used to improve network training and examine model learning.
2. We present gradient-based methods to learn commentaries by optimising network's 's validation loss, leveraging recent work implicit differentiation to scale larger models.
3. We use commentaries to define example- weightin curricula, a common method of teaching neural networks. We show that these learned commentaries hold interpretable insights, lead to speedups in training, and improve performance few-shot learning tasks.
Work done while interning Google.

---

Published as a conference paper at ICLR 2021
4. We define data augmentation policies with label-dependent commentaries, and obtain insights into the design of effective augmentation strategies and improved performance on benchmark tasks as compared to baselines.
We parameterise commentaries as attention masks to find important regions of images. Through qualitative and quantitative evaluation, show these masks identify salient image regions and can be used to improve the robustness of neural networks to spurious background correlations. We show that learned commentaries can generalise: when training new models, reusing learned commentaries can lead to learning speed/performance improvements. This suggests a use-case for commentaries: being stored with a dataset and leveraged to improve training of new models.
2 TEACHING WITH COMMENTARIES
Definition: We define a commentary to be learned information helpful for (i) training model on a task or (ii) providing insights on the learning process. We envision that commentaries, once learned, could be stored alongside a dataset and reused as is to assist in the training of new models. Appendix A explores simple instantiation of commentaries for Celeb-A (Liu et al., 2015), to provide intuition of the structures that commentaries can encode.
Formally, let f(: y, () denote commentary that is function of data point J. prediction target and iteration of training i, with parameters ø. The commentary may be represented in a tabu- lar fashion for every combination of input arguments, or using neural network that takes these arguments as inputs. The commentary is used to train a student network n(x;; 8) with parameters A.
2.1 LEARNING COMMENTARIES
We now describe algorithms to learn commentaries' Throughout, we denote the training set as Dr, the validation set as Dv and the loss function (e.g. cross-entropy) as L. With 0 denoting the parameters of the student network and & denoting the commentary parameters, we let 0, á be the respective optimised parameters. We seek to find & such that the student network's validation loss, Lv, minimised. As the commentary is used during the training of the student network, Lv implicitly depends on é. enabling the use of gradient- based optimisation algorithms find ő.
Algorithm 1: Backpropagation Through Training: When student network training has small memory footprint, we optimise commentary parameters by iterating the following process, detailed in Algorithm 1: (1) train student and store the computation graph during training; (2) compute the student's validation loss; (3) calculate the gradient of this loss w.r.t. the commentary parameters by backpropagating through training; (4) update commentary parameters using gradient descent.
By optimizing the commentary parameters over the entire trajectory of student learning, we encour- age this commentary to be effective when used in the training of new student networks. This supports the goal of the commentary being stored with the dataset and reused in future model learning.
Algorithm 2: Large-Scale Commentary Learning with Implicit Differentiation: When train- ing the student model has a large memory footprint, backpropagating through training to obtain exact commentary parameter gradients is too memory expensive. We therefore leverage the Im- plicit Function Theorem (IFT) and efficient inverse Hessian approximation to obtain approximate gradients, following Lorraine et al. (2020).
The gradient of the validation loss w.r.t the commentary parameters can be expressed as:
dCy
aLv a8
a x
(3)
06
06
The first term on the right hand side in equation 3 is simple to compute, but the second term is expensive. Under fixed-point and regularity assumptions on student and commentary parameters ab as the following product:
(f()), o). the IFT allows expressing this second term as
ae
LL 30 00T
PLT 30 T
(4)
06
((d)
'Code httph:tt////// httpaa//p///ggtb....o/gggggeeeee
2

---

Published as a conference paper at ICLR 2021
Algorithm 1 Commentary Learning through Backpropagation Through Training.
1: Initialise commentary parameters ø
2: for / /=1, l,..., T meta-training steps do
3: Initialise student network n f)) with parameters 8g
4: Train student network with N steps of gradient descent to optimise:
Cr Φ) = Ey [ (z; (:::))
(1)
where É is loss function adjusted from & incorporate the commentary, and (T ()) is the expected adjusted loss over the training data. Output: 0, the optimised parameters of student network (implicitly a function of d, (())). Compute validation loss:
5:
(v( Be [e (m(a;(()))))
6: Compute (Lv(d) by backpropagating through the N steps of student training, and update Φ
7: end for
8: Output: o the optimised parameters of the commentary.
Algorithm 2 Commentary Learning through Implicit Differentiation.
1: Initialise commentary parameters ó and student network parameters 0
2: for t = 1,........ .M do
3: Compute the student network's training loss, (r((f, ; Φ). equation 1.
4: Compute the gradient of this loss w.r.t the student parameters 0.
5: Perform single gradient descent update on the parameters obtain Ô (implicitly a function
of é) (())..
6: Compute the student network's validation loss, (v))), equation 2.
7: Compute acy
36
8:
Approximately compute De term and implicit Jacobaaa products (Lorraine al., 2020).
8ú with equation using a truncated Neumann series with single
Compute the overall derivative acv using steps (7) and (8), and update di d6
9:
10: Set f + θ.
11: end for
Output: d. the optimised parameters of the commentary.
i.e., a product of an inverse Hessian and matrix of mixed partial derivatives. Following Lorraine al. (2020), we efficiently approximate this product using truncated Neumann series and implicit vector-Jacobian products. Leveraging this approximation then yiclds second method for commen- tary learning, described in Algorithm 2. Since a single term in the Neumann series is sufficient for learning, each iteration of this algorithm has similar time complexity to single iteration of training.
In this method, commentary parameters are learned jointly with student parameters, avoiding train- ing a single student model multiple times. This approach therefore scales to millions of commentary parameters and large student models (Hataya et al., 2020; Lorraine et al., 2020). However, since the commentary is not directly optimised over the entire trajectory of learning, its generalisability new models is not ensured. We examine this in our experiments, demonstrating that commentaries learned in this manner can indeed generalise training new student networks.
COMMENTARIES FOR EXAMPLE WEIGHTING CURRICULA
We explore our first main application of commentaries: encoding separate weight for each training example at each training iteration. Since the commentaries are a function of the training iteration, they encode curriculum structure, refer them as curriculum commentaries.
3

---

Published as a conference paper at ICLR 2021
Non-Overiapping Dataset: Example Digits
Overlapping Dalaset: Example Diglts
Rotation Angle Distribution
Rotation Angle Distribution
class_1
class_]
Learned Example Weights
Learned Example Weights
Weightt
Weight
-20
20
-20
20
30
Rotation
Rotation
Figure 1: Learned per-example training weights on Rotated MNIST. We visualise rotated digits and ro- tation distributions from both the datasets. We plot the learned example weights at iteration 500 of student (left),
trainigg function of rotation after discretising based digit rotation. In the con-ovellapping examples closer the decision boundary are weighted most. When classes overlap (right), the more representa- tive examples with greater rotation are weighted highly and examples in overlap region are downweighted, which is sensible as examples in the overlap region are less indicative of the class label.
We specify these weights using a commentary neural network (or teacher network) t(z, ó) + [0, 1] that produces weight for every training example every iteration of training of the student network. When training student network, using the notation of $2.1, the commentary is incorpo- rated in the training loss as: L = t(x, i; ø) [n((x; 0), y). where [(-) is the original loss function for the task. The validation loss is unweighted.
3.1 SYNTHETIC EXAMPLE: ROTATED MNIST DIGITS
We first learn example weight curriculum commentaries a synthetic MNIST binary classification problem. Each example in the dataset is a rotated MNIST digit '1', with variable rotation angle that defines the class. We generate datasets: the non-overlapping dataset and the overlap- ping dataset. In the non-overlapping dataset, the rotation angle for each example from class 1 and class 0 is drawn from non-overlapping distributions Uniform[13 45) and Uniform[-45, 15] re- spectively. In the overlapping dataset, the rotation angles are drawn from overlapping distributions Uniform 55, 30] and Uniform[ 30, respectively (Figure 1).
We use two block CNNs as both the commentary neural network and student network. The commen- tary network takes as input the image and the iteration of student training, and outputs weight for each example in the batch. We learn commentary parameters by backpropagating through student training (Algorithm 1, $2.1). and use 500 gradient steps for inner optimisation (i.e., N 500). Im- plementation is with the hig library (Grefenstette et al., 2019). Further details in Appendix B.
Results: Figure visualises the two datasets and plots the learned example weights as a function of rotation at iteration 500 of the student training. When classes do not overlap (left), the example weights are highest for those examples near the decision boundary (small rotation magnitude). When the classes do overlap (right), the more representative examples further from the boundary are upweighted and ambiguous examples in the overlap region are downweighted: a sensible result. We perform further analysis of the learned example weighting curriculum in Appendix B.I, demon- strating that the learned curricula in both cases are meaningful. Overall, these results demonstrate that the learned commentaries capture interesting and intuitive structure.
3.2 COMMENTARIES FOR CIFAR1O AND CIFAR
We now learn example weighting curriculum commentaries on CIFAR10 and CIFAR100. The com- mentary network is again the two block CNN architecture, and when training the commentary net- work, the student network is also a two block CNN. We use Algorithm $2.1 once more, with 1500 gradient steps in the inner optimisation: N = 1500. For evaluation, the trained commentary net- work is used produce example weights for (i) 2 block CNN (ii) ResNet- 18 (iii) ResNet-34 student networks, all trained for 25000 steps, considering 3 random initialisations, to assess generalisability. Further details in Appendix B.2.
4

---

Published as a conference paper at ICLR 2021
Commentary Generalisation Over Training Steps
CIFAR1O
50
Accuracy
Accuracy
20
Inner Loop Train Step Limit
Inner Loop Train Step Limit
SmalicHN-commeetenn
SmaliCHN-commeeten
20
SmallCHN curriculum commaniary
SmallCHN curriculum commontary SmallCNN
SmallCNN
15000 20000 25000
15000 20000 25000
Training Step
Training Step
Figure 2: Example weight curricula can speed up training. Test-set accuracy curves on CIFAR 10/100 when using curriculum commentaries, non-curricllum commentaries, and no commentaries, during student network training. The learned curriculum commentary network which generates per-iteration example weights results in learning speed improvements. This learning speed improvement holds when the student network trained for many more steps than the number of inner loop update stops used during commentary network training (1500 steps). This demonstrates that the commentaries generalise to longer training times.
Commentary Generalisation Acress Architectures
CITAR10
DTAR10
DFAR2C0
Lass Tran Lind tomelll commenia tonellik cetallieeetttatmmmmmmmoooo
Iner Losp Tran LITE. no1e.36c3treeee no1e none:3.ccmtoourr cermanito
20
Laap Than Jert.
Laap Than Latt noneec-carmnn nane wwm.mmmmmmmmmmmeee....
nonelll-cur cuterrrerr surmentaryy
1030 L203 15830 28060 29030
$030 Lnban 15200 20060 29030
(030 1500 Ze0eo 25030
5292 (os0: 1500 20060 26000
Figure Example weight curricula generalise across network architectures. Using a learned curriculum commentary network trained with a simple block CNN student, we apply these example weights to train two ResNet architectures. This gives improved lest set accuracy curves for ResNet students also, indicating that the commentaries generalise across architectures.
Example weighting commentaries improve learning speed. Figure 2 shows accuracy curves on the test sets of CIFAR1O/100 for the two block CNN student with example weight curricula (or- ange line), baseline (green line, example weights) and an ablation (blue line, example weights without curriculum structure, meaning the commentary network only takes the image x and not the training iteration as argument when outputting weights). On both datasets, the networks trained using the curriculum commentaries obtain better performance than the baseline and ablation over approximately 25000 steps of training (10 epochs), and have superior learning curves.
Example weighting commentaries generalise to longer training times and across architectures. At training time, the commentary network was leamned to produce example weights for the two block CNN student for 1500 inner update steps (N = 1500, Algorithm 1). Figure shows that the learned example weights lead student network learning speedups well-beyond this point, suggesting gen- eralisability of the commentaries to longer training times. In addition, when the same commentary network is used to produce example weights for ResNet- students (Figure 3), we also observe a learning speedup, suggesting that the commentaries can generalise across architectures.
3.3 COMMENTARIES FOR FEW-SHOT LEARNING
Finally, we use example weight commentaries for few-shot learning. We start with the MAML algorithm (Finn et al., 2017), which learns student parameter initialisation that allows fast adap- tation on new learning task using support set of examples for that task. To incorporate example weighting MAMI., at training time, we jointly learn the MAML initialisation and a commentary network to provide per-example weights for each example in the support set, as function of the inner loop step number, At test time, we follow the standard MAML procedure, and also incorporate the example weights when computing the support set loss and the resulting updates. Both networks use the 4-conv backbone structure from the original MAML paper. Details are in Appendix B.3.
We evaluate standard MAML baseline and commentary variant standard few-shot learning benchmarks: (i) training/testing on MinilmageNet (MIN); and (ii) training on MIN and testing on CUB-200-2011 (CUB). Results shown in Table 1, specifying the experimental setting (N-way
5

---

Published as a conference paper at ICLR 2021
Mode: Train Data -> Test Data
MAMI, Baseline
Example weighting + MAMI
5 5-w I-shot: MIN- +MI
43.64 + 0.61
45.07 + 0.61
5-way 1-shot: MIN-+CUB
37.45 + 0.55
38.00 + 0.54
5-way 5-shot: MIN-
61.85 0.52
62.99 + 0.52
5-way 5-shot: MIN+
57.75 0.54
59.06 + 0.55
Table 1: Mean accuracy and 95% confidence interval across 1000 test-time tasks. On common few-shot learning benchmarks, using example weighting for the support set improves the MAML baseline's accuracy. Improvements also observed when evaluating out-of-distribution datasets.
Lower Error Rate Blending:
Higher Error Small Blendinng:
into 1 Ifrection 0.39), into (fraction 0.43)
0 into (fraction = 0.02), 2 into 7 lfrection = 0.02)
Figure 4: The learned blending augmentation is related to the class error rate. On digit 1s, class with maximise the
low average error ratc, the learned augmentation blends in large amounts of other digits learning signal. On digits with higher average error rates, such as 8s, the level of blending very low so as to focus on classifying the class alone (rather than making the task more complex by adding another digit).
K KC-shot,, and the dataset used for training/testing. all experiments, incorporating example weight- ing can improve on the MAML baseline, suggesting the utility of these commentaries in few-shot learning. Further experiments on other benchmark datasets (CIFAR-FS/SVHN) showing similar trends are in the appendix (Table B.1).
4 COMMENTARIES FOR DATA AUGMENTATION
We now investigate lapen-dependent commentaries that parameterise data augmentation policies. We consider augmentation scheme where pairs of images are blended together with proportion dependent on the classes of the two examples. At each training iteration, we:
Sample two examples and their labels, (x1, M/1 and (I22, 3/2) from the training set.
Obtain the blending proportion = t(y1, 3/2 Φ). and form a new image I. = Az + (1 A)22, and class Vm equivalently.
Use this blended example-label pair (Im, 3m) when computing the training loss.
To compute the validation loss, only unblended examples from the validation set.
For classification problems with N classes, the teacher t U2: Φ) outputs an N x N matrix. This augmentation scheme is inspired by mixup (Zhang et al., 2018). However, blend with deter- ministic proportion, depending on pairs labels, rather than drawing blending factor from a Beta distribution. In doing so, we more finely control the augmentation policy.
Augmentation Commentaries on MNIST: We learn an augmentation commentary model on MNIST by direct backpropagating through the training of 2-block CNN student network (Al- gorithm 1, 12.1). In the learned augmentation, the етгоr rate on class is correlated (Pearson correlation= 00.54) with the degree of blending of other digits into an example of class lower eтror on class i implies that other digits are blended more heavily into it. On MNIST, this means that the class that has on average the lowest error rate (class 1) has other digits blended into sig- nificantly (Figure left); classes that have on average higher error rate (e.g., class 7, class 8) rarely have other digits blended in (Figure 4 right). Further details in Appendix C.I.
4.1 AUGMENTATION COMMENTARIES FOR CIFAR10 AND CIFAR 100
We next learn and evaluate augmentation commentaries CIFARIO and CIFARI00. We evalu- ate the effect of these augmentation commentaries on improving generalisation performance for a standard student network architecture. Since this is memory intensive process, use the implicit differentiation method (Algorithm 2, & 2.1) to ensure computational tractability. We lear the com- mentaries jointly with ResNet-18 18 student network. Once the commentary is learned, fix it and

---

Published as a conference paper at ICLR 2021
CIFARIO
CIFAR100
Test Accuracy
Test Loss
Test Accuracy
Test Loss
No commentary
93.84 + 0.22
0.239 + 0.005
74.79 + 0.39
1.05 + 0.03
Random commentary
93.84 + 0.30
0.385 + 0.007
73.39 + 0.58
1.14 + 0.01
mixup (Zhang al., 2018)
94.42 + 0.12
0.313 0.010
75.89 + 0.55
1.01 + 0.02
Learned commentary
94.12 0.19
0.225 + 0.007
76.25 + 0.11
0.97 + 0.01
Table Learned augmentation commentaries result in competitive improved model accuracy and loss on CIFAR10/100, Using the learned, label-dependent augmentation commentary during training results in models that are competitive with mixup and superior to other baselines. We show mean/standard deviation over three different initialisations.
Original
Origial Image Class: Top 5 Blerded Classes
Original Image Class: Truck
Original Class: Cat
0.50
Top 5 BBended Classes
0.45
Sandrn tracien
9.15
Blonding frection
1033
Blerding frection
0.48
Bgoding
0.10
0.44
0.35
8.05
6.42
0.00
0ame
3650
210
dee
609
horre
she
plamo mr0 Drd
ong
frog
horse
ship
buuck
Ison
treetor
tiper
b306
Vor
Figure 5: Blending proportions for selected CIFARIO and CIFAR100 classes reveal interesting insights. Visualizing the learned blending proportions on two CIFARIO classes (left), we see that both classes are most blended with others that are visually similar (truck-automobiiee and cat-dog), which may help the network differentiate between them. On CIFARI00 (right), considering the top 5 blended classes in two cases, we observe again the presence of visually similar classes that may be confused (seal-otter, and squirrel-mouse) but also visually unrelated classes. These may provide extra learning signal with each example.
train three new students with different random initialisations evaluate the commentary's 's efficacy, assessing the commentary's 's generalisability.
Results: Table shows test accuracy for different augmentation policies on CIFARIO and 100. We compare the learned commentary to using only standard data augmentations for CIFAR10/100 (No commentary) and a random initialisation for the commentary matrix (Random commentary). We also compare mixup (Zhang et al., 2018). We observe that the learned commentary is competitive with mixup and improves on other baselines across both tasks. the appendix, we compare to an ablation that destroys the structure of the learned commentary grid by shuffling it, and find that the unshuffled commentary does better (Table C.I).
Further Analysis: In Figure 5, we visualise: (i) for two CIFARIO classes, the blending fractions (defined as (1 - A)) associated with the other classes (left); and (ii) for two CIFAR 100 classes, the blending fractions associated with the five most highly blended classes. For CIFAR10, we that other classes that are visually similar and therefore sources of misclassification are blended more significantly. On CIFAR 100, we that within the top blended classes, there are classes that visually similar, but there are also blended classes that have far less visual similarity, which could be blended to obtain more learning signal per-example. Further analysis in Appendix C.2.
ATTENTION MASK COMMENTARIES FOR INSIGHTS AND ROBUSTNESS
We study whether commentaries can learn identify key (robust) features in the data, formalising this problem as one of learning commentaries which act attention masks. We learn commentary attention masks on a variety of image datasets: an MNIST variant, CIFAR10/100 medical chest X-rays, and Caltech-UCSD Birds (CUB)-200-2011. where we find that the learned commentaries identify salient image regions. Quantitatively, we demonstrate the effectiveness of attention mask commentaries over baselines in ensuring robustness to spurious correlations.
Formally, we learn a commentary network ((:; () - ji to output the centre of a 2D Gaussian that is then computed and used (with predefined standard deviation depending the input image size, see Appendix D) as pixelwise mask for the input image before feeding it through a student network. We denote the mask based on ((x; d)) mxx Our goal is to learn masks that highlight the most
7

---

Published as a conference paper at ICLR 2021
Coloured MNIST
Chest X Ray Data
CIFARIO
CIFAR100
Image
Learned
Mask
6
Masked Image
Figure 6: Learned attention masks highlight salient image regions for classification. We learn com- mentary network to produce image attention masks several image datasets, and the masks qualitatively sensible in all cases. On Coloured MNIST, where the image label is determined by the red digit, the masks focus on the red digit. On dataset of chest X-rays, the masks focus on the chest cavity, which is the appropri- ate reason for detecting the condition in question (cardiomegaly). On CIFARIO/ important regions, such as the faces of animals, the hump of the camel, and the bodies of vehicles.
the masks are focused on
important regions of the image for training and testing, so the masks are used both at train time and test time. We therefore have that C = x-ent (n (x O m (x, 0) y). The commentary network is U-Net (Ronneberger al., 2015) with output layer from KeypointNet (Suwajanakorn et al., 2018). Commentary parameters are learned using Algorithm 2, 32.1, for a ResNet- student.
5.1 QUALITATIVE AND QUANTITATIVE ANALYSIS ON IMAGE DATASETS
We learn masks on dataset where each image has two MNIST digits, coloured red and blue, with the red digit determining the label of the image. As seen in Figure left, the commentary selectively attends to the red digit and not the blue digit.
Masks for Coloured MNIST:
Masks for Chest X-rays: Using a dataset of chest X rays (Irvin et al., 2019), we train a student network to detect cardiomegaly, a condition where an individual's heart is enlarged. Learned masks are centered on the chest cavity, around the location of the heart (Figure 6), which is clinically relevant region. These masks could be used medical imaging problems to prevent models relying on spurious background features (Badgeley et al., 2019; Winkler et al., 2019).
Masks for CIFAR10/100: The learned masks on CIFAR10/100 (Figure 6) attend to important image regions that define the class, such as the faces of animals/the baby, wheels/body of vehicles, and the humps of the camel. In the appendix (Table D.1) we show quantitatively that the learned masks are superior to other masking baselines, and also provide further qualitative examples.
5.2 MASK COMMENTARIES FOR ROBUSTNESS TO BACKGROUND CORRELATIONS
Using the task introduced in Koh et al. (2020), we now demonstrate that mask commentaries can provide robustness spurious background correlations. We take the CUB-200-200 200-2011 1 dataset (Welin- der et al., 2010) and the associated fine-grained classification problem to classify the image as one of 200 bird species. Using the provided segmentation masks with the dataset, the background of each image is replaced with a background from the Places dataset (Zhou et al., 2017). For the training and validation sets, there background, but for the testing set, this mapping is permuted, so that the background features are now spuriously correlated with the actual image class.
specific mapping between the CUB class and the Places class for the
We first learn an attention mask commentary network, and for evaluation, use this learned network when training new ResNet-18 student (pretrained on ImageNet, as Koh et (2020)). We assess student performance on the validation and test sets (with the same and different background mappings as the training set, respectively), considering three random seeds.
Results: Leamedd masks are shown in Figure 7; the masks are mostly focused on the bird in the image. a quantitative evaluation (Table 3), we see that using the masks helps model performance significantly on the test set compared to bascline that does not masks. This suggests that the masks are indeed helping networks to rely on more robust features in the images. The validation accuracy drop is expected since the model input is limited region of the image.
8

---

Published as a conference paper at ICLR 2021
Image
Learned
Mosk
Image
Figure 7: Learned masks on CUB dataset are primarily focused on the bird, rather than the background.
Validation Set Accuracy Top
Test Set Accuracy (Distribution Shift) Top
Top
Top 5
Baseline (No masks)
78.82 1.19
93.95 0.28
25.81 0.60
56.98 + 0.69
With Masking Network
74.73 + 0.76 92.45 0.23
30.46 + 0.27
61.57 + 0.70
Table 3: Learned attention masks offer improved robustness to spurious background correlation. We train a commentary network produce attention masks on a version of the CUB-200-201 datasct that contains spurious background correlations, with the correlations present in the training/validaaionn set not present in the set (the test set has distribution shift). On hoth top 1 and top accuracy, using the masks results in improved model performance on the shifted test set.
6 RELATED WORK
Learning to Teach: Neural network teaching and curriculum learning have been proposed as early as Bengio et al. (2009); Zhu (2015). Recent work examining teaching includes approaches to: select, weight, and manipulate training examples (Fan et al., 2018; Liu et al., 2017; Fan et al., 2020; Jiang et al., 2018; Ren et al., 2018; Shu et al., 2019; Hu et al., 2019); adjust loss functions (Wu et al., 2018); and learn auxiliary tasks/labels for use in training (Liu et al., 2019; Navon et al., 2020: Pham et al., 2020). In contrast to most of these methods, to unify several related approaches and serve as a general learning framework for teaching. This enables applications in both standard settings such as example weighting, and also novel use-cases (beyond model performance) such as attention masks for interpretability and robustness. These diverse applications also provide insights into the training process of neural networks. Furthermore, unlike many earlier works that jointly learn teacher and student models, we also consider a different use case for learned commentaries: instead of being used in the training of single model alone, we demonstrate that commentaries can be stored with dataset and reused when training new models.
work commentaries aims
Learning with Hypergradients: Our algorithm for learning commentaries uses hypergradients derivatives of a model's validation loss with respect to training hyperparameters. Prior work has proposed different approaches to compute hypergradients, including memory-efficien exact com- putation in Maclaurin et al. (2015), and approximate computation in Lorraine et al. (2020); Lorraine and Duvenaud (2018); MacKay et al. (2019). We build on Lorraine al. (2020), which utilises the implicit function theorem and approximate Hessian matrix inversion for efficiency, to scale com- mentary learning to larger models.
CONCLUSION
In this paper, we considered general framing for teaching neural networks using commentaries, defined as meta-information learned from the dataset/task. We described two gradient-based meth- ods to learn commentaries and three methods of applying commentaries to assist learning: example weight curricula, data augmentation, and attention masks. Empirically, we show that the commen- taries can provide insights and result in improved learning speed and/or performance of datasets. addition, we demonstrate that once learned, these commentaries can be reused to improve the training of new models. Teaching with commentaries is of-on-e-- idea, and we hope that this work will motivate larger scale applications of commentaries and inspire ways of automatically re-using training insights across tasks and datasets.
variety

---

Published as a conference paper at ICLR 2021
REFERENCES
M. A. Badgeley, J. R. Zech, I. Oakden-Raynes B. S. Glicksherg, M. Liu, W. Gale, M. V. McConnell, B. Percha, T. M. Snyder, and Dudley. Deep learning predicts hip fracture using confounding patient and healthcare variables. NPJ digital medicine, 2(1):1-10, 2019.
D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541 6541-6549, 2017.
Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
Y. Fan, F. Tian, T. Qin, X.-Y. Li, and T.-Y. Liu. Learning to teach. arXiv preprint arXiv:1805.0366 2018.
Y. Fan, Y. Xia, L. Wu, S. Xie, W. Liu, J. Bian, T. Qin, X.-Y. Li. and T.-Y. Liu. Learning to teach with deep interactions. arXiv preprint MoXiv:2007.06649 2020.
C. Finn, P. Abbeel, and S. Levine. agnostc meta-leariig for fast adaptation of deep net- works. In ICMI, 2017.
E. Grefenstette, B. Amos, D. Yarats, P. M. Htut, A. Molchanov, F. Meier, D. Kiela, K. Cho, and
S. Chintala. Generalized inner loop meta-learning. arXiv preprint arXiv:1910.01721 2019.
R. Hataya, J. Zdenek, K. Yoshizoe, and H. Nakayama. Meta approach to data augmentation opti- mization. arXiv preprint urXiv:2006.07900 2020.
Z. Hu, B. Tan, R. R. Salakhutdinov, M. Mitchell, and P. Xing. Learning data manipulation for augmentation and weighting. In Advances in Neural Information Processing Systems, volume 32, 2019.
Irvin, Rajpurkar, M. Ko, Yu, Ciurea-lleus, C. Chute, H. Marklund, Haghgoo, R. Ball, K. Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 590-597, 2019.
L. Jiang. Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pages 2304-2313, 2018.
P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang. Concept bottleneck models. arXiv preprini arXiv:2007.04612. 2020.
S. Kornblith, J. Shlens, and Q Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2661-2671, 2019.
H. B. Lee, H. Lee, D. Na, S. Kim, M. Park, E. Yang, and J. Hwang. Learning to balance: Bayesian meta-learning for imbalanced and out-of-distributi tasks. In International Conference on Learning Representations, 2019.
Liu, A. Davison, and F. Johns. Self-supervised generalisation with meta auxiliary learning. In Advances in Neural Information Processing Systems, pages 1679-1689, 2019.
W. Liu, B. Dai, A. Humayun, C. Tay, C. Yu, L. B. Smith, J. M. Rehg, and L. Song. Iterative machine teaching. In International Conference on Muchine Leurning, pages 2149-2158, 2017.
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
L. Long. Maml-pytorch implementation. Pytorh,, 2018.
nttps://github.com http://hhhhu..mm///aegeeeeeeeeee
Lorraine and D. Duvenaud. Stochastic hyperparameter optimization through hypernetworks. arXiv preprint arXiv: 2018.
10

---

Published as a conference paper at ICLR 2021
J. Lorraine, P. Vicol, and D. Duvenaud. Optimizing millions of hyperparameters by implicit differ- entiation. In International Conference on Artificial Intelligence and Statistics,, pages 1540-1552, 2020.
M. MacKay, P. Vicol, J. Lorraine, D. Duvenaud, and R. Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. In International Con- ference Learning Representations (ICLR), 2019.
D. Maclaurin, D. Duvenaud, and R. Adams. mradient-basedd hyperparameter optimization through reversible learning. In International Conference on Machine Learning, pages 2113-2122, 2015.
A. Madry, A. Makelov, Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant adversarial attacks. arXiv preprint arXiv: 10:4v:7706.0000 170600003606 2017.
A. Navon, I. Achituve, H. Maron, G. Chechik, and E. Fetaya. Auxiliary learning by implicit differ- entiation. 2020.
H. Pham, Q. Xie, Z. Dai, and Q. V. Le. Meta pseudo labels. arXiv preprint arXiv:2003.10588 2020.
A. Raghu, Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards under- standing the effectiveness of maml. In International Conference Learning Representations, 2019a.
M. Raghu, C. Zhang, Kleinberg, and Bengio. Transfusion: Understanding transfer learning for medical imaging. In Advances in neural information processing systems, pages 3347-3357, 2019b.
M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. arXiv preprint arXiv: 1803009000 2018.
O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
J. Shu, Q. Xie, L. Yi, Q. Zhao. S. Zhou, Z. Xu, and D. Meng. Meta weitht-net: Learning explicit mapping for sample weighting. In Advances in Neural Information Processing Systems, pages 1919-1930, 2019.
S. Suwajanakorn, N. Snavely, J. Tompson, and M. Norouzi. Discovery of latent 3d keypoints via end-to-end geometric reasoning. In Advances in neural information processing systems, pages 2059-2070, 2018.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-uesd birds 200. 2010.
J. K. Winkler, C. Fink, Toberer, A. Enk, T. Deinlein, R. Hofmann Hofmann-Wellenhof, L. Thomas, A. Lal- las, A. Blum, W. Stolz, et al. Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recog- nition. JAMA dermatology. 155(10): 135-1 1141, 2019.
L. Wu, F. Tian, Xia, Y. Fan, T. Qin, L. Jian-Huang, and T.-Y. Liu. Learning teach with dynamic loss functions. In Advances in Neural Information Processing Systems, pages 6466-6477, 2018.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprinl arXiv:/611.00 1.03530, 2016.
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 1452-1464, 2017.
X. Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. 2015.
11

---

Published as a conference paper at ICLR 2021
blond hair, smiling
black hair
wearing glasses, not smillng
brown hair, wearing hat
black hair, wavy hair
pale skin, wearing
Figure A.1: Dimensions of the metalearned commentaries show correlations with CelebA attributes.
A ILLUSTRATIVE EXAMPLE: VECTOR COMMENTARIES ON CELEB-A
As an illustrative example, we describe how to learn real vector valued commentaries on the CelebA dataset, and then analyse the resulting commentaries.
The CelebA dataset contains images of faces, with each image also accompanied by 40 dimensional attribute labels. We learn a commentary neural network f(x; Φ) [-1,1] produce real vector valued commentary for each training example. We train a student network n(x;; 0) (Ú. f) to predict both the attributes and regress the commentary. The training and validaiton losses are as follows:
= (t - t(z:; (z: + (n(x; 0),
(5)
C = x-exeen((ng 0),
(6)
For both commentary and student networks, the CNN architectures from (Finn et al., 2017)). We learn commentary parameters using backprop- agation through student training (Algorithm $2.1). The inner optimisation for student learning consists of 100 iterations of Adam optimzer updates on the training loss using a batch size of 4. We use the higher library (Grefenstette al.. 2019). We use 50 meta-iterations (outer optimisation iterations) to train the commentary network.
4-block CNN with 3 filters (similar to
Results: Figure A.1 visualises the examples that are most positively/neeaaively correlated with cer- tain dimensions of the vector valued commentaries. We see clear correspondence between inter- pretable attributes and the commentary dimensions. Attributes such as hair colour, wearing glasses, and wearing hats are well-encoded by the commentaries. This suggests that such commentaries could function as an auxiliary task to give a network more signal during the learning process. Stor- ing the commentary network with the dataset could then allow these auxiliary labels to be re-used when training future models.
B EXAMPLE WEIGHTING CURRICULA
provide further details about the experiments using commentaries to define example weighting curricula.
B. ROTATED MNIST
Dataset: Both the overlapping and non-overlapping datasets are generated to have 10000 training examples, 5000 validation examples, and 5000 test examples.
Network architectures: CNNs with two blocks of convolution, batch normalisation. ReLU, and max pooling are used for both the student network and commentary network. The commentary network additionally takes in the iteration student training, which is normalised and passed as extra channel in the input image (i.e., the MNIST image has two channels. instead of a single channel). The commentary network uses sigmoid activation to produce an example weight for every data point in the batch.
12

---

Published as a conference paper at ICLR 2021
Non-Overlapping Dataset: Example Digits
Overlapping Example Digits
Rotation Angle Distribution
Rotation Angle Distribution
class_ class_1 __
class_0 0
class_1 __
Learned Example Weights
Learned Example Weights
Weight
Weight
40
-20
20 40
-30
-20
-10
0
20
30
Rotation
Rotation
Figure B.I: Learned per-example training weights Rotated MNIST. Visualizing the relationship be- tween example weights rotations at the end of training (left) reveals that for non-overlapping dataset, examples with the largest weights are located close to the decision boundary, which is expected, as these provide the most learning signal. In the overlapping case, weights for examples within the overlap region are lower, again expected as these examples ambiguous and do not assist in learning.
Example Rotated Digits
Commentary Curriculum:
Commentary Curriculum:
Non-Overlapping
Overlapping
Rank Corr (abs(rotation), ex-weight)
Ratation Distribution Overiagpin))
Rank Corr (abs (rotaatio)) ex-weigght)
0.0
-0.2
0.6
Rotation Angle Distribution (Dverispping)
-20
-0.4
0.4
-0.6
200
400
400 Training Iteration
-00
20
Rotation Angle
26
40
Training Iteration
Figure B.2: A learned curriculum of per-example training weights. Example rotated digits and the two classes are shown on the left. We plot the rank correlation between the magnitude of digit rotation and the example weight curriculum weights examples which are closer to the decision boundary more as training goes on. This resem- bles tcaching the student to focus on easy examples at first, and then moving to progressively harder examples as training progresses. When classes overlap, the curriculum weights more representative examples (near class centroids) more highly training progresses.
the course of student training for both datasets. When classes non-overlapping, the
Training details: We train both networks using the Adam optimiser, with a learning rate of le-4 for the student, and 1e-3 for the commentary network. The student network is trained for 500 inner optimisation steps, with batch size of 10. We train for 20 commentary network iterations. Training implemented using the higher library (Grefenstette et al., 2019).
Figure B. visualises for both datasets: the relation between the learned example weight and the rotation of the digit at the end of student training (iteration 500) following binning. Con- sidering these final example weights. for the non-overlapping case, examples with higher weight are those with smaller rotation and thus closer the decision boundary (i.e.. resembling support vectors) these provide the most information are those with greater rotation. When classes overlap, the weights are maximised for examples that better characterize each class, and not in the overlap region these are most informative of the class label.
Results:
separate the classes. Lower weighted examples
Now considering the learned curriculum (Figure B.2), for the non-overlapping dataset, observe that the rank correlation between the example weight and the rotation magnitude decreases over the course of student training iteration. This is intuitively sensible, since the example weights first prioritise the easy examples (large rotation), and then learn to focus on the harder ones (small rota- tion) that provide most information to separate the classes. By contrast, in the overlapping case, the
13

---

Published as a conference paper at ICLR 2021
curriculum has examples that are further from the boundary (larger rotation magnitude) consistently weighted most highly, which is sensible as these are the most informative of the class label.
Overall, the results this synthetic case demonstrate that the learned commentaries capture inter- esting and intuitive structure.
B.2 CIFAR 1IFAR10/100
Network architectures: We use the two block CNN from the MNIST experiments for the commen- tary network, and as the student network when training the commentary network. We employ the same strategy for encoding the iteration of training. At testing time, we evaluate this commentary network by teaching three different student network architectures: two block CNN, ResNet- 18, and ResNet-34.
Training details: We train both networks using the Adam optimiser, with a learning rate of 1e-4 for the student, and 1e-3 for the commentary network. During commentary network learning, the student network is trained for 1500 inner optimisation steps, with batch size of 8, and is reset to random intialisation after each commentary network gradient step. We train for 100 commentary network iterations. Training is implemented using the h gher library (Grefenstette et al., 2019). At testing time, we use a batch size of 64; the small batch size at training time is due to GPU memory constraints.
B.3 FEW-SHOT LEARNING (FSL)
The MAML algorithm finds a network parameter initialisation that can then be used for adaptation to new tasks using support set of examples. The commentary network here is trained to provide example weights for each example in the support set, at each iteration of inner loop adap- tation (i.e., the example weights are not used at meta-testing time). We jointly learn the MAML initialisation and the commentary network parameters; intuitively, this represents learning initial- isation that able to adapt to new tasks given examples and associated weights.
Setup:
Dataset details:
We use standard datasets used to evaluate FSL methods, and the associated splits between training and testing tasks from prior work (Lee al., 2019; Long. 2018). We evaluate on two out-of-disrribttin settings, namely: training the few-shot learner on CIFAR-FS and testing on SVHN; and training the few-shot learner on MinilmageNet and testing on CUB-200.
Network architectures: Both the commentary and the student networks use a 4-block CNN archi- tecture commonly seen in prior work (Finn et al., 2017). The student network takes a given image as input. The commentary network takes as input the support set image and the class label. The one-hot labels are converted into a 64 dimensional vector using a fully connected layer, concatenated with input image representations, then passed through two more fully connected layers to produce the output. This output is passed through sigmoid to ensure example weights lie in the range [0, 1]. These weights normalised to ensure a mean weight of across the entire support set, which helped stability.
We use Adam with learning rate of le-3 to learn both the commentary network parameters and the student network initialisation point for MAML. A meta-batch size of 4 is used for meta training. We use SGD with a learning rate of le-2 for the inner loop adaptation. At meta- training time, we use 5 inner loop updates. At meta-test time, we use 15 inner loop updates (similar to some other methods, to allow for more finetuning). For evaluation, we create 1000 different test time tasks (randomly generated) and we compute mean/95% CI accuracy on this set of tasks. We use the h gher library (Grefenstette et al., 2019).
Training details:
We evaluate standard MAML baseline and our commentary variant on standard few-shot learning benchmarks: (i) training/testing on MinilmageNet (MIN) and CIFAR-FS (in-distribution testing); and (ii) training MIN and CIFAR-FS and testing CUB-200-201 (CUB), and SVHN diit-f--isttibbution testing). Results are shown in Table B.I. Each row specifies the experimental setting -wway K-shot), the dataset used for training, and the dataset used for testing. In all experi- ments, incorporating example weighting can improve on the MAML baseline, suggesting the utility of these commentaries in few-shot learning.
Results:
14

---

Published as a conference paper at ICLR 2021
Mode
MAMI. Baseline
Example weighting MAMI.
5-way 1-shot: -CIIRAR--
46.27 = 0.71
47.07 + 0.72
5-way 1-shot: SEAAR-FS-SVHIE
24.89 + 0.36
26.26 + 0.38
5-way 5-shot: CIFAR-FS CIFAR-FS-:CIFAR-ES -CIFAR-FS
65.40 + 0.61
66.00 + 0.63
5-way 5-shot: CIFAR-FS-SSHHN
37.28 0.43
37.70 + 0.43
5-way 1-shot: MIN
43.64 0.61
45.07 + 0.61
5-way 1-shot: MIN-CUB
37.45 + 0.55
38.00 + 0.54
5-way 5-shot: MIN-+MIN
61.85 + 0.52
62.99 + 0.52
5-way 5-shot: MIN->CUB
57.75 + 0.54
59.06 + 0.55
Table B.1:
Mean accuracy and 95% confidence interval across 1000 test-time tasks. On common few-shot learning benchmarks, using example weighting for the support set improves the MAML baseline's accuracy. Improvements are also observed when evaluating on out-of-distribnn oif-iiitiitttion datasets.
C DATA AUGMENTATION
We provide further details about the experiments using commentaries to define data augmentation policies.
C.I MNIST
Network and training details: The 2 block CNN is used as the student network. Denoting each entry of the commentary grid as di.j, we initialised each entry to O. The blending proportion is formed as: Xr = 1 0.5 sig This is to ensure that the blending proportion is between 0.5 and I; this implies that blended image contains more of the first image (class i) than the second (class j). Without this restriction, certain blending combinations could 'flip,, making the results harder to interpret. The inner optimisation uses SGD with learning rate of le-3, and had 500 gradient steps. We used 50 outer optimisation steps to learn the commentary parameters, using Adam with a learning rate of le-1. The commentary parameters were learned with the hi higher library (Grefenstette al., 2019).
Further Detail on MNIST Augmentation Commentaries: We learn an augmentation commentary model / on MNIST, represented as x0x 10 matrix. This commentary learned by backpropagating through the inner optimisation, using a 2-block CNN student network. For each outer optimisation update, we use 500 steps of inner loop training with the augmented dataset, and then compute the validation loss the unblended data obtain commentary parameter gradients.
We find a trend in the learned augmentation relating the error rates and blending proportions. Con- sider single image Tú, label i, and other images Tz. label Vj # i. Averaging the blending pro- portions (computed as 0.5 x (igmoid(@i..)) over j. the error rate on class is correlated (Pearson correlation= -0.54) with the degree of blending of other digits into example of class i; that is, lower eтror rate on class implies that other digits are blended more heavily into it. On MNIST, this means that the class that has on average the lowest eиror rate (class has other digits blended into it most significantly (seen in Figure 4 left). On the other hand, classes that have on average higher error rate (e.g., class 7, class 8) rarely have other digits blended in (Figure 4 right).
C.2 CIFAR 10/100
Network and training details: The student network is a ResNet18. We use the method from Lor- raine et al. (2020) to learn the commentary parameters. The commentary parameters are initialised in the same way as for the MNIST experiments. These parameters are learned jointly with a student, and we alternate updates to the commentary parameters and the student parameters. We use Neu- mann step to approximate the inverse Hessian when using the IFT. For commentary learning, we use Adam with LR of 1e-3 as the inner optimiser, and Adam with LR of le-2 as the outer optimiser.
For evaluation, we train three randomly initialised students using the fixed commentary parameters. This training uses SGD with common settings for CIFAR (starting LR 1e-1, weight decay of 5e-4,
15

---

Published as a conference paper at ICLR 2021
CIFARTO
CIFAR100
Test Accuracy
Test Loss
Test Accuracy
Test Loss
No commentary
93.84 + 0.22
0.239 + 0.005
74.79 + 0.39
1.05 + 0.03
Random commentary
93.84 + 0.30
0.385 0.007
73.39 0.58
1.14 0.01
mixup (Zhang al., 2018)
94.42 + 0.12
0.31 0.01
75.89 0.55
1.01 0.02
Learned commentary (shuffled)
94.01 + 0.19
0.235 + 0.005
75.95 0.27
0.99 + 0.01
Learned commentary (original)
94.12 0.19
0.225 0.007
76.25 0.11
0.97 0.01
Table C.1: Learned augmentation commentaries result in competitive or improved model accuracy and loss CIFAR10 and 100. Compared to not using commentary and using randomised label-depende commentary, the original learned commentary results models that competitive with mixup and improve compared other baselines. show mean/standard deviation over three different initialisations.
airplane
automobile
bird
cat
deer
dog
frog
horse
ship
truck
airplane aunomobu
bird
deer
dog
frog
horse
ship
tuck
Figure C.I: Augmentation commentary policy CIFARIO. We observe interesting emergent patterns of the learned blending proportions; classes that visually similar and potentially confused (cat/dog, automo- bile/truck) are blended relatively significantly.
decaying LRR after 30, 60, and 80 epochs by a factor 10). We use standard CIFAR augmentations in addition the learned augmentations this evaluation phase.
We compare to using no commentary (just the standard CIFAR augmentation policy, random crops and flips), random commentary (where augmentation grid constructed by uni- formly sampling blending proportions in the range [0.5, 1)), mixup (Zhang et al., 2018) with blending proportion drawn from B,) and shuffled version of our method where the commentary grid is shuffled at the start of evaluation (destroying the structure, but preserving the scale of augmenta- tion).
Baselines:
Results: Table C.1 shows model accuracy for different augmentation policies on CIFAR10 and 100. We compare the learned commentary to using only standard data augmentations for CIFAR 10/100 (No commentary). shuffling the learned commentary grid, using random initialisation for the com- mentary tensor, and mixup (Zhang al., 2018). We observe that the learned commentary com- petitive with mixup and improves other baselines. The fact that the learned commentary does better than the shuffled grid implies that the structure of the grid is also important, not just the seale of augmentations learned.
Visualising the policy: For CIFAR10, we visualize the full augmentation policy in the form of a blending grid, shown in Figure C.I. Each entry represents how much those two classes blended, with scale on left. This corresponds to 0.5 x sigmoid( (p);), with di,5 representing an entry in the commentary grid.
16

---

Published as a conference paper at ICLR 2021
CIFARIO
CIFARTO0
Test Accuracy
Test Loss
Test Accuracy
Test Loss
Random mask
92.15 + 0.25
0.274 + 0.008
71.27 + 0.10
1.18 + 0.02
Permuted learned mask
91.69 + 0.07
0.340 + 0.08
71.50 + 0.07
1.22 + 0.01
Learned mask
92.99 + 0.11
0.265 + 0.003
73.16 = 0.44
1.12 + 0.01
Table D.1:
Performance of different masking strategies on CIFARIO and 100. Using the appropriate per- image learned masks improves (in both loss and accuracy) permuting the masks across the entire dataset and randomly selecting mask regions of the appropriate scale.
D ATTENTION MASKS
Datasets and Mask Information: We use number of datasets to evaluate the learned masks, in- cluding: Coloured MNIST (a synthetic MNIST variant), CheXpert (a dataset chest X-ray images from (Irvin et al., 2019)), CIFARIO/100, and CUB-200-2011. More details:
The Coloured MNIST dataset is formed by randomly sampling two MNIST digits from the dataset, and choosing one to be red and one to be blue. The red digit determines the image label. The two digits are randomly placed on two different quadrants of 56 x 56 grid. The standard deviation of the mask is set to be 15 pixels.
The CheXpert dataset has large X-ray radiograph images. Each image is resized to be 320 x 200. The mask standard deviation is 50 pixels.
CIFARI0/100 masks are be 15 pixels standard deviation.
CUB-200-2011 images are resized to be 224 224, and the mask standard deviation 50 pixels.
Network architectures: The student network was a ResNer18, and the commentary network was U-Net (Ronneberger et al., 2015) with an output layer from KeypointNet (Suwajanakorn et al., 2018). This takes a probability mass function defined spatially, and the (z, y) centre the mask is computed the mean in both spatial dimensions. Producing the mean in this manner significantly helped stability rather than regressing a real value.
We the method from Lorraine et al. (2020) to learn the commentary parame- ters. These parameters are learned jointly with a student, and we alternate updates to the commentary parameters and the student parameters. We use Neumann step approximate the inverse Hessian when using the IFT. When the commentary network is learned, we use Adam with LR le-4 for both inner and outer optimisations. We found balancing this learning rate to important in the resulting stability of optimisation and quality of learned masks.
Training details:
When evaluating the learned masks, we trained three new ResNet- 18 students with different random seeds, fixing the commentary network. For CIFAR for evaluation, we use SGD with common settings for CIFAR (starting LR 1c-1, weight decay of 5c-4, decaying LR after 30, 60, and 80 epochs by factor of 10). We use standard CIFAR augmentations for this also.
Visualizing Masks: Figure D.I shows masks from the main text and further additional examples.
Baselines for CIFAR experiments: For the random mask baseline, for each example in a batch, we select centre point randomised over the whole image, then form the mask by considering gaussian centered at that point with standard deviation 15 (same size masks from commentary network). This resembles very aggressive random cropping. For the permuted learned mask, we use the learned commentary network to predict masks for all images. Then, permute the mask- image pairs, so that they no longer match up. We then train with these permuted pairs and assess performance. Our goal is understand what happens when we have random masks with similar overall spatial distribution to the real masks.
CIFAR Masking Quantitative Analysis: We compare masks from the leamned commentary net- work to two baselines: randomly chosen mask regions for each image (of the same seale as the learned masks, but with the centre point randomised over the input image), and permuted masks
17

---

Published as a conference paper at ICLR 2021
Coloured MNIST
Chest Ray Data
CIFAR10
2
Image
Learned Mask
2
Masked Image
CIFAR100
Image
Learned Mask
Masked
Image
Figure D.1: Learned attention masks highlight salient image regions for classification. We learn commentary network to produce image attention masks four datasets, and the masks are qualitatively effective in all cases. On Coloured MNIST, where the image label is determined by the red digit (blue is distractor), the masks focus on the red digit. On dataset of chest X-rays, the masks focus on the chest cavity, which is the appropriate reason for detecting the condition in question (cardiomegaly). On CIFAR10/100, the masks focused important regions of the object the image, such the faces of animals//abies, and the bodies of vehicles.
(where we shuffle the learned mask across all the data points). Table D.I shows the results. Espe- cially on CIFAR 100, learned mask improves noticeably on the other masking methods both test accuracy and loss. This suggests that overall, the masks highlighting more informative image regions. We do not expect using the masks on standard problems to result in improved held-out per- formance, because the backgrounds of images may contain relevant information to the classification decision.
Further details robustness study: The dataset generated using the open source code from Koh et al. (2020). The student network for this study was pretrained on ImageNet, as in Koh et al. (2020). To train student models at the evaluation stage, used SGD with learning rate of 0.01, decayed by a factor of 10 after 50 and 90 epochs. We used nesterov momentum (0.9) and weight decay of 5e-5.
18


Reviews:
[{"review_id": "zsOFBP8eQpo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a framework for learning a commentary or teacher model\nthat provides helpful information during training. Under this framework, a\nteacher model learns to provide meta-information, referred to as a commentary,\non a given training example, with the goal of improving the student network\n(e.g., improving convergence speed or robustness). The learning of the teacher\nmodel is done by minimizing the validation set loss of the student model under\nparameterization from a training loop that used the commentaries.\n\nThe optimization of the teacher parameters involves backpropagating\nthrough gradient descent steps. The authors propose doing this directly\nin small toy examples, or using an approximation using the Implicit Function\nTheorem for more realistic problems.\n\nThe authors then demonstrate that the commentary learning framework can\nbe used in a variety of ways:\nlearning to provide example weights, learning a blending policy for data augmentation, and learning to provide an attention mask for image classification.\nIn the latter case, they show that using the attention masks leads to a more\nrobust student when the backgrounds are modified to spuriously correlate\nparticular class labels on the training/validation sets but not on the\ntest set.\n\nThis paper has many strengths. It introduces an interesting meta-learning\nframework and demonstrate its flexibility to implement three different\nkinds of commentary for improving a student network. Additionally, it supports\nits statements with a variety of interesting experiments. While I would\nlike to see this paper accepted, it could be improved \nin its organization and explication.\n\nFor example, it is not immediately clear that the commentary network is learned with equations 1 & 2, but the student network in this process is ultimately discarded and only after the finished teacher network is obtained, is a\nstudent network actually trained for evaluation purposes. An \"illustrative example\" of commentaries is put in the appendix instead of the main paper which\nfeels like a lost opportunity to holistically explain the algorithm and\nclear up these misconceptions.\n\nIn the example weighting curricula it is not clear what is meant by curricula.\nIn an ablation study, example weights produced by a teacher network with and\nwithout curricula are compared, but what constitutes the curricula here is\nnever defined. I'm inferring that in the curricula case, the teacher network\nproduces weights using both the example x and the current training iteration\ni, while the non-curricula case only uses x. This should be stated explicitly\nsomewhere. \n\nIn equations 1 and 2 the authors write\n\noptimize(phi/theta)[loss func]\n\nwhen\n\nargmin_phi/theta loss func\n\nwould convey what the authors intend while using standard notation.\n\nOn the masking experiments, the masks are used both at training and test time\nwhich seems somewhat odd. The commentary formalism proposed is supposed to\nbe limited to learning a commentary model to facilitate training. In this case,\nthe commentary model seems less like a separate teaching model and more like\nanother component of the classification model, similar to \"Recurrent Models of Visual Attention\" where a component of the classifier determines where to\nfocus in an image. How well does the student network trained with masks provided by the teacher do on the validation/test sets without masks?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "yZOwY3-Wu9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary] This paper introduces a gradient-based meta-learning method to learn weighting for each training sample, called \"commentary\" in the paper's description, as a form of auxiliary learning, updated on training loss to accelerate training speed, and improve generalisation.\n\n[Strength] The motivation is interesting, as to balance training by automatically computing training sample weighting. The toy example on rotated MNIST clearly shows that the teacher network would put a higher weighting on samples closing to the decision boundary, which is expected and confirms the authors' assumption.\n\n[Weakness] The paper has several weaknesses which I will outline below.\n-- Curriculum v.s. Non-curriculum learning. The authors present two versions of \"commentary\", one is with a curriculum and the other is without a curriculum. The curriculum version, which I assume is to update training sample weighting iteratively with the updated teacher network. However, the authors did not mention any details on the non-curriculum learning version, on how they compute the training sample weighting in a fixed way.\n\n-- Inconsistent results on ResNet18. In Fig 3., ResNet 18 seems to perform unreasonably low on CIFAR-10/100 with roughly ~70 and ~45 of accuracy, respectively. However, with the same network of ResNet18, they perform in a reasonable performance in Table 2.\n\n-- Marginal performance. Using commentary to generate mix-up style sample weighting seems reasonable and interesting. But the results are quite marginal, or do not consistently improve over the standard mix-up baseline as shown in Table 2.\n\nFurther questions: What is the distribution of training and validation data in CIFAR10/100 results? Is it possible that the teacher network simply put higher weights on training samples that look most similar to validation data, as a form of a trivial/degenerated solution? I assume this might happen particularly in the rotated MNIST example? And how authors generate the labels for those overlapping samples in rotated MNIST? \n\nI hope the authors could elaborate on these questions and further refine the paper. I think in general the idea is interesting and worth exploring, but this paper is not ready yet.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TNh0tCoKLp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a general framework for boosting CNNs performance on different tasks by using'commentary' to learn meta-information. The obtained meta-information can also be used for other purposes such as the mask of objects within spurious background and the similarities among classes. The commentary module would be incorporated into standard networks and be iteratively optimized with the host via the proposed objective. To effectively optimize both the commentary and the standard network, this paper adopts the techniques including implicit function theorem and efficient inverse Hessian approximations. \n\nThis paper studies three kinds of commentary named weight curriculum commentary that learns individual weights for data samples, augmentation commentary that learns data augmentation strategy similar to mix-up, and attention mask commentary that learns to focus on objects of images. All three commentaries have been examined via extensive experiments on small-scale benchmarks and shown improvements. \n\nThe studied direction is very interesting that how to obtain meta-information with standard data annotations. Although, the reviewer thinks all three commentaries' tasks are studied in the literature. This paper integrates them into a general framework. Also, the weight curriculum commentary can be naturally compatible with MAML. There may be possible to derive more different auxiliary tasks via properly designing the commentary structure. The experimental details are described in detail and make the experiments are easy to replicate. Overall the paper is easy to follow.\n\n-------------------------\n\nHowever, the proposed framework needs human efforts to specifically design the formula of the commentary structure to involve proper inductive bias for learning the targeted meta-information. There're works that also only use the standard annotations to study adjusting the weights for data samples, data augmentation (like the mix-up), and focusing the object.  Compared to them, the reviewer does not see the clear advantages of the proposed method. Also, the author does not provide many comparisons. \n\nThe reviewer also concerns about scalability. Currently, the method is only tested on several small-scale benchmarks including MNIST, CIFAR, TINY-ImageNet, and CelebA, X Ray Data. However, the objective would need a lot of computation resources, though approximately computing the inverse Hessian. The reviewer doubts that it is hard to apply in real-world tasks and the method can not bring benefits when the data is big. But, this concern may be beyond the scope of this paper and can be considered in the following works.\n\nIn conclusion, the reviewer would rate 7 due to it nicely integrates several meta-information tasks into a unified framework and hope this work can bring some new understandings to the communities. \n\n---------------after rebuttal----------\n\nI've read all reviews and the rebuttal. I think overall this is a good paper and would like to keep my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4qqqQxtiBja", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a framework for incorporating and optimising meta-information as additional parameters when optimising the validation loss for a specific network on a given task. More precisely, they instantiate a teacher network that supplies the meta-information (could be anything from attention maps, to importance weights over sample points, to parameters of a data-augmentation scheme) that is then used by a student network, and propose a formulation whereby the two are trained in tandem so as to optimise the output loss of the student network. \n\nGeneral remarks: The paper is well-written and well-presented. The idea is clear, language is effective and visualisations/tabulations are accessible. \n\nPros (Originality/Significance)\n(a) I believe the biggest contribution of the paper is in the unifying of different existing approaches that explicate and optimise meta-information as part of the network pipeline in a bid to improve performance (accuracy or robustness). Ideas such as example reweighting, example blending (e.g. mixup), describing attention masks, or transforming input data (rotation, etc) are treated as special cases of a particular student-teacher formulation. To the best of my knowledge, this unified treatment is unique and comes at a crucial time when same techniques seem to reappear under different names and in different forms and prevent a wholecome understanding of the approach and its wider implications.\n\nCons (Originality/Significance)\n(a) The maths of the reunification is rather simple, teacher outputs values that a student uses and optimises, and could lead to a time-consuming training.\n(b) The instantiations of meta-information studied in this paper have been explored before and don't seem to offer any significant advantages over existing methods (the benchmarks in some tables are not up-to-date, for example, consider referring to [1] for sample reweighting benchmark and [2] wherein an attention based classification network for CUBS has been proposed.\n(c) Some use cases are shown to work only for small datasets, such as CIFARs for example reweighting. Newer techniques that are able to scale example reweighting to ImageNet exist [3]. Their method seems to need curriculum learning on top of example reweighting for full benefit, please comment.\n\n\n[1] Learning to Reweight Examples for Robust Deep Learning, Mengye Ren et.al.\n[2] Learn to pay attention, Jetley et.al. \n[3] Sample Balancing for Deep Learning-Based Visual Recognition, Chen et.al.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an interesting unified framework for meta-learning with commentaries, which contains information helpful for learning about new tasks or new data points. The authors present three kinds of different instantiations, i.e., example weighting, example blending, and attention mask, and show the effectiveness with the extensive experiments. The proposed method has a potential to be used for a wide variety of tasks.
Decision: Accept (Poster)