Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method, TRAM, that integrates the privileged information into the learned network weight through weight sharing at training time and approximately marginalizes over the privileged information at test time. Experiments on both realistic and synthetic datasets demonstrate the effectiveness of TRAM in improving model performance on noisy data.
main_review: Strength:
1. the authors present a clear explanation of the underlying intuition of using PI and the motivation behind TRAM. 
2. the idea of integrating privileged information through weight sharing is interesting, and the two-step design is novel (as far as I am concerned).
3. the proposed method can (in principle) apply to any neural network model and has zero overhead at prediction time.

Weakness/Concerns:

1. In the title and abstract, the authors state that privileged information helps explain away label noise. However, the latter investigations only focus on the accuracy improvement without an in-depth discussion about how and how much the label noise can be "explained away." in realistic deep neural network models.
2. there lacks quantitative (or even qualitative) evidence about how, how much, and what kind of privileged information is transferred through weight sharing in realistic deep neural network models.
3. In the discussion of Table 2, it is overstated that "TRAM provides significant performance improvement" since a ~5% increase in accuracy does not seem significant at all.

Continuing point 2 the above, I have a question: Is it possible to control the amount of privileged information transferred to the shared network block by designing a specific training scheme? Say (just a rough idea), we may gradually increase the weight of the shared network block during backpropagate through the $q(y|x,a)$ branch.
summary_of_the_review: The paper proposes TRAM, a method of integrating privileged information in the network weights during training. Experiments on both realistic and synthetic datasets show that TRAM can help improve the model accuracy on noisy data. However, the improvement is significant. Further, the lack of detailed investigations about how the privileged information is integrated into the network weights and how the privileged information help explain away the label noise in realistic deep neural network models limits the contribution of the present work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes classification methods utilizing privileged information (PI, which is available only at training time). The main idea is to directly approximate the conditional marginal distribution, i.e., p(y|x) \approx q(y|\pi(x), w), where \pi(x) is a feature extractor and w is the weight parameter. Another main point is to learn the representation (\pi(x)) of the features with access to PI at the training time. To do this, the proposed methods employ knowledge transfer by weight sharing. The proposed methods include several variants by adopting heteroscedastic classification (Het-TRAM) or distillation (Distilled-TRAM).  
main_review: Strengths 
1. The proposed methods have simple architectures (not requiring specific modules, e.g., Gaussian dropout [Lambert et al., 2018], for the marginalization). 
2.  Making prediction for a test point (PI is not available) has the same cost with a standard network trained without PI. 

Weaknesses
The manuscript could be improved in presentation. I am listing some points which are unclear and need to be addressed.
- What is the formal definition of label noise for classification? It seems that the current version of the manuscript provides an example for label noise for regression (Section 2.2), but there is not given a clear definition or an example for label noise for classification. 
- I think that the transition from the theoretical analysis using the linear regression model in Section 2 to the motivation of approximating the conditional marginal distribution is not that smooth. It seems that the analysis just supports that PI can be useful. In fact, the motivation of approximating the conditional marginal distribution is directly explained by the equation (2) (the idea of the representation learning with PI is explained by the example in 2.2). I think that the section 2.1 could be revised to address how this theoretical analysis can motivate the approximation of the conditional marginal distribution.
- In the introduction, it is stated “We provide empirical evidence suggesting that the representations learned with access to PI are more robust against label noise”. What does “empirical evidence” refer to? The motivating example in Section 2.2?  If the answer is yes, I am not sure if this toy example can be convincing empirical evidence. I think that this is just a motivating example to explain the intuitive idea of the representational learning with PI. In addition, the example is for regression (the assumptions here would not be realistic settings for classification).   

summary_of_the_review: I think that the main idea, directly approximating the conditional marginal distribution without requiring any sampling methods or additional specific architectures, is interesting (although I think the proposed methods are not super novel). I also like the way the authors build their methods from theoretical analyses and motivating examples. However, I at the same time think that the manuscript could be improved. Please see the detailed comments above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper suggests a notion that using additional information called privilege information (PI) from annotators will help explain away the noise of the label they annotated and propose a way to implement that idea. The authors give the intuition from a simple linear model and a non-linear sine model that PI is useful. Then, an implementation of that idea called TRAM is proposed to make use of the information from PI through a stop gradient operator in training.  Since there are scant large-scale datasets with PI annotation, to evaluate the proposed approach, the authors have to create a synthetic one relabeled from ImageNet. 
main_review: Strengths:
1. Propose the notion of using PI to improve the performance of a network by explaining away the noise from annotators. 
2. Propose a new way to implement that idea using a stop gradient operator

Weaknesses:
1. Theoretically, the authors do not give a formal proof of why using PI in a practical network like ResNet helps to improve performance. The authors just give the proof for the very simple edge cases like linear and non-linear sine models. How about the classification case that this paper is targeted for? These simple cases are only true when the feature extraction network is kept unchanged during training. 

2. Empirically, the experiments are conducted in a limited and counter-intuitive: (a) When testing on CIFAR-10, it is quite unintuitive that the authors train on the test set and evaluate on the training set. To address the lack of training examples since the test set is used for training, they use the pretrained model on ImageNet; (b) When testing on the synthetic dataset relabeled from ImageNet, the information of the probability of the label assigned by a pretrained model is used as a proxy for the confidence of a human annotator. This information tells a lot about the quality of the label, which is unrealistic in real-world scenarios since, for each example, the annotators are required to additionally answer the confidence question resulting in a significant cost and ambiguity. 

My suggestion is that the authors should collect a real-world large-scale dataset such that the PI is insignificantly expensive to obtain along with the main annotations to prove their claim empirically. Otherwise, this paper has a limited impact on practical scenarios.
summary_of_the_review: This paper proposes an intuitive phenomenon that using PI is helpful and improves the performance of the model. However, the authors have not given any proof for the general case (not the edge cases) theoretically or empirically successfully.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper develops a simple and efficient method for training with privileged information and testing without privileged information. The training with privileged information focuses on transferring via sharing the weights of the encoder, which is the knowledge learned with privileged information.
The testing without privileged information addresses the practical case where test data may not be equipped with useful privileged information. The corresponding solution is to marginalize over privileged information at test time.
main_review: Strengths:

1. The proposed framework uses privileged information in deep neural networks efficiently.  

2. The method is well-demonstrated with both theoretical analyses and numerical experiments. Thus the result is convincing.

3. Using privileged to deal with label noise may help to better characterize the instance-dependent label noise. Thus this work could contribute a new idea to learning with label noise.


Weaknesses:

1. The notations could be better defined. For example, in Page 2, in addition to saying "Denoting by $\prod_x$ the orthogonal projector associated with $X$", it would be better to show the equation as well. In Page 3, $u$ in $q(y|x,a;u)$ is not clearly defined.It is also confusing why $a \cdot u$ is the noise term. Should it be $a \cdot v$? 

2. The privileged information in the toy example may be too strong since $\epsilon$ is too small compared with $v$ and knowing $a$ means knowing which point is corrupted by label noise. The privileged information is often side information to help better decisions, rather than critical knowledge as shown in this example.

3. The feature extractor trained with privileged information is supposed to be better than the one trained without privileged information. But the current experiment did not explicitly show it. An ablation study that compares the performance of fine-tuning the last layer using clean data with fixed feature extractors (trained with PI or not) would be helpful.
summary_of_the_review: The paper proposes a new perspective on the noisy learning literature. The proposed framework is simple but useful.
review_title: null

Meta-review: The manuscript proposes (TRansfer and Marginalize) TRAM method that integrates the privileged information into the learned network weight through weight sharing at training time and approximately marginalizes over the privileged information at test time. TRAM can also be combined with methods for dealing with noisy labels, distillation (Distilled-TRAM) and heteroscedastic output layers (Het-TRAM). Experiments are performed on both realistic and synthetic datasets including CIFAR-10H, re-labeled ImageNet, and Civil Comments Identities.

Reviewers agreed on several positive aspects of the manuscript, including:
1. The proposed methods have simple architectures (not requiring specific modules, e.g., Gaussian dropout [Lambert et al., 2018], for the marginalization);
2. The proposed method can in principle be applied to any neural network model and has zero overhead at prediction time.

Reviewers also highlighted several major concerns, including:
1. The analysis is performed on edge cases such as linear and non-linear sine models. There is no analysis for the classification case that this manuscript is targeted for. The simple cases are only true when the feature extraction network is kept unchanged during training;
2. Empirically, the experiments are conducted in a limited and counter-intuitive; 
3. Lack empirical evidence suggesting that the representations learned with access to privileged information are more robust against label noise;
4. Lack quantitative (or even qualitative) evidence about how, how much, and what kind of privileged information is transferred through weight sharing in realistic deep neural network models.

Several new experiments have been added to show, among others: representations learned with privileged information outperform representations learned without access to privileged information (using a linear classification model on ImageNet), better quantitatively and qualitatively understanding how and how much privileged information is transferred in realistic deep networks.

Post-rebuttal, reviewers stayed with borderline ratings, and they have suggested further improvements: simulating with more annotators by using different checkpoints and/or different hyperparameters, collecting a real-world large-scale dataset such that the privileged information is insignificantly expensive to obtain along with the main annotations, and disentangling the effect of the pretraining model on the denoising method.
Decision: Reject