Equally Critical: Samples, Targets, and Their Mappings in Datasets

Authors: Runkang Yang, Peng Sun, Xinyi Shang, Yi Tang, Tao Lin
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Data-efficient Learning

Abstract:
Neural scaling laws highlight the trade-off between test error reduction and increased resources in machine learning, revealing diminishing returns as data volume, model size, and computational power increase.
This inefficiency poses sustainability challenges, as marginal performance gains necessitate exponential resource consumption.
Recent works have investigated these laws from a data-efficient standpoint, primarily concentrating on sample optimization, while largely neglecting the influence of target.
In this study, we first demonstrate that, given an equivalent training budget, employing soft targets on a 10% subset can outperform the use of one-hot targets on the full dataset. Building on this observation, we review existing paradigms in the sample-target relationship, categorizing them into distinct sample-to-target mapping strategies.
Subsequently, we propose a unified loss framework to assess their impact on training efficiency. Finally, we conduct a comprehensive analysis of how variations in target and sample types, quantities, and qualities influence training efficiency across three training strategies, providing six key insights to enhance training efficacy.

Decision: reject
Reviewer scores: 6, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1729069984294

thread_type:
official_review

<thread_object>
creator: reviewer_trqj

<content>
summary:
The paper investigates the effects of soft target formats, teacher model performance, data quantity, and data augmentation on the training process, within a framework similar to knowledge distillation. The paper primarily presents extensive experiments on CIFAR outputs and summarizes the key findings derived from these experiments.

strengths:
1. Studying neural scaling laws and exploring how to overcome them to achieve a balance between efficiency and performance is both practical and meaningful.
2. The paper presents extensive experimental results from multiple perspectives.

weaknesses:
1. For scaling law research, the experimental scale is too small, even with the so-called “large-scale” Tiny-ImageNet dataset mentioned in the paper.
2. The paper merely lists observations without extracting the underlying insights or potential implications for practical applications.
3. I have concerns about the experimental results. For instance, a ResNet-18 model trained on CIFAR-10 typically achieves over 95% accuracy, yet the best result reported in the paper is only around 80%.
4. The so-called key findings are also trivial. For example, “Soft targets expedite early-stage training”: it’s well-known that knowledge distillation from a teacher model accelerates model convergence. Additionally, “one-hot targets yield superior final accuracy” is unsurprising, as the teacher models in the paper are weak and unable to match the performance of traditional supervised learning, thus hindering final accuracy. I don’t believe this conclusion would hold for more challenging tasks and strong teacher models.
5. I believe the statement “this study is the first to emphasize the critical role of targets in breaking neural scaling power laws” is an overclaim. As mentioned in lines 50-57, there are already existing works on it.
6. The paper does not provide the experimental setup for Figure 1, and the conclusions drawn are inconsistent with those shown in Figure 6(a), where even a 100% subset of soft targets fails to outperform hard targets.
1. Some works regarding offline and online data selection on efficient learning, such as those listed below, should be discussed.

[1] Spanning training progress: Temporal dual-depth scoring (TDDS) for enhanced dataset pruning. CVPR 2024.

[2] Data selection for language models via importance resampling. NeurIPS 2023.

[3] Diversified batch selection for training acceleration. 2024. ICML 2024.

[4] Towards accelerated model training via bayesian data selection. NeurIPS 2023.

[5] Coveragecentric coreset selection for high pruning rates. ICLR 2023.

[6] Beyond neural scaling laws: beating power law scaling via data pruning. NeurIPS 2022.

I’m not sure if I have misunderstood certain parts of the paper. However, based on my current assessment, I believe this paper is not suitable for publication at ICLR. I will adjust my score accordingly, depending on the authors’ clarifications and modifications during the rebuttal phase.

questions:
My questions that need clarification are included in the weaknesses section.


After Rebuttal: I realize that this paper may fall outside my area of expertise. Therefore, I have adjusted my evaluation to a rating of 6 with a confidence level of 1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730614669836

thread_type:
official_review

<thread_object>
creator: reviewer_pm2n

<content>
summary:
The authors study how three different categories of data augmentation (dubbed Strategies A, B, and C) affect the rate of improvement in model test accuracy with respect to training budget and training dataset size. The methods are grouped by how the data augmentation affects the sample *label* rather than the features: the first group uses standard one-hot labels, the second uses soft labels from a teacher model but recomputed on each augmented feature vector, and the final group computes a soft label only on the pre-augmented feature vector and uses this for all augmented samples. They conduct experiments to determine which augmentation strategy is optimal in different settings and provide general recommendations based on these results.

strengths:
Data and computational efficiency are highly relevant practical problems. As we reach fundamental upper limits on the possible size of training datasets, finding ways to improve the neural scaling laws that have been observed until now will be essential for continuing to improve model capabilities. Thus, the stated problem under study is relevant to the ICLR community.

weaknesses:
The main thrust of the paper is that, in the context of improving neural scaling laws, their "finding underscores the significance of the exploration the target component, a frequently overlooked aspect in the deep learning community." The discussion of neural scaling laws centers of the fact that exponentially larger datasets are needed to achieve only marginal performance improvements; in particular, these scaling laws are a problem only once we have reached the "extremely large dataset regime." On the other hand, the insights the authors provide showing the advantage of augmented targets all occur in low data regimes. As they return to the size of the full CIFAR-10 dataset, regular 1-hot labels have the best performance. Thus, it's unclear what relevance the insights in the paper have to the stated practical problem of interest, i.e., datasets at a scale far beyond that of CIFAR-10.

Related to this first point, Claim 3 on the efficacy of the different strategies is misleading. In particular, the best overall performance is in fact obtained by Strategy B with a 90% accurate teacher model (Table 2 in the appendix).

The choice to separate the model training into the "backbone" (feature extractor) and classifier is motivated by the claim that the cross-entropy loss cannot handle soft labels, but this is not true. The cross-entropy can be computed between any two discrete distributions with the same support (https://en.wikipedia.org/wiki/Cross-entropy). In fact, the KL divergence and cross-entropy differ by a quantity which is constant w.r.t. the trained model, so the gradients for the proposed training strategy for the backbone are the same as the standard CE loss. It should also be noted that previous data augmentation methods which use soft labels (such as MixUp) also apply CE with the soft labels directly.

The description of the results in Fig. 5a seems to be incorrect. This paragraph states that MixUp > standard augmentation > no augmentation, but the plot has standard augmentation with the highest performance with MixUp and no augmentation approximately equal for the relevant purple and blue lines.

Minor: A more descriptive name than "Strategy A/B/C" would make it easier for the reader to remember the salient features of the different augmentation methods.

questions:
Can the authors explain why the results/insights from the paper are relevant to neural scaling laws at the practically relevant scales discussed in the introduction?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714469646

thread_type:
official_review

<thread_object>
creator: reviewer_apkj

<content>
summary:
The main goal of the paper is to investigate the influence of different target encodings on learning efficiency in an exploratory way that generates some interesting new hypotheses.
The authors categorize three types of target encodings: hard labels (A), soft labels based on the un-augmented input (B) and soft labels from the augmented input (C).
To empirically compare the three approaches, they define a teacher-student training setup.
The teacher is trained using approach A and is used to generate soft labels for approaches B and C. Student networks are then trained using approaches A, B and C and they are evaluated with respect to top-1 accuracy on three image classification tasks. The network structure consists of a backbone and two heads, one for the soft labels and one for the hard labels respectively. For approaches B and C, the hard labels are only used to train the network head for the hard labels while the backbone and head for the soft labels are trained using the soft labels. As the authors are interested in disentangling the influence of the soft and hard labels on the representational capacity of the backbone, this training setup prevents the hard labels from influencing the structure of the backbone. In the remainder of the paper they evaluate the three methods in different experimental setups, where they use teachers with different quality, select different numbers of observations per class and also vary the data augmentation schemes for the teacher and the student. From these experimental results they are able to derive six findings that relate the varied experimental factors to the accuracy of the student model. For example, soft targets can speed up early training and student networks – when training using approach B and C – are limited by the capacity of the teacher.

strengths:
They pose an interesting question how different target encodings influence training efficiency of neural networks
Interesting experiments are designed that investigate questions such as how the quality of labels affects the accuracy of the student during different stages of the training, whether better teacher performance always entails better student performance or the interplay of data augmentation for the student and the teacher.
All experiments are repeated at least five times.
The paper is well structured and easy to read

weaknesses:
The experiments fail to consider other possibly relevant factors. For example, it is possible that strategy A in the results from figure 3a) simply needs a different learning rate
While the experiments are repeated at least five times, no uncertainty quantification (such as standard error) is included in the plots or the analysis.
Research data:
No code is provided
Experiment results are not included, e.g. as csv files
While interesting experiments are designed and phenomena are observed, little explanation is offered as to why these patterns are being observed.
For example, in the context of Figure 7 it would be interesting to discuss the effects the different augmentation strategies have on the class probabilities, which might explain why some augmentation methods perform worse with strategy C than with strategy B. 
When an augmentation method actually changes the class probability of an image (such as random cropping), using strategy C to train the network does not reduce noise but instead feeds the network wrong soft labels.

questions:
Figure 1: How are the results obtained? For example, how were the number of epochs determined?
Section 3.2: I do not fully understand the motivation behind the design of a unified loss function. It is said that CE is unable to exploit information from soft targets, but CE can be used with soft targets, so it’s not clear to me what is meant with that statement. Also, why is it not a valid evaluation strategy to evaluate strategies B and C by training the network exclusively using the soft labels with cross-entropy?
Later it is said that KL divergence is used to leverage the information in the soft targets, but this seems to contradict what was said above, i.e. that CE cannot make use of soft labels, but minimizing CE and KL divergence is equivalent.
Figure 5:
Which strategy was used to train the student?
On line 405 it says: “MixUp-trained teacher models achieve superior performance compared to standard augmentation”, but this is not supported by Figure 5a?
Why are the results of the experiments from the appendix not summarized in the main paper, i.e. whether they support the six key findings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731121404591

thread_type:
official_review

<thread_object>
creator: reviewer_wgq7

<content>
summary:
This paper investigates the often-overlooked role of targets in data-efficient learning, particularly in the context of neural scaling laws. Neural scaling laws indicate that achieving lower test errors typically requires exponentially more data and computational resources, leading to inefficiencies and sustainability challenges.

The authors observe that using soft targets on a smaller subset of data can outperform using one-hot targets on the full dataset under the same training budget. Motivated by this, they explore the impact of different sample-to-target mapping strategies on training efficiency. They categorize these strategies into three types:

STRATEGY A: Multiple augmented samples within the same class are mapped to a single one-hot target (conventional supervised learning).
STRATEGY B: Each augmented sample is mapped to a unique soft target generated by a teacher model (knowledge distillation).
STRATEGY C: Multiple augmented views of a single sample are mapped to the same soft target (proposed method to reduce noise in soft targets).

strengths:
Comprehensive Analysis: The paper provides a thorough investigation of how different sample-to-target mappings and data augmentation strategies affect training efficiency, offering valuable insights.

Novel Perspective on Targets: By highlighting the often-neglected role of targets in dataset design, the paper contributes to a more holistic understanding of data-efficient learning.

Unified Loss Framework: The introduction of a unified loss function that separates the backbone training from the classifier training allows for a clearer evaluation of the representational capacity influenced by different strategies.

Practical Implications: The findings offer actionable guidance on selecting target types, teacher models, and augmentation strategies to enhance training efficiency, which can be beneficial for practitioners.

Extensive Experiments: The use of multiple datasets and varied experimental settings strengthens the validity of the conclusions drawn.

weaknesses:
Theoretical Analysis: It would be ideal to provide a theoretical framework or intuition to explain the empirical observations, especially concerning why weaker teacher models can aid early learning and why STRATEGY C effectively reduces noise.

This addition would be a nice enhancement rather than any requirement, but I am not allowed to leave this section blank.🥸

questions:
Applicability to Larger Datasets: Have you considered applying STRATEGY C to larger-scale datasets

============ Revise according to the Associate Program Chair's comments ================ $\searrow$

"Have you explored applying STRATEGY C to larger datasets like ImageNet? What computational or methodological challenges do you anticipate in scaling up this approach?"

Teacher Model Selection: How does the choice of teacher model architecture impact the student model's performance under STRATEGY B and STRATEGY C? $\searrow$

"Have you considered comparing the impact of different teacher model architectures (e.g., ResNet vs. Vision Transformer) on student performance under STRATEGY B and STRATEGY C?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732526546313

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all reviewers for their constructive feedback and thoughtful evaluations. We are encouraged by their recognition of various strengths in our work and have taken their insights into account to enhance the **clarity and impact** of our paper. Below, we summarize the key points acknowledged by the reviewers:

1. **Novel Perspective on Neural Scaling Laws and Training Efficiency**:
    - We highlight the often-overlooked role of targets in dataset design and how **redefining sample-to-target mappings can improve data efficiency**. This introduces a **meaningful and new training framework** that, to the best of our knowledge, has not been proposed before. This perspective on targets has been acknowledged as **novel and insightful** by Reviewer wGQ7 and aligns with the relevance of neural scaling laws noted by Reviewers pm2n and trqj.
2. **Comprehensive Evaluation of Target-Mapping Strategies**:
    - We systematically compare three strategies (A, B, C), revealing:
        - Strategy A exhibits **long-term advantages** but struggles in early training.
        - Strategy B **accelerates early convergence** but is constrained by the teacher model's capacity.
        - Strategy C **effectively addresses the short-term limitations** of Strategy A and the **long-term deficiencies** of Strategy B.
    - These findings are validated on **diverse datasets** (e.g., CIFAR-10/100, ImageNet) and **different architectures** (ResNet50, ViT), as noted by Reviewer wGQ7, who appreciated the robustness of our experiments.
3.  **Broader Applications And Strong Practical Implications**:
    - The proposed strategies, particularly Strategy C, offer a **viable approach to mitigating inefficiencies** in traditional training paradigms.
    - To the best of our knowledge, systematic studies focusing on the benefits of weaker teacher models are scarce in knowledge distillation, and we are the first to reveal that **weaker teacher models can significantly enhance student model performance,** an insight validated by Reviewer apkJ’s emphasis on the impact of teacher quality.
    - Our findings provide ***ACTIONABLE INSIGHTS*** for deep learning community on how to select ***TARGET TYPES, TEACHER MODELS***, and ***AUGMENTATION STRATEGIES*** to enhance training efficiency, as highlighted by Reviewers wGQ7 and apkJ.
4. **Future Impact and Directions**:
    - By introducing a novel focus on target-mapping strategies, our work opens up a **promising** direction for further research, including **theoretical studies** and **applications to other domains**.
    - We believe this work is **a step** toward **understanding how target mappings influence training dynamics** and addressing the inefficiencies highlighted by neural scaling laws, a concern emphasized by Reviewers pm2n and trqj.

To address the reviewers' concerns, we have made significant revisions, including:

- **Scaling to Larger Datasets and Architectures**: We extended our experiments to larger datasets, such as **ImageNet**, and employed additional backbones **(ResNet50 and ViT)**. These results are included in Figure 15 and Table 6, 7.

While theoretical analyses are valuable, we emphasize that our primary objective is to ***SYSTEMATICALLY*** analyse the ***CRITICAL*** and often ***OVERLOOKED*** role of ***TARGETS AND THEIR MAPPINGS*** in a dataset, thus propose and validate a ***MEANINGFUL*** and ***NOVEL*** training paradigm that benefits **efficient learning**, with theoretical exploration left as a **promising direction for future work**.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper studies the effectiveness of training with soft labels. Authors position their paper in the context of neural scaling laws, where diminishing returns of various kinds have been established. The authors conducted an interesting set of experiments and generated valuable insights from those. However, the experiments, including the additional ones with other backbones (ResNet50 and ViT), do not show that strategy C, the one advocated by the authors, exhibits the best of both worlds as Strategy A (the conventional strategy) remains superior when more training budget is available. Hence, while the observation that soft labels can boost the training speed initially, one could argue that they negatively impact the final performance of the model. The authors do not propose an explanation or a more theoretical justification of the assumed benefits of Strategy C. In addition, questions remain how Strategy C is actually implemented (and the impact of the actual implementation): for instance, depending on the augmentation family (eg, MixUp), it is not clear how a single soft label would be produced for all augmented variants of a data point. Overall, the paper is well written and the direction interesting, but the paper is lacking evidence to support the main claims.