reviews: 
Comments: This paper endeavors to comprehend the underlying characteristics of noise within pre-training datasets and seeks to mitigate its influence on downstream tasks. The study reveals that the noise present in pre-training datasets exerts distinct effects on in-domain (ID) and out-of-domain (OOD) tasks. In the case of ID tasks, slight noise during pre-training can yield improvements in in-domain transfer performance. However, for OOD tasks, noise consistently degrades out-of-domain performance. To substantiate their findings, the authors employ Singular Value Entropy (SVE) and Largest Singular Value Ratio (LSVR) to capture the behavior of the trained features in both ID and OOD tasks. Subsequently, the authors devise a loss function that enhances the SVE and LSVR of these features, resulting in superior overall performance.
Strengths: - The impact of noise within pre-training datasets on subsequent tasks has not been thoroughly investigated in the existing literature. The insights presented in this paper, such as the distinct effects of noise on in-domain (ID) and out-of-domain (OOD) tasks, are novel and intriguing.

- The design of the loss function is a direct consequence of the insights gained from observations, and experiments demonstrate that this designed loss outperforms the Cross-Entropy (CE) baseline, including LP and MLP structures.

- Experiments encompass a wide range of tasks, including both image and image-language tasks. Furthermore, various popular base model structures are used in this paper.
Weaknesses: -  I find Figure 3 challenging to interpret. It consists of numerous data points for each configuration, lacking connecting lines. Consequently, I struggle to draw the conclusions reached by the authors based on this figure.

-  I've observed a potential contradiction between SVE and LSVR. For instance, in the case of two-dimensional features, [1.0, 0.0] exhibits the highest LSVR while having the lowest entropy. It would be beneficial if the authors could provide further clarification regarding this inconsistency.

-  I would appreciate it if the authors could offer more detailed explanations as to why a slight amount of noise can benefit in-domain (ID) tasks. This is somewhat contradictory to the existing literature on learning with noisy labels, and additional insights would be valuable.

-  In the paper, the authors claim that the proposed method enhances SVE and LSVR. However, Figure 5 (d) indicates that the proposed method does not yield superior LSVR compared to the LP and MLP models. Furthermore, it seems that as the noise ratio increases, LSVR does not drops significantly for all the settings.

- It's worth noting that some related work, such as [R1], also employs Singular Value Decomposition (SVD) to address noisy label problems. It would be beneficial for the authors to discuss the distinctions between your approach and previous work.

- I would like to point out that the improvements achieved by the proposed method appear to be relatively modest. According to the experimental results, the proposed method only demonstrates an approximately 1% improvement compared to the MLP model.

- For ResNet-50, it might be worthwhile to explore the feasibility of fine-tuning all layers rather than constraining the encoder. It would be valuable if the authors could conduct experiments to determine if the proposed loss function is effective in cases where all layers are fine-tuned.

[R1]: FINE Samples for Learning with Noisy Labels
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good