Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: SPML uses a single positive label for each image, which is used to train the image classifier. At the same time, this method approaches the performance of traditional multilabel classifiers, its unlikely to give us behavior similar to that of our annotators. 
strengths_and_weaknesses: 
This work operationalizes certain notions of bias in the framework of SPML. The work appears to give evidence for P_uniform being a surrogate measure of bias. 

Why does it matter that we emulate the behavior of that of our annotators? What happens if they are biased or the 'annotators' happen to be using biased models, i.e., ChatGPT, BERT models, etc.? 

suggested_changes: Explain a bit more about why we wish to emulate behavior of our annotators perhaps. 

I wonder if using multiple terms and which combination of multiple terms would help increase performance.
review_title: null

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: Introduce bias on single positive multi-label annotation without reproducible code
strengths_and_weaknesses: Strengths
1. paper introduces bias on SPML(single positive multi-label )annotation
2. table compares the performance on different label bias(uniform, size, location & semantic) & different loss

Weakness
1. the code is not provided where we are unable to determine if results are reproducible or the validity of the paper
2. Interpretation of the different loss subscripts mentioned in the table is missing 
suggested_changes: 1. Adding code helps the reviewers to determine the validity and verify the research is reproducible.
2. Below the table provide a expansion/meaning of the different losses with the subscripts : AN−LS , EM, AN, ROLE
review_title: null

Meta-review: metareview: The paper explores the effect of label bias in single positive multi-label (SPML) learning. The authors argue that the standard benchmarks for SPML, which choose one positive label per image uniformly at random, may not be representative of real-world annotation settings. The paper introduces bias on SPML annotation and compares the performance of different label biases and loss functions.
The tackled problem of this paper seems to be noted by one reviewer as high potential.
Yet the other reviewer identified the lack of reproducible code and clear explanations for some terms as weaknesses.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to archive