Venue: NeurIPS 2022 Workshop TSRML

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary: This paper studies the '*data externalities phenomenon*' --- adding more data from certain input sources will reduce performance on key evaluation groups. The authors conduct a set of interesting experiments for understanding the data externalities phenomenon. 
strengths: 1. The findings on *data externalities phenomenon* is very interesting, where this paper shows that this phenomenon could occur in both toy and real-world datasets. Meanwhile, the experiments are well-designed and provide valuable insights on understanding this phenomenon (e.g., complicated effect on magnitude of data externalities). 
weaknesses: N/A
overall_recommendation: I recommend acceptance, because this paper provides interesting results on the data externalities phenomenon.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary: This paper studied a very interesting problem, data externalities, that adding more data from some input sources reduces performance on key evaluation groups. The authors showed that certain 'data inefficiency' implied space for model improvement, and focused on understanding how and when data externalities manifest across learning settings and training procedures. Extensive real data analysis validate their findings.
strengths: - Overall, the paper is well written and the motivation is clearly clarified.

- The studied problem is interesting and important for accurate and trustworthy machine learning.

- Extensive experiments are provided and results are impressive, which show the usefulness of the framework. 

weaknesses: - The paper is not easy to follow with too many terminologies which may be benefit from detailed explanations and additional references. 

- The authors may consider a figure to illustrate the whole framework of data externalities on group performance.
overall_recommendation: This paper delivered an interesting idea and useful framework with extensive experiments, but terminologies need more detailed explanations.
review_title: Review

Meta-review: Following the unanimous recommendations from reviewers, the submission is accepted.
Decision: Accept