Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper presents a study of how ethics considerations related to crowdsourcing are disclosed in the AI community compared to psychology. The study includes ICLR, NeurIPS and Springer journals, which is a reasonable representative sample of AI publications. The primary findings are that AI lags behind psychology in ethics disclosures, but has improved greatly since 2018 particularly at NeurIPS. The paper is unique, important and very clearly written.
strength_and_weaknesses: Strengths

The paper tackles a very timely topic with a thorough, well-designed study of the AI community in comparison to psychology. Choosing psychology as a baseline is well founded, as it performs much HSR and has an established practice of rigorous disclosure requirements in its publications.

The paper reveals important and clear differences between ICLR and NeurIPS, largely stemming from NeurIPS adoption of an explicit checklist of ethics considerations for authors and reviewers. While ICLR has a similar ethics policy as NeurIPS, the introduction of explicit mechanisms to state and review compliance appears to make a huge difference.

Section 6.3 makes an excellent point about the lack of consideration and disclosure of demographic data in AI. This gap has become a source of mistrust of the AI community by the general public in high-profile cases regarding bias in facial recognition systems, for example.

Weaknesses

The paper has few weaknesses, and these are minor points.

In 4.1, it would be clearer to state that ICLR does not have an explicit ethics criterion as part of its review process. The reality is that many or most authors and reviewers do not read conference policy documents, nor are they necessarily aware when these are changed. However authors and reviewers will pay much attention to requirements that are encoded into the submission and review processes, such as a checklist that must be filled out, and associated text.

There is no general discussion of ethics as a review criterion for conference papers, but this is a major recent development that should lead to greater enforcement of ethics policy. It would strengthen the paper to include a table of major AI conferences vs. ethics policies and how they are enforced, even for those that are not included in the detailed study such as CVPR, ICML, AAAI.

The finding that conference papers “overwhelmingly utilised the MTurk platform compared with others”, sec. 5.1, could be biased by the nature of the paper selection criteria, which searched for MTurk explicitly. What about papers that leveraged labeling service companies, which are increasingly abundant and often comparable in price to MT?

It would be helpful to add a summary table in sec. 5.4 with comparative numbers between the two conferences and the journal so that the reader does not have to skip back and forth between tables on different pages in order to see their differences.

A major difference between AI and psychology research is the much higher proportion requiring IRB review in the latter, as shown in Table 5. This difference could explain virtually all of the other differences, as IRB review forces researchers to consider many criteria such as potential harm to participants that may not be considered thoroughly otherwise. Data labeling tasks are usually not HSR in the US, and many researchers (correctly) do not seek IRB review for them. Considering only behavioral tasks, table 7, is an insightful way to break down the data as such tasks are more likely to require IRB review. However, I don’t think the data supports the statement that “AI behaviour studies still do not meet the same standards as Psychology experiments” for NeurIPS. IRB review was reported for 45% vs. 73% in psychology, but this could easily reflect a true difference in the nature of the behavioral science being conducted. For ICLR and AI in general the statement is more supportable.

clarity,_quality,_novelty_and_reproducibility: The paper is very clear, easy to read, and compelling. The arguments and studies are laid out in a narrative structure that is easy to follow.

While there is no algorithm proposed, the paper is very novel and unique. Papers about our research processes, rather than the research itself, are rare but often the most important.

summary_of_the_review: This paper is important and would likely generate much discussion and debate at the conference. The study could be deeper by considering more conferences, but in its current form it is sufficiently justified and complete to warrant publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper looks at papers in AI submitted at ICLR and NeurIPS for adherence to ethical declaration standards when elements of work has been crowdsourced.This has been compared to the standards met by papers in Psychology and Springer journals.
The evidence clearly demonstrates a significant gap in disclosures across all standards including ethics reviews, payment data, demographic data.

strength_and_weaknesses: Strengths:
- Addresses an important aspect of appropriateness of disclosure and details provided.
- Clearly demonstrates a gap.

Weaknesses:
- Although there is a requirement to disclose and they found missing reporting, it is unclear if this is significant. 
- As stated in the paper, while the requirements for disclosure are present now but these requirements seem have been made in 2021-22 for ICLR and NeurIPS.But the data is from 2018-2021. Clearly, voluntary disclosure standards should or could have  be higher than what had been required previously. Hence, not sure if this study is too early. 
clarity,_quality,_novelty_and_reproducibility: Clarity: Clearly written.
Quality: High
Novelty: Good work addressing an important aspect of ethical disclosure,although in related work there is reference to similar work on NLP papers(Shmueli et al.) 
Originality: Good work again comparing with Psychology and other journals.
summary_of_the_review: Although the work is good and important, it is unclear to me if this is ahead of time, if the requirements for disclosure for ICLR and NeurIPS were changed in 2021-22.It is hard to interpret this data for ambiguity when there was no requirement,although authors could have volunteered it.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigated the ethics disclosure in the AI papers that use crowdsourcing as a data collection method published in ICLR, NeurIPS, and Springer journals in the past four years. The main finding of the investigation is that compared with the Psychology paper, ethics disclosures are far less common. Also, the paper has some other interesting findings, such as the journals and conferences have the power to influence engagement with research ethics.
strength_and_weaknesses: Strength:
- The topic of the paper is interesting. As crowdsourcing platforms have been widely used in AI research, the researchers in this field have not paid enough attention to the ethics in the usage of crowdsourcing. This paper will inspire the thinking of this issue.
- The authors chose the Psychology papers as a baseline, which is sound and can disclose the difference between psychology research that involves ethics as a tradition and AI research.
- The venues where the AI papers were published were chosen soundly. The findings of this research are convincing and interesting.

Weaknesses:
- The main weakness of the paper is that some findings lack deep insights and thorough discussion. For example, in 2020, the disclosure of the IRB reviews started to appear at AI conferences. Why did the research community begin to pay attention to the IRB reviews? What benefits can the IRB disclosure bring to the crowd workers and the research itself? Another example is in Table 6. For behavior research, the ethical disclosure of AI research is far less than that of psychology but for the other research type, the former is higher than the latter.  For what reason, in behavior research, the ethical disclosure of AI research is rather low?
- The venues of papers can be extended. As far as we know, there are also many AI papers that utilize crowdsourcing published in AAAI and IJCAI. Also, the papers that utilize crowdsourcing in a more complicated way can be found in WWW, CHI, and HCOMP. Why did the authors not include the paper in these venues?


clarity,_quality,_novelty_and_reproducibility: The paper is clearly presented and easy to follow. The paper has somewhat novelty but lacks more deep insights. The paper should be easy to reproduce.
summary_of_the_review: This paper investigated the ethics disclosure in the AI papers that use crowdsourcing as a data collection method published in ICLR, NeurIPS, and Springer journals in the past four years. The topic of the paper is interesting. As crowdsourcing platforms have been widely used in AI research, the researchers in this field have not paid enough attention to the ethics in the usage of crowdsourcing. This paper will inspire the thinking of this issue. The paper is clearly presented and easy to follow. However, the paper lacks more deep insights. Some critical issues were not deeply touched and analyzed. Thus, it is below the acceptance bar of this venue.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper reviews the information reported around human subjects experiments in articles focused on artificial intelligence published at ICLR, at NeurIPS, and in Springer Nature. The authors identity papers that use crowdsourced labor (specifically papers that meet three criteria that they define), and on this set of works they analyze whether IRB is mentioned, payment terms are outlined, workers' consent and demographics are disclosed. They then analyze how reporting varies between "evaluation" and "behavioral" tasks, and compare it with trends in psychology. They also analyze how trends were affected by policy changes at ICLR and NeurIPS, e.g., the introduction of the checklist. The authors conclude that "AI research does not meet the standard of psychology research for research ethics disclosures" and advise that demographic data, when collected, should be disclosed. 

strength_and_weaknesses: The topic is certainly interesting and I agree with the authors that this is something that should be clearly discussed (more) in the machine learning community. However, I have a few concerns. First, I am not totally sure that the topic is appropriate for ICLR (cf the call for papers). While the topic is of interest to the community, this work seems to be more appropriate for CSCW or HCOMP, venues where researchers have deeply thought about this issue. Second, there are several details that are not clear from the paper, described below:
* How is "AI research" defined? There are so many venues (HCOMP/CSCW/CHI etc.) where AI research is done and authors in my experience tend to be careful about IRB reporting. Even other conferences that are noteworthy (CVPR/ACL/FAccT/ICML/AISTATS etc.) are not taken into account here. The authors seem to focus on ICLR and Neurips because of their use of OpenReview, but this is a huge limitation and narrow view of the field. 
* Related to my previous concern, there are subfields within AI/ML where authors often report the demographics of study participants. This is the case for almost all of the papers with Human-AI experiments, which likely fall in the "behavioral" tasks (and a clear definition of what is "behavior" of study participants should be given), but only some of them are published at the venues considered. The simple partition defined by the authors does not recognize this. 
* There is a very thin and blurry line between what does and does not constitute human subjects research. For example, see the thorough discussion of https://arxiv.org/pdf/2206.04039.pdf. This is not discussed in the paper but is certainly relevant. 
* Sample size in table 1 (and in many other) is very small. Although I think the few trends are large enough that this issue does not undermine the message of the paper, it should be mentioned in the limitations of the analysis. 
* For certain applications, demographics of the participants are important. For some others, they are simply not. For example, in a task where subjects are asked to draw bounding boxes around objects, knowing the demographics of the workers is not fundamental. The authors do not seem to acknowledge this. In addition, (Lum and Isaac, 2016) and (Raghavan et al, 2019) do not mention at all crowdsourcing: They deal with biases in data collection that are totally unrelated to crowdsourcing.
* In table 8, the analysis should be done conditioning on the venue and on the type of task. The observed patterns could be explained by these factors.  
clarity,_quality,_novelty_and_reproducibility: The writing is clear and the analysis is likely novel, but the insights it offers are limited. There are enough details about the methods that the experiments are likely reproducible. In my opinion, the central message of the paper ("AI research does not meet the standard of psychology research for research ethics disclosures") is not well supported by the analysis. 
summary_of_the_review: In my review, I have raised several concerns and pointed to the lack of clarity around some details. The paper is not ready for publication at the current stage. The authors may address some of my concerns in the rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a comparative analysis of ethics disclosure practices and norms applied in AI and Psychology research dissemination venues such as ICLR, NeurIPS, Springer Journals, and CHI around human subject experiments carried out by crowdsourcing workers on MTurk by analyzing disclosed information on published/submitted papers selected based on specific criteria. The paper concludes that ICLR and NeurIPS does not meet the ethics disclosure standards of Springer Journals on Psychology and CHI, which have applied ethics policy on human subject research for a long time. However, the findings show that NeurIPS is doing better than ICLR due to its earlier adoption of a policy related to ethics on human subjects in 2021. 

Although the study is important, its findings are similar to that of (Santy et al., 2021) and (Shmueli et al., 2021). The sample set used in the paper to represent AI venues might not be a representative set to make generalizations as the authors used data from ICLR and NeurIPS, which recently adopted Ethics Guidelines or Code of Ethics. This study might be a bit early to be able to make generalizations given in the paper. It would be interesting to see an analysis of the rate of adoption of "Ethics Guidelines" or "Code of Ethics" in ICLR and NeurIPS and how it compares to other disciplines in the next five years. 
justification_for_why_not_higher_score: Although the study is important, its findings are similar to that of (Santy et al., 2021) and (Shmueli et al., 2021). The sample set used in the paper to represent AI venues might not be a representative set to make generalizations as the authors used data from ICLR and NeurIPS, which recently adopted Ethics Guidelines or Code of Ethics. This study might be a bit early to be able to make generalizations given in the paper.
Decision: Reject