Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors propose a framework to detect epistemological biases and stereotypes in text and use this to assist with detecting injustice in text. They fine-tune a BERT based model for bias detection and Co-STAR and Social Bias Frames to detect stereotypes and stereotype concepts. They also propose an interactive interface that can be used by editors and journalists to learn about potential bias in the news media text they are publishing.


strength_and_weaknesses: Strengths:
1. Authors tackle an important and interesting problem
2. They propose an approach with a real-world application in mind and propose a framework that can be used in production to detect bias in text

Weakness:

1. I fail to see why the methodology has not been proposed as a generic approach to detect epistemological bias but has been restricted to news media.
2. The problem that the authors are trying to capture is slightly confusing. Are they trying to detect epistemological biases? Are they trying to detect injustice in text? How are the two different?
3. Authors mention detecting TMI but I couldn't find the process that they are using for the same
4. The comparative test seems to have a scope of only 20 headlines which doesn't seem enough
clarity,_quality,_novelty_and_reproducibility: While the end to end proposal of framework is interesting, the technical contributions can be strengthened. I also felt that the details about using Co-STAR models and SBF models could be more elaborative making it easier to reproduce the results. 
summary_of_the_review: Please see the sections above
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a pipeline that detects epistemologically biased words (e.g., ripped - negative) and stereotypes (e.g., women should be dressed like brides) in texts. The authors develop an interactive user-interface to show the detected output with a goal of helping journalists and editors. The backend of this interface (i.e., the detection part) consists of three previous models. The paper shows a study about two subjects, Meghan Markle and Kate Middleton, demonstrating the output from the system about them. 
strength_and_weaknesses: Strengths:
- I find the topic is interesting.
- The paper throws big questions (in Sec 2), e.g., what kinds of implications does our use of these words impact society?
- It shows how the pipeline is from the input text to the output. 

Weaknesses:
- The paper doesn't describe the motivation in a convincing way.
- The technical details are missing. It is difficult to know how the actual detection part works unless we read the previous papers. 
- There is no discussion about the accuracy of those models.
- It seems that the propose pipeline uses existing models (with a small modification in the first model, which I was not really convinced of), and the interactive interface is a contribution. However, there is no screenshot of or an emphasis on the interface. The paper just describes what it does, and that's all.

Detailed comments are below. Major and minor comments are mixed.
- use citations in a right format for readability. In addition, there are many incomplete sentences. 
- "give journalist and editors a tool which will help them learn" <= They are experts, and they sometimes/often use the biased words intentionally to present their "opinions" and influence the public. In this case, how could your tool be helpful?
- The paper seems to assume that the existing models give correct output, and there is no discussion about the accuracy of those models.
- "it is difficult for common people to identify a biased word" <= Then, how was the training data for the existing models created? Was there any training involved?
- "difficultly" => difficulty
- expert linguistic and epistemological bias features in the tagger model<= Were these features manually extracted/annotated by experts? what kinds of features are they? 
- When you extend the model by including TMI, is this also manual feature extraction?
- I'm confused with the TMI features because they look like they are already indicating biases, i.e., the output. So, the input are bias features, and the output are the probabilities indicating biases? It feels like the features are part of the intended output.
- Why did you make a debiasing weight 1.3 as opposed to previous work? Any particular reason?
- In 3.2, "We rank these outputs based on their semantic similarity to the headlines." <= what is the semantic similarity between streotypes and new headlines? I'm not sure if the term "semantic similarity" is correct here. "The stereotype and stereotype concept most correlated with the sentence" <= how do you compute this? 
- Does the tagger model take into account contexts where a word is used? How is the output of this model different from using the epistemological lexicon directly to find biased words? It seems that the words that are not in this lexicon are not included for the study (the output) anyway. 
- interactive interface seems like the biggest contribution of this paper, but no screenshot or any study regarding that?
- Table 1 uses a lot of space, but I'm not sure what the point is here showing all those. This space should be used for describing more technical details. 
- need a brief description of "Meghan Markle and Kate Middleton". not all the readers know them. 
- data: Is the data of Mechan Markle and Kate Middleton from the bias data corpus (Pryzant et al., 2020b)? How many headlines? <= Section 6 says it was only 20. How did you get this data?
- "if the measures and metrics suggested here become targets, they will cease to be useful." <= what does this mean?
- I think it would be more helpful to design/interpret the current comparative study to show interesting/meaningful findings (e.g., in social sciences). 
clarity,_quality,_novelty_and_reproducibility: - The work used previous existing models. But there are no details to provide justification about why the models are used (i.e., whether using the models are good choices) and how accurate they are.

summary_of_the_review: Although the topic of the paper is very interesting and important, I think the paper needs more work - little technical contribution and no deeper discussion/study about the efficacy of the proposed interactive interface. In addition, no technical details are provided to let readers understand how those biased words are detected.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work used news media dataset to detect character, testimonial, and framing injustices. The models  employed are fine-tuned BERT model, CO-STAR model and Social-Bias Frames model. The framework is able to automatically detect epistemological bias using a fine-tuned tagger and a lexicon lookup. The result demonstrated 10 article headlines of Meghan Markle and 10 articles of Kate Middleton through the framework. In essence, journalists can use the system to detect potential bias types and injustices that might be present in their headlines before publishing. 
strength_and_weaknesses: Strengths: 
1. The paper was well written; the flow and arrangements of the sections are excellent 
2. The approaches discussed and used are current and detailed to some extent 
3. The presentation of the overall result is impressive and relevant 

Weaknesses: 
1. The framework developed was not shown - a schematic diagram or an algorithm would have sufficed. 
2. The approach of the development of the tagger-UI was not presented - what software development methods were employed?
3. The tagger-UI was not presented in the work - a screenshot would have sufficed. 
4. No mathematical model was demonstrated to prove the theory behind the entire process. 
5. How the sentiments (pos, neg) were derived was not shown clearly 
clarity,_quality,_novelty_and_reproducibility: The paper was very clear and it shows a reasonable novelty and quality. 
summary_of_the_review: This is a good paper and if the weaknesses are addressed, it will show good reproducibility and would benefit the community immensely in other domains away from the news media.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a framework for epistemological bias detection, by combining different (existing) tools and features from previous work. By doing so, they claim to detect three types of bias: character, testimonial, and framing injustices. This bias is associated to single words, while exploiting context on the sentence level. Once a word is tagged it is linked to an entry in an epistemological lexicons from social sciences. This helps to explain the decision of the system to a user and to associate it to potential stereotypes. The authors see journalists and editors as the main target group of their tool, since it can be used to reduce unwanted bias in their articles.
The model is qualitatively evaluated on a data set with 20 headlines to reveal sentiment and bias type.
strength_and_weaknesses: + The text analysis tasks proposed in this paper go beyond most established tasks with respect to the linguistic complexity required to solve the task. 
+ The application area is very timely and tackles a pressing societal issue.
+ The authors are aware of the limitations of the work and make them explicit.
- The paper is hard to access from a computer science oriented perspective, since a lot concepts from social sciences are used without definition or intuitive examples.
- The technical contribution is restricted to the conceptual framework. No novel algorithmic contributions are provided, nor quantitative empirical insights.
- While the task of bias detection is being addressed in a more fundamental way than before, the paper misses a larger vision, like using the tool to analyze bias quantitatively on large scale media data.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written with some minor issues. However, a lot of terminology is used that might not be known to the ICLR community. Some more gentle introductions or intuitive examples would have been helpful.
Quality: The qualitative research is good. However, the quantitative part is not sufficient. The paper presents convincing anecdotal evidence, but how this approach works on large sets of non-handpicked headlines is unclear.
Novelty: The concepts presented and computational framework proposed are novel, but not from a technical contribution and not with respect to empirical insights.
Reproducibility: Since most components of the framework are openly accessible, the work is mostly reproducible.
summary_of_the_review: I believe the paper's contribution is to push NLP more towards fundamental concepts from social and communication science. However, the lack of algorithmic novelty or statistical empirical evidence on the performance of the model makes it not yet ready for acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposes a combined model to detect epistemological biases and stereotypes in text. Reviewers agreed this work tackles an important and interesting question, with explicit limitation discussion and an interactive interface. On the other side, reviewers questioned the technical contribution given much detail is missing, and found the quantitative part weak. There are also concerns about motivation and how this tool could be used more broadly and quantitatively on social media. After reviewing the strengths, weaknesses, and authors' responses, I recommend not accepting the current version and encouraging the authors to improve it in accordance with the reviewers' recommendations.
Decision: Reject