Socratic Reasoning Improves Positive Text Rewriting
Authors: Anmol Goel, Nico Daheim, Christian Montag, Iryna Gurevych


Reviewer paper summaries: The paper proposes an LLM-prompting based cognitive reframing method based on asking socratic questions, called SocraticReframe. Additionally, the authors propose a dataset for cognitive reframing rewriting annotated with the generated socratic questions. Automated and human expert evaluations show that the socratic questions as rationales can boost positive rewriting performance.
Summary of strengths: The paper is well motivated and also well organized.

The automated evaluation includes two aspects key to understanding text rewriting performance, content preservation and transfer strength.

The authors conduct human evaluation with domain experts.

The authors provide a dataset that may be useful to research in mental health and counseling.
Summary of weaknesses: Are there any human evaluation of the rewriting quality? As in, were the human annotators asked to evaluate how useful and faithful the rewritten statement is, with respect to the original negative framing? I understand that the authors included content preservation in the automated evaluation, but since the task is text rewriting, it is important to have a clear understanding if the rewritten text can be considered a rewriting of the original text. If we don't care about that aspect, that we might as well generate from scratch. 

I ask this question because from Figure 2's example, I don't really see how "We will not be able to complete the work that needs to be done" is meaningfully related to (or rewritten to)"I can find a solution to this". They feel marginally related and seem like two wholly unrelated responses to a given situation. What is the content preservation performance in this case in terms of BLEU, BLEURT? If there are situation descriptions (such as in COGREF), then how does from-scratch rewriting of positive frames perform? To reiterate my point, I want the authors to make a stronger case why the rewriting (from negative to positive) is useful, as opposed to just generating positive framing. That way I believe the paper will be stronger. Also it would be great if there can be a human evaluation of the rewriting aspect, because I don't think the proposed automated content preservation metrics adequately capture that aspect.

The methodology is LLM prompting and doesn't provide any interesting scientific or engineering contributions.
Comments, suggestions and typos: For datasets where human annotation of reframes exists, can you compare the model performance against them?
Overall assessment scores: 3