I'm sure you're a real scholar yourself: Exploring Ironic Content Generation by Large Language Models
Authors: Pier Felice Balestrucci, Silvia Casola, Soda Marem Lo, Valerio Basile, Alessandro Mazzei


Reviewer paper summaries: In this paper, the authors investigate the performance of irony generation using large generative language models (here Mistral 7B). Concretely, they generate ironic replies to a previous post. In their work, they also train models for specific demographic groups and evaluate whether human annotators prefer the model trained for their demographics. For the generated outputs, the authors provide manual and linguistic analysis and compare to the human replies.
Summary of strengths: 1. Clear conceptual definition of irony. Important to include this for the task.

2. Clever conceptualization. It's a good idea to account for age-specific recognition of irony detection.

3. Interesting results for the linguistic analysis, bridging computational and linguistic information/knowledge.
Summary of weaknesses: 1. The paper lacks a lot of clarifications and argumentation for the experimental setup (see detailed comments below). If this point is properly addressed, the scores can definitely be increased.

2. The paper only investigates a single generative model (Mistral 7B) and does not provide any argumentation for selecting this one in particular. This makes it hard to believe the results of this paper can be generalised to all/most generative models, especially because the selected model is rather small compared to some of the alternatives, so some of the issues may not be representative for larger models or models that have undergone more thorough instruction tuning (which helps the model stick to the prompt instructions).

3. Some of the methodological choices could be considered sub-optimal (see detailed comments below).
Comments, suggestions and typos: General Remarks/questions/comments:

1. One point that is not sufficiently discussed in the experimental setup, is what the impact is of using the perspectivist corpus. The perspectivist corpus contains multiple annotations for the same text that indicate whether it is ironic or not. For model fine-tuning, it was not entirely clear what was done with these disaggregated labels? Is only one selected (depending on the age of the annotator?).

2. Conceptually, It seems a bit strange to train a model specifically for generating non-ironic replies, if the goal is to investigate whether generative models can produce irony in a realistic manner. This should be discussed in the introduction/methodology of the paper. Ideally, the default replies of the model should not be ironic or should be less ironic than the specific irony model.

3. Recent related work, Human and System Perspectives on the Expression of Irony: An Analysis of Likelihood Labels and Rationales (Maladry et al., 2024) proposes the use of a likert scale for "irony likelihood". As it is used in this paper for the evaluation, the related work should probably be cited to support the use of a Likert scale, especially because this deviates from the annotation on the source dataset.
In addition, it is a bit strange to use this type of annotation (annotating the two generated outputs individually) if the goal is to directly compare the two. Is there any reason for not using best-worst ranking, and directly comparing which is perceived as more ironic compared to the other? If the research question is intended to be comparative, having comparative annotations would make more sense.

4. A lot of the annotation details, such as annotation time, whether they failed attention checks and payment could be moved to the appendix as they do not directly contribute to the argumentation or findings of the paper.

5. In these experiments, the authors fine-tuned "young" and "old" models based on the demographics. Although the data for finetuning is different for the two groups, the pre-training data is the same. Likely, this already makes sure that the foundation model already possesses the required knowledge to generate the ironic reply. For finetuning, it would seem like models are mostly learning what type of output they are meant to generate for a prompt. Whilst this helps them to learn how you are formulating or investigating a task, they likely already acquired a lot of knowledge about ironic texts and real-world knowledge.

Detailed comments: 

034: The authors give a solid introduction into the concept of irony, starting mostly from a detection perspective, but they do not directly connect this to the generation of irony. It would be a good addition to also include why generation, in addition to detection, can be an interesting task. Some recent work that could be an inspiration for providing a motivation is "When It Pays to Be Insincere: On the Benefits of Verbal Irony" by 
Valeria A. Pfeifer1 and Penny M.. 

Something else that may still be interesting to explain is in what sense irony detection is different from humor detection. 


049-050: "annotate how ironic the reply is according to their perception." → I believe this phrasing can be improved. A statement is either intended as ironic, which means that the literal interpretation should be inverted, or it is not. As a result, "how ironic" is not a possibility, but it would rather refer to "how likely" it is that something can be considered as ironic or it would refer to how confident the annotators are about their judgement.

146-150: "The annotators were asked to determine whether the reply was ironic in the context of the post; however, we noticed that, in some cases, the positive instances contain non-ironic humorous content.

→ Although this is an interesting finding, a simple mention of this has no meaningful contribution or place in the paper. This should either be elaborated upon with an example and then excluded from/ updated as label for that sample or not mentioned at all.

158: There are a lot of available open source models, is there any particular reason for choosing Mistral 7B over any other models? I believe at this parameter count LLama3 8B should perform better for most tasks, but it is possible it was not available yet when the experiments started. It would be interesting to include some argumentation for model selection.


163 - 164 "You are given in input (Input) a Post extracted from social media conversation" → What is going on with the capitalized Input between brackets and the capitalization of post?. The formatting seems a bit strange.


Line 203: "Thus, we trained four distinct models …" This statement does not seem to connect to the previous sentence very well. There seems to be some argumentation missing here. It should be made explicit why 4 models are trained for generation of ironic and non-irnic content.


206+: In this section, authors mention investigating a subset of 100 pairs, but it is not clear where these samples originate from, ideally, one assumes this is another held out development set that was not used for training or testing.

216: The authors mention investigating the model outputs, but it is not clear whether the annotators share the "target demographics" of the model they are evaluating. It should also be mentioned which of the 4 models is evaluated.

224: "The annotators used a spreadsheet with different colors for the annotation" → This does not seem like an entirely empirical method to do this manual evaluation. There are a lot of missing details: could the annotators see each other's annotations? If the annotators are the other authors of this paper, they are aware that the outputs were generated by a fine-tuned generative model, which would also influence their judgement.

227: "We collected 553 posts from the original dataset and manually removed those not in English or where no semantic content was found (for example,only containing mentions and hashtags)." → It is not clear where these posts come from? Is this the held-out test set from the 80-20 split?

233- 238: "To choose the two models, we opted for a trade-off between satisfactory and problematic replies, favoring the models that performed better. The models resulting in better performances were those where the input was masked for training, and the user mentions were removed." → It is not clear what is meant by "problematic replies" and how this trade-off was achieved or decided upon. 

At this point, the authors only mention 2 models, but earlier they report training 4 models for ironic and not-ironic content. It should be explained how the authors selected the right model from these 2x4.

271: the authors use named entities as a proxy for world knowledge, but this is lacking any motivation or argumentation. World knowledge can also mean very different things for different people. This can either be very generic knowledge that we expect everyone to have or very specific knowledge about particular events or people. Only for the latter NER would be relevant.

309: why did the authors choose a word2vec model for calculating the similarity between the replies? This is one of the potential approaches, but one of the downsides relies strongly on lexical overlaps. Were there any other, more recent alternatives that provide a more sentence-level semantic description of the sentences, such as embeddings for a sentence-transformer?

329: The authors propose the use of a likert scale for irony detection, but they do not explain where this decision originates from? This is the first mention in the paper where irony is not considered as a binary classification issue. The original dataset does not make use of any scalar labels for irony detection. 

365: it is interesting to see that in 197 of samples (almost  40% of cases), the ironic model was not perceived as more ironic than the non-ironic model. In line 371, the authors mention that the output of their ironic model is "significantly perceived" as ironic. However, these scores are not too impressive and indicate that there is a lot of room for improvement. 

In line 367, the authors mention using TextBlob for calculating the sentence polarity. Why was TextBlob chosen to provide this information? Sentiment polarity has an interesting relationship with irony detection, as superficially positive sentiments (literal expressions) should often be interpreted as negative (due to the ironic/sarcastic meaning inversion). A simple lexical dictionary would work best for calculating such superficial, literal sentiment. More advanced systems for sentiment analysis might already understand some irony/sarcasm and therefore correctly predict sarcastic comments as "negative". In this paper it is not clear what the authors are looking for in "sentiment", does this refer to positive sentiment words or the actual intended sentiment?

429: In the original dataset, the demographics were split into multiple age groups based on generation. Why do the authors for this paper combine this into 2 groups for young and old people? 

433: "We discarded instances for with no majority was reached"

439: Why are there only 455 samples here? At times, the different numbers for data sizes make the paper hard to follow. It seems like a lot of data is lost because there was no appropriate output. Why was it not possible to fill in this gap? Given that all generative models have some inherent randomness, would it not have been possible to run them again and see if a different output would meet the minimum criteria?

500-501: "Looking at the overall results, we noticed that, on average, older individuals preferred outputs generated by the O-model, while younger individuals displayed a similar preference" → this is too ambiguous, it is not clear whether it means the young people prefer the same model (the older model) or their own (young) model.

Sections for Limitations and Ethical Considerations should not be numbered (as far as I know).
Overall assessment scores: 3.5