The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation
Authors: Tyler Loakman
Aaron Maladry
Chenghua Lin
Venue: EMNLP 2023 Findings (2023)
Keywords: human evaluation
humour
sarcasm
irony
natural language generation
position paper
critical survey
sociology
TL;DR: A review and analysis of humour, sarcasm and irony in the context of NLG, particularly regarding human evaluation procedures.

Abstract: Human evaluation in often considered to be the gold standard method of evaluating a Natural Language Generation system. However, whilst its importance is accepted by the community at large, the quality of its execution is often brought into question. In this position paper, we argue that the generation of more esoteric forms of language - humour, irony and sarcasm - constitutes a subdomain where the characteristics of selected evaluator panels are of utmost importance, and every effort should be made to report demographic characteristics wherever possible, in the interest of transparency and replicability. We support these claims with an overview of each language form and an analysis of examples in terms of how their interpretation is affected by different participant variables. We additionally perform a critical survey of recent works in NLG to assess how well evaluation procedures are reported in this subdomain, and note a severe lack of open reporting of evaluator demographic information, and a significant reliance on crowdsourcing platforms for recruitment.

The Iron(ic) Melting Pot:
Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation
Tyler Loakman', Aaron Maladry² and Chenghua Lin 'Department of Computer Science, The University of Sheffield, UK LLT Ghent University, Belgium
'Department of Computer Science, The University of Manchester, UK tcloakman tcloakman1@sheffieeel ac.elldacccu.. ac.uk aaron. maladry@ugent..b chenghua. lin@manchester.ce ac.
Abstract
more esoteric, such as humour, irony, and sarcasm where interpretations of such language are highly inter- twined with more broad demographic characteristics, as well as individual and idiosyncratic factors.
Human evaluation is often considered to be the gold standard method of evaluating Natural Language Generation system. However, whilst importance is accepted by the community at large, the quality of its execution is often brought into question. In this position paper, we argue that the generation of more esoteric forms of language humour, irony and sarcasm constitutes subdomain where the characteristics of selected evaluator panels are of utmost importance, and ev- ery effort should be made to report demographic characteristics wherever possible, in the interest of transparency and replicability. We support these claims with an overview of cach language form and an analysis of examples in terms of how their interpretation is affected by different participant variables. We additionally perform critical survey of recent works NLG to assess how well evaluation procedures are reported in this subdomain, and note a severe lack of open reporting of evaluator demographic information. and a significant reliance on crowdsourcing platforms for recnuitment.
Human evaluation is viewed the foremost important form of evaluation for any NLG system (Howcroft et al., 2020), owing largely to its direct cap- ture of the opinions of the (usually) human end-user. This is made even more prominent when considering the low concurrent validity that many easily deployed automatic evaluation metrics have with human judgements (Reiter, 2018; Alva-Manchego et al., 2021; Zhao et al., 2023a). Whilst such evaluation procedures are considered preferable over automatic metrics (particularly in the cases where no reliable automatic metrics are available), this does not mean that they are infallible. Numerous recent works have outlined the inconsistency with which human evaluation procedures are executed, and increasing attention is being brought to the knock-on effects that such oversights may cause (Zajko, 2021).
In this paper, we present an overview of 3 particular types of language: humour, irony and sarcasm and demonstrate how different evaluator demographies may affect the interpretation of such language, and therefore its evaluation in relevant systems. Further- more, we present a critical survey and assessment of 17 NLG research papers from between 2018 and 2023 from the ACL Anthology that concern the generation of these types of language, and analyse behaviour in regard to the level of transparency used when reporting details of the human evaluation procedure.
1 Introduction
The aim of Natural Language Generation (NLG) system is to generate coherent and well-formed text of a particular type, usually given an input such as prompt (Raffel et al., 2020; Wang et al., 2022), outline (Goldfarb- Tarrant al., 2020; Tang et al., 2022), topic (Van de Cruys, 2020), or data (Kale and Rastogi, 2020; Chen et al., 2020). As a result, NLG contains many sub-domains such as summarisation (Goldsack et al., 2022, 2023), data-to-text generation (Kale and Rastogi, 2020; Chen et al., 2020), dialogue generation (Tang et al., 2023), story generation (Goldfarb-Tamm)) et al., 2020; Tang et al., 2022), and humour generation (Sun et al., 2022b; Loakman et al., 2023). Whilst it may be possible for any competent speaker of the target language sufficiently understand generated outputs, many language forms may be considered
Related Works
In recent years, the execution of human evaluation in NLG has become with myriad top-tier venues hosting dedicated tracks and workshops specifically focused on the evaluation
more significant topic,
Sometinues NLG outputs are used inputs to further NLG models (Zhang et al., 2022). PIn what we describe as "cultural melting pot" in the title.
Conesponding author

---

of evallaation itself. such as the HumEEval workshops at EACL 2021, ACL 2022 and RANLP 2023.3 As a product of such venues, multiple recent works have been proposed that aim to highlight common inadequacies in multiple aspects of human evaluation, including the design of assessment criteria, the recruit- ment and training of participants, the use of biased response elicitation techniques (Parmar et al., 2023), and the transparency of reporting these processes.
that is only somewhat overcome with the provision of training materials standardise criteria definitions among evaluators). In order to partially mitigate these evaluation pitfalls, van der Lee et al. (2019) suggest a range of best practices, including the reporting of con- fidence intervals for results, using multiple ratings of each criteria, and using randomisation and counterbal- ancing when assigning evaluators to conditions (c.g. human- model- generated text). Additionally, due to many of the problems that we outline in this work, the domain is currently starting to demonstrate the pro- gression of a paradigm shift towards understanding the role of human disagreement as a valuable feature, rather than noise, which is also resulting in the rise of perspectivist approaches (Pavlick and Kwiatkowski, 2019; Meaney, 2020; Plank, 2022; Uma et al., 2022). Additionally, whilst the use of such language can lead to different responses from different people, this ef- feet is compounded in humann computer interactions, where the communicative divide is even more pro- nounced (Shani et al., 2022; Shapira et al., 2023).
Howcroft et al. (2020) assess a 20 year time-span of work within NLG (165 papers between 2000-2019) in regard to how evaluation procedures are performed, finding that there is substantial variation in how evalu- ation criteria are named and portrayed. Resulting from this, they produce normalised mappings between dif- ferent criteria deseriptions and their underlying evalua- tion focus and call for human evaluation criteria to be standardised, in addition to the creation of evaluation sheets outlining best practices for response elicitation from evaluators. Amidci et al. (2019) also assess NLG research in terms of response elicitation methods with a particular focus on the use of Likert scales and com- mon deviations from best-practice within 135 papers from a 10 year period (c.g. whether or not to view them as ordinal or interval in nature, and consequently how to perform statistical significance testing), Fur- ther issues have also been identified by Schoch et al. (2020) with the framing of evaluation questions and participant recruitment methods, including the pres- ence of demand characteristics in evaluators being more common crowdsoureing platforms as partici- pants vie for limited positions in HIT's and wish give researchers the results they believe they desire (under the belief it will lead to more future work). Further- more, these issues are exacerbated by cases wherein re- searchers reject annotations outright due to not falling in line with their expectations, or pay inadequately for services on crowdsourcing platforms (Huynh et al., 2021; Shmueli et al., 2021). However, whilst auto- matic metrics may have many drawbacks for evalu- ating NLG, human evaluation has been suggested to be imperfect by its very nature, with human judge- ments being inconsistent. For example, Clark et al. (2021) find that human evaluators range significantly in their preconceptions of the quality of text NLG systems can produce, resulting in evaluators having biased perspectives when assessing text quality (a bias
3 Introducing Humour, Sarcasm, and Irony
In the following section we discuss three primary examples of language types where interpretations and opinions are highly affected by a range of nuanced participant variables. For each type, we outline the nature of this language alongside range of examples, and reflect upon how interpretation of these examples may differ across demographics.
3.1 Humour
As the first language form to discuss, humour is also one of the most complex. The general consensus within cognitive and linguistic theories of humour is that an essential element for something to be found hu- morous is the existence of some degree of incongruity between expectations and reality (Raskin, 1985). Consequently, the experience of humour arises from the resolution of these incongruities within someone's 's mind, This, in turn, is how puns may elicit humour. when the initial interpretation of a text primes the interpretation of one semantic sense of a word, before that expectation is violated when the text is viewed as a whole. For example, in "the greyhound stopped to get hare cut", the interpretation of "hare" as in the rabbit-like animal is primed by the mention of the "greyhound" which chases hares, whilst "hare cut" forces the reader to reinterpret "hare" as the homo- phone "hair", which resolves the incongruity from the non-collocational phrase "hare cut" (He et al., 2019).
Pht "https://humee" ///une/alleval.... gi thub.
'Demand characteristics are cues that may allow participant to infer the nature and desired outcome of a study, often resulting in the participant then altering their behaviour to align with the perceived expectation, thereby invalidating results.

---

Type
Example
Humour
Q: "Why would Trump change lightbulb?" A: "He was told Obama installed it."
2. Q: "How does change lightbulb?" A: "He holds it and waits for the world revolve around him."
Non-humour
3. "Colourless green ideas sleep furiously."
Table 1:
Examples of humorous jokes with different types of assumed world knowledge.
Whilst puns are the most frequently studied form hurouur in the area of NLG (He et al., 2019; Mittal et al., 2022; Tian et al., 2022), primarily due to their simple exploitation of phonetics and word senses, humour more generally can arise from more complex incongruities which are more likely to be resolved by individuals with specific assumed world knowledge. Table presents examples of such humour. In Example 1, the perception of humour is dependent on factors including an individual's knowledge of politics and their ideological leaning, as well as region. The reader is therefore assumed to have external background knowledge of US polities in order to infer that the joke concerns Donald Trump's record of reversing actions made by his predecessor, Barack Obama. Additionally, due the joke using Trump as the target and effect calling him petty, whether or not this joke is seen as humorous also dependent on one's political leanings. This is due to Republicans, the political party that Trump represented, being less likely to find comments mocking him to be humorous. In Example 2, the joke is based around the shared knowledge of individuals in patriarchal society which has, of course, traditionally benefited men. As a result, the joke uses men as the target, and comments on a presumed level of arrogance. In addition, this example requires the reader to interpret both literal and metaphorical meanings of something "revolving around" something else. Consequently, factors that could affect the perception of humour here are one's familiarity with a patriarchal society and culture (as it can be imagined that such assumptions about male arrogance are not shared by members of a matriarchal culture). Additionally, due to the joke targeting men, there the possibility for some men to be offended at the assumptions that it is making, and it is known that gender generally affects the perception of certain types of jokes such as these (Lawless et al., 2020). Furthermore, large body of work has investigated the different tendencies of particular genders towards different patterns of language use,
which in some cases may also affect the perception of different language types (Coates, 2016; Rabinovich et al., 2020). Finally, Example 3 (Chomsky. 1957) is an instance of an incongruous sentence which is semantically nonsensical (yet grammatically valid), where this incongruity does not give rise to humour (in the general case). In addition to the examples outlined and analysed here, numerous works have also ascertained link between different demographic variables and humour perception. For example, Jiang et al. (2019) explore the literature on how individuals in the Western and Eastern worlds view humour as suitable under different conditions, and differ also in the exact forms of humour they opt to produce.
3.2 Irony
Although the exact definition of irony still remains a topic of discussion, NLP research has reached the general consensus that verbal irony is form of figurative language where the producer of the language intends to convey the opposite meaning of the literal interpretation (Grice, 1978; Burgers, 2010; Camp, 2012). Consequently, similar to within the humour domain, an ironic statement violates the true beliefs of the producer and the expectations of the audience. is for this reason that many ironic (and sarcastic) statements may also be considered humorous a well known feature of British comedy programming (Gervais, 2011). This means that correct understanding of irony presupposes true accepted knowledge about whatever thing, person or situation is being referred to. Such presupposed knowledge can assume multiple forms. A range of ironic texts are presented in Table 2. Trony can, for example, be factual, where it is generated by making factually incorrect statement such as in Example where human alludes to having wings. Alternttivly, it can also rely on social conventions, habits or shared knowledge within members of a community. For instance, in order to understand the irony in Example 2, the receiver expected to know that most people would not genuinely applaud the actions of someone they call a dictator. Whilst some knowledge and social
'In this cuse, US polities is quite well known world-over, but similar joke for a different govemment could be imagined, where universal knowledge would be far less likely.

---

Type
Example
Irony
1. "Just found about the train strike today, guess I'll have to my wings and fly there myself."
2. 2I's's been great week for dictators. Congrats North Korea and Cuba."
3. "Never knew something could be 80% halal... I've learned something new I guess."
"A local fire station has burnt down."
Non-irony
5. "I went to the hospital yesterday. It was big building."
Table 2: Examples of different forms of verbal irony (excluding sarcasm).
conventions are ubiquitous, many ironic expressions also assume specific (in-group) knowledge that is often only shared within particular social group. For example, within the statement seen in Example 3, the receiver of the message is required understand that "halal" is a most often used as binary concept, and therefore something being a "percentage" halal is infeasible, thus requiring additional background knowledge about Islam and its conventions. A single individual can, of course, be a member of several different social groups that share common ideals and knowledge, such as political conviction, free-time activities, or music taste, without even having to consider major cultural or geographical divergences. This simple fact, that irony strongly relies on common(sense) knowledge, makes an inherently subjective phenomenon, both for detection and the evaluation of generative models. addition, ironic statements can also rely on situational or contextual knowledge, in which case it is then classified as "situational irony" as opposed to "verbal irony". An example of situational írony is demonstrated in Exam- ple 4, where fire station burning down is unexpected due containing multiple resources explicitly for the extinguishing of such fires. According Wallace et al. (2014), contextual information may even be essential for most humans to recognise most forms of irony. Finally, Example 5 presents a non-ironic statement, where expectations about concepts are not violated, hospital is usually assumed to be large building. Recent works have taken into account the plethora of variables affecting the interpretation of irony, with Frenda al. (2023) developing corpus of irony from a perspectivist approach, finding that irony perceptions vary with generation and nationality (even within speakers of the same language).
negative connotation and aggressive tone, this form of verbal irony is intended to ridicule someone or some- thing (Filik et al., 2019). Table 3 presents selection of sarcastic texts. The clearly aggressive intent inherent in sarcasm is presented in Examples I and 2. In Ex- ample 1, the intention is to express that someone with seemingly no special qualities is able to have sense of pride, whilst in Example 2, the choice of the word "marinate" implies that the target of the comment is wearing too much perfume, similar to how meat may marinate in a glaze for many hours before being ready. Examples 3 and 4 present non sar-saacattic utterances, with Example 3 being blatant insult, and Example 4 demonstrating the expected gratitude someone would have if they had been bought supercar (assuming no deeper contextualised meaning). As sarcasm is consid- ered to be subcategory irony, research into irony detection or generation may often include sarcasm, and treats them as one and the same (Jijkoun and Hof- mann, 2009; Filatova, 2012; Van Hee, 2017: Maladry et al., 2022). Moreover, studies by Gibbs (1986) and Bryant and Fox Tree (2002) indicate that there is an on- going semantic shift, where verbal irony is perceived more often sarcasm in popular use. Recent papers on sarcasm generation (Mishra et al., 2019; Oprea et al., 2021) 1) do not systematically discuss their defi- nition of sarcasm relation to the concept of verbal irony. However, one has to assume here that sarcasm is synonymous with verbal irony, since it would be sub- optimal for generative systems to create content that might ridicule people in hurtful way. In short, both for pragmatic reasons as well conceptual changes, makes sense consider irony and sarcasm be the same thing. Thus, the same motivations that make irony subjective apply to sarcasm as well. On top of that, detection and generation tasks concerning sar- casm specifically also demand that evaluators are able to discem the aggressive tone and retrieve the intention of the author or speaker, which can vary significantly. For example, Phillips al. (2015) find age to be
3.3 Sarcasm
Traditional definitions usually consider sarcasm to be a sub-category of írony. In addition to having a strong

---

Type
Example
Sarcasm
1. "It's astonishing how someone average and unremarkable manage to feel good about themselves."
2. "Nice perfume. How long did you marinate in it?"
Non-sarcasm
3. "You're te an idiot."
4. Thank you so much for buying me Ferrar!"
Table 3: Examples of sarcasm and non-sarcasm.
significant contributing factor in an individual's ability to correctly perceive intended sarcasm, with older in- dividuals struggling more than younger: Furthermore, Rockwell and Theriot (2001) find gender differences in the patterns of sarcasm use to both be affected by the gender of the speaker, well the gender of the recipient interfocuter. Additionally, the use of sar- casm has been shown to vary alongside socio-cultural contexts, with Blasko et (2021) noting that sar- casm use more prevalent individualist countries with a lower power distance such as the U.S than col- Icctivist countries with higher power distance such as China. 6 It logically follows, therefore, that these differ- ences may also impact someone's ' ability to perceive intended sarcasm, and consequently how they would react sarcastic utterances in different contexts.
4.1 Scope and Selection
The papers included in this survey pool were all taken from the ACL Anthology for the venues of *ACL,, EMNLP, COLING, LREC, INLG, and CoNLL, between 2018 and 2023. 78 Initially, all papers concering humour, irony and sarcasm were selected. This was performed by keyword matching titles on the following word stems: "hum-", "sarc-", "iron-", "fun-", "jok-" and "pun-"."10 The selected papers include main conference, Findings, and workshop submissions, in addition to one system demonstration, resulting in 259 papers. Submissions to shared-tasks then excluded, reducing the pool to 135 papers. The resulting papers were then checked over to ensure relevance, and categorised broadly into generation or detection (the latter of which may be binary. multi-class, or estimation). These criteria resulted in 135 remaining papers, of which 22 were focused on the generation of humour, sarcasm and/or irony, and 108 on detection (with the remaining 5 being classed as "other"). We then excluded 4 of the remaining 22 generation papers that were surveys dataset papers where human evaluation was performed on non-computationally generated examples. This resulted in a final selection of generation papers, all of which are listed in Appendix A.1.
4 Critical Survey
Following cxisting works that critically review human evaluation procedures in Natural Language Generation, we present critical survey of works in the area of computational humour, irony, and sarcasm. We begin by reviewing the scope and range of literature search for this review and the prerequisite criteria for incorporation. Following this, we motivate the need for increased care in this specific domain of language by briefly exploring the prevalence of research in the area, both in terms of generative works, as well as those on the detection of such language. As previously discussed, irony and sarcasm highly related language forms, resulting in the common use of the terms interchangeably. However, to ensure that the intended definition is respected (the correctness of which cannot always be inferred with complete certainty) we report irony and sarcasm separately based on the titles of the papers.
4.2 Subject Prevalence
As seen in Figure 1, there has been significant number of accepted papers these topics within this 6 year scope of collection. In addition to these papers, this search also revealed shared tasks within this period, including sarcasm detection shared task
Wheee the relevant venues have occurred at the time of writing.
"INLG and CoNLL contributed 0 papers that met our criteria for inclusion, but were nevertheless surveyed.
"To capture the most common form of humour generation. "In order capture "humour", "sarcasm", "irony". "funny", "joke", and "puns". including all derivations and regional spelling.
"Powwr Distance" refers to the distribution of perceived power (political, financial etc.) across a society. A lower power distance means that members of a society are more "equal", for example, when comparing a blue collar worker to politician.
"This is due to shared task submissions often attracting non-novel approaches, and all following a standardised evaluation procedure as determined by the task organisers. Their inclusion would therefore bias the results.

---

All but of the 18 papers found also contained human evaluation, and therefore the remaining 17 papers will form the basis of the following analysis. Following on from earlier sections which motivated the significance of participaat variables in the interpretation of these language forms, Table 4 presents the percentage of surveyed papers that report on the following evaluator demographic information: language proficiency, location and nationality, age, gender, education level, and social class.
HUMMOUR BARCAFM
SARCAIM IRONY
50
40
#
30
30
14
16
10
2
DETECTION
CENERATION
OTHER
TASK
Participant Demographics As Table shows, less than half of the assessed works (7 17) reported on any demographic information regarding their selected human evaluator panels. Of those that were more transparent in regard to evaluator demographies, language proficiency and location (usually proxy for nationality) were the most common detail given. single paper (Mishra al., 2019) reported evaluator education.
Figure I: Breakdown of selected papers by NLP task and specific language type (based on the paper titles). These values exclude shared task papers, but include generation papers that are surveys or datasets, which were later removed. The "other" task category refers to papers primarily presenting content such as annotation schema (Pamulapati et al., 2020) and cognitive analyses (Zeng and Li, 2022), among other forms of research in this area we are not directly investigating.
Importantly, even in those papers which did outline some demographic information, this reporting was frequently crucially inadequate. For example, Oprea et al. (2022) provide the full participant information sheet that was presented, but reduce the demographic information to "We target everyone registered as living in (country) Academic platform.", where the valuable information of which country (or which countries) redacted. Additionally, the one paper that reported education (Mishra et al., 2019) was only to the extent that the two evaluators "had a better understanding of the language and socio-culural diversities" statement that may range from meaning native speakers of the target language in the desired culture, to people with doctorates in sociology. None of the surveyed papers outlined evaluator age, gender or social class.
at the Figurative Language Processing workshop at ACL 2020 (Ghosh et al., 2020), Task of SemEval 2020 and 2021 on humour detection (Hossain et al., 2020; Faraj and Abdullah, 2021), a WANLP 2021 shared task on sarcasm detection (Abu Farha et al., 2021), and SemEval-2022 Task 6, also sarcasm detection (Abu Farha al., 2022). 12
the Prolific
In addition the works captured by our literature search and the aforementioned shared tasks, it reasonable to think that the prevalence of recently released Large Language Models (LLMs) such as ChatGPT and GPT-4 would result in further increases in research into this field, now that models are widely available that able to better handle the generation and explanation of complex topics such as humour. Not explicitly included in this survey, but nevertheless relevant, are also more general works on anthropo- morphism language-based Al agents (Abercrombic et al., 2021; Gros et al., 2022), in which the ability to detect, generate, and respond to humour, irony and sarcasm are paramount if anthropomorphism is deemed desirable (Oprea et al., 2022).
Recruitment As can seen in Table 5, 12 of the 17 surveyed papers contained explicit mention of participant recruitment methods. Upon further investigation into these cases, we find that all 12 instances concern the use of crowdsourcing platforms, of which 9 are Amazon Mechanical Turk (AMT), are Prolific Academic (Oprea et al., 2021, 2022). and one does not explicitly mention the platform (Tian al., 2022). Of these 12 papers that report recruitment, compensation amounts are reported less than half (5 papers), ranging from approximately
4.3 Reporting Transparency
Of the 135 remaining papers, we now move our attention to the 18 focused on generation specifically.
12 As expected, all shared tasks were on detection rather than generation, due to the more simple nature of evaluation for a shared task environment.

---

Type
Papers
Percentage
uation panels are not conducive to easy comparison across studies." Whilst the substamtial reliance on crowdsoureing platforms, particularly Mechanical Turk, may result in a different set of participants to the typical WEIRD demographic social science and health research, the lack of reliable worker filters and allowances on what demographic data can be collected by requesters nevertheless results in biased sample. Early works the the use of crowdsoureing outside of this survey have even touted the low cost as an affordance of the method, whilst paying as low as 1/10th of a single US cent per individual HIT task 2009). Perspectives such as these therefore result in biased demographics being used for evaluation and annotation at best, and the outright exploitation of people from particular socio-economic backgrounds and regions at worst.
Demographies?
7
<42%
Language Proficiency
4
<24%
Location/Nationlliy
4
<24%
Age
0
0%
Gender
0
0%
Education
<6%
Social Class
0%
Counts of papers containing reporting of demographic variables in the area of humour, sarcasm and irony generation. "Demographics" refers to the number/percentage of papers that report any demographic information.
Table 4:
(Callison-Buri)
9.23 US (Oprea et al., 2021, 2022) to 20 USD per hour (Tian et al., 2022: Mittal et al.. 2022).
In terms of inter-annotator agreement (IAA), we find that just over half of the surveyed papers include explicit mention of statistical tests for agreement levels. IAA is one method through which the variation amongst evaluator opinions can best be analysed, in order to understand how subjective task is, and consequently how best to assess the quality of an NLG system's output where there is no "ground truth" for a given criteria (e.g. the humour, irony and sarcasm discussed here). Finally, we note that only 4 (of 17) papers convey any information regarding the level of training given to participants, whether that constitute the presence of qualifier tasks on crowdsourcing platforms (Sun et al., 2022a.b) or extensive documentation (Oprea et al., 2021, 2022). In tasks that are inherently subjective, the provision of adequate training materials can help to account for variation the understanding of different individuals by orientating opinions, and ensuring that all evaluators are working from the same definition of particular phenomenon (such distinguishing between general verbal irony and sarcasm specifically). Especially in cases where IAA is not reported, the inclusion of the extent of this background training given to evaluators is useful the research community for judging to what extent the evaluators were in-line with each other's perspectives.
One reason that transparent reporting of recruitment methods is important is due to allowing inference of potential participant characteristics that may not be reported elsewhere. For instance, research in the social sciences has often been criticised for focusing primarily on WEIRD participants, Western and Educated individuals from Industrialised, Rich and Democratic societies (Henrich et al., 2010). However, as access to language technology becomes more democratised, cheaper to access and easier to use (Simons et al., 2022; de Dios-Flores et al., 2022) the majority of end users of these technologies going to be well represented by this set of character- isties that participants often have. This tendency is further reflected by the overwhelming bias towards work on the English language, which leads to the demotion of other non-digitally accessible languages, and reinforces existing inequalities (Søgaard, 2022). In addition to these characteristics, there has also historically been bias towards male participants in scientific research (Holdcroft, 2007: Ganzáa-González al.. 2019), something that is also common in AI due to the field of computer science still being dominated by men overall (European Commission and Dinerta for Research and Innovation, 2019; Breidenbach et al., 2021). Consequently, not only does this result in Al research not involving peo- ple from key demographics who will be affected by these technologies, but the research performed on the generated language will lack important replicability for the evaluation process, where heterogeneous eval-
not
the nature of the task
Discussion and Proposed Action Points
Our analysis has shown a distinct lack of reporting on potentially relevant demographic information in ad-
This study paid 0.38 GBP for approximately 3 minutes of work. The USD conversion is the exchange rate at the time of writing this survey.
'We exclude the cases where the research was intended for a specific demographic group to start with, in which case more homogencous evaluator group may be desirable.

---

Factor
Papers
Percentage
While the language proficiency of the participants relevant all tasks, we recognise that other charac- teristics such as gender and cultural background (e.g. religion and political leaning) may not always apply. Still, it would be beneficial to report which facets were taken into consideration, as they may become relevant in hindsight, and in the case of dataset annotations, facilitate more extensive use of any collected data in fu- ture works. Similarly, task proficiency may sometimes seem irrelevant for subjective and intuitive tasks. How- ever, Iskender al. (2021) and Lloret al. (2018) demonstrate the preference of expert evaluators for complex linguistic tasks. Additionally, authors may be reluctant to present only moderate agreement for sub- jective tasks due to concerns regarding evaluation qual- ity. However, this criterion is highly relevant as it de- scribes the expected degree of subjectivity for task, concept the community should be able to embrace. In addition to these points, we also encourage increased interest in the area of perspectivist approaches to the study of these language forms, and in NLP as a whole. Finally, the intentional omission of many of these criteria (such as IAA and language proficiency, in par- ticular) may be considered intentionally deceptive, and therefore odds with the foundations of open science.
Recruitment
12
<71%
Compensation
<30%
IAA
<53%
Training Materials
<24%
Table
Counts of papers containing explicit mention of recruitment methods, compensation details, inter-annotator agreement (IAA), and the provision of training materials.
dition details conceming the evaluation procedure. To tackle this issue, we suggest future work on evaluation of (especially subjective) generative tasks to include an "evaluation statement", akin to the data statements proposed by Bender and Friedman (2018). In line with their work, we suggest the inclusion of the following points for such statements:
evaluation logistics
number of evaluators used
the split of samples across evaluators (e.g. does everyone rate every sample)
recruitment method used (e.g. internal emails, posters or crowdsourcing)
compensation given to evaluators (e.g. gift cards and cash additionally, the choice to not monetarily compensate evaluators should also be divulged)
Conclusion
In conclusion, we reiterate the importance of the role that Human Evaluation plays in assessing the outputs of Natural Language Generation systems. We have further demonstrated why particular types of language such as humour, irony, and sarcasm demand additional attention to be paid during the construction of human evaluation processes, particularly as it pertains to the demographic variables of selected evaluators. In no way do we discourage the use of the "non-native" or L2+ speakers of target language in such evaluation (or any other minority demographic), nor do we likewise encourage the use of "native" LI target-language speakers (or any other majority demographic) as the true "gold standard". Rather, we aim to encourage additional thoughts and reflections in the community when selecting panels of human evaluators for such language, analysing results in light of their demographic characteristics as feature, rather than bug, whilst openly reporting such decisions wherever possible. To this end, we provide list of suggested points that can be included part of an "evaluation statement". Additionally, we advocate for the use of stratified sampling techniques to gain more varied evaluator panels.
evaluator demographic information
age (e.g. range, group or generation)
language proficiency & "native" language (L.)
gender
cultural background (e.g. religion, political leaning, country of origin and general education level)
task proficiency (i.e. expert or layperson)
evaluation process
clear and operationalised definitions of the evaluated phenomena (with explicit men- tion if term with a largely accepted defi- nition is being used in a non-standard way) amendments made in light of potential pilot studies
training materials and examples of criteria given to evaluators
agreement study (i.c. reporting inter-annotator agreement)

---

Limitations
References
The limitations of this work are as follows. We only survey papers from 2018-2023 in order to capture more recent and modern behaviours in the research discipline. However, this results in a small number of available papers. This is naturally exacerbated by the choice of focusing solely on humour, sarcasm and irony, rather than language generation more generally. However, believe that surveys of NLG broadly have been performed in recent years high standards, and instead wish to use these niche language types as specific examples of types of language that are immensely affected by these factors, whilst conveying that transparent reporting and consideration of evaluation practices is entirely applicable to NLG whole, and not just the subdomains focus on. Furthermore, focus exclusively the use of human evaluators in generation tasks. However, much of the same ideas apply equally to the selection of annotators during dataset creation for detection tasks. All statements and opinions expressed in this work are by nature in good faith, and we accept, appreciate and welcome perspectives from other individuals from different backgrounds to support, challenge, or revise any of the claims made herein.
Gavin Abercrombie, Amanda Cereas Curr, Mugdha Pandya, and Verena Rieser. 2021. Alexa, Google, Siri: What are your pronouns? gender and anthropomor- phism in the design and perception of conversational assistants. In Proceedings of the 3rd Workshop on Gen- der Bias in Natural Language Processing, pages 24-33, Online. Association for Computational Linguistics.
Ibrahim Abu Farha, Silviu Vlad Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 task 6: iSarcasmEval, intended sarcasm detection in English and Arahic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022) pages 802-8 014. Seattle, United States. Association for Computational Linguistics.
Ibrahim Abu Farba, Wajci Zaghouami, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296-305, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021. The (un)suitabiliity of automatic evalu- ation metries for text simplification. Computational Linguisticx, 1884).61-889.
Jacopo Amidei, Paul Piwek, and Alistair Willis. 2019.
The use of rating and Likert scales in natural language generation human evaluation tasks: A review and In Proceedings of the
some recommendations. 12th International Conference on Natural Language Generation, pages 397-402, Tokyo, Japan. Association for Computational Linguistics.
Ethics Statement
We believe in and firmly adhere the ACL Code of Conduct in the performance of this survey and the expression of opinions. We review only information that was publicly available in the published versions of the papers we discussed. However, whilst we encourage the collection of a lot of metadata about participants in NLG research, we emphasise that this data should only be collected where the correct ethics application procedures have been followed
Emiil M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Associaiion for Computational Linguistics, 6:587-604.
Dawn G. Blasko, Victoria A. Kazmerski, and Sharif- fah Sheik Dawood. 2021. Saying what you don't mean: A cross-cultural study of perceptions of sarcasm. Cana- dian Journal of Experimental Psychology / Revue cana- dienne de psychologie expérimentale, 75(2):114-11,
by the respective researchers, and that collected information should be only as fine-grainet as is needed to accurately characterise the participant panel (for example, someone's hometown is not necessary when their nationality or cultural identity would suffice, and neither is someone's exact age when 10-year age brackets may be sufficient).
Aurélie Breidenbach, Caroline Mahlow, and Andreas Schreiber. 2021. Implicit gender bias computer science a qualitative study.
Gregory Bryant and Jean E Fox Tree. 2002. Recognizing verbal irony in spontaneous speech. Metaphor and symbol, 17(2)99-119.
A Acknowledgements
C. Burgers. 2010. Verbal irony: Use and effects in written discourse. Ph.D. thesis, UB Nijmegen.
Tyler Loakman is supported by the Centre for Doc- toral Training in Speech and Language Technologies (SLT) and their Applications funded by UK Research and Innovation [grant number EP/S023062/1). Aaron Maladry is supported by Ghent University [grant BOF22422001.0019.001
Chris Bullison-Burch.. 2009, Fast, cheap, and creative: Evaluating translation quality using Amazon's Me- chanical Turk. In Proceedings of the 2009 Conference Empirical Methods in Natural Language Pro- cessing, pages 286-295, Singapore. Association for Computational Linguistics.

---

E. Camp. 2012. Sareasm, pretense, and the seman- tics/pragmatics distinction. Nous, 46:587-634 -344.
Elena Filatova. 2012. Irony and sareasm: Corpus generation and analysis using crowdsourcing. In Lrec. pages 392-398. Citeseer.
Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng, 2020. R"3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7976-7986, Online. Association for Computational Linguistics.
Ruth Filik, Alexandra Turcan, Christina Ralph-Nearman, and Alain Pitiot. 2019. What is the difference between irony and sarcasm? an finri study. Cortex, 115:112-122.
Simona Frenda, Alessandro Pedrami, Valerio Basile, Soda Marem Lo, Alessandra Teresa Cignarella, Raffaella Panizzon, Cristina Marco, Bianca Scarlini, Viviana Patti, Cristina Bosco, and Davide Bernardi. 2023. Epic: perspeeeer annotation of a corpus of irony. ACL 2023.
Arjun Chandrasckaran, Devi Parikh, and Mohit Bansal. 2018. Punny captions: Willy wordplay in image descriptions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putatioonll Linguistics: Human Language Technologies. Volume 2 (Short Papers), pages 770-775, New Orleans, Louisiana. Association for Computational Linguistics.
Judit Garzfa-Gonzzlez, Patricia Forcén, and Maria Jimenez Jimenez-Sanchez. 2019. Men and women differ in their perception of gender bias in research institutions. PloS one, 14( 122::02272776022555666
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020. KGPT: Knowledge-groumdee pre-training for data-to-text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8635-8648, Online. Association for Computational Linguistics.
Apama Garimella, Carmen Banea, Nabil Hossain, and Rada Mihalcea. 2020. "judge me by my size (noun), do you?" YodaLib: A demographic-aware humor generation framework. In Proceedings of the 28th International Conference on Computational Linguis- rics, pages 28 1814-2825, Barcelona, Spain (Online). International Committce on Computational Linguistics.
Noam Chomsky, 1957. Syntactic structures, Syntactic structures. Mouton, Oxford, England.
Ricky Gervais. 2011. Ricky gervais: The difference between american and british humour.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021. All that's 's 'human' is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7282-7296, Online. Association for Computational Linguistics.
Debanjan Ghosh, Avijit Vajpayee, and Smaranda Muresan. 2020. A report on the 2020 sarcasm detection shared task. In Proceedings of the Second Workshop Figurative Language Processing, pages 1-11, Online. Association for Computational Linguistics.
Raymond W Gibbs. 1986. On the psycholinguistics of sarcasm. Journal of experimental psychology: general, 115(1):3.
Jennifer Coates. 2016.
Women, men and language: A sociolinguistic account of gender differences in language. Routledge.
Seraphina Goldfarb-Tarann Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. 2020. Content planning for neural story generation with aristotelian rescoring. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4319-43336 Online. Association for Computational Linguistics.
Iria de Dios-Flores, Carmen Magariños, Adina Ioana Vladu, John E. Ortega, José Ramom Pichel, Marcos Gareía, Pablo Gamallo, Elisa Feránndez Rei, Alberto Bugarin-Diz, Manuel González González, Senén Barro, and Xosé Luis Regueira. 2022. The nós project: Opening routes for the Galician language in the field of language technologies. In Proceedings of the Workshop Towards Digital Language Equality within the 13th Language Resources and Evaluation Conference, pages 52-61, Marseille, France. European Language Resources Association.
Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2022. Making science simple: Corpora for the lay summarisation of scientific literature. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
European Commission and Directorate-General for Research and Innovation. 2019. She figures 2018. Technical report, European Commission.
Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2023. Domain-driven and discourse- guided scientific summarisation. In European Conference on Information Retrieval, pages 361-376.
Dalya Faraj and Malak Abdullah. 2021. SarcasmDet at SemBval-2021 task 7: Detect humor and offensive based on demographic factors using RoBERTa pre- trained model. In Proceedings of the 15th International Workshop Semantic Evaluation 1SemExal-20211 pages 527-533, Online, Association for Computational Linguistics.
P.H. Grice. 1978. Further notes on logic and conversation, volume 9. pages 113-127. P. Cole, Syntax and Semantics, New York: Academic Press.

---

David Gros, Yu Li, and Zhou Yu. 2022. Robots-dont-ery: Understanding falsely anthropomorphic utterances in dialog systems, In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 3266-3284, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Student Research Workshop), pages 334-351, Toronto, Canada. Association for Computational Linguistics.
Mihir Kale and Abhinav Rastogi. 2020. Text-to-text pre-training for data-to-text tasks. In Proceedings of the 13th International Conference on Natural Language Generation, pages 97-102, Dublin, Ireland. Association for Computational Linguistics.
He He, Nanyun Peng, and Percy Liang. 2019. Pun generation with surprise. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 7734,1744, Minneapolis, Minnesota. Association for Computational Linguistics.
Tiffany J. Lawless, Conor J. O'Dea, Stuart S. Miller, and
Donald A. Saucier. 2020. Is it really just a joke? gender differences in perceptions of sexist humor. Humor (Berlin, Germany), 33(2):9113315
Elena Lloret, Laura Plaza, and Ahmet Aker. 2018. The challenging task of summary evaluation: an overview. Language Resources and Evaluation, 52:101-148.
Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3);61-83.
Tyler Loakman, Chen Tang, and Chenghua Lin. 2023. TwistL Resources and baselines for tongue twister generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 579-589, Toronto, Canada, Association for Computational Linguistics.
Anita Holdcroft. 2007. Gender bias research: how does it affect evidence based medicine? Journal of the Royal Society of Medicine, 100(1):2-3, 3. PMID: 17197669.
Nabil Hossain, John Krumm, Michael Gamon, and Henry Kautz. 2020. SemEyal-2020 task 7: Assessing humor in edited news headlines. In Proceddiggs of the Fourteenth Workshop on Semantic Evaluation, pages 746-758, Barcelona (online). International Committee for Computational Linguistics.
Fuli Luo, Shunyao Li, Pengcheng Yang. Lei Li, Baobao Chang, Zhifang Sui, and Xu Sun. 2019. Pun-GAN: Generative adversarial network for pun generation. In Proceedings of the 2019 Conference ods Natural Language Processing and the 9th Interna tional Joint Conference on Natural Language Process ing (EMNL.P-JCNIIP) P), pages 3388-3393, Hong Kong, China, Association for Computational Linguistics.
Empirical Meth-
David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg. Sashank San- thanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, pages 169-182, Dublin, Ireland. Association for Computational Linguistics.
Aaron Maladry, Els Lefever, Cynthia Van Hee, and Véronique Hoste. 2022. The limitations of irony detection in dutch social media. Language Resources and Evaluation.
J. A. Meaney. 2020. Crossing the line: Where do demographic variables fit into humor detection? In Proceedings of the 58th Annud Meeting of the Associ- ation for Computational Linguistics: Student Research Workshop, pages 176-181. Online. Association for Computational Linguistics.
Jessica Huynh, Jeffrey Bigham, and Maxine Eskenazi. 2021. A survey of nlp-related crowdsourcing hits: what works and what does not.
Neslihan Iskender. Tim Polzehl, and Sebastian Möller. 2021. Reliability of human evaluation for text summarization: Lessons learned and challenges ahead. In P Proceedings of the Workshop on Human Evaluation of NLP Systems (HumnEval), pages 86-96.
Abhijit Mishra, Tarun Tater. and Karthik Sankaranarayanan. 2019. A modular architecture for unsupervised sarcasm generation. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference natural lan- guage processing (EMNLP-IJCNLP) P),, pages 61 4445544.
Tonglin Jiang, Hao Li, and Yubo Hou. 2019. Cultural dif- ferences in humor perception, usage, and implications. Frontiers in Psychology, 10.
Anirudh Mittal, Yufei Tian, and Nanyun Peng. 2022. AmbiPun: Generating humorous puns with ambiguous context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 1053-1062, Seattle, United States. Association for Computational Linguistics.
Valentin Jijkoun and Katja Hofmann. 2009. Generating a monnonngee engleenlllhhh subjectivity lexicon: Relations that matter. Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 398-405.
Faria Binte Kader, Nafisa Hossain Nujat, Tasmia Binte Sogir, Mohsinul Kabir, Hasan Mahmud, and Md Kam- rul Hasan. 2023. "when words fail, emojis prevail": A novel architecture for generating sarcastic sentences with emoji using valence reversal and semantic incon- gruity. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4:
Silviu Oprea, Steven Wilson, and Walid Magdy. 2021. Chandler: An explainable sarcastic response gen- erator. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 339-349, Online and

---

Punta Cana, Dominican Republic. Association for Computational Linguistics.
Stephanie Schoch, Diyi Yang, and Yangfeng Ji. 2020. "this is problem, don't you agree?" framing and bias in human evaluation for natural language generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pages 10-16, Online (Dublin, Ireland). Association for Computational Linguisties.
Silviu Vlad Oprea, Steven Wilson, and Walid Magdy, 2022. Should a chatbot be sarcastic? understanding user preferences towards sarcasm generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7686-7700, Dublin, Ireland, Association for Computational Linguistics.
Chen Shani, Alexander Libov, Sofia Tolmach, Liane Sewin-Eytan. Yoelle Maarek, and Dafna Shahaf. 2022. "alexa, do you want to build a snowman?" characterizing playful requests to conversational agents. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA '22, New York. NY, USA. Association for Computing Machinery.
Vaishnavi Parulaaatii Gayatri Purigilla, and Radhika Mamidi. 2020. novel annotation schema for conversational humor: Capturing the cultural nuances in kanyasulkam. In Proceedings the 14th Linguistic Annotation Workshop, pages 34 47, Barcelona, Spain. Association for Computational I Linguisties.
Natalie Shapira, Oren Kalinsky, Alexander Libov, Chen Shani, and Sofia Tolmach. 2023, Evaluating humorous response generation to playful shopping requests. In ECIR 2023.
Mihir Parmar, Swaroop Mishra, Mor Geva, and Chitta Baral, 2023. Don't blame the annotator: Bias already starts in the annotation instructions. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1779-1789, Dubrovník, Croatia. Association for Computational Linguistics.
Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyond fair pay: Ethical implications of NLP crowdsoureing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 3758-3769, Online. Association for Computational Linguistics.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transac- tions of the Association for Computational Linguistics, 7:677-694 6644
Gary F. Simons, Abbey L. L. Thomas, and Chad K. K. White. 2022. Assessing digital language support on a global scale. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4299-4305, Gyeongju, Republic of Korea, Intemational Committee on Computational Linguistics.
Louise H. Phillips, Roy Allen, Rebecca Bull, Alexandra Hering, Matthias Kliegel, and Shelley Channon. 2015. Older adults have difficulty decoding sarcasm. Developmental Psychology, 51(12)1180-188)
Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671- 10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Anders Søgaard. 2022. Should we ban English NLP for year? In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing. pages 5254-525500 Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Ella Rabinovich, Hila Gonen, and Suzanne Stevenson. 2020. Pick fight bite your tongue: Investigation of gender differences in idiomatic language usage. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5181-5192, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Jiao Sun, Anjali Narayan- Chen, Shereen Oraby, Alessandra Cervone, Tagyoung Chung, Jing Huang, Yang Liu, and Nanyun Peng. 2022a. ExPUNations: Augmenting puns with keywords and explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4590 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-tex transformer. Journal of Machine Learning Research. 211140)1-67
Jiao Sun, Anjali Narayan-Cheri Shereen Oraby, Shuyang Gao, Tagyoung Chung, Jing Huang, Yang Liu, and Nanyun Peng. 2022b. Context-siuated pun generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4635- 4648, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Victor Raskin. 1985. Semantic mechanisms of humor. Synthese language library: 24, Reidel, Dordrecht.
Ehud Reiter. 2018. A structured review of the validity of BLEU, Computational Linguistics, 44(3):993-44
Chen Tang. Hongbo Zhang, Tyler Loakman, Chenghua Lin, and Frank Guerin. 2023. Terminology-awar medical dialogue generation. In ICASSP 2023 2023 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Patricia Rockwell and Evelyn M. Theriot. 2001. Culture, gender, and gender mix in encoders of sarcasm: A self-assessment analysis. Commanication Research Reports, 18( (12)-44-5..

---

Chen Tang, Zhihao Zhang, Tyler Loakman. Chenghua Lin, and Frank Guerin. 2022. NGEP: A graph- based event planning framework for story generation. In Proceed- ings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Compulational Linguistics and the 12th International Joint Conference Natural I Language Processing (Volume 2: Short Papers), pages 186-193,, Online only. Association for Computational Linguistics.
Zhiwei Yu, Hongyu Zang. and Xiaojun Wan. 2020. Homophonic pun generation with lexically constrained rewriting. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 2870-2876, Online. Association for Computational Linguisties.
Mike 7ajko. 2021. Conservative Al and social inequality: conceptualizing altematives to bias through social theory. Al & SOCIETY, 3663):1047-1055.
Yufei Tian, Divyanshu Sheth, and Nanyun Peng, 2022. A unified framework for pun generation with humor principles. In Findings of the Association for Compu- tational Linguistics: EMNLP 2022, pages 3253-3261,
Qingcheng Zeng and An-Ran Li. 2022. A survey in automatic irony processing: Linguistie, cognitive, and multi-X perspectives. In Proceedings of the 29th International Conference on Computational Linguis- tics, pages 824-836, Gyeongju, Republic of Korea. International Committee on Computational Linguisties.
Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alexandra N. Uma, Tommaso Fomaciari, Dirk Hovy. Silviu Paun, Barbara Plank, and Massimo Poesio. 2022. Learning from disagreement: A survey, Journal of Artificial Intelligence Research, 72:1385-1470.
Yue Zhang, Hongliang Fei, Dingcheng Li, and Ping Li. 2022. PromptGen: Automatically generate prompts using generative models. In Findings of the Association for Computaaional Linguistics: NAACL 2022, pages 30-37, Seattle, United States, Association for Computational Linguistics.
Tim Van de Crays. 2020. Automatic poetry generation from prosaic text. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 2471-2480, Online. Association for Computational Linguistics.
Kun Zhao, Bohao Yang, Chenghua Lin, Wenge Rong. Aline Villavicencio, and Xiaohui Cui. 2023a. Eval- uating open open-domain dialogues in latent space with sentence prediction and mutual information. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 562-577.
Chris van der Lee, Alhert Gau,, Emiel van Miltenburg, Sander Wubben, and Fmiiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pages 355-368. Tokyo, Japan. Association for Computational Linguistics.
Wenye Zhao, Qingbao Huang, Dongsheng Xu, and Peizhi Zhao. 2023b. Multi-modal sarcasm generation: Dataset and solution. In Findings of the Association for Computational Linguisties: ACL 2023, pages 5601- 5613, Toronto, Canada. Association for Computational Linguistics.
Cynthia Van Hee. 2017. Can machines sense irony? exploring automatic irony detection on social media. Ph.D. thesis, Ghent University.
Byron C Wallace, Laura Kertz, Eugene Charnaa, et al. 2014. Humans require context infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Arnual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 512-516.
A Appendix
A.1 Surveyed Generation Papers
Boshi Wang, Xiang Deng, and Huan Sun. 2022. Iteratively prompt pre-trained language models for chain of thought. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 2714-2730. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Orion Weller, Nancy Fulda, and Kevin Seppi. 2020. Can humor prediction datasets be used for humor generation? humorous headline generation via style transfer. Proceedings of the Second Workshop on Figurative Language Processing, pages 186-191, Online. Association for Computational Linguistics.
Zhiwei Yu, Jiwei Tan, and Xiaojun Wan. 2018. A neural approach to pun generation. In Proceedings of the 56th Arnual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1650-1660. Melbourne, Australia. Association for Computational L Linguistics.

---

Title
Author (Year)
Neural Approach to Generation
Yu et (2018)
Punny Captions: Witty Wordplay in Image Descriptions
Chandnssekran al. (2018)
A Modular Architecture for Unsupervised Sarcasm Generation
Mishra et al. (2019)
Pun- Ban:GAN: Generative Adversarial Network for Pun Generation
Luo al. (2019)
Pun Generation with Surprise
He (2019)
Can Humor Prediction Datasets be used for Humor Generation?
Weller et al. (2020)
Humorous Headline Generation via Style Transfer
R²: Reverse, Retrieve, and Rank for Sarcusm Generation with Commonsense Knowledge
Chakrabarty et al. (2020)
Homophonic Pun Generation with Lexically Constrained Rewriting
Yu et al. (2020)
"Judge me by my size (noun), you?"
Garimella (2020)
YodaLib: Demographic-Awwrr Aware Humor Generation Framework
Chandler: An Explainable Sarcastic Response Generator
Oprea al. (2021)
Should Chutbot be Sarcastic? Understanding User Preferences Towards Sarcusm Generation
Oprea et al. (2022)
Context-Situated Pun Generation
Sun (2022b)
ExPUNations: Augmenting Puns with Keywords and Explanations
Sun et al. (2022a)
A Unified Framework for Pun Generation with Humor Principles
Tian et al. (2022)
AMBIPUN: Generating Puns with Ambiguous Context
Mitall al. (2022)
"When Words Fail, Emojis Prevail":
Kader al. (2023)
A Novel Architecture for Generating Surcastic Sentences With Emoji
Using Valence Reversal and Semantic Incongruity
Multi-modal Sarcasm Generation: Datuset and Solution
Zhao et al. (2023b)
Table 6: List of covered generation papers in the critical survey.


Reviews:
[{"review_id": "zsWVjU1KfU", "reviewer": "Reviewer_4dTM", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critical review. Firstly, we accept that the community is indeed larger than what was surveyed, However, in line with standard methodology for performing critical reviews (Moher et al., 2009 - The PRISMA Statement), we defined our scope before performing the review, in which we selected a subset of the ACL Anthology which included all *ACL venues, EMNLP, LREC and COLING. As we state, this resulted in 244 papers across main conference, findings, system demos, and workshops, which was filtered down to 15 as we deemed detection papers to be out of scope, in addition to removing shared task submissions due to using a unified evaluation strategy, and finally, we remove 4 generation papers due to not clearly using human evaluation. It is also not unusual for small sample sizes to be used when supporting a point about the inadequacy of an approach, for example, Reiter (2018, in Computational Linguistics) use 34 papers to analyse the use of BLEU, which is only a subset of all papers that would have been reporting BLEU (as is acknowledged in their paper). We also present our work primarily as a position/opinion paper that identifies a problem and offers justification via the small survey, rather than being primarily a survey paper that is intended to represent the entire domain.\n\nWe intentionally focus on generation instead of detection because the human evaluation for detection tasks is more static. By static, we mean that the human evaluation is unified for each dataset and remains the same for any system that is evaluated on that specific dataset (ideally, the publication of a dataset also includes a data statement, such as the one proposed by Bender and Friedman, 2018). This makes it somewhat easier and more reliable to compare system performance. For generation, each system generates its own samples and is evaluated by different annotators, likely with different demographics.\n\nOur aim was to provide evidence that current evaluation is frequently sub-optimal, and the fact we were able to find so many significant oversights in the small remaining sample from top-tier venues we believe strengthens our main points, rather than weakens them. We of course agree that a wider catchment area would better represent the state of the research domain in its entirety, but we do not feel as though this undermines the essential points we have tried to convey about the special consideration that esoteric language should be given at evaluation time.\n\nAdditionally, we agree that experimentation to demonstrate the effects of these variables empirically would be valuable, but regretfully admit that time and resource constraints made this untenable due to the large participant pool that would be required (as having single individuals represent an entire demographic is problematic, and many characteristics may co-occur). \n\nFinally, our examples were designed to be accessible by the majority of the audience so that it is clear how variables that are often taken for granted actually cause differences in interpretations. As we state, simply changing elements of our examples such as through using government officials from less widely known nations than the U.S would immediately present an example of humour where the meaning would be lost on many individuals. We will aim to include a few examples of subculture specific language in the final version, particularly for humour (as sarcasm and irony often require more context specific information, rather than world knowledge, and therefore the difference between a genuine compliment and a sarcastic retort can simply be the person uttering it).\n\nWe also thank you for the recommended reading, and look to incorporate these into the final version where appropriate (especially in regards to Meaney, 2020).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There is a summary of the topics of humor, irony, and sarcasm.\nI believe that the claim (that it is important to refer to demographics) is correct and it is important to raise awareness.", "reasons_to_reject": "The importance of demographic variables within humor is known (e.g., meaney 2020). The survey is based on only 15 papers and is very unrepresentative. The NLP computational humor community is larger than was taken into consideration. For example NLP humor detection in HCI (Shani et al.,  2022) or NLP humor generation in ECIR (Shapira et al., 2023). There are much more venues to take into account.\n\nThere is room for improvement in validating the position with more effective efforts. For example, the inclusion of experiments showcasing the impact of demographics. For instance, employing humor-related datasets and having diverse groups assess them could illustrate varying outcomes.\n\n---\n\nAfter reading the author's response, my main criticism remains.\nI would also want to respond regarding the authors' claim _\"provide evidence that current evaluation is frequently sub-optimal\"_\n\nPointing to a study that did not report demographics  \n1. Does not mean that the study did not take into account the matter of demographics\n2. Does not prove the importance of the report\n\nThe authors did not provide _any_ empirical experiment regarding the importance of demographics. The anecdotal examples the authors provided not convincing enough in my opinion.\n\nThe importance of demographic variables within humor is known (e.g., meaney 2020). The survey is based on only 15 papers and is very unrepresentative. The NLP computational humor community is larger than was taken into consideration. For example NLP humor detection in HCI (Shani et al.,  2022) or NLP humor generation in ECIR (Shapira et al., 2023). There are much more venues to take into account.\n\nThere is room for improvement in validating the position with more effective efforts. For example, the inclusion of experiments showcasing the impact of demographics. For instance, employing humor-related datasets and having diverse groups assess them could illustrate varying outcomes.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The Iron(ic)Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation\n\nThis position paper claims the importance of reporting demographics when using human evaluation for NLG tasks in the domain of humor, irony, and sarcasm. \nThe authors presented an overview of the topics (humor, irony, sarcasm) and gave 1-5 examples for each. \nThe authors review the literature (critical survey), showing that most papers do not include demographics.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Meaney, J. A. (2020, July). Crossing the Line: Where do Demographic Variables Fit into Humor Detection?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (pp. 176-181).\n\nShani, C., Libov, A., Tolmach, S., Lewin-Eytan, L., Maarek, Y., & Shahaf, D. (2022, April). “Alexa, do you want to build a snowman?” Characterizing playful requests to conversational agents. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (pp. 1-7).\n\nShapira, N., Kalinsky, O., Libov, A., Shani, C., & Tolmach, S. (2023, March). Evaluating humorous response generation to playful shopping requests. In European Conference on Information Retrieval (pp. 617-626). Cham: Springer Nature Switzerland.", "Typos_Grammar_Style_and_Presentation_Improvements": "Regarding the examples in Tables 1-3: It would be better if you could emphasize examples that are really impossible to understand without a demographic context. Personally, I was able to understand the humor/sarcasm in all provided 12 examples. It could have been much more powerful and convincing if there had been examples that I would not have understood without explanation. Maybe even divide it by different demographics - which examples will get different scores in different demographics and also prove it with an experiment."}}, {"review_id": "XjvGo4TACn", "reviewer": "Reviewer_Txig", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review and highly appreciate your kind comments towards our work. Thank you for identifying clarity issues, particularly where we have adopted terms from the social sciences and not made sure to define them. To quickly summarise:\n \nL105) “demand characteristics” are clues in an experiment that may accidentally inform a participant of the “real” aim of a study and desirable results, therefore making the participant alter their behaviour so that it falls in line with what they believe the research is trying to find. This may effectively lead to research coming to whatever conclusions the researchers wanted to reach, because the evaluators are not necessarily answering objectively. This is particularly true for crowdsourcing platforms, where workers (whether correctly or not) assume that they are more likely to be chosen for future tasks if they give researchers “better” results.\n\nL120) Here we mean that training materials/guidelines can help to converge participants on a common definition of a certain concept (e.g. sarcasm), but this cannot entirely overpower individual differences, and are therefore not a fool proof solution.\n\nL333) “Power Distance” refers to the distribution of “power” (political, financial etc.) across a society. A lower power distance means that members of a society are more “equal”. Take, for example, a middle class person in a democratic nation such as the US. The power distance between this member of the public and a local politician is likely to be lower than the same social distance would be in a nation run by an authoritarian dictatorship (on average, anyway). We hope to clarify and reformulate these points in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is a position paper but falls very well within the scope of the conference. It is mostly a survey, but it justifies very well the motivation for the study, and results, even if restricted to the particular niche of NLG of humor/irony/sarcasm language, provide useful insight for the NLP community as a whole.", "reasons_to_reject": "I have no reason to reject this paper. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This position paper discusses the significance of human evaluation in Natural Language Generation (NLG) systems, particularly for humor, irony, and sarcasm. In the case of these \"more esoteric forms of language\", the authors perform a critical survey on recent works in NLG to assess how well evaluation procedures are reported in this subdomain, and stress the need for transparency by reporting evaluator demographics and note the overreliance on crowdsourcing for recruitment.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are some minor aspects that could be clarified and improve the paper's already remarkable clarity and insightfulness:\n\n(l. = line)\nl.105: \"demand characteristics in evaluators\": unclear formulation (would you elaborate?)\n\nl.120-3: \"resulting in biased perspectives when evaluating text quality that is only somewhat *overcome* with the provision of training materials.\" : unclear formulation\n\nl.333-5: \"lower power distance (i.e. social hierarchy)... higher power distance\": unclear formulation, particularly, without any qualification social hierarchy.\n"}}, {"review_id": "ve5VVAllpJ", "reviewer": "Reviewer_bfpd", "summary": "", "questions": "While your paper is particularly focused on the assessment of humor, irony and sarcasm, how do you think your recommendations translate to the design of user studies for other domains?\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review, and appreciate your kind words towards our work. We agree that formulating more concrete recommendations or “action points” would be a beneficial way to end the paper, and we intend to include such recommendations in the final version with the remaining space (see the end of our response).\n\nIn terms of the effects on other NLG tasks, the need for transparency is universal, and increased information about evaluator panels would prove beneficial in any domain. Our selected language forms (such as humour) facilitate these claims to be made in the most obvious setting however. We additionally believe our main points translate over to other domains in terms of properly operationalising evaluation criteria (as other studies have looked at), as it is not always only the object of evaluation that is subject to personal interpretation, but often also what evaluators are specifically asked to rate language on (i.e. what is “funny” or “poetic” etc.). The design of specific NLG tasks in a way that avoids particular types of bias and interpretation differences is highly reliant on the exact task at hand, so it’s difficult to make blanket statements other than calling for transparency. \n\nSome of our suggested take-home points are as follows: Provide a short data evaluation statement, inspired by the data statements, such as the one proposed by Bender and Friedman (2018) for datasets annotated for detection tasks. Although a unified template for such statements could be useful, not all demographics are relevant for each task.\n\nSuggested Action Points:\n- Outline no. of evaluators, number of samples seen by each, and the method of recruiting them.\n- For each evaluator, describe demographics:\nage (range/group/generation)\nnative language and proficiency in the target language\nGender\nCultural background (religion, political leaning, education level - The exact types should be chosen as necessary depending on the type of content being evaluated)\nTask proficiency/education -  For binary humour evaluation, one can assume a linguistic education does not have any significant benefits (due to being less representative of the general population) but this may be more relevant when evaluating specific types of humour that a linguist may be better placed to properly identify (for example, Iskender, Polzehl and Moller [2021, HumEval] and Lloret, Plaza and Aker [2018, LRE] demonstrate the preference of expert evaluators in more complex tasks).\n\n- Present the guidelines given to the evaluators\n- Definitions of the evaluated phenomena\n- Modifications made in light of potential pilot studies\n- Conduct a (at the very least minimal) agreement study to investigate the subjectivity of the evaluation task. Although presenting modest agreement may come across as limited evaluation quality, the community should be able to embrace the concept of subjectivity.\nPromote the use of perspectivist approaches that use the different demographic distributions of evaluator panels as a dimension of the research itself, rather than as a footnote detail. These systems are to eventually be used in the real world, and the real world is not homogenous, and nor should our NLP systems be expected to be.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is timely and addresses an important aspect (i.e. lack of transparency in the human evaluation process), it is well written and informative. The conference will certainly benefit from accepting this relevant piece of work. I highly enjoyed reading it!\n", "reasons_to_reject": "While the discussion is very helpful and informative, it would be beneficial for the authors to include a list of concrete recommendations and takeaways for the reader on the design of future user studies that increase transparency in reporting assessments. \n\nIt would also be beneficial if the authors discuss how their findings could be informative for the design of user studies in other NLP tasks. ", "questions_to_authors": "While your paper is particularly focused on the assessment of humor, irony and sarcasm, how do you think your recommendations translate to the design of user studies for other domains?\n", "paper_topic_and_main_contributions": "The paper discusses the role of human evaluation in the assessment of humor, irony and sarcasm generation, leaving from the observation that while human evaluation is preferable to automated evaluation metrics, it is not infallible. The authors discuss how different evaluator demographics and user characteristics affect the interpretation of more esoteric forms of language, nevertheless the reporting of details which highly impact the transparency and replicability of user studies is often overlooked in research papers. The authors present a detailed description of humor, irony, and sarcasm, highlighting subtleties and explaining why they demand special attention in human evaluation studies. Then they survey papers from ACL Anthology published between 2018-2022, with a particular focus on 15 generation papers, with regards to the transparency of human evaluation in their reporting. The authors find that less than half of the papers (only 6 out of 15) report any demographic information, language proficiency, location, or evaluator level of education. Moreover, only 12 out of 15 papers report details on the recruitment and compensation of the annotators, while less than half include statistical tests for agreement levels. The authors conclude with the recommendation to pay special attention to the construction of human evaluation processes.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a position paper arguing that characteristics of evaluators (i.e., demographic characteristics) are especially important for generation of language such as humour, irony, and sarcasm.

Overall, this paper makes a valuable contribution. Reviewers agree that the paper’s core claim is well-motivated, and that it addresses a critical issue (lack of transparency with regards to human evaluators) and raises important awareness on the issue. Reviewers’ concerns are generally well-addressed by the authors’ responses; one reviewer suggests that the work would be strengthened by a list of concrete takeaways, which the authors provide in their response, while another reviewer’s concern about the scope of the literature review is addressed by an extended response detailing the review methodology.

Finally, one reviewer raises concerns that the position paper requires an empirical experiment in order to justify its claim (i.e., that evaluators’ demographic characteristics impact outcomes), on the grounds that the paper’s analysis — which provides examples showing that different demographic characteristics impact interpretations of humour, irony, and sarcasm, and further shows that most papers in the meta-analysis do not mention demographic characteristics — is not convincing. While I agree that such an empirical experiment might be interesting, I see it it as a different analysis (for another paper, perhaps) and not one that is required for this paper. While it’s perhaps true that a study that does not report demographics (as is the case for most of the studies examined by this paper) may still have taken demographics into account or may have deemed demographics irrelevant to its analysis, the burden is on such a study to report this thinking, and not on readers to assume that it was present — therefore, in my view this paper’s analysis showing a distinct lack of reporting in the literature, as well as examples illustrating that demographics are indeed often relevant, seems to justify its claim.
Decision: Accept-Findings