SlowGuess/NAACL_2022_OCR huggingface dataset:
[SQ33] In general, for the following statements, please answer if you agree or disagree with each statement for NLG systems' evaluation. Strongly agree, agree, neither agree or disagree, disagree, strongly disagree
- It's possible to develop automatic metrics that can reliably evaluate the fairness and inclusion of a system.
- Fairness and inclusion metrics are only relevant for NLG systems that are deployed to real users.
- Fairness and inclusion is a separate research area from NLG evaluation.
- The language an NLG system or model generates should adhere to standard grammar rules of the language the content is generated in. E.g., an English model should adhere to the grammar rules of standard English.
- Fairness and inclusion is less important for academic prototypes.
- Fairness and inclusion of a system can only be effectively measured in a real deployed setting.
[SQ34] Do the training or evaluation dataset(s) you use for your system or model capture a particular voice? By voice we mean that it may capture a particular style of speaking or writing, or a particular persona.
- They capture a non-neutral, specific voice (a particular persona or style).
- They capture an assemblage of voices.
- They capture a neutral voice.
- They do not capture any voice.
- They could be capturing a voice but I'm not sure what it is.
[SQ35] Do you aim for your system or model to capture a particular voice?
- We aim for it to capture a non-neutral, specific voice (a particular persona).
- We aim for it to capture an assemblage of voices.
- We aim for it to capture a neutral voice.
- We aim for it to capture no voice.
- We do not aim for it to capture any voice.
[SQ36] Does your system or model capture a particular voice?
- It captures a non-neutral, specific voice (a particular persona or style).
- It captures an assemblage of voices.
- It captures a neutral voice.
- It does not capture any voice.
- It could be capturing a voice but I'm not sure what it is.
[SQ37] Is neutrality of voice something your system or model should embody?
- Definitely yes, yes, maybe, no, definitely no, I don't know
B.9 Final thoughts
[SQ38] How do you think NLG evaluation could be improved? R: Write-in
[SQ39] How would you characterize your experience with practices and metrics aimed at ethical AI? This does not have to be related to your NLG work.
- No, I have no experience, I have a bit of experience, I have worked on ethical AI related issues, Ethical AI is the focus of my work
[SQ40] If you would like enter the raffle drawing for one of the ten $50 amazon gift cards, for anonymity purposes, after submitting this form you will be provided with a link to another form to fill in your email address and enter the raffle. For this, please also write down a key phrase here, which you will also be asked to re-enter on the raffle form. We will only be use this key phrase to validate that the raffle participants have completed the survey. Please don't use a key phrase that is associated any accounts. R: Write-in
C Comprehensive Survey Results
Figure 1: Question 6: What type of NLG system(s) or task(s) have you worked on or with? Check all that apply.
Figure 2: Question 13: When you evaluate your system/model, what are the goals of this evaluation? Check all that apply.
Figure 3: Question 14: Of the following, which are the most important performance criteria to evaluate for your NLG system or task? Select up to three or write your own. For more detailed definitions, please refer to the appendix here: https://aclanthology.org/2020.inlg-1.23.pdf
Figure 4: Question 16: Is your work (or will your work be) part of an NLG system that is deployed to users in real settings (users who use the system for their own reasons, self-incentivized, and are not paid as test subjects/crowd workers)?
Figure 5: Question 17: Is your goal or one of your future goals to personalize the voice of your system for users? By voice we mean that it may capture a particular style of speaking or writing, or a particular persona.
Figure 6: Question 18: For the following statements, please answer how often you or others on your team do the following.
Figure 7: Question 20: For the system or task, you described earlier, do you agree or disagree that the following evaluation methods are reliable ways to assess performance?
Figure 8: Question 22: In general, please answer if you agree or disagree with the following statement about evaluating NLG systems or tasks
Figure 9: Question 23: In general, please answer how often the following statements are true:
Figure 10: Question 24: Rank which of the following resources currently limit the evaluation for your particular system or task the most. Top choice being the most limiting resource. Visualizing top 3 choices.
Figure 11: Question 26: Rank which of the following considerations currently guide your evaluation practices the most. Top choice being the most influential consideration. Visualizing top 3 choices.
Figure 12: Question 30: Do you intentionally try to block certain type of content from your generation? (e.g., by using blocklists or classifiers, cleaning the training data, etc.)
Figure 13: Question 31: For the following questions, for your system, how do you measure the following? Fairness and Inclusion states that NLG systems should treat all people equally, empowering and engaging everyone by providing equal benefit and access to e.g., opportunities and resources.
Figure 14: Question 33: In general, for the following statements, please answer if you agree or disagree with each statement for NLG systems' evaluation.
Figure 15: Questions 34 - 37 on whether a NLG system or model captures a particular voice. By voice we mean that it may capture a particular style of speaking or writing, or a particular persona.
Figure 16: Question 39: How would you characterize your experience with practices and metrics aimed at ethical AI? This does not have to be related to your NLG work.