SlowGuess/NAACL_2022_OCR huggingface dataset:
For note-taking purposes, I will record the audio of this call. I will ask you to share your screen while you are trying out our new tools, and I will also record your screen. Feel free to turn off your camera if you don't like your image to be recorded.
Any questions before we start?
Is it okay if I start studying now?
[if they say yes, start recording]
3. Interview
3.1 Opening
To start, could you please tell me a little bit about yourself?
What experience do you have with writing and editing?
3.2 Reflection on warm-up
Could you please walk me through your experience writing the summaries?
- How did you approach the tasks?
What were some strategies that you took?
How did you decide what to include?
○ How did you decide if a summary is done and good to go?
What went well?
What did not go well?
What were some challenges?
How do you feel about summarizing the different articles?
Which ones were easy? Why
Which ones were difficult? Why?
- How did you approach the different articles differently?
[Pick a the task that they thought as most difficult]
Could you tell me more about your experience finishing this task? Walk me through how you did it.
- Why was this task particularly difficult?
- Did you approach this task differently than other ones?
If yes, how? And why?
If not, how did your strategy with previous articles work or not work?
What went well?
What did not go well?
What are some challenges?
What kind of support do you wish to have?
Do you want to have the same support for the other tasks? Why or why not?
Did you take any notes or use any external tools or resources?
In general, what kind of support do you wish you could have while summarizing articles?
- Do you want to have different support for different types of articles? Why or why not?
3.3 Interaction with prototypes
Now I am going to have you try out SumAssists, which is a collection of digital tools that we developed to support people to write summaries. I will ask you questions as you try out the tools. Note that some features of the tools are not fully developed yet, so for some parts I need to ask you to imagine your interaction with some features.
I'm now asking you to share your screen to show me your interaction with the tools.
[send them the link to the Figma prototype]
Now you are seeing the original article. This is the same news article about the Massachusetts heath website that you have worked on in Part 1 of the study. You can see 5 different tools that are designed to support you to write the summary for this article.
3.3.1 Selecting or rating model output
[ask the participant to interact with the tool]
What's your first impression of this tool?
- What do you think it is for?
- How do you want to use it?
This tool provides you three options of AI-generated summaries of this article. You will be able to select the one that you think is the best.
Now please imagine that you are summarizing the article using this tool. Please talk through step-by-step how you are going to do it.
What do you like about this tool? Why?
What do you dislike about this tool? Why?
What is helpful with your summarization process? In what ways?
- What is unhelpful with your summarization process? In what ways?
- How much power of control with the summarization do you feel that you have using this tool? Do you like it this way? Why?
- How much will you rely on this tool while doing the summarization task? Do you like it this way? Why?
What type of documents will you rely on this tool?
How much would you use it
Any cases that you wouldn't want to use it
What case you want to totally override it
- How much will you trust the AI-generated summaries using this tool? Do you like it this way? Why?
If we are making this tool available in real life, in what situation do you see yourself using this tool?
Imagining you are summarizing the Reddit posts using this tool, how would your experience be different or similar?
Imagining you are summarizing the government bills using this tool, how would your experience be different or similar?
How would you like to improve this tool?
Let's go back to the main page by clicking on the arrow on the upper right corner.
3.3.2 Post-editing
[Debrief them about the tool] This tool provides you with an AI-generated summaries of this article that you can edit on. In the real world you will be able to type and delete any words in this text box on the right just as you are doing normal editing. Right now our implementation doesn't support that. Just imagine you can type and edit the summary.
What do you like about this tool? Why?
What do you dislike about this tool? Why?
- What is helpful with your summarization process? In what ways?
- What is unhelpful with your summarization process? In what ways?
- How much power of control with the summarization do you feel that you have using this tool? Do you like it this way? Why?
- How much will you rely on this tool while doing the summarization task? Do you like it this way? Why?
- Compared to the previous tool, which one do you think would be more helpful? Why?
- Imagining you are summarizing the Reddit posts using this tool, how would your experience be different or similar?
- Compared to the previous tool, how do you think this tool would be more or less helpful in this situation?
- Imagining you are summarizing the government bills using this tool, how would your experience be different or similar?
- Compared to the previous tool, how do you think this tool would be more or less helpful in this situation?
- If the AI is doing a reasonable job, for longer docs that you may not know as much
- Are there other text that this tool will be helpful for you to summarize
Let's go back to the main page
3.3.3 Interactive editing
[Debrief them about the tool] This tool provides you with an AI-generated summaries of this article. You will be able to edit the first sentence of the summary by selecting in the drop down menu. Once you make the selection and click predict, the rest of the summary will be updated based on your edits on the first sentence.
What do you like about this tool? Why?
What do you dislike about this tool? Why?
- What is helpful with your summarization process? In what ways?
- What is unhelpful with your summarization process? In what ways?
- How much power of control with the summarization do you feel that you have using this tool? Do you like it this way? Why?
- How much will you rely on this tool while doing the summarization task? Do you like it this way? Why?
- How much will you trust the AI-generated summaries using this tool? Do you like it this way? Why?
- Compared to the previous tool, which one do you think would be more helpful? Why?
3.3.4 Guiding model output
[Debrief them about the tool] This tool provides you the power to tell the AI model to generate the kind of summaries that you'd like to see. You can change the length of the summary, the style of the summary, and also highlight parts of the original sentences that you'd like to see in the summary by highlighting them in the original text.
3.3.5 Writing with model assistance
Now this tool is embedded in the Google doc. For this one, please type the summary that you'd like to write for this article.
[Participant type summary]
[SumAssist Bot copies a sentence in gray color once they finish the first sentence]
[SumAssist Bot adds a comment on their first sentence]
If you think the sentence in gray is good, go ahead and keep writing. If you want to edit it, also go ahead. If you don't like it, you can just delete it and write.
[ask the participant to stop]
This tool is a bot that we built in Google doc and it can suggest next sentences in gray text. It can also comment on the sentence that you wrote as you are writing.
Imagine that you are writing the entire summary with this bot. What do you think the experience would be like?
- What is helpful with your summarization process? In what ways?
- What is unhelpful with your summarization process? In what ways?
What do you like about this tool? Why?
What do you dislike about this tool? Why?
How do you feel about the auto suggestions?
How could the auto suggestion be more useful?
How do you feel about the auto commenting?
How could the auto commenting be more useful?
- Would the auto suggestions and comments be annoying? Why and why not?
- How much power of control with the summarization do you feel that you have using this tool? Do you like it this way? Why?
- How much will you rely on this tool while doing the summarization task? Do you like it this way? Why?
- How much will you trust the AI-generated summaries using this tool? Do you like it this way? Why?
- Compared to the previous tool, which one do you think would be more helpful? Why?
3.4 General Reflection
In general, which tools that you would most likely be using when you are summarizing articles? Why?
In general, which tools that you would least likely be using when you are summarizing articles? Why?
- For Reddit, news, and government bills respectively?
If you could snap your fingers and create a summarization assistant to help you, how would you like to combine some of these tools?
- And what other features would you like to have?