SlowGuess/NAACL_2022_OCR huggingface dataset:
For all our experiments, we used a p3.2xlarge EC2 instance. This instance has one GPU with 16GB capacity and 61GB of RAM. Out of all of our experiments, simulated long-term success rate experiments took the most amount of GPU hours to run. Altogether it took somewhere between 24-36 GPU hours to complete all the experiments.
A.4 Human evaluation setting details
We recruited four trained annotators to evaluate generated conversations on the following five metrics, each on a scale of 1 to 5. We split up the examples across the four annotators such that each example was judged by two annotators. We included the ground truth conversation as an additional baseline to act as an upper bound. To ensure the ratings would be high-quality, we provided a rubric, included below, for each metric with examples for different ratings, did an initial pilot for a few sample conversations, and provided a reference sheet to help calibrate the ratings across annotators.
A.4.1 Rubric
Evaluate generated conversations on a few metrics, each on a scale of 1 to 5:
[utterance-level] Fluency: Is this response fluent and grammatical?
• 1: Generated responses do not make any sense, English-wise and grammar-wise, which could include misspelled words, no transition words, limited punctuation, skipped words, etc (e.g., “the fight help order”)
- 3: Generated responses have some good English so you can make out what is being said but it's not well-formed sentences (e.g., "will you help order")
- 5: Generated responses have perfect English and perfect grammar. Customers can use lower-case text as less-formal style so first-letter capitalization is not necessary (e.g., "can you help me refund my order?")
[utterance-level] Relevance: Is this response a natural reply to the previous utterance in the conversation?
1: The generated response is not at all relevant to the conversation context/history (e.g., when asked for account id: "I can't get my promo code")
- 3: The generated response is somewhat relevant to the conversation context/history but not the best fit (e.g., when asked for account id: “No”)
- 5: The generated response is perfectly relevant and a great response to the conversation context/history (e.g., when asked for account id: “Account ID: 3425435”)
[utterance-level] Control-consistency: If control words appear in this response, are they appropriately used?
- 1: When used, the control words (which are uppercased) make no sense in the generated responses. They are fully forced into the responses (e.g., "TODAY account id: 435650")
3: When used, the control words (which are uppercased) make some sense in the generated responses but are not super smooth (e.g., "I need help with my order, can you help TODAY?")
- 5: When used, the control words (which are uppercased) are perfectly and naturally used in the generated responses (e.g., "TODAY, I want to buy a shirt. Can you help me?")
[conversation-level] Diversity: Is there diversity in the customer responses of the conversation?
1: Almost all of the responses are repetitive and have no diversity (e.g., “ok” “ok” “ok” “thanks”)
- 3: Some of the generated responses provide diversity while many do not (e.g., "I want to buy a shirt" "can you help me with this?" "ok" "thanks")
- 5: All of the generated responses are diverse and provide a variety of interesting words through the conversation. The customer can still say ok and thanks but it shouldn't happen all the time and has to be appropriate for that point in the conversation (e.g., "I want to buy a shirt" "can you help me with this?" "ok sure" "thank you very much for your help!")
[conversation-level] Coherence: Are all of the customer responses in the conversation coherent with respect to each other?
1: All of the responses have very little relation when seen together (e.g., "I want to buy a shirt", "can you help me with my promo code", "my subscription expired")
- 3: Some of the generated responses in a conversation make sense together and some don't (e.g., "I want to buy a shirt" "I have a promo code for buying this shirt" "my subscription expired")
- 5: All generated responses make complete sense together (e.g., "I want to buy a shirt", "I have a promo code for buying this shirt", "thank you for helping me with this purchase!")
FOP-guided
Control words to generate: valid credit make minute try card purchase trying worked
Speaker	Utterance
AGENT	Hello, thank you for contacting AcmeCorp, how may I help you today?
CUSTOMER	Hi I was trying to make a purchase but for some reason my credit card keeps getting rejected
AGENT	I would happy to help.
CUSTOMER	I got an email from you guys about a great sale and it listed this code and it said it was invalid
AGENT	Lets try a few things. First can you check the expiration date on your card.
CUSTOMER	No its valid thats why I am confused.
AGENT	Can you try to re-enter the card information one more time?
CUSTOMER	Ok just a minute please
AGENT	Alright, let me know if that helps.
CUSTOMER	Ok I did its s ordering right now. That worked and it says it is correct.
AGENT	Great, is there anything else I can help you with today? Is there anything else?
CUSTOMER	That is all I needed thanks
AGENT	Your welcome. Thank you for contacting us. Have a great day.
CUSTOMER	you too
Table 5: Example simulation on the ABCD dataset using the FOP-guided method
FOP-retrieval
Control words to generate: valid credit make minute try card purchase trying worked
Speaker	Utterance
AGENT	Hello, thank you for contacting AcmeCorp, how may I help you today?
CUSTOMER	Hi! I was trying to make a purchase but for some reason my credit card keeps getting rejected
AGENT	I would happy to help. Lets see if we can troubleshoot this issue.
CUSTOMER	ok let me know what i should try
AGENT	Lets try to make sure your card is not expired.
CUSTOMER	no. its valid. That's why I am confused
AGENT	First can you try to log out and log back in.
CUSTOMER	Ok let me give that a shot.
AGENT	Okay, next can you check the expiration date on your card.
CUSTOMER	Ok hang on one minute let me try that
AGENT	No problem.
CUSTOMER	Tried that. It didn't work either
AGENT	Can you try to check the expiration on your card?
CUSTOMER	Let me try that
AGENT	Alright.
CUSTOMER	Well that kinda helped. I do not know why
Table 6: Example simulation on the ABCD dataset using the FOP-retrieval method
DBS
Control words to generate: valid credit make minute try card purchase trying worked
Speaker	Utterance
AGENT	Hello, thank you for contacting AcmeCorp, how may I help you today?
CUSTOMER	I would like to know if there is any reasonable way for validating an order?
AGENT	I would happy to look into this for you.
CUSTOMER	I already got a credit for a return and would like to correct it.
AGENT	May I have your full name or account id please?
CUSTOMER	Crystal Minh
AGENT	Thank you Crystal.
CUSTOMER	Can you make sure that this is correct?
AGENT	One moment please.
CUSTOMER	ok
AGENT	What is your username, email address, and Order ID?
CUSTOMER	crystalm minute
AGENT	Thanks Crystal. I need to verify your purchase.
CUSTOMER	crystalm minute@email.com
AGENT	Your refund is in progress and you initiated it online.
CUSTOMER	Okay. I will try it out now.
Table 7: Example simulation on the ABCD dataset using the DBS method
CGMH
Control words to generate: valid credit make minute try card purchase trying worked
Speaker	Utterance
AGENT	Hello, thank you for contacting AcmeCorp, how may I help you today?
CUSTOMER	Myvalid credit card make me purchase minute we try to credit card to purchase. Have been trying this worked
AGENT	I would happy to look into this for you.
CUSTOMER	I was wondering about this method ofvalid
AGENT	May I have your full name or account id please?
CUSTOMER	sure its avalid
AGENT	Thank you may I please have your full address?
CUSTOMER	9 Guessvalid
AGENT	What method of payment would you like to use?
Table 8: Example simulation on the ABCD dataset using the CGMH method
Prompt
Control words to generate: valid credit make minute try card purchase trying worked
Speaker	Utterance
AGENT	Hello, thank you for contacting AcmeCorp, how may I help you today?
CUSTOMER	Hi I'm having a problem with my account
AGENT	I would happy to help.
CUSTOMER	The credit card keeps getting rejected
AGENT	Lets see if we can try a few things.
CUSTOMER	ok
AGENT	Can you try entering the information again?
CUSTOMER	Okay I did but it still doesn't work
AGENT	Next can you try logging out and back in?
CUSTOMER	Ok hold on
AGENT	Let me know if that helps.
CUSTOMER	hey that worked!
Table 9: Example simulation on the ABCD dataset using the Prompt method