SlowGuess/NAACL_2022_OCR huggingface dataset:
Figure 6 shows our annotation interface to add knowledge-grounded chit-chat to TOD. The left part shows the full dialogue, where the annotators can click and expand each turn to make the chitch chat enrichment. The right part shows all the entities with the associated knowledge snippets. The annotators can click on each entity name to expand
Metrics	SimpleToDPlus win (%)	SimpleToD win (%)	Tied (%)
Engagingness	40.0	30.3	29.8
Interestingness	31.8	19.5	48.8
Knowledge	38.0	18.3	43.8
Humanness	38.3	26.8	35.0
Table 9: Human evaluation of SimpleToDPlus vs. SimpleToD.
Metrics	Combiner win (%)	SimpleToD win (%)	Tied (%)
Engagingness	34.8	33.5	31.8
Interestingness	27.0	22.5	50.5
Knowledge	32.5	23.0	44.5
Humanness	27.8	32.5	39.8
Table 10: Human evaluation of Combiner vs. SimpleToD.
the textbox to see the knowledge snippets. We add index number to each knowledge snippet (shown in green brackets), and the annotators are asked to write down the indexes of the knowledge snippets they used for writing the knowledge grounded chitchat. Figure 7 shows one example annotation turn using our interface.
Appendix B: Model and Training Details
All the implementations are based on the Huggingface Transformers library<sup>3</sup>. For all models, we use the Adam optimizer (Kingma and Ba, 2015). For the knowledge selection model, we use BERT-base with learning rate of 3e-5 and batch size of 16. For the baseline SimpleToD model, SimpleToDPlus model, and Combiner model, we all use learning rate of 1e-4 and batch size of 16. All the experiments are done using TESLA M40 GPU cards.
Appendix C: Evaluation Details
Table 9 and 10 show the human evaluation results of SimpleToDPlus vs. SimpleToD, and Combiner vs. SimpleToD, respectively.
Enrich Task-Oriented Dialog With Entity Knowledge
Dialog 15_00100
Pre-check: (required)
Turn 1
User: Can you get me a hotel?
System: Where are you going.
Rewrite knowledge-enriched response for turn 1
Turn 2
User: I need a room in Los Angeles.
System: Try this 2 star one: AC Hotel by Marriott Beverly Hills.
Rewrite knowledge-enriched response for turn 2
Rewrite knowledge-enriched response
Knowledge snippets selected
Turn 3
User: Where exactly are they? How much will this set me back? System: $120 a night. Go to 6399 Wilshire Boulevard.
Rewrite knowledge-enriched response for turn 3
User: Alright. Thanks.
System: Do you want me to got ahead with the reservation?
Rewrite knowledge-enriched response for turn 4
Turn 5
User: No. Get me a full-size car there until March 12th. System: From March 6th? Time?
Rewrite knowledge-enriched response for turn 5
About: Los Angeles
[1] The written history of Los Angeles city and county began with a small Mexican town that changed little in the three decades after 1848, when California became part of the United States.
[2] Much greater changes came from the completion of the Santa Fe railroad line from Chicago to Los Angeles in 1885.
[3] Immigrants flooded in, especially white Protestants from the Midwest.
[4] Los Angeles had a strong economic base in farming, oil, tourism, real estate and movies.
[5] It grew rapidly with many suburban areas inside and outside the city limits.  
[6] Hollywood made the city world famous, and World War II brought new industry, especially high-tech aircraft construction.
[7] Politically the city was moderately conservative, with a weak labor union sector.
[8] Since the 1960s growth has slowed—and traffic delays have become famous.
[9] LA was a pioneer in freeway development as the public transit system deteriorated.
[10] New arrivals, especially from Mexico and Asia, have transformed the demographic base since the 1960s.
[11] Old industries have declined, including farming, oil, military and aircraft, but tourism, entertainment and high tech remain strong.
[12] Los Angeles, Spanish for "The Angels"), officially the City of Los Angeles and often known by its initials L.A., is the second-most populous city in the United States (after New York City), the most populous city in California and the county seat of Los Angeles County.
[13] Situated in Southern California, Los Angeles is known for its mediterranean climate, ethnic diversity, sprawling metropolis, and as a major center of the American entertainment industry.
[14] Los Angeles lies in a large coastal basin surrounded on three sides by mountains reaching up to and over.
[15] Historically home to the Chumash and Tongva, Los Angeles was claimed by Juan Rodríguez Cabrillo for Spain in 1542 along with the rest of what would become Alta California.
[16] The city was officially founded on September 4, 1781, by Spanish governor Ettino de Navas.
Figure 6: Our annotation interface example 1.
Valid example
by Marriott Beverly Hills. People can often meet celebrities there!
User: Where exactly are they? How much will this set me back?
System: $120 a night. Go to 6399 Wilshire Boulevard.
About: Ac Hotel By Marriott Beverly Hills
[28] Beverly Hills is a city in Los Angeles County, California, United States, surrounded by the cities of Los Angeles and West Hollywood.
[29] Originally a Spanish ranch where lima beans were grown, Beverly Hills was incorporated in 1914 by a group of investors who had failed to find oil, but found water instead and eventually decided to develop it into a town. [30] By 2013, its population had grown to 34,658.
[31] Sometimes referred to as "90210", one of its primary ZIP codes, it was home to many actors and celebrities throughout the 20th century.
[32] The city includes the Rodeo Drive shopping district and the Beverly Hills Oil Field.
[33] Gaspar de Portola arrived in the area that would become Beverly Hills on August 3, 1769, travelling along native trails which followed the present-day route of Wilshire Boulevard.
The Beverly Hills Hotel
[34] The Beverly Hills Hotel, also called "The Beverly Hills Hotel and Bungalows", is located on Sunset Boulevard in Beverly Hills, California.
[35] One of the world's best-known hotels, it is closely associated with Hollywood film stars, rock stars and celebrities.
[36] The hotel has 208 guest rooms and suites, and 23 bungalows, each designed in the peachy pink and green colors which are a trademark of the hotel.
Figure 7: Our annotation interface example 2.