SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 1 lists the number of intents, slots and users goals in the three datasets used in the evaluation. And Table 3 shows all annotated dialogue acts and slots in details. Task-oriented dialogue systems are designed to help users to complete a specific goal  $G$ . Even though the dialogue system knows nothing about the user goal explicitly, the whole dialogue progresses around this user goal  $G$  implicitly. In order to explain the user goal better, we take a user goal as an example from the movie domain:
$$
\begin{array}{r} \mathbf {G o a l} = \left(C = \left[ \begin{array}{l} \text {m o v i e n a m e} = \text {E n t e r} \\ \text {t h e D r a g o n} \\ \text {a c t o r} = \text {B r u c e L e e} \\ \text {d a t e} = \text {t o d a y} \end{array} \right], \right. \\ R = \left[ \begin{array}{l} \text {t h e a t e r} = \\ \text {s t a r t t i m e} = \end{array} \right]. \end{array} \tag {10}
$$
In this user goal, a user inquires the dialogue system about the theater and start time of a today's movie about the Enter the Dragon by Bruce Lee. The user goals are generated from the annotated datasets mentioned in Table 3. The user goals extracted from the same dataset are then aggregated into a user goal set for that task. The user goals extracted from the same dataset are then aggregated into a user goal set for that task. When running a dialogue, the user simulator (Li et al., 2016) randomly samples a user goal from the user goal set to converse with the dialogue system. Helping the user to achieve specific user goals is the task to complete for dialogue systems. In this paper, we use the success rate and averaged reward as our main evaluation criteria. We do not use averaged turns into our criteria because overestimation bias mainly prevents the dialogue system from completing a task in a dialogue. This is explicitly with success rate and averaged reward, and this is not directly related with averaged turns. If and only if the dialogue system recognizes all constraints provided by users and informs all information that users want, and finally books the desired tickets successfully, the user goal is viewed as successful, and the dialogue policy received positive reward for success. The averaged reward means the averaged cumulative discounted reward received by dialogue system per dialogue.
The size of the experience replay pool in the movie domain and other domains is set to 8000 and 10000, respectively. The number of target networks in Averaged DQN, Maxmin DQN and SUNRISE is set to 4. The temperature parameter of SUNRISE is set to 2. The target network update period for Averaged DQN is set to 4. In the experiment, we use a user simulator to interact with dialogue systems. In the movie domain, the dialogue system receives a 2L reward if the dialogue finishes successfully and receives -L if it fails. Also, a fixed reward (-1) is given to the dialogue system for each dialogue turn. In the restaurant and taxi domains, the dialogue system receives a 2L reward if the dialogue finishes successfully and receives 0 if it fails. Also, a fixed reward (0) is given to the dialogue system for each dialogue turn. Under this setup, the dialogue datasets for experiments have varieties (Li et al., 2016).
Task	Intents	Slots	Dialogues
Movie	request,inform, confirm_question, confirm_answer, greeting,closing, deny,not_sure, multiple_choice, thanks,welcome	city,closing, data,greeting, distanceconstraints, movename,price, numberofpeople, starttime,state, taskcomplete,theater, teater_chain,ticket, video_format,zip	280
Restaurant	request,inform, confirm_question, confirm_answer, greeting,closing, deny,not_sure, multiple_choice, thanks,welcome,	address,atmosphere, choice,city,closing, cuisine,date,food, dress_code,greeting, distanceconstraints, numberofkids,mealtype, numberofpeople, other,personfullname, phonenumber,pricing, rating,restaurantname, restauranttype,seating, starttime,state,zip, result,occasion, taskcomplete,reservation	4103
Taxi	request_inform, confirm_question, confirm_answer, greeting,closing, deny,not_sure, multiple_choice, thanks,welcome,	car_type,city,speed, closing,car_level,date, distanceconstraints, dropout_location, zip,result,numberofkids, greeting,name,driver_id, numberofpeople,other, pickup_location,state, dropout_location_city, pickup_location_city, pickup_time,cost, taxi_company,mc_list, taskcomplete,taxi,budget, emergency degree,drive_level	3094
Table 3: The details of the datasets. (Li et al., 2016, 2018)