SlowGuess/NAACL_2022_OCR huggingface dataset:
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020b. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8689-8696.
Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm.
Yu-Ping Ruan, Zhen-Hua Ling, Jia-Chen Gu, and Quan Liu. 2020. Fine-tuning bert for schema-guided zero-shot dialogue state tracking.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems.
Yang Zhang, Vahid Noroozi, Evelina Bakhturina, and Boris Ginsburg. 2021. Sgd-qa: Fast schema-guided dialogue state tracking for unseen services.
Jeffrey Zhao, Raghav Gupta, Yuan Cao, Dian Yu, Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. 2022. Description-driven task-oriented dialog modeling.
Jeffrey Zhao, Mahdis Mahdieh, Ye Zhang, Yuan Cao, and Yonghui Wu. 2021a. Effective sequence-to-sequence dialogue state tracking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7486-7493, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021b. Calibrate before use: Improving few-shot performance of language models.
A Prompt Design
We experimented with various formats for the SDT prompt before arriving at the final format. Below, we list alternative designs that we tried and their impact on JGA, as evaluated on the SGD test set.
A.1 Categorical value strings vs. multiple choice answers
We found that JGA dropped  $-2\%$  when we tasked the model with decoding categorical values instead of multiple choice answers - e.g. payment_method  $\equiv$ debit card instead of payment_method  $\equiv$ b (where b is linked to the valuedebit card in the prompt as described in Section 2). When tasking the model to decode categorical values, it would often decode related yet invalid values, which we counted as false in our evaluation. For example, instead ofdebit card, the model might decode bank balance.
A.2 Slot IDs vs. slot names
When we delexicalized slot names with slot IDs, JGA dropped  $-5\%$ . One downside of this approach is that the model lost access to valuable semantic information conveyed by the slot name. Another downside is that the model could not distinguish two slots that had the same value in the prompt. For example, if the prompt was "I would like a pet-friendly hotel room with wifi" and the corresponding slots were  $1 = \text{True}$  (has_wifi) and  $2 = \text{True}$  (pets Allowed), it is ambiguous which ID refers to which slot.
The potential upside of using slot IDs was to remove dependence on the choice of slot name, but this did not succeed for the reasons above.
A.3 Decoding active slots vs. all slots
We experimented with training the model to only decode active slots rather than all slots with none values when they were inactive. JGA dropped  $-0.4\%$ , which we hypothesized might be a result of greater dissimilarity between the slot-value string in the prompt (which contained all slots by construction) and the target, which only contained a subset of slots.
A.4 In-line annotations vs. dialogue+slots concatenated
We hypothesized that bringing the slot annotation in the prompt closer to where it was mentioned in the dialogue might help the model better understand the slot's semantic meaning. We changed the format as follows:
- Original: [ex] [user] I would like a pet-friendly hotel room with wifi [system] I found ...
[slot] has_wifi=True
- In-line: [ex] [user] I would like a pet-friendly hotel room with wifi [has_wifi=True] [system] I found ...
However, this decreased JGA by more than  $-20\%$ . We hypothesized that this was likely due to a mismatch between the prompt's annotations and the target string format, which we did not change.
B SDT Model Details
We used the publicly available T5 checkpoints<sup>2</sup>. For all experiments, we used a sequence length of 2048,  $10\%$  dropout and a batch size of 16. We used a constant learning rate of  $1e - 3$  or  $1e - 4$ . All models were trained for 50k steps or until convergence, and each experiment was conducted on either 64 or 128 TPU v3 chips (Jouppi et al., 2017).
C Baseline Models
For SGD, we compare against SGP-DST (Ruan et al., 2020), MRC+WD-DST (Ma et al., 2019), T5-seq (Zhao et al., 2022) and T5-ind (Lee et al., 2021a).
For MultiWOZ, we compare against TRADE (Wu et al., 2019), SUMBT (Lee et al., 2019), TransferQA (Lin et al., 2021a), and T5-seq. Transfer QA is based on T5-large.
Model	All	Seen	Unseen
SDT-seq + desc	88.6±0.9	95.7±0.5	86.2±1.0
SDT-seq	88.8±0.5	95.8±0.2	86.4±0.7
Table A1: We experiment with prompting using both descriptions and demonstrations (SDT-seq + desc) vs. demonstrations-only (SDT-seq) and find that adding descriptions does not improve performance.
Figure 3: Results of secondarily finetuning T5-seq with dialogues, to help understand whether prompting or finetuning is more effective. The examples used for finetuning are derived from the set of dialogues used as prompts across the 5 trials of SDT-seq. From this, we observe that prompting with a single dialogue demonstration outperforms few-shot finetuning.
$^{2}$ https://github.com/google-research/text-to-text-transfer-transformer/blob/main/releases_checkpoints.md
Figure 4: The original schema for a Payment service alongside its closest  $(v_{1})$  and farthest  $(v_{5})$  SGD-X variants, as measured by linguistic distance functions. For the SGD-X benchmark, models are trained on the original SGD dataset and evaluated on the test set, where the original test set schemas are replaced by SGD-X variant schemas.