SlowGuess/NAACL_2021_OCR huggingface dataset:
While the above experiments focus on proving the base performance of the proposed response generation models (HIER, HIER++, HIER-CLS, and ablations), HT-Encoder can be applied to any model that uses a standard transformer encoder. Hence, in a final experiment (Table 6), we integrate HT-Encoder with an existing state-of-the-art model Marco. We replace the standard transformer in
$^{7}$ https://github.com/budzianowski/multiwoz  
 $^{8}$ A system with two Tesla P100 GPUs were used for training.
Models	Evaluation Metrics
BLEU	Entity-F1	Inform	Success	Score
HRED	17.50	-	70.7	60.9	83.3
TokenMoE	16.81	-	75.30	59.70	84.31
Transformer	19.1	55.1	71.1	59.9	84.60
SET	18.67	51.61	76.80	57.69	85.92
MAT	18.86	54.89	71.9	52.5	81.06
HIER	20.91	54.45	73.60	60.10	87.76
Table 2: Simplest Baselines in absence of both Belief or Policy / Dialog Act annotations
Models	Pretraining	Annotations	Evaluation Metrics
Belief	DB	Policy	BLEU	Entity-F1	Inform	Success	Score
SimpleTOD	GPT-2	Oracle	Oracle	Oracle	17.78	-	93.4	83.2	106.08
SimpleTOD	GPT-2	Oracle	-	Oracle	18.61	-	92.3	85.8	107.66
HDSA	-	Oracle	Oracle	Oracle	30.4	86.2	87.9	78.0	113.4
DAMD	-	Oracle	Oracle	Oracle	27.3	-	95.4	87.2	118.5
SET++	-	-	-	Oracle	25.56	82.27	85.7	74.3	105.56
HIER++	-	-	-	Oracle	29.54	85.01	88.3	85.4	116.39
HIER-CLS	-	-	-	Oracle	29.29	84.23	88.3	85.9	116.39
Table 3: Context-to-Response generation with Oracle Policy. Superior Performance of DAMD: DAMD always receives an extra input of  $B_{t - 1}$  annotation, while predicting for  $B_{t}$  or response  $R_{t}$ , which helps in NLU of the subsequent utterances. This is not available in any other models.
Models	Pretraining	Annotations	Evaluation Metrics
Belief	DB	Policy	BLEU	Entity-F1	Inform	Success	Score
AttLSTM	-	Oracle	Oracle	-	18.80	54.8	71.2	60.2	84.50
SimpleTOD	GPT-2	Oracle	Oracle	Gen	16.9	-	84	72.8	94.5
HDSA	-	Oracle	Oracle	BERT	23.6	68.9	82.9	68.9	99.50
DAMD	-	Oracle	Oracle	Gen	18.60	-	89.20	77.90	102.15
SOLOIST	GPT-2, DC	Oracle	Oracle	-	18.03	-	89.60	79.30	102.49
Marco	-	Oracle	Oracle	Gen	19.45	-	90.30	75.20	102.20
Marco-BERT	-	Oracle	Oracle	BERT	20.02	59.99	92.3	78.6	105.47
SET++	-	Oracle	Oracle	BERT	22.08	65.33	86.2	76.3	103.33
HIER++	-	Oracle	Oracle	BERT	23.04	64.15	86.5	76.6	104.59
HIER-CLS	-	Oracle	Oracle	BERT	22.89	64.57	85.2	76.8	103.89
Table 4: Context-to-Response: For this experiment only belief-states are given. GPT-2,DC means a second pretraining phase using extra dialog corpus (DC) starting from GPT-2 model parameters.
Marco with an HT-Encoder and rerun the context-to-response generation experiment. Introducing HT-Encoder into Marco helps improve in terms of inform (minor), success and the combined score metric. The results of this experiment show that HT-Encoder is suitable for any model architecture.
Overall, our experiments show how useful the proposed HT-Encoder module can be for dialog sys
tems built upon transformer encoder-decoder architecture. It is also applicable to tasks where the input sequence can be split into an abstract set of subunits (e.g., search history in Sordoni's application). We believe that our proposed approach for hierarchical encoding in transformers and the algorithm for converting the standard transformer encoder makes it an invaluable but accessible resource for
Models	Pretraining	Annotations	Evaluation Metrics
Belief	DB	Policy	BLEU	Entity-F1	Inform	Success	Score
DAMD	-	Gen*	Oracle	Gen	16.60	-	76.40	60.40	85.00
SimpleTOD	GPT-2	Gen	-	Gen	15.01	-	84.4	70.1	92.26
SOLOIST	GPT-2, DC	Gen	Gen	-	16.54	-	85.50	72.90	95.74
HIER-Joint	-	Gen	-	Gen	19.74	53.94	80.5	71.7	95.84
Table 5: End-to-End: Belief State predicted by model itself. *In the End-to-End setting also, DAMD will need to use the oracle  $B_{t - 1}$  for predicting the current belief  $B_{t}$ .
Models	Act Prediction	Response Generation
Precision	Recall	F1	BLEU	Inform	Success	Score
Marco	72.61	74.98	73.72	19.16	88.45	73.5	100.14
Marco + HTEncoder	73.23	74.11	73.68	19.05	91.72	75.8	102.81
Marco-BERT	-	-	-	19.82	90.86	76.66	103.58
Marco-BERT + HTEncoder	-	-	-	19.53	90.99	78.41	104.23
Table 6: Comparison between vanilla Marco model and Marco + HT-Encoder with proposed HT-Encoder. Bold-faced results denote statistically significant improvement with  $p < 0.05$ . We didn't observe any significant improvement in act-prediction F1-Score or BLEU scores for response generation. The numbers given in the table are means of 10 different runs of each algorithm.
future researchers working on dialog systems or similar problem statements with transformer-based architectures.
Task Oriented Dialog Systems Researchers identify four different subtasks for any task-oriented dialog system (Wen et al., 2017), natural language understanding (NLU), dialog state tracking (DST), dialog act or policy generation, and Natural Language Generation (NLG). Before the advent of large scale Seq2Seq models, researchers focused on building feature-rich models with rule-based pipelines for both natural language understanding and generation. It usually required separate utterance-level and dialog-level NLU feature extraction modules. These NLU features decide the next dialog act that the system should follow. This act is then converted into a natural language response using the NLG module. Young et al. (2013) modeled this problem as a Markov Decision Process whose state comprised of various utterance and dialog features detected by an NLU module. However, such models had the usual drawback of any pipelined approaches, error propagation. Wen et al. (2017) proposed using neural networks for extracting features like intent, belief states, etc. and
training the NLU and NLG modules end-to-end using a single loss function. Marco (Wang et al., 2020) and HDSA (Chen et al., 2019) used a finetuned BERT model as their act predictor as it often triumphs other ways to train the dialog policy network (even joint learning). HDSA is a transformer Seq2Seq model with act-controllable self-attention heads (in the decoder) to disentangle the individual tasks and domains within the network. Marco uses a soft-attention over the act sequence during the response generation process.
Hierarchical Encoders The concept of Hierarchical Encoders have been used in many different context in the past. It has been most well known in the area of dialog response generation as the HRED model. Many open domain dialog systems have used the hierarchical recurrent encoding scheme of HRED for various tasks and architectures. Hierarchical Encoder was first proposed by (Sordoni et al., 2015a) for using in a query suggestion system. They used it to encode the user history comprising multiple queries using an Hierarchical LSTM network. Serban et al. (2016) extended this work to open domain dialog generation problems and proposed the HRED network. HRED captures the high level features of the conversation in a context RNN. Several models have adopted this approach later on,
e.g. VHRED (Serban et al., 2017), CVAE (Zhao et al., 2017), DialogWAE (Gu et al., 2018), etc. Another area in which researchers have proposed the use of hierarchical encoder is for processing of paragraph or long documents. Li et al. (2015) used a hierarchical LSTM network for training an autoencoder that can encode and decode long paragraphs and documents. Zhang et al. (2019) proposed HIBERT where they introduced hierarchy into the BERT architecture to remove the limitation on length of input sequence. HIBERT samples a single vector for each sentence or document segment (usually contextual embedding of CLS or EOS token) from the sentence encoder to be passed onto the higher level transformer encoder. Liu and Lapata (2019) applies a similar approach for encoding documents in a multi-document summarization task.
This paper explored the use of hierarchy in transformer-based models for task-oriented dialog system. We started by proposing a generalized framework for Hierarchical Transformer Encoders (HT-Encoders). Using that, we implemented two models, one new model called HIER, and another HIER-CLS model by adapting the existing HIBERT architecture into our framework. We thoroughly experimented with these models in four different response generation tasks of the MultiWOZ dataset. We compared the proposed models with an exhaustive set of recent state-of-the-art models to thoroughly analyze the effectiveness of HT-Encoders. We empirically show that the basic transformer seq2seq architecture, when equipped with an HT-Encoder, outperforms many of the state-of-the-art models in each experiment. We further prove its usefulness by applying it to an existing model Marco. This work opens up a new direction on hierarchical transformers in dialogue systems where complex dependencies exist between the utterances. It would also be beneficial to explore the effectiveness of the proposed HT-Encoder when applied for various other tasks.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge
Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pages 2623-2631. ACM.
Pawel Budzianowski and Ivan Vulić. 2019. Hello, it's GPT-2 - how can I help you? towards the use of pretrained language models for task-oriented dialogue systems. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 15-22, Hong Kong. Association for Computational Linguistics.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. 2019. Semantically conditioned dialog response generation via hierarchical disentangled self-attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3696-3709, Florence, Italy. Association for Computational Linguistics.
Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic. 2018. Multimodal affective analysis using hierarchical attention strategy with word-level alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2225-2235, Melbourne, Australia. Association for Computational Linguistics.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Dan Jurafsky. 2000. Speech & language processing. Pearson Education India.
Jiwei Li, Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1106-1115, Beijing, China. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Hierarchical transformers for multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070-5081, Florence, Italy. Association for Computational Linguistics.
Jiahuan Pei, Pengjie Ren, and Maarten de Rijke. 2019. A modular task-oriented dialogue system using a neural mixture-of-experts. arXiv preprint arXiv:1907.05346.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan deh, Lars Liden, and Jianfeng Gao. 2020a. Soloist: Few-shot task-oriented dialog with a single pretrained auto-regressive model. arXiv preprint arXiv:2005.05298.
Shuke Peng, Xinjing Huang, Zehao Lin, Feng Ji, Haiqing Chen, and Yin Zhang. 2019. Teacher-student framework enhanced multi-domain dialogue generation. arXiv preprint arXiv:1908.07137.
Shuke Peng, Feng Ji, Zehao Lin, Shaobo Cui, Haiqing Chen, and Yin Zhang. 2020b. Mtss: Learn from multiple domain teachers and become a multi-domain dialogue expert. arXiv preprint arXiv:2005.10450.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C. Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 3776-3784. AAAI Press.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 3295-3301. AAAI Press.
Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and JianYun Nie. 2015a. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management, CIKM 2015, Melbourne, VIC, Australia, October 19 - 23, 2015, pages 553-562. ACM.
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015b. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 196-205, Denver, Colorado. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Kai Wang, Junfeng Tian, Rui Wang, Xiaojun Quan, and Jianxing Yu. 2020. Multi-domain dialogue acts and response co-generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7125-7134, Online. Association for Computational Linguistics.
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449, Valencia, Spain. Association for Computational Linguistics.
Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160-1179.
Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5059-5069, Florence, Italy. Association for Computational Linguistics.
Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020a. Task-oriented dialog systems that consider multiple appropriate responses under the same context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9604-9611.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online. Association for Computational Linguistics.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654-664, Vancouver, Canada. Association for Computational Linguistics.