SlowGuess/NAACL_2022_OCR huggingface dataset:
tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Chunyi Liu, Peng Wang, Jiang Xu, Zang Li, and Jieping Ye. 2019a. Automatic dialogue summary generation for customer service. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pages 1957-1965. ACM.
Zhengyuan Liu, Angela Ng, Sheldon Lee, Ai Ti Aw, and Nancy F. Chen. 2019b. Topic-aware pointer-generator networks for summarizing spoken conversations. In ASRU2019, pages 814-821.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Association for Computational Linguistics.
Gabriel Murray, Steve Renals, Jean Carletta, and Johanna Moore. 2006. Incorporating speaker and discourse features into speech summarization. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 367-374, New York City, USA. Association for Computational Linguistics.
Tatsuro Oya, Yashar Mehdad, Giuseppe Carenini, and Raymond Ng. 2014. A template-based abstractive meeting summarization: Leveraging summary and source text relationships. In Proceedings of the 8th International Natural Language Generation Conference (INLG), pages 45-53, Philadelphia, Pennsylvania, U.S.A. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Kochen Qin, Lu Wang, and Joseph Kim. 2017. Joint modeling of content and discourse relations in dialogues. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 974-984, Vancouver, Canada. Association for Computational Linguistics.
Dragomir R. Radev, Hongyan Jing, Magorzata Sty, and Daniel Tam. 2004. Centroid-based summarization of multiple documents. Inf. Process. Manag., 40(6):919-938.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Osman Ramadan, Paweł Budzianowski, and Milica Gašić. 2018. Large-scale multi-domain belief tracking with knowledge sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 432-437, Melbourne, Australia. Association for Computational Linguistics.
Korbinian Riedhammer, Daniel Gillick, Benoit Favre, and Dilek Hakkani-Tur. 2008. Packing the meeting summarization knapsack. In INTERSPEECH 2008, 9th Annual Conference of the International Speech Communication Association, Brisbane, Australia, September 22-26, 2008, pages 2434-2437. ISCA.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1978. A simplest systematics for the organization of turn taking for conversation. In Studies in the organization of conversational interaction, pages 7-55. Elsevier.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorre. 2018. Unsupervised abstractive meeting summarization with multisentence compression and budgeted submodular maximization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 664-674, Melbourne, Australia. Association for Computational Linguistics.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.
Sansiri Tarnpradab, Fei Liu, and Kien A. Hua. 2017. Toward extractive summarization of online forum discussions via hierarchical attention networks. In FLAIRS 2017, Marco Island, Florida, USA, May 22-24, 2017, pages 288-292. AAAI Press.
Lucy Vanderwende, Hisami Suzuki, Chris Brockett, and Ani Nenkova. 2007. Beyond sumbasic: Task-focused summarization with sentence simplification and lexical expansion. Inf. Process. Manag., 43(6):1606-1618.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810-1822, Florence, Italy. Association for Computational Linguistics.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. 2020. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10524-10533. PMLR.
Lin Yuan and Zhou Yu. 2019. Abstractive dialog summarization with semantic scaffolds. ArXiv preprint, abs/1910.00825.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117, Online. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328-11339. PMLR.
Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020. A hierarchical network for abstractive meeting summarization with cross-domain pretraining. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 194-203. Association for Computational Linguistics.
In TANET, the token encoder, utterance encoder, and decoder all have 6 layers, i.e.,  $N = 6$ . Each multi-head attention sub-layer has 12 heads, i.e.,  $h = 12$ . The size of feed-forward layer is 3072.
The hidden size  $d_h$  is 768. We employ the same vocabulary as BART (Lewis et al., 2020), which has 50265 tokens. TANET has 180M parameters in total. We use a dropout probability of 0.1 for all layers. In the thread-aware attention layer, we define 20 learnable embeddings, i.e.,  $k = 9$ . For optimization, both pretraining and downstream finetuning use AdamW optimizer (Loshchilov and Hutter, 2019) with  $\beta_1 = 0.9$ ,  $\beta_2 = 0.999$ , and  $\epsilon = 1e - 8$ . We pre-train TANET with an accumulated batch size of 256. The initial learning rate is set as  $5e - 5$  and linearly decreased to 0 after 500k steps. We use beam search with the commonly used trigram blocking (Paulus et al., 2018; Lewis et al., 2020) to select the best candidate during inference for the downstream tasks. To improve the pretraining efficiency, we set the maximum number of utterances to 124, each utterance has a maximum of 200 tokens, and the pseudo-summary has a maximum of 256 tokens. BART<sub>base</sub> and PEGASUS<sup>large</sup> are implemented with the codes provided by HuggingFace at https://github.com/huggingface/transformers/tree/v4.1.1/examples/seq2seq. The initialized pre-trained models are available at https://huggingface.co/facebook/bart-base and https://huggingface.co/google/pegasus-large. We implement LED<sup>large</sup> by referring to the code at https://colab.research.google.com/gibthp/patrickvonplaten/ notebooks/blob/master/Fine_tune_Longformer_Encoder_Decoder_(LED)_for_Summarization_on_pubmed.ipynb. All experiments are conducted on a machine with 4 NVIDIA Tesla V100 GPUs.
Finally, we randomly sample two cases from the test sets of MultiWOZ and FORUM to conduct qualitative analysis. We show the conversation and the summaries generated by HMNet,  $\mathsf{LED}_{\text {large }}$ ,  $\mathsf{Bart}_{\text {base }}$ ,  $\mathsf{TANET}$ , as well as the ground truth in Table 7 and Table 8. We can see that TANET can produce a more concise summary with less incorrect facts and has a higher level of abstraction ability about the conversations. In contrast, the summaries generated by HMNet and  $\mathsf{Bart}_{\text {base }}$  cannot cover all the important information and are prone to some mistakes.  $\mathsf{LED}_{\text {large }}$  tends to copy content from the original conversation, especially as in Table 8.
Table 7: A case from MultiWOZ. We underline some vital facts in the conversation. Red denotes the incorrect content in the generated summaries. Blue indicates what appears in TANET's summary but is not covered by the ground truth.
Hi, I'm hoping a local expert can help us out, we're traveling over to New York (on route to Florida) on Wednesday 28th March, flying out on Saturday 31st, this gives us around 2 and a half days to see the city, below is the list of places we are looking to visit/see. My only thoughts at the moment are to go to the ESB first thing around 8am and TOTR around dusk. I was hoping to use the subway to get around and our hotel is The Belvedere just off 8th Ave on 48th street, in which order should we visit these sights? is it possible? any info on which subway lines to take would be fantastic. We are two families of 4 - 4 adults 4 kids, aged 7 to 13. Maceys - browse for say 2 hours 5th Ave - stroll down and people watch Brooklyn Bridge - wander over, check out the skyline Central Park - relax Top of the Rock - watch the transition from day to night Ground Zero - must go and pay respect. Times Square - sense the hustle and bustle Staton Island Ferry - relax a little Statue of Liberty - view from the ferry? Empire State Building - must do! Grand Central Station - pass through and see the architecture Ellis Island - not sure about this? Carnegie Deli - take in a cheesecake. Have we missed anything? Given we are a party of 8 will we need to book restaurants? any suggestions nearby the hotel offering good steaks and pizza (sorry I know this is a very subjective question). Thanks very much in advance - counting down the days. N16E
SummerSh...Wow, that's a lot to try and squeeze into such a short visit! I'd eliminate Ellis Island and Macy's. Ellis Island just takes up too much time, and Macy's - even though it's the original and the world's largest store- is just a dept. store, you can find a branch in Florida too. You shouldn't have to make dinner reservations if you avoid the pre-theater crowd in the area. Just go at 8:00. The best pizza is at John's, on W. 44th (four blocks from your hotel). It's in a converted church, so very interesting. I will advise you to make reservations NOW for Carmine's. Even though the place is enormous, it's really tough to get a reservation. That's where I'd suggest you get your steak fix. Their Flintstone-sized Porterhouse is yummy! Everything in Carmine's is served family style, so you'll only need two steaks for your entire group. And one big bowl of pasta will feed everyone. If you go to the World Trade Center site, please also visit St. Paul's Chapel, it's just a block away: www.saintpauls chapel.org And don't go to the Carnegie JUST for the cheesecake (even though it's fantastic!). You've gotta try their pastrami!
N16E: Thanks Bettina, What order do you think we should visit these places in? Took a look at Carmines - the pictures on the website make it look a little formal? Thanks again. N16E.
HMNet: traveling over to new york on the route to florida on wednesday 28th march. he is hoping to use the subway to take the belvedere . they are two families of 4 -4 adults and they should visit around 8 am . is also looking for advice on what to do on which subway lines to take and their hotel is fantastic . mentions that they are a party of 8 in advance and they will need to book down ellis island . they also recommend going to carnegie deli and take pizza in carnegie deli . recommends to make reservations for 44th.
LEDlarge: N16E is traveling to New York on Wednesday 28th March, flying out on Saturday 31st, gives them around 2 and a half days to see the city. They are two families of 4 - 4 adults 4 kids, aged 7 to 13. They have a list of places they are looking to visit/see. They would like to visit ESB first thing around 8am and TOTR around dusk. They were hoping to use the subway to get around and their hotel is The Belvedere just off 8th Ave on 48th street. They ask which order should they visit these sights? In which order? Any info on which subway lines to take would be fantastic. They also ask about steaks and pizza nearby the hotel offering good steaks. SummerShowers says to eliminate Ellis Island and Macy's. Ellis Island just takes up too much time, and Macy's - even though it's the original.
Bartbase: N16E needed help to plan for a two day trip to New York. N16E was traveling with his family of four and had planned to visit the Empire State Building, Central Park, Top of the Rock, Times Square, Statue of Liberty, Grand Central Station, Ellis Island, Carnegie Deli, St. Paul's Chapel, Macy's, and John's all in the same short time frame. He/She also wanted to know the order of things to do in the city, and how to get around. Bettina said that they should make reservations NOW for Carmine's.
TANET: N16E will be traveling to New York and needs help to plan for a two day stopover. They will be staying at the Belvedere Hotel, and would like to visit the Top of the Rock, and the Empire State Building. They are two families of four and will be going to Macy's. They plan on going to Ground Zero, Central Park, Times Square, Statue of Liberty, and Grand Central Station. They also ask about dinner reservations, and suggestions on places that offer good steak and pizza. SummerShowers mentions that John's has a lot of steak and may be a little too formal. He/She suggests Ellis Island, and to avoid the pre-theater crowd. Paul's and Carmine's are two family of four. N16E asks about a cheesecake in the Carnegie Deli. Summershowers responds that Carmine's is a good option as well and recommended a place called Le Pain de St. Paul's. For steak Carmine, go to Paul's Chapel and get pastrami for cheesecake.
Ground N16E needed help to plan for a two day stopover in NYC en route to Florida, and wanted opinion on his/her itinerary. Truth: N16E said that they were staying at the Belvedere, and wanted to start at the Empire State Building and end at the Top of the Rock. N16E said that they had a party of 8 and that the Empire State Building was a must stop. N16E planned to browse at Macy's, stroll down the Brooklyn Bridge, go to Central Park to relax, visit the Top of the Rock, and see Ground Zero on the first day. On the second day N16E said that they would go to Times Square, ride the Staten Island Ferry, and see the Statue of Liberty. N16E wanted to know if they could fit in Ellis Island and a visit to Grand Central Station, and still get a cheesecake for Carnegie Deli. Summersshowers responded saying that was a lot to do in two days, and recommended that they drop Macy's and Ellis Island. Summersshowers said that John's had the best pizza and recommended that N16E get reservations for Carmine's immediately. Summersshowers said that Carmine's was family style and recommended the Flintstone-sized Porterhouse steak. Summersshowers said that if N16E goes to the World Trade Center, to stop by St. Paul's Chapel, and to get pastrami a Carnegie Deli as well as a cheesecake. N16E thanked Bettine, and asked for a recommended order in visiting places, and said the Carmine's might be a little too formal.
Table 8: A case from FORUM. The conversation's domain is trip. We underline some vital facts in the conversation. Red denotes incorrect content in the generated summaries. Blue indicates what appears in TANET's summary but is not covered by the ground truth.