SlowGuess/NAACL_2021_OCR huggingface dataset:
R. Thomas McCoy, Tal Linzen, Ewan Dunbar, and Paul Smolensky. 2019. RNNs implicitly implement tensor-product representations. In International Conference on Learning Representations.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Computational Natural Language Learning.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
H. Palangi, P. Smolensky, X. He, and L. Deng. 2018. Question-answering with grammatically interpretable representations. In AAAI.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multireward reinforced summarization with saliency and entailment. In 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, USA.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, Vancouver, BC, Canada.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Imanol Schlag and Jürgen Schmidhuber. 2018. Learning to reason with third order tensor products. In Advances in Neural Information Processing Systems, pages 10002-10013.
Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jurgen Schmidhuber, and Jianfeng Gao. 2019. Enhancing the transformer with explicit relational encoding for math problem solving. arXiv preprint arXiv:1910.06611.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596-4604. PMLR.
Paul Smolensky. 1990. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159-216.
Kaiqiang Song, Lin Zhao, and Fei Liu. 2018. Structure-infused copy mechanisms for abstractive summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1717-1729, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Paul Soulos, Tom McCoy, Tal Linzen, and Paul Smolensky. 2019. Discovering the compositional structure of vector representations with role learning networks. arXiv preprint arXiv:1910.09113.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning.
A TP-TRANSFORMER Architecture
We provide Fig. 3 to visualize the full encoder-decoder architecture of our TP-TRANSFORMER.
B Summarization Experimental Setup
The Transformer and the two TP-TRANSFORMERS all have 6 layers, 8 heads per layer, dimension per head  $d_{k} = 64$ , model dimension  $d_{m} = 512$ , and feedforward dimension  $d_{f} = 2048$  for the encoder and decoder. Our TP-TRANSFORMER with discrete roles has  $N_{r} = 50$  role embeddings of dimension  $d_{r} = 64$  at every layer. We search the optimal  $N_{r}$  from \{20, 50, 100, 200, 1000\} and select the one with the best validation set performance. For each of the
Figure 3: TP-TRANSFORMER model architecture.
dataset above, we train the all three models from scratch using an Adafactor Optimizer (Shazeer and Stern, 2018) with square root learning rate decay and dropout rate of 0.1. The total number of parameter of the Transformer, TP-TRANSFORMER with continuous roles, and our TP-TRANSFORMER with discrete roles are 60506880, 65234688, and 64258080 respectively. Every model is trained on 4 NVidia V100 GPUs (32GB) with a batch size of 32 per GPU.
We conduct human evaluation to compare the summaries generated by the original Transformer and our TP-TRANSFORMER. We randomly sample 120 examples from the test sets of XSum and Wikihow datasets with the corresponding beam-searched model summaries. For every example, we show the source document, the reference summary, and two model summaries shuffled in order to three human evaluators, and ask them to decide which summary is better in six different aspects: grammar, coherency, factuality, saliency, redundancy, and an
overall preference. We then take the majority vote of every examples from its three human annotators.
C Probing Experimental Setup
As there is no existing dataset for probing decoders, we create our own training and evaluation data by running off-the-shelf models on the summarization datasets. Specifically, to probe a decoder trained on the XSum dataset on the POS task, we run an POS tagger on the reference summaries from the XSum training set and the model-generated summaries for the XSum dev set to create the ground-truth labels for the training set and model-specific dev set. We use Stanford CoreNLP (Manning et al., 2014) to get the labels for POS, dependency and named entity probing tasks. We use a BERT-base model (Devlin et al., 2019) from AllenNLP (Gardner et al., 2018) to get the ground-truth labels for SRL. We restore the model trained on a summarization dataset and freeze its parameters during the probing. We simply add a linear layer on top of the target representation to project it onto the output label space.
D Related Works
Implicit TPR Encodings in Neural Networks McCoy et al. (2019) showed that, in GRU-based (Cho et al., 2014) encoder-decoder networks performing fully-compositional string manipulations, trained on extensive data that fully exemplifies the range of possible compositions, the medial encoding between encoder and decoder could be extremely well approximated by TPRs. Soulos et al. (2019) presented the ROLE model that learns its own role scheme to optimize the fit of a TPR approximation to a given set of internal representations in a pre-trained target neural network, removing the need for human-generated hypotheses about the role schemes the network might be implementing. While this work successfully interprets the Tensor Product Representation in fully compositional tasks, abstractive summarization, as well as most other NLP tasks, are only partially compositional and the symbolic rules in language are much more complex. Although these two works showed that Tensor Product Representation can naturally emerge in a unstructured representations, we argue that standard models only learn TPRs without any special bias to do so when the compositional structure of the task is simple and blatant and when the training set makes that painfully clear
by providing a good sample of the compositional possibilities. That is possible for the simple string tasks addressed in the two previous works, but not in the abstractive summarization as well as other real-world NLP tasks, where we show that having explicit TPR helps in modeling the structure information.
Sequence Models Encode Implicit Structure. Several recent works have shown that the pretrained Transformer-based BERT (Devlin et al., 2019) embeddings implicitly encode structural linguistic relations with various interpretation methods. The first, and also the most popular method (Tenney et al., 2019a) is to train an auxiliary classifier to probe the model's hidden representations for specific linguistic information. The second method (Lin et al., 2019) abstracts the Transformer model into a graph based on the attention weights, and explores syntactic structures based on the graph's structure. The third method (Hewitt and Manning, 2019) sees the hidden representations of BERT as in a metric space and directly connect the distance between representations to the distance between elements in a symbolic structure (e.g., a dependency-parse tree) to extract the implicit structures without extra training. The interpretation method deployed here falls under the probing family, but future work will also pursue other interpretation methods.
E Examples of Generated Summary
We provide examples generated by the Transformer baseline and our TP-TRANSFORMER in Table 5 and Table 6.
Datasets	Summary
Source	Nottinghamshire Police said it would expand its categories to include misogynistic incidents. It means abuse or harassment which might not be a crime can be reported to and investigated by the police, and support for the victim put in place. Nottingham Women's Centre said it hopes it will help give more victims the courage to report incidents. Chief Constable Sue Fish claimed it will make the county a safer place for women. "What women face, often on a daily basis, is absolutely unacceptable and can be extremely distressing," she said. "Nottinghamshire Police is committed to taking misogynistic hate crime seriously and encourages anyone who is affected by it to contact us without hesitation. Work on the idea first started with the Nottinghamshire Safer for Women Conference last year, co-hosted by the police with the Nottingham Women's Centre. BBC TV reporter Sarah Teale was harassed in the street while reporting on the conference. The force defines misogyny hate crime as: "Incidents against women that are motivated by an attitude of a man towards a woman and includes behaviour targeted towards a woman by men simply because they are a woman. "The classification now means people can report incidents which might not be considered to be a crime and the police will investigate. Nottingham Women's Centre has been helping train call centre, force control staff and officers on the beat to recognise misogynistic hate crime and ways to tackle it. These officers will also examine if and how a victim can be supported or if anything can be done to help prevent them being targeted again. Domestic abuse will not be recorded as a misogyny hate crime because it has its own procedure, the force said. Melanie Jeffs, centre manager at Nottingham Women's Centre, said: "We're pleased to see Nottinghamshire Police recognise the breadth of violence and intimidation that women experience on a daily basis in our communities. "She added: "Recording this as a hate crime will give us a detailed picture of how often, when and where it is happening. "It has been very difficult to build that picture before but we will now get detailed data to analyse. "Showing that the police take it seriously will also give people the confidence to come forward and report offences. "A crime that the victim or any other person perceives to be motivated by hostility or prejudice towards any aspect of a person's identity. Police forces in England, Wales and Northern Ireland annually monitor five strands of hate crime: Forces can include their own definition of a hate crime with several recently adding sub cultures.
Reference	Harassment of women is to be recorded as a hate crime in a bid to tackle sexist abuse.
Transformer	Women who commit misogyny and harassed a woman are to be asked to take part in an anti-Semitic conference.
TP-TRANSFORMER	A police force has launched a national drive to combat misogyny and hate crimes in Nottinghamshire.
Table 5: An example from the XSum dev set and the summaries generated by the Transformer baseline and TP-TRANSFORMER.
Datasets	Summary
Source	Sixty patrol boats will protect the UK's two new aircraft carriers which are due to arrive at Portsmouth Naval Base in 2017. The first carrier, HMS Queen Elizabeth, is expected to be operational in 2020. </br> "We are going to see a bigger Royal Navy and the flagship... will be here in Portsmouth," Michael Fallon said. The 60 Pacific 24 rigid-hulled inflatable boats will be built by BAE systems to "guard the carriers in the harbour and our new frigates and destroyers", Mr Fallon said. He said they will also enhance security by providing a rapid response in rescue, anti-piracy and counter-narcotics missions in the area. Mr Fallon said: "Through the defence review, defence spending is going to go up every April for the rest of this parliament. He said as part of the larger investment, the government will also be able to provide the new aircraft carriers with sufficient fighter jets. </br> "We have said we will maintain a minimum fleet of 19 destroyers and frigates, but as the older frigates are retired we also hope to add a lighter frigate between the offshore patrol vessel and Type 26 and to build more of those as well. </br> "Mr Fallon's visit to Portsmouth Naval Base comes as work has begun to rebuild the jetty for the arrival of HMS Queen Elizabeth in 2017. Floating cranes are also dredging Portsmouth harbour to prepare deeper channels for the aircraft carriers to sail from the base, which are the largest ships ever built for the Royal Navy. </br> "This is a huge financial investment in making sure the channel is wide enough, in enlarging the jetty here so they can take the carriers and in making sure the carriers are properly guarded," Mr Fallon said. Taller than Nelson's Column and longer than Portsmouth's Spinnaker Tower laid on its side, the new carriers will displace 65,000 tonnes of water. To make room for the carriers three million cubic metres of clay, sand and gravel will be removed from a two-mile stretch of Portsmouth Harbour covering an area the size of 200 football pitches.
Reference	Increased spending will result in a "bigger" Royal Navy, the defence secretary has said, as he announced a new £13.5m shipbuilding contract.
Transformer	The Royal Navy's new aircraft carriers will be patrolling the Portsmouth harbour this year, the defence secretary has said.
TP-TRANSFORMER	Plans for a new Royal Navy aircraft carriers to be built in Portsmouth have been unveiled.
Table 6: An example from the XSum dev set and the summaries generated by the Transformer baseline and TP-TRANSFORMER.