Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models
Year: 2023

Abstract:
Natural language contains rich logical structures and logical information, and correctly detecting and accurately understanding these logical structures and information underlying natural language texts is very crucial for NLP models’ performance on many important NLU and NLG tasks. Existing pre-trained language models based on the transformer architecture mostly adopt a classical design for constructing their input embeddings that ignores the logical structures underlying natural language texts, thus limiting their ability to better capture and encode key logical information in the input sequences. To overcome such limitations, in this paper we first propose a novel approach to construct logic-aware input embeddings for transformer language models through a combination of logic detection, logic mapping and hierarchical logical projections, and then develop a corresponding new modeling paradigm that can upgrade existing transformer language models into logical transformers to boost their performance on different NLU and NLG tasks. Our empirical experiments on four important and challenging NLU and NLG tasks demonstrate that our proposed logical transformer language models can achieve superior performance over their baseline transformer models through a deeper understanding of the logical structures of texts.

heading: 1 Introduction
text: Natural language contains rich logical structures and logical information (Lakoff, 1970; Van Benthem, 1986) that are crucial to a deep and accurate understanding of its meaning. Therefore, the ability to correctly detect and accurately understand the logical structures and information within natural language texts is very crucial for NLP models’
∗ This work was done when Borui Wang was a research intern at Microsoft Research.
† This work was done when Daniel McDuff was at Microsoft Research.
performance on many important Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks.
The types of logics contained in natural language are very diverse, including not only mathematically well-defined propositional logic and firstorder logic (Lu et al., 2022; Han et al., 2022), but also more general types of natural and structural logical relationships that people frequently use in natural language texts to convey and communicate their ideas and meanings more effectively and clearly.
In recent years we have witnessed huge progress and success in many fields of natural language processing brought about by the introduction of all different kinds of pre-trained language models (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020; Yang et al., 2019; Clark et al., 2020; Lewis et al., 2020; Raffel et al., 2020; Zhang et al., 2020) based on the transformer architecture (Vaswani et al., 2017). Most existing pre-trained language models adopt the classical approach for constructing the input embeddings that are fed into the encoder parts of the language models, which can be summarized as the summation of the following three key components (Devlin et al., 2019):
(1) Token Embeddings - that are used to encode and represent the semantics and meaning of each token in the vocabulary;
(2) Position Embeddings - that are used to encode the positional information of each token in the input sequence;
(3) Segment Embeddings - that are used to indicate which segment of the input sequence each token belongs to.
This classical design of the input embeddings has been proven to be very effective at capturing important semantic and positional features from
1762
natural language texts and helping pre-trained language models to learn good contextualized representations of the input textual sequences (Devlin et al., 2019). However, it also has a very important limitation - it doesn’t consider or try to explicitly encode the logical structures underlying the text inputs, which are also very crucial for the deep and accurate understanding of the meaning of the text inputs.
Therefore, in order to overcome this limitation and to enable pre-trained language models to better capture and understand the important logical structures underlying natural language texts, in this paper we propose a novel approach to construct logicaware input embeddings for transformer-based pre-trained language models and a corresponding new modeling framework that can upgrade existing transformer language models into logical transformers to boost their performance on different NLU and NLG tasks.
Our new approach consists of two major modules: (1) logic detection and mapping, and (2) multi-layer hierarchical logical projections. It has the following key advantages:
• Strong Generalizability: Our proposed new approach for constructing logic-aware input embeddings doesn’t alter the main architecture of transformer language models and only modifies the input embeddings at the front end before they are fed into the encoder part of the language models. Therefore, our new approach enjoys strong generalizability and can be smoothly added to many different pretrained language models based on the transformer architecture.
• Consistent Boost in Model Performance: Our proposed new approach is empirically shown to consistently boost the performance of different transformer language models on different NLU and NLG tasks.
• Negligible Increase in Model Size: Our proposed new approach will only increase the number of parameters of transformer language models by a negligible amount.
• Low Overhead on Training Time: Our proposed new approach will not significantly increase the training time of transformer language models by a large amount. The majority of the overhead in training time will come
from the initial text processing steps of logic detection and logic mapping, which only need to be executed once before the actual training epochs start.

heading: 2 Logical Relationships and Keywords
text: In this work, we consider logical relationships in natural language texts as the underlying relationships among different language constituents that carry meaningful information regarding logical understanding and reasoning of the texts. In natural language, such logical relationships are usually indicated by logically-connective keywords and phrases. In this paper, we define a taxonomy of 12 most commonly seen types of logical relationships and their corresponding sets1 of logical keywords (including phrases2) for natural language3:
1. Conjunction: a conjunction logical relationship indicates that the two language constituents involved are presented jointly in addition to each other. Its logical keywords are:
and, as well, as well as, also, at the same time.
2. Disjunction: a disjunction logical relationship indicates that the two language constituents involved are presented alternatively next to each other. Its logical keyword is:
or.
3. Negation: a negation logical relationship indicates that the meaning of the language constituent mapped by it is negated. Its logical keywords are:
not, no, none, n’t, nothing.
4. Conditional: a conditional logical relationship indicates that the content of one language constituent is the premise of the content of another language constituent. Its logical keywords are:
1The sets of logical keywords listed here are not necessarily the most exhaustive sets that contain all possible keywords in each category, but rather serve as the preliminary and exemplar sets that can already cover the majority of the most frequently appearing logical keywords in real-world texts. These sets are open to extension.
2For conciseness, in this paper we will use the term ‘logical keywords’ to refer to both logical keywords and logical key phrases.
3Here the logical keywords are all defined in English, but similar categorization of logical relationships and sets of logical keywords can also be defined in other languages as well.
themforeasytoo
them
for themtoo easy
be too easy for them
be
would be too easy for them
wouldit
it
it would be too easy for thembecause
because it would be too easy for them
that waygo
way
go that way
to go that way
to
not to go that way
not
that
very hard
hardvery
try
try very hard not to go that way because it would be too easy for them
I
IBut
But I try very hard not to go that way because it would be too easy for them.
Contrastive
Negation Causal
.
As an example, we sample a news article from the training set of the CNN/Dailymail dataset (Nallapati et al., 2016) and manually annotate the appearances of the above defined types of logical relationship in the article. See Figure 6 for the annotation of the logical relationships in this example article, where the logical keywords associated with different logical relationships are highlighted with different colors.

heading: 2.1 Categorization of Logical Relationships
text: According to how many logical components (in the form of text spans) are associated with each logical keywords and how different logical components are mapped by the logical keywords, we categorize the set of all logical keywords into three different categories:

heading: 2.1.1 Unary Logical Relationships
text: The logical keywords indicating unary logical relationships are those that each only maps to one single logical component (text span). For example, most keywords of negation relationship and example relationship are indicating unary logical relationships, such as not, for example, such as, etc.

heading: 2.1.2 Intrinsically-Mapped Binary Logical Relationships
text: The logical keywords indicating intrinsicallymapped binary logical relationships are those that each maps to two separate logical components (text spans) that are both contained within the parent sentence constituent of the logical keyword itself. For example, most keywords of conjunction rela-
Algorithm 1 Logic Detection and Mapping Input: Sentence s
Constituency parser C : S → T Set of logical keywords K
Output: List of logic mapping dictionariesM 1: Run C over s to obtain its constituency parse tree T (s) 2: Nkey(s)← [ ] 3: M← [ ] 4: for each constituent node n in T (S) do 5: if str(n) ∈ K then 6: Nkey(s)← Nkey(s) + n 7: for nk inNkey(s) do 8: Dk ← {} 9: Dk[‘keyword’] = str(nk)
10: if str(nk) ∈ KU then 11: Dk[‘α’] = str( pa(nk) \ nk ) 12: else if str(nk) ∈ KBin then 13: Use str(nk) to segment str( pa(nk) ) into 3 segments: str( pa(nk) ) = A + str(nk) + B 14: Dk[‘α’] = A, Dk[‘β’] = B 15: else if str(nk) ∈ KBex then 16: if ∃ pa(pa(nk)) then 17: Dk[‘α’] = str( pa(pa(nk)) \ pa(nk) ) 18: Dk[‘β’] = str( pa(nk) \ nk ) 19: else if ∃ another sentence s′ right before s then 20: Dk[‘α’] = s′, Dk[‘β’] = str( pa(nk) \ nk ) 21: else 22: Dk[‘α’] = ∅, Dk[‘β’] = str( pa(nk) \ nk ) 23: M←M+Dk 24: return M
tionship and disjunction relationship are indicating intrinsically-mapped binary logical relationships, such as and, as well as, or, etc.

heading: 2.1.3 Extrinsically-Mapped Binary Logical Relationships
text: The logical keywords indicating extrinsicallymapped binary logical relationships are those that each maps to two separate logical components (text spans) where one is contained within the parent sentence constituent of the logical keyword itself while the other is outside (usually appears before) the span of this parent sentence constituent. For example, most keywords of conditional, comparative, temporal and causal relationships are indicating extrinsically-mapped binary logical relationships, such as if, but, during, because, etc.

heading: 3 Logic Detection and Mapping
text: In this section, we describe our logic detection and mapping module based on keyword detection and constituency parsing. For each sentence s in the source text, we first perform constituency parsing (Kitaev and Klein, 2018) over s to obtain its constituency parsing tree T (s). In this paper, we use the Berkeley Neural Parser (Kitaev and Klein, 2018) to perform constituency parsing.
But
But I try ……. them.“People …… months.
keyword
parent
<latexit sha1_base64="c16HAYs8sCFzjk6BmE3oMl7QOLk=">AAAB6nicbVDLSgNBEOyNrxhfUY9eBoPgKeyK+LgFvHiMaB6QrGF20psMmZ1dZmaFsOQTvHhQxKtf5M2/cZLsQRMLGoqqbrq7gkRwbVz32ymsrK6tbxQ3S1vbO7t75f2Dpo5TxbDBYhGrdkA1Ci6xYbgR2E4U0igQ2ApGN1O/9YRK81g+mHGCfkQHkoecUWOle/k46pUrbtWdgSwTLycVyFHvlb+6/ZilEUrDBNW647mJ8TOqDGcCJ6VuqjGhbEQH2LFU0gi1n81OnZATq/RJGCtb0pCZ+nsio5HW4yiwnRE1Q73oTcX/vE5qwis/4zJJDUo2XxSmgpiYTP8mfa6QGTG2hDLF7a2EDamizNh0SjYEb/HlZdI8q3oX1fO780rtOo+jCEdwDKfgwSXU4Bbq0AAGA3iGV3hzhPPivDsf89aCk88cwh84nz9TEI3P</latexit> nk
<latexit sha1_base64="yRo1ANztdtvHJ8g6hRyreBPT7qI=">AAAB9XicbVBNS8NAEJ34WetX1aOXYBHqpSRS/LgVvHisYD+gTctmu2mXbjZhd6KW0P/hxYMiXv0v3vw3btsctPXBwOO9GWbm+bHgGh3n21pZXVvf2Mxt5bd3dvf2CweHDR0lirI6jUSkWj7RTHDJ6shRsFasGAl9wZr+6GbqNx+Y0jyS9ziOmReSgeQBpwSN1O0ge8I0JpOS7I7OeoWiU3ZmsJeJm5EiZKj1Cl+dfkSTkEmkgmjddp0YvZQo5FSwSb6TaBYTOiID1jZUkpBpL51dPbFPjdK3g0iZkmjP1N8TKQm1Hoe+6QwJDvWiNxX/89oJBldeymWcIJN0vihIhI2RPY3A7nPFKIqxIYQqbm616ZAoQtEElTchuIsvL5PGedm9KFfuKsXqdRZHDo7hBErgwiVU4RZqUAcKCp7hFd6sR+vFerc+5q0rVjZzBH9gff4AeAeSeA==</latexit>
pa(nk)
the previous sentence <latexit sha1_base64="3VILl0VJCn+UrVoMU98Lpger7F4=">AAAB6XicbVDLSgNBEOz1GeMr6tHLYBA9hV0JPm4BLx6jmAckS5id9CZDZmeXmVkhLPkDLx4U8eofefNvnCR70MSChqKqm+6uIBFcG9f9dlZW19Y3Ngtbxe2d3b390sFhU8epYthgsYhVO6AaBZfYMNwIbCcKaRQIbAWj26nfekKleSwfzThBP6IDyUPOqLHSgz7rlcpuxZ2BLBMvJ2XIUe+Vvrr9mKURSsME1brjuYnxM6oMZwInxW6qMaFsRAfYsVTSCLWfzS6dkFOr9EkYK1vSkJn6eyKjkdbjKLCdETVDvehNxf+8TmrCaz/jMkkNSjZfFKaCmJhM3yZ9rpAZMbaEMsXtrYQNqaLM2HCKNgRv8eVl0ryoeJeV6n21XLvJ4yjAMZzAOXhwBTW4gzo0gEEIz/AKb87IeXHenY9564qTzxzBHzifPz9gjSg=</latexit> s0 <latexit sha1_base64="6MmxRebE4Cm2FawSQk71fs/islo=">AAACBXicdVDLSgNBEJz1bXxFPephMAh6CZsHbnITvHiMYFTIxjA76dUhs7PLTK8Ylly8+CtePCji1X/w5t84eQgqWtBQU9XNdFeQSGHQdT+cqemZ2bn5hcXc0vLK6lp+fePMxKnm0OSxjPVFwAxIoaCJAiVcJBpYFEg4D3pHQ//8BrQRsTrFfgLtiF0pEQrO0Eqd/LaPcItZwgZ76rK3T30DGAmVGmqfnXzBLXpevVz1qFt0R7CkUqtX6jVamigFMkGjk3/3uzFPI1DIJTOmVXITbGdMo+ASBjk/NZAw3mNX0LJUsQhMOxtdMaC7VunSMNa2FNKR+n0iY5Ex/SiwnRHDa/PbG4p/ea0Uw1o7EypJERQffxSmkmJMh5HQrtDAUfYtYVwLuyvl10wzjja4nA3h61L6PzkrF0sHxepJtXBYn8SxQLbIDtkjJeKRQ3JMGqRJOLkjD+SJPDv3zqPz4ryOW6ecycwm+QHn7RPZvpjM</latexit> pa(nk) \ nk
<latexit sha1_base64="/FWAmq+z3GXrvBxyQ76jnWxzyzU=">AAAB7HicdVBNS8NAEN3Ur1q/qh69LBbBU0jaYtpbwYvHCqYttKFsttt26WYTdidCCf0NXjwo4tUf5M1/47aNoKIPBh7vzTAzL0wE1+A4H1ZhY3Nre6e4W9rbPzg8Kh+fdHScKsp8GotY9UKimeCS+cBBsF6iGIlCwbrh7Hrpd++Z0jyWdzBPWBCRieRjTgkYyR+EDMiwXHFsz2tW6x52bGcFQ2qNZq3ZwG6uVFCO9rD8PhjFNI2YBCqI1n3XSSDIiAJOBVuUBqlmCaEzMmF9QyWJmA6y1bELfGGUER7HypQEvFK/T2Qk0noehaYzIjDVv72l+JfXT2HcCDIukxSYpOtF41RgiPHyczziilEQc0MIVdzciumUKELB5FMyIXx9iv8nnartXtn123ql1czjKKIzdI4ukYs81EI3qI18RBFHD+gJPVvSerRerNd1a8HKZ07RD1hvnzH1jvA=</latexit>
component <latexit sha1_base64="09EUrHkbMFZHIm6QLxcEqraGpd8=">AAAB7XicdVBNS8NAEN3Ur1q/qh69LBbBU0jaYppbwYvHCrYW2lAm2027drMJuxuhlP4HLx4U8er/8ea/cdtGUNEHA4/3ZpiZF6acKe04H1ZhbX1jc6u4XdrZ3ds/KB8edVSSSULbJOGJ7IagKGeCtjXTnHZTSSEOOb0NJ5cL//aeSsUScaOnKQ1iGAkWMQLaSJ0+8HQMg3LFsT3Pr9Y97NjOEobUGn7Nb2A3VyooR2tQfu8PE5LFVGjCQame66Q6mIHUjHA6L/UzRVMgExjRnqECYqqC2fLaOT4zyhBHiTQlNF6q3ydmECs1jUPTGYMeq9/eQvzL62U6agQzJtJMU0FWi6KMY53gxet4yCQlmk8NASKZuRWTMUgg2gRUMiF8fYr/J52q7V7Y9et6penncRTRCTpF58hFHmqiK9RCbUTQHXpAT+jZSqxH68V6XbUWrHzmGP2A9fYJ+ZGPZA==</latexit>↵ component
(a) Execution of Algorithm 1 over the logical keyword ‘But’.
because
because it would be too easy for them
try very hard not to go that way because it would be too easy for them
parent
parent
keyword
<latexit sha1_base64="c16HAYs8sCFzjk6BmE3oMl7QOLk=">AAAB6nicbVDLSgNBEOyNrxhfUY9eBoPgKeyK+LgFvHiMaB6QrGF20psMmZ1dZmaFsOQTvHhQxKtf5M2/cZLsQRMLGoqqbrq7gkRwbVz32ymsrK6tbxQ3S1vbO7t75f2Dpo5TxbDBYhGrdkA1Ci6xYbgR2E4U0igQ2ApGN1O/9YRK81g+mHGCfkQHkoecUWOle/k46pUrbtWdgSwTLycVyFHvlb+6/ZilEUrDBNW647mJ8TOqDGcCJ6VuqjGhbEQH2LFU0gi1n81OnZATq/RJGCtb0pCZ+nsio5HW4yiwnRE1Q73oTcX/vE5qwis/4zJJDUo2XxSmgpiYTP8mfa6QGTG2hDLF7a2EDamizNh0SjYEb/HlZdI8q3oX1fO780rtOo+jCEdwDKfgwSXU4Bbq0AAGA3iGV3hzhPPivDsf89aCk88cwh84nz9TEI3P</latexit> nk
<latexit sha1_base64="yRo1ANztdtvHJ8g6hRyreBPT7qI=">AAAB9XicbVBNS8NAEJ34WetX1aOXYBHqpSRS/LgVvHisYD+gTctmu2mXbjZhd6KW0P/hxYMiXv0v3vw3btsctPXBwOO9GWbm+bHgGh3n21pZXVvf2Mxt5bd3dvf2CweHDR0lirI6jUSkWj7RTHDJ6shRsFasGAl9wZr+6GbqNx+Y0jyS9ziOmReSgeQBpwSN1O0ge8I0JpOS7I7OeoWiU3ZmsJeJm5EiZKj1Cl+dfkSTkEmkgmjddp0YvZQo5FSwSb6TaBYTOiID1jZUkpBpL51dPbFPjdK3g0iZkmjP1N8TKQm1Hoe+6QwJDvWiNxX/89oJBldeymWcIJN0vihIhI2RPY3A7nPFKIqxIYQqbm616ZAoQtEElTchuIsvL5PGedm9KFfuKsXqdRZHDo7hBErgwiVU4RZqUAcKCp7hFd6sR+vFerc+5q0rVjZzBH9gff4AeAeSeA==</latexit>
pa(nk)
<latexit sha1_base64="nwBqW746ehyTlRPecCnOdhJUuVk=">AAACAnicbZDLSgMxFIYz9VbrbdSVuAkWod2UGSledgU3LivYC7RjyaRpG5rJDMkZsQzFja/ixoUibn0Kd76NaTugtv4Q+PjPOZyc348E1+A4X1ZmaXlldS27ntvY3NresXf36jqMFWU1GopQNX2imeCS1YCDYM1IMRL4gjX84eWk3rhjSvNQ3sAoYl5A+pL3OCVgrI590AZ2D0lExoUfkrfDYrFj552SMxVeBDeFPEpV7dif7W5I44BJoIJo3XKdCLyEKOBUsHGuHWsWETokfdYyKEnAtJdMTxjjY+N0cS9U5knAU/f3REICrUeBbzoDAgM9X5uY/9VaMfTOvYTLKAYm6WxRLxYYQjzJA3e5YhTEyAChipu/YjogilAwqeVMCO78yYtQPym5p6XydTlfuUjjyKJDdIQKyEVnqIKuUBXVEEUP6Am9oFfr0Xq23qz3WWvGSmf20R9ZH99H65dS</latexit>
pa(pa(nk)) <latexit sha1_base64="P+68BN5f6KEe2nuR729SPQreEq8=">AAACG3icdVDLSgNBEJz1bXxFPXoZDIJell1NYvYmePEYwaiQxDA76eiQ2dllplcMS/7Di7/ixYMingQP/o2Th/hACxqKqm66u8JECoOe9+5MTE5Nz8zOzecWFpeWV/Kra6cmTjWHGo9lrM9DZkAKBTUUKOE80cCiUMJZ2D0c+GfXoI2I1Qn2EmhG7FKJjuAMrdTK7zYQbjBLWH/7i6mL7s4ObRjASKjU0J9OK1/wXK9cDkpF6rml0p5fDCwJgkrJ96jvekMUyBjVVv610Y55GoFCLpkxdd9LsJkxjYJL6OcaqYGE8S67hLqlikVgmtnwtz7dskqbdmJtSyEdqt8nMhYZ04tC2xkxvDK/vYH4l1dPsVNpZkIlKYLio0WdVFKM6SAo2hYaOMqeJYxrYW+l/IppxtHGmbMhfH5K/yenu65fdovHxcJBMI5jjmyQTbJNfLJPDsgRqZIa4eSW3JNH8uTcOQ/Os/Myap1wxjPr5Aectw9CMKIr</latexit> pa(pa(nk)) \ pa(nk)
<latexit sha1_base64="BuT9ZmLmagCyRtDmRN68DIgOo3w=">AAACBXicdVA9SwNBEN3z2/gVtdRiMQjaHHeaxFwn2FhGMCokMextJrpkb+/YnRPDkcbGv2JjoYit/8HOf+PmQ1DRBwNv35thZ16YSGHQ8z6cicmp6ZnZufncwuLS8kp+de3MxKnmUOOxjPVFyAxIoaCGAiVcJBpYFEo4D7tHA//8BrQRsTrFXgLNiF0p0RGcoZVa+c0Gwi1mCevvqMvuLm0YwEio1FD7bOULnuuVy0GpSD23VNr3i4ElQVAp+R71XW+IAhmj2sq/N9oxTyNQyCUzpu57CTYzplFwCf1cIzWQMN5lV1C3VLEITDMbXtGn21Zp006sbSmkQ/X7RMYiY3pRaDsjhtfmtzcQ//LqKXYqzUyoJEVQfPRRJ5UUYzqIhLaFBo6yZwnjWthdKb9mmnG0weVsCF+X0v/J2Z7rl93iSbFwGIzjmCMbZIvsEJ8ckENyTKqkRji5Iw/kiTw7986j8+K8jlonnPHMOvkB5+0T7LaY2Q==</latexit> pa(nk) \ nk
component<latexit sha1_base64="fGl0e7nb19o0zIPEejxqfbip4uQ=">AAAB7XicdVDLSgMxFM34rPVVdekmWARXQ0ZnamdXcOOygn1AO5RMmraxmWRIMkIZ+g9uXCji1v9x59+YPgQVPXDhcM693HtPnHKmDUIfzsrq2vrGZmGruL2zu7dfOjhsapkpQhtEcqnaMdaUM0EbhhlO26miOIk5bcXjq5nfuqdKMyluzSSlUYKHgg0YwcZKzS7m6Qj3SmXkokolDHyI3CC48PzQkjCsBh6CnovmKIMl6r3Se7cvSZZQYQjHWnc8lJoox8owwum02M00TTEZ4yHtWCpwQnWUz6+dwlOr9OFAKlvCwLn6fSLHidaTJLadCTYj/dubiX95ncwMqlHORJoZKshi0SDj0Eg4ex32maLE8IklmChmb4VkhBUmxgZUtCF8fQr/J81z16u4/o1froXLOArgGJyAM+CBS1AD16AOGoCAO/AAnsCzI51H58V5XbSuOMuZI/ADztsnDJiPcQ==</latexit>↵ <latexit sha1_base64="1t+eZUYFJcW/98Ub0gkUDmryg+o=">AAAB7HicdVDLSsNAFJ3UV62vqks3g0VwFSaa1GZXcOOygmkLbSiT6aQdOpmEmYlQQr/BjQtF3PpB7vwbpw9BRQ9cOJxzL/feE2WcKY3Qh1VaW9/Y3CpvV3Z29/YPqodHbZXmktCApDyV3QgrypmggWaa024mKU4iTjvR5Hrud+6pVCwVd3qa0TDBI8FiRrA2UtCPqMaDag3ZqF73PRci2/MuHdc3xPcbnoOgY6MFamCF1qD63h+mJE+o0IRjpXoOynRYYKkZ4XRW6eeKZphM8Ij2DBU4oSosFsfO4JlRhjBOpSmh4UL9PlHgRKlpEpnOBOux+u3Nxb+8Xq7jRlgwkeWaCrJcFOcc6hTOP4dDJinRfGoIJpKZWyEZY4mJNvlUTAhfn8L/SfvCduq2e+vWmv4qjjI4AafgHDjgCjTBDWiBABDAwAN4As+WsB6tF+t12VqyVjPH4Aest09E7Y79</latexit>
component
(b) Execution of Algorithm 1 over the logical keyword ‘because’.
not to go that way
not
keyword
<latexit sha1_base64="UFwEZuk2FaxYq1aloAXzOin9fss=">AAAB83icbVDLSgNBEOz1GeMr6tHLYhCSS9iVoJ4k4MVjBPOAZA2zk9lkyOzsMNMrhJDf8OJBEa/+jDf/xkmyB00saCiquunuCpXgBj3v21lb39jc2s7t5Hf39g8OC0fHTZOkmrIGTUSi2yExTHDJGshRsLbSjMShYK1wdDvzW09MG57IBxwrFsRkIHnEKUErdQ3qkiIl+Tgql3uFolfx5nBXiZ+RImSo9wpf3X5C05hJpIIY0/E9hcGEaORUsGm+mxqmCB2RAetYKknMTDCZ3zx1z63Sd6NE25LoztXfExMSGzOOQ9sZExyaZW8m/ud1UoyugwmXKkUm6WJRlAoXE3cWgNvnmlEUY0sI1dze6tIh0YSijSlvQ/CXX14lzYuKf1mp3leLtZssjhycwhmUwIcrqMEd1KEBFBQ8wyu8Oanz4rw7H4vWNSebOYE/cD5/APXskPo=</latexit> str(pa(nk))
<latexit sha1_base64="6UTCCfj3P0ZZiqAfdHVTj0kw6TQ=">AAAB9XicbVBNS8NAEJ34WetX1aOXYBHqpSRS1JMUvHisYD+gTctmu2mXbjZhd6KW0P/hxYMiXv0v3vw3btsctPXBwOO9GWbm+bHgGh3n21pZXVvf2Mxt5bd3dvf2CweHDR0lirI6jUSkWj7RTHDJ6shRsFasGAl9wZr+6GbqNx+Y0jyS9ziOmReSgeQBpwSN1O0ge8I0JpOS7I7OeoWiU3ZmsJeJm5EiZKj1Cl+dfkSTkEmkgmjddp0YvZQo5FSwSb6TaBYTOiID1jZUkpBpL51dPbFPjdK3g0iZkmjP1N8TKQm1Hoe+6QwJDvWiNxX/89oJBldeymWcIJN0vihIhI2RPY3A7nPFKIqxIYQqbm616ZAoQtEElTchuIsvL5PGedm9KFfuKsXqdRZHDo7hBErgwiVU4RZqUAcKCp7hFd6sR+vFerc+5q0rVjZzBH9gff4AeYiSfQ==</latexit>
pa(nk)
<latexit sha1_base64="yRo1ANztdtvHJ8g6hRyreBPT7qI=">AAAB9XicbVBNS8NAEJ34WetX1aOXYBHqpSRS/LgVvHisYD+gTctmu2mXbjZhd6KW0P/hxYMiXv0v3vw3btsctPXBwOO9GWbm+bHgGh3n21pZXVvf2Mxt5bd3dvf2CweHDR0lirI6jUSkWj7RTHDJ6shRsFasGAl9wZr+6GbqNx+Y0jyS9ziOmReSgeQBpwSN1O0ge8I0JpOS7I7OeoWiU3ZmsJeJm5EiZKj1Cl+dfkSTkEmkgmjddp0YvZQo5FSwSb6TaBYTOiID1jZUkpBpL51dPbFPjdK3g0iZkmjP1N8TKQm1Hoe+6QwJDvWiNxX/89oJBldeymWcIJN0vihIhI2RPY3A7nPFKIqxIYQqbm616ZAoQtEElTchuIsvL5PGedm9KFfuKsXqdRZHDo7hBErgwiVU4RZqUAcKCp7hFd6sR+vFerc+5q0rVjZzBH9gff4AeAeSeA==</latexit>
pa(nk)
<latexit sha1_base64="c16HAYs8sCFzjk6BmE3oMl7QOLk=">AAAB6nicbVDLSgNBEOyNrxhfUY9eBoPgKeyK+LgFvHiMaB6QrGF20psMmZ1dZmaFsOQTvHhQxKtf5M2/cZLsQRMLGoqqbrq7gkRwbVz32ymsrK6tbxQ3S1vbO7t75f2Dpo5TxbDBYhGrdkA1Ci6xYbgR2E4U0igQ2ApGN1O/9YRK81g+mHGCfkQHkoecUWOle/k46pUrbtWdgSwTLycVyFHvlb+6/ZilEUrDBNW647mJ8TOqDGcCJ6VuqjGhbEQH2LFU0gi1n81OnZATq/RJGCtb0pCZ+nsio5HW4yiwnRE1Q73oTcX/vE5qwis/4zJJDUo2XxSmgpiYTP8mfa6QGTG2hDLF7a2EDamizNh0SjYEb/HlZdI8q3oX1fO780rtOo+jCEdwDKfgwSXU4Bbq0AAGA3iGV3hzhPPivDsf89aCk88cwh84nz9TEI3P</latexit> nk
parent
<latexit sha1_base64="NtTTl0iLeaz13zfZPKc9PgrG9jQ=">AAACAnicbVDLTgIxFO3gC/GFujJuGsHEFZkhILAjceMSE3kkQEinFGjotJP2jpFMiBt/xY0LjXHrV7jzbyyPhaInaXJyzj25vccPBTfgul9OYm19Y3MruZ3a2d3bP0gfHjWMijRldaqE0i2fGCa4ZHXgIFgr1IwEvmBNf3w185t3TBuu5C1MQtYNyFDyAacErNRLn2Q7RIQjksUdYPcQUxWESjIJ01464+bKxVKxUMBuruDlK+WKJe4c2FuSDFqi1kt/dvqKRoENU0GMaXtuCN2YaOBUsGmqExkWEjomQ9a2VJKAmW48P2GKz63SxwOl7ZOA5+rPREwCYyaBbycDAiOz6s3E/7x2BINyN+YyjIBJulg0iAQGhWd94D7XjIKYWEKo5vavmI6IJhRsaylbgrd68l/SyOe8y1zhJp+pVpZ1JNEpOkMXyEMlVEXXqIbqiKIH9IRe0Kvz6Dw7b877YjThLDPH6Becj2+ltZeb</latexit> ↵ component
<latexit sha1_base64="eVkztoFe5zAyggC3GUB8oOuMoq4=">AAAB7XicbVBNS8NAEN34WetX1aOXxSJ4CklJbXsrePFYwX5AG8pku2nXbjZhdyOU0P/gxYMiXv0/3vw3btsctPpg4PHeDDPzgoQzpR3ny9rY3Nre2S3sFfcPDo+OSyenHRWnktA2iXksewEoypmgbc00p71EUogCTrvB9Gbhdx+pVCwW93qWUD+CsWAhI6CN1BkATyYwLJUdu16tVT0PO7bnVhr1hiHOEtjNSRnlaA1Ln4NRTNKICk04KNV3nUT7GUjNCKfz4iBVNAEyhTHtGyogosrPltfO8aVRRjiMpSmh8VL9OZFBpNQsCkxnBHqi1r2F+J/XT3VY9zMmklRTQVaLwpRjHePF63jEJCWazwwBIpm5FZMJSCDaBFQ0IbjrL/8lnYrtXtvenVduNvI4CugcXaAr5KIaaqJb1EJtRNADekIv6NWKrWfrzXpftW5Y+cwZ+gXr4xvsWo9a</latexit>↵ component
<latexit sha1_base64="Dsco2EY4KjZJ3ZkeSDD+Q8MR0XE=">AAACBXicbVA9SwNBEN3z2/h1aqnFYhBiE+7k1KQL2FhGMFFIYtjbTHTJ3t6xOyeGI42Nf8XGQhFb/4Od/8bNR6HRBwNv35thZ16YSGHQ876cmdm5+YXFpeXcyura+oa7uVU3cao51HgsY30VMgNSKKihQAlXiQYWhRIuw97p0L+8A21ErC6wn0ArYjdKdAVnaKW2u9tEuMcsYYOCuu4d0KYBjIRKDbXPtpv3iqWjk6MgoF4x8A/LpbIl3gjUn5A8maDadj+bnZinESjkkhnT8L0EWxnTKLiEQa6ZGkgY77EbaFiqWASmlY2uGNB9q3RoN9a2FNKR+nMiY5Ex/Si0nRHDWzPtDcX/vEaK3VIrEypJERQff9RNJcWYDiOhHaGBo+xbwrgWdlfKb5lmHG1wORuCP33yX1I/LPrHxeA8yFfKkziWyA7ZIwXikxNSIWekSmqEkwfyRF7Iq/PoPDtvzvu4dcaZzGyTX3A+vgHMh5jC</latexit> pa(nk) \ nk
(c) Execution of Algorithm 1 over the logical keyword ‘not’.
Figure 3: An example execution of Algorithm 1 on the example sentence ‘But I try very hard not to go that way because it would be too easy for them.’ over the three detected logical keywords ‘But’, ‘because’, ‘not’.
Then we search through all the constituent nodes in T (s) to detect the ones that exactly matches the keyword strings of the logical keywords as defined in Section 2. Let Nkey(s) denote the set of constituent node in T (s) that matches logical keywords. Then for each logical keyword node nk ∈ Nkey(s), we fetch its parent constituent node pa(nk). Now we have three different cases:
1. If nk corresponds to a unary logical relationship (i.e. negation and example), then the α component of nk is detected as: pa(nk) \ nk.
2. If nk corresponds to a binary logical relationship and the relationship is intrinsically mapped, then str(pa(nk)) will be divided by str(nk) into three different segments: str(pa(nk)) = A + str(nk) + B. Now the α component of nk is detected as A and the β component of nk is detected as B.
3. If nk corresponds to a binary logical relationship and the relationship is extrinsically mapped, then the α component of nk is detected as: pa(pa(nk))\pa(nk), and the β component of nk is detected as: pa(nk) \ nk.
Our proposed methods for logic detection and mapping described above are summarized in Algorithm 1. See Figure 3 for an example of executing Algorithm 1 on an example sentence taken from the example article in Figure 6, based on the constituency parsing tree depicted in Figure 1.

heading: 3.1 Sense Disambiguation of Logical Keywords
text: In English, certain logical keywords have multiple meanings and can indicate different logical relationships under different contexts. For example, the logical keyword ‘since’ has two different meanings: (1) ‘from a time in the past’, which indicates a temporal logical relationship; (2) ‘because’, which indicates a causal logical relationship. In our categorization of logical relationships and keywords (described in Section 2), there are a total of 3 keywords that can have multiple logical meanings: since, as, and while. Therefore, in order to increase accuracy of our proposed logic detection module, we need to first perform accurate logical sense disambiguation when we detect these logically ambiguous keywords.
In our empirical experiments over a set of randomly sampled sentences that contain ambiguous logical keywords, each manually-labelled with its ground-truth logical relationship under the context,
we found that different uses of ambiguous logical keywords have very strong clustering tendency and are largely linearly-separable under the contextualized encoding of transformer language models. For example, we use the ALBERT model (Lan et al., 2020) to encode 20 different occurrences of the logical keyword ‘since’ randomly sampled from the CNN/Dailymail dataset (Nallapati et al., 2016), and project the last-layer hidden state vectors for these 20 ‘since’ onto their first two principal components using Principal Component Analysis (PCA) (Hotelling, 1933), which is depicted in Figure 2. As we can see from Figure 2 the contextualized embeddings of the logical keyword ‘since’ are largely linearly separable between the two different logical meanings.
Therefore, in order to improve the accuracy of our logic detection module, we first manually collected logical relationship annotations for the set of ambiguous logical keywords in English. Then we encode them using the ALBERT model (Lan et al., 2020) and train individual support vector machine (SVM) (Cortes and Vapnik, 1995) classifiers for each of the ambiguous logical keywords to accurately disambiguate their different logical meanings.

heading: 4 Logical Transformers

heading: 4.1 Logical Embedding Vectors
text: The major new parameters that we introduce in our proposed modeling framework of logical transformers are a set of parametrized and trainable logical embedding vectors. These logical embedding vectors share the same dimensionality, but their dimensionality doesn’t necessarily equal to the dimensionality of the transformer language model’s token embedding vectors. Below we describe how to construct these logical embedding vectors in detail.
First of all, the 12 types of logical relationships we defined in Section 6 can be classified into two different categories: (1) ‘unary logical relationship’ that maps to only one logical component; (2) ‘binary logical relationship’ that maps to two logical components. More specifically, negation and example are unary logical relationships and all the other 10 types are binary logical relationships.
For each unary logical relationship U , we construct two parametrized logical embedding vectors: vUkey and v
U . In the logical embedding layer of U , we assign vUkey to each token detected to be part of
an appearance of some logical keyword in U , and assign vU to all the tokens that are within some text span mapped by some logical keyword in U .
For each binary logical relationship B, we construct three parametrized logical embedding vectors: vBkey, v B α and v B β . In the logical embedding layer of B, we assign vBkey to each token detected to be part of an appearance of some logical keyword in B, assign vBα to all the tokens that are within some left text span mapped by some logical keyword in B, and assign vBβ to all the tokens that are within some right text span mapped by some logical keyword in B.
And finally we construct another special parametrized logical embedding vector vE that corresponds to empty logical association. For each token that doesn’t belong to any logical relationships in a logical embedding layer, it will be assigned vE
for this layer. See Table 1 for a concrete example of assigning multiple layers of logical embedding vectors to tokens in an input sequence based on the results of logic detection and mapping.
Therefore, based on the 12 different types of logical relationships that we defined in Section 6, we will construct a total of 2×2+10×3+1 = 35 different logical embedding vectors for our logical transformers.

heading: 4.2 Multi-Layer Hierarchical Logical Projections
text: Now we describe how to compute the logic-aware input embeddings through multi-layer hierarchical logical projections using the set of logical embedding vectors that we defined in Section 4.1. Let Nlogic denote the dimensionality of the logical embedding vectors, and let N denote the dimensionality of the token embedding vectors of the transformer language model. We first define a parametrized and trainable linear transformation layer L that projects a (N +Nlogic)-dimensional vector into an N -dimensional vector.
Then for each token t in the input token sequence, we collect all the logical embedding vectors assigned to it during the logic detection and mapping process and sort them in order according to their associated logical keywords’ depth in the constituency parse tree of the input sentence. Let’s denote this sorted set of all the logical embedding vectors assigned to token t as: {v1t , ..., vKt }, where K is the maximum number of logical layers to be considered and should be treated as a hyperparameter.
Now let’s denote the original token embedding vector for token t as wt, then to compute a logicaware token embedding vector wlogict for t, we first initialize u0t = wt, and then recursively apply the following computation4:
4This series of (linear projection + nonlinear activation) can also be replaced by a series of multilayer perceptrons.
uit = f(L(ui−1t ⊕ vit)), for i = 1, ...,K, where ⊕ denotes vector concatenation and f is some non-linear activation function, such as GELU (Hendrycks and Gimpel, 2016). Then we have:
wlogict = wt + u K t .
Now let pt denote the position embedding vector of token t and st denote the segment embedding vector of token t, then the final logic-aware input embedding vector for each token t in the input sequence would be computed as: wlogict + pt + st. Then at the front end of our proposed logical transformers, we use these logic-aware input embeddings to replace the traditional input embeddings and feed them into transformer encoders to help language models better encode and learn logical information from the textual inputs. See Figure 5 for an illustration of multi-layer hierarchical logical projections for an example token with logic depth K = 3.

heading: 4.3 Model Training
text: During the training of our proposed logical transformers, we set both the set of 35 logical embedding vectors and the linear transformation layer L to be fully parametrized and trainable, and then initialize them with random values. All these added new parameters will be updated together with the original trainable parameters in the transformer language models during the model training process.

heading: 4.4 Negligible Increase in Model Size
text: The only new parameters introduced in our proposed logical transformers, compared with their corresponding baseline transformer language models, are the set of 35 logical embedding vectors and the linear transformation linear L used in hierarchical logical projections. Let Nlogic denote the dimensionality of the logical embedding vectors, then
the total increase in model size can be calculated as: Nlogic×35+(N+Nlogic)×Nlogic+Nlogic = N2logic +N ·Nlogic + 36Nlogic.
For all the recently proposed transformer language models, this increase in model size is rather small and negligible compared with their very large number of parameters. For example, for the RoBERTa-large model (Liu et al., 2019), its total number of parameters is 355M and the dimensionality of its embedding vectors is 1024. If we set Nlogic = 1024 as well, then after we use our proposed new modeling paradigm to upgrade RoBERTa-large into Logical-RoBERTa-large, the percentage of increase in model size is only: (10242 + 1024 × 1024 + 36 × 1024) ÷ 355M ≈ 0.601%, which is almost negligible. This efficiency in model size guarantees that the logical transformers take roughly the same amount of computation time during both training and inference as their baseline transformer language models.

heading: 5 Experiments
text: In order to evaluate our proposed logical transformer architecture’s performance boost on different NLU and NLG tasks with different transformer language models, in our experiments, we test it on three NLU datasets and one NLG dataset.

heading: 5.1 Natural Language Understanding Tasks
text: In the NLU part of our experiments, we test the RoBERTa model (Liu et al., 2019) and our Logical-
RoBERTa model on three logically-challenging natural language understanding tasks over three corresponding datasets: (1) reading comprehension on the ReClor dataset (Yu et al., 2020); (2) question answering on the LogiQA dataset (Liu et al., 2020); and (3) dialogue-based reading comprehension on the DREAM dataset (Sun et al., 2019). All of these three datasets require logical reasoning.

heading: 5.2 Natural Language Generation Task
text: In the NLG part of our experiments, we test the BART model (Lewis et al., 2020) and our LogicalBART model on the task of dialogue summarization over the DialogSum (Chen et al., 2021) dataset.

heading: 5.3 Results
text: The results of our three NLU experiments are shown in Table 2, and the results of NLG experiment are shown in Table 3. As we can see from Table 2 and Table 3, the accuracy scores and the ROUGE scores of our logical transformer language models are consistently higher than their corresponding baseline transformer language models across all the different NLU and NLG tasks. This consistent boost demonstrates that the important logical structures and information extracted and captured by our proposed logical transformers are indeed very effective and useful in further improving transformer language models’ performance on logically-challenging NLU and NLG tasks.

heading: 6 Related Work
text: Recently there has been increasing interest in improving pre-trained language models’ logical reasoning ability (Xu et al., 2022; Pi et al., 2022). For example, Lu et al. (2022) proposed a new method for parsing natural language into the forms of propositional logic and first-order logic using dual reinforcement learning. Pi et al. (2022) proposed a new unsupervised adversarial pre-training method, called LogiGAN, in order to enhance language models’ abilities of logical reasoning. Xu et al. (2022) proposed a new Logiformer architecture based on a two-branch graph transformer network to improve language models’ performance on interpretable logical reasoning.
In contrast to these previous work that mostly focus on introducing new training methods or constructing complex model architectures, our proposed method in this paper only modifies the input embeddings and is thus more straightforward
and easily generalizable to different types of transformer language models.

heading: 7 Conclusion
text: In this paper we introduced a new modeling paradigm for transformer language models that detects and extracts important logical structures and information from input texts and then integrates them into the input embeddings through carefully designed multi-layer hierarchical logical projections to infuse logical structures into pretrained language models. Our empirical experiments on four important and challenging NLU and NLG tasks showed that our proposed logical transformer language models consistently perform better than their corresponding baseline transformer language models through a deeper understanding of the key logical structures underlying natural language texts.

heading: 8 Limitations
text: In theory, the method proposed in this paper can be applied to different types of transformer language models for both pre-training and fine-tuning. Due to limit of computational resource, we currently haven’t had the chance to test our proposed method in the very promising setting of large-scale language model pre-training yet. In future work,
we plan to further test our proposed logical transformer architecture on large-scale language model pre-training to see how much performance boost it can achieve.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 8
7 A2. Did you discuss any potential risks of your work? There are no potential risks of our work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.

heading: B1. Did you cite the creators of artifacts you used?

heading: No response.

heading: B2. Did you discuss the license or terms for use and / or distribution of any artifacts?

heading: No response.

heading: B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.

heading: B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.

heading: B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be.

heading: No response.
text: C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3 and Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.

heading: D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? No response.

heading: D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.

heading: D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? No response.

heading: D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?

heading: No response.

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: No response.

heading: Future Work
text: In future work,
we plan to further test our proposed logical transformer architecture on large-scale language model pre-training to see how much performance boost it can achieve