PhraseTransformer: Self-Attention using Local Context for Semantic Parsing
Authors: Phuong Minh Nguyen
Vu Tran
Minh Le Nguyen
Venue:  (2021)
Keywords: phrase transformer
logical form
semantic parsing
neural machine translation
meaning representation

Abstract: Semantic parsing is a challenging task whose purpose is to convert a natural language utterance to machine-understandable information representation. Recently, solutions using Neural Machine Translation have achieved many promising results, especially Transformer because of the ability to learn long-range word dependencies. However, the one drawback of adapting the original Transformer to the semantic parsing is the lack of detail in expressing the information of sentences. Therefore, this work proposes a PhraseTransformer architecture that is capable of a more detailed meaning representation by learning the phrase dependencies in the sentence. The main idea is to incorporate Long Short-Term Memory (LSTM) into the Self-Attention mechanism of the original Transformer to capture more local context of phrases. Experimental results show that the proposed model captures the detailed meaning better than Transformer, raises local context awareness and achieves strong competitive performance on Geo, MSParS datasets, and leads to SOTA performance on Atis dataset in methods using Neural Network.

Under review as a conference paper at ICLR 2021
PHRASETRANSFORMER SELF-ATTENTION USING Lo- CAL CONTEXT FOR SEMANTIC PARSING
Anonymous authors Paper under double-blind review
ABSTRACT
Semantic parsing is a challenging task whose purpose is to convert a natural lan- guage utterance to solutions using Neural Machine Translation have achieved many promising re- sults, especially Transformer because of the ability to learn long-range word de- pendencies. However, the one drawback of adapting the original Transformer to the semantic parsing the lack of detail in expressing the information of sen- tences. Therefore, this work proposes Phrase Transforre architecture that is capable of a more detailed meaning representation by learning the phrase depen- dencies in the sentence. The main idea is to incorporate Long Short- Memory (LSTM) into the Self-Attention mechanism of the original Transformer to capture more local context of phrases. Experimental results show that the proposed model captures the detailed meaning better than Transformer, raises local context aware- ness and achieves strong competitive performance on Geo, MSParS datasets, and leads to SOTA performance on Atis dataset in methods using Neural Network.
information representation. Recently.
INTRODUCTION
Semantic parsing is an important task which can be applied for many applications such as Ques- tion and Answering systems or searching systems using natural language (Woods 1973 Waltz & Goodman 1977). For example, the sentence "which state borders hawaii" can be represented as logical form (LF) using X-calculus syntax "(lambda SO (and (state:t $0) (next to:t SO hawaii)))" There are various strategies to address the semantic parsing task such as constructing handcraft- rules (Woods 1973 Waltz& & Goodman 1977 Hendrix al. 1978), using Combinatory Categorial
Grammar (CCG) (Zettlemoyer & Collins 2005 2007 Kwiatkowski et al. [011]. adapting statistical
machine translation method (Wong & Mooney 2006:
2007) or Neural Machine Translation (Dong
& Lapata 2016 Jia & Liang 2016 Dong & Lapata 2018 Cao et al., 2019). The major factor of the CCG method is based on the alignments of sub-parts (lexicons or phrases) between natural sentence and corresponding logical form and to learn how best to combine these subparts. In more detail, the phrase "borders hawai" aligned to "(nexi_to: SO hawaūzz)" LF. Conversely, the methods using Neural Machine Translation learn the encoder representing sentence into vector and decode that vector into LF. The current SOTA models are Sequeence-o-Sequeeee using LSTM (Seq2seq) (Dong & Lapata 2018 Cao al.) 2019) Geo, Atis and Transformer (Ge et al. 2019) on MSParS. The methods using Neural Network almost work effectively without any handcrafted features. However, there is still room improve the performance based the meaning of local context in phrases.
According CCG methods, the semantic representation of sentence is the combination of sub- meaning representation generated by phrases in a sentence. However, Transformer architecture
Phrase Transformer
Encoder
1:0 cil one waybefor uo
hi could i get one way ticket from ci0 to you pleas give fare from c00 coach way
way
to
Decoder
( (lamdda $0 (and oneway $D El departure time 50 to from cid to c0
Figure 1: Pharse alignments in ChraseTransformer

---

Under review as a conference paper at ICLR 2021
only learns the dependencies between single words without considering the local context by the phrase. Therefore, we propose a new architecture named PhraseTramsffrmec that focuses on learning the relations of phrases in sentence (Figure T. To do this, modify the Multi-head Attention VVaswani et al. 2017) by applying the self-attention mechanism into phrases instead of single words. Firstly, we use n-gram split sentence into phrases. Then, we use the final hidden state of LSTM architecture to represent the local context meaning of those phrases.
Our contributions are: (1) proposing a novel model based on Transformer that works effectively for semantic parsing tasks, (2) conducting experiments to confirm the awareness capacity of the model, (3) achieving competitive performance on Geo, MSParS datasets and new SOTA performance on Atis dataset in the methods using Neural Network.
2 RELATED WORK
In Semantic Parsing task, recent works have shown that using the deep learning approach achieved potential results. results. These methods are divided into three groups:
Decoder Customization. Dong & Lapata apply the Seq2seq model to semantic parsing task and introduce Sequence-to-tree (Seq2tree) (Dong & Lapata| (0166) model constructing the tree struc- ture of the LF. This model focuses on modifying the decoding method based on bracket pairs to start a new decoding level. On an other aspect, Dong & Lapata (2018) continue to introduce new architecture Coarse-to-Fine (Coarse2Finee) based on a rough sketch of meaning to improve the structure-awareness of Seg2seq model. Similarly, [Li et al. (2019) also use the sketch meaning mech- anism on BERT model (Devlin et al. 2019) by two steps: classify the template of I.F and fill the low-level information to that template. our opinion, the main problem is to improve the under- standing capacity of the model because semantic parsers need to capture the complicated in the natural sentences before decoding. Therefore, our work focuses on designing the Encoder architec- ture improve the understanding capacity of the model.
Data Augmentation. There are numerous works that focus on data augmentation to improve the performance of the semantic parsing model (Jia & Liang] 2016 Ziai 2019: Herzig & Berant 2019). 2019). Jia & Liang propose three rules based on Synchronous Context-Free Grammar to recombine data. This step increases the size of the training data and grows the performance of the model (Tia & Liang| 2016). Similarly, Ziai] proposes a method that automatically augments data based on the co- occurrence of words in the sentence. The author separates the training process into two phases: (1) use augmented data to train for BERT (Devlin et al. 2019) and (2) fine-tuning on original data.
Weak Supervision. Some methods use semi-supervised learning for semantic parsing task such as (Kočiský et al.] 2016; Yin et al.] 2018 Goldman et al., 2018 Cao et al.) 2019: 2020). These works are promising approaches for the data- data-hungry problem because of the ability extract latent information such as unpaired logical forms. In our proposed model, we aim to construct the latent representation for phrases and learn these representations via the self-attention mechanism of the Transformer. We hypothesize that complicated sentences are constructed from various phrases, so learning to represent these phrases makes the model more generalizable.
In Neural Machine Translation task, the approach using phrase information or constituent tree is proved that effective and attracts many works (Wang et al. 2017] Wu et al, 2018 Wang et al. 2019
Hao et al. 1019.Nuue et al.) 2020). The points that make the difference in work are: (1) our model is capable of learning without any additional information (e.g. constituent tree), (2) in the training process, although we do not force the attention limit the scope of the dependencies, model is able to pay high attention to the important phrase automatically. Compare with Yang et al.) 2018). the purpose of using local context information similar but different in localness modeling: based on the distance, Yang et al. (2018) cast a Gaussian bias to change attention score while our method is simpler by incorporating multi different n n-gram views as the various local contexts.
MODEL ARCHITECTURE
Our novel architecture (Figure 2) is based on the Decoder of Transformer (Vaswani et al. 2017]. We define a new model named PhraseTransformer to improve the encoding quality of Trans- former by enhancing the Encoder architecture while keeping the original Decoder.
2

---

Under review as a conference paper at ICLR 2021
Legends
Neural Layer
Vector or matrix
Hidden vector (th
X N
Encoder
Transtormer
Add Norm
Multi-Head Attention
Phrase Transtormer
Phrase
Ai- gramL
9 oulpui
Identical
Layer
Feed Forward
Linear
V3
VS
V
V FB
Add Norm
1
2
R3
RA
K5
7
Concal
Multi-Haad
1
43.4
9
9
Allention
Scaled Dot-Product
vf
Attention
2-gram LSTM
3-gram LSTM
4-gram LSTM
R5
Positional
Z-gram
3-gram) LSTM
4-gram
LSTM
LSTM
Encoding
Input Embedding
V1
V2
V3
V4
V5 M6
V7
VB
Ho H1
H6 Ho
Linear
Linear
K1
k2
kg
K4
kg 16
ky
kg
one Dalor TO
Inputs (ol0 a'1 way
heads
41
92
43
94
45
96
47
98
45 inpurn
1951 96
Phrase Transformer Encoder architecture using n_gram LSTM in MultiHead Layer. In this case, n-gramLSTM layer is built with n [0, 0, 0, 2, 4, 4], 2-gram, 3-gram, 4-gram models apply to every two heads from head 3 to head 8.
Figure 2:
Transformer Encoder (original). In Transformer Encoder architecture, Vaswani et al. (2017) proposed stack of N Identical I Layers; each layer consists of two sub-layers: Multi-Head Attention layer and Position-wis wiie Feed--d layer. Let æ be an input vector synthesized from the vector word embedding and positional encoding
[Z1,.... T|s]] where [S] is sentence length.
In the Multi-Head Attention layer Vaswani et al. use the Linear layer to get multi-views for the inputs. This layer processes the input vector (x) and generates H distinct featured vectors (Hf is the number of heads) and forwards to Self-Attentio layer using Scaled-Do Product. After that, all heads processed by Concat and Linear layers to compute the output of the Multi-Head layer.
4i, k;, U; x W head, Attention(ge k; v.;)
hMUHH h head W
(3)
hNorm LayerNorm(ha..) (h + x))
(4)
hout Layer (Frerooonnneddeoooaaa(aaooooll + hNorm)
(5)
where Attention Scaled Dot-Product Attention:
Attention(q.: k, v,) = softmax q.k. vdn
66)
where da is dimensions per head, i is the identical index of head (0 < H), W is parameters, NorerNorm, FeedForward are the functions that are used similar to Vaswani et (2017).
Phrase Transformer Encoder. The Encoder is enhanced from the original model in Multi-Head Attention layer because this layer is the major factor to extract the features of inputs sequence. More detail, after H heads are generated by Linear layer, grams and use Bidirectional LSTM (Hochreiter & Schmidhuber 1997) to extract the local context information of these grams (Figure 2). Besides, assume that the meaning phrases are usually composed by difference length, therefore we use various n-gram models. To do this, the Phrase function is in Equation
n-gram model to split the sentence into
Phrase(s:) ( (m.ggaanLSM((s) if n4 40 otherwise
where is sequential hidden state of sentence of head i (0 < i H) in N is gram size vector for H heads; n, is the gram size corresponding to head i; n. gramLSTM is procedure that splits the sequential input into grams by n4 gram model, and applies Bidirectional LSTM for each gram of s,:
layer;
n, W-ggraLSSM((ss) [n (s.)
3

---

Under review as a conference paper at ICLR 2021
where n4- PSg is the Bidirectional LSTM computed by sum of forward and backward final hidden state:
T4 ww.gramLSTMMyc..
LSTM/ (nt gramy, (
(9)
Ri-gramng (8:) [1 -- Ha k Hx]
(10)
where H3 is the hidden state corresponding to word index & in a sentence, n, fi-gram is the gram index that is list of n, continuous hidden states (paddings zero for first words), Si-gramISTM, (s.) is the vector to capture local context of the gram index k. After that, the query (qi), key (k;), value (v;) matrixes (Equation 2 are replaced by Phrase function:
q. k/, v/ Phraso(q:)) Phraso(k;). Phraso(e,)
(11)
head; Attention(() q/ k v/)
(12)
Residual Connection. Similar to the original Transformer architecture, also employ residual connection as an extension aim to effectively integrate local context features and current word in- formation. We used sigmoid((r) function to adjust the rate of context (n, gramLSTM, (s.)) and
current word (8t,.) information. The hidden state n, gramLSTM, in Equation g is replaced as following:
T1 ni-gramLSTM, (s.) = (81,k)
n..gramLSTM, - + (1 (8.,k)) 8i,
(13)
Finally, M NoNoms hout are computed similarly to Transformer architecture.
Model variantion. We replace the method representing local context (Bidirectional LSTM) by an other simple method that is Sum of all hidden state of words in the phrase. More detail, we customize the Phrase function by replacing n gramLSTM (Equation
7
with n; grannaum:
ni ni-gramSum( &;) [nu gramSum,
(14)
ni-gramSump (s,) Σ
(n4 Tr4-gram, (s:)
(15)
where (i-gram.(ss) function is computed similar to the Equation
10
Training method is to maximize the Log-Likelihood function of the probabilities generate the LF (y) given a sentence (z) from annotated dataset (D);
maximize:
Σ <>D>>D
log Po (y/x)
(16)
Metric measurement. al datasets, we compute sentence-level accuracy by using exact match- ing (EM) and logic matching (LM) that developed by Dong Lapata (2018). LM metric measures the performance better than the EM method because it is probable for comparing the variant of ex- pression. For example, the predicted LFs in different order and logic: and ( (nneway $0 ) <( departure time $0 ti0) is equal and <{ departure time $0 ni oneway $0 )
4 EXPERIMENTS
The purpose of experiments is to compare the performance of TarassTransfrmmrr and extension models with the original Transformer. Besides, we explore the awareness about the phrase alignment between sentence and the generated LF by ThraseTrranffrmes
4.1 DATATSETS
We conduct the experiments on three datasets Geo (Zelle Mooney 1996), Atis (Dahl et al. 1994) and MSParS (Duan) 2019). Table shows the observation of these datasets. Geo and Atis datasets are small size but more complicated in information relations than the MSParS dataset. The average length of LFs Atis dataset (28.4) is about twice longer than that MSParS dataset (14.7). The
original MSParS dataset have large vocabulary (around 40k) because it consists various entities name in the open domain. Therefore, preprocess this dataset similarly to Ge et al. (2019) by replacing character by " and using byte-pairs-a pairs-encodinng (BPE) (Sennrich et al. (016) to deal with rare-word problem.

---

Under review as a conference paper at ICLR 2021
Geo consists of queries about geography information of the U.S. and LFs in lambda- calculus syntax. We use the version preprocessed by Dong & Lapata (2016) by replacing all entities by numbered markers (e.g. "new york" "s(").
Atis consists of queries about flight information and LFs in lambda-caleulus syntax. We also use the version preprocessed by Dong & Lapata (2016) similar to Geo dataset.
MSParS is large-scale open domain dataset with LFs in lambda-DG syntax (Liang et al., 2011). This dataset contains 12 question types (Duan 2019) such as single-relations multi- turn-entity, etc. for Knowledge-baie Question Answering system.
Table 1: Statisties information of three datasets. The MSParS dataset (BPE6k) is preprocessed by BPE 6000 operations. Vocabulary size and average length of source (Src) and target (Tgt) side are computed train set.
Total examples
Vocab size
Avg. Src
length Tgt
Datatset
Train Dev Test
Src
Tgt
Geo
600 0 280
433
51
10.6
18.7
Atis
3434 491 448
120
166
7.3
28.4
MSParS (BPE6k)
63826 9000 9000
4965
5854
12.8
23.9
4.2 SETTINGS
In training processes of the MSParS and Atis datasets, to prevent overfitting, we use the early stop- ping conditioned on metries word-level or sentence-level accuracy dev set.
Because Transformer is quite sensitive in hyper- parameters, we keep most the same as Transformer- of layers N 6 and number of heads in Multi-Head layer H 8; hidden size dmmade
model (Vaswani et al.) 2017) such as the number
512;
dropout is selected in f0.1,0,3 0,33: Adam optimizer with 31 = 0.9, 82 0.998, c = 10-9 The weights of models are initialized with Xavier initialization (Glorot Bengio 2010). The embedding vectors are shared among the source-side and target-side, between the input-to-embedding layer and output-to output-to-softmax layer in Decoder. We also retain the learning rate decay method: Ir(ste))
d dmoded -iidd--min(st wwt-mminnttte step 0.5, step-warmup_stteee size for each head is selected in {0, 2, 3, 4]. The weights of Bidirectional LSTM layers in the heads using the same n-gram model (e.g. heads 3, 4) are shared. Besides, the experimental dataset sizes are quite different, therefore we use three papee--prrmmter sets"] Geo: warmup step 100 learning rate init selected from (0.05, 0.1), satchhssize 128 (the batch size using number of tokens), the maximum traning steps mar_steps 15000; Atis: warmuup _steppssepppppppp 100 learning rate init selected from (0.1, 0.2). batch_size = 4096. the maximum traning steps maa _stesss 250000; MSParS: warmup_step = 8000, learning rate init selected from {0.5, 1.0, 2.0), batch_size 8192. the maximum traning steps max steps = 250000. On this dataset, we conducted experiments to check the number of BPE operations impacting performance (Figure Based those results, the MSParS dataset preprocessed by BPE 6000 operations for all other experiments.
1.5) where step is the current step number. The n-gram
4.3 RESULTS AND ANALYSIS
4.3.1 PERFORMANCE
Model setting We conducted experiments to find the best gram sizes for PhraseTransformer on Atis and MSParS (Table 2) because the size of those datasets are larger than Geo that make the results are more stable. We hypothesize that performance increases when applying various gram sizes to the Atis dataset. By using various gram sizes, PhraseTrannffrmerr can see different linguistic features in various local context sizes in Multi-head layers. For domain adaptation, the gram sizes be chosen depending observing the number of words in meaningful phrases. Using various gram sizes makes Phrase Transformer more generalized. Besides, using LSTM to represent spans on all layers helps PhraseTTrasformin capture more sequential information than Transformer.
Thee model using bold value achieved better performance than other values our experiments.
5

---

Under review as a conference paper at ICLR 2021
Residual Connection. On MSParS dataset, the performance is not so different when changing gram sizes (models 4 Table 2 We observe that because this dataset has diversity in object name with more than 75% words in vocabulary appearing less than 4 times in train set. One of the challenge of this dataset is to recognize the object name and type, so capturing original word features is im- portant. These words are usually splited into many word pieces by BPE, the 14.gramI.STM component lose original word information when intergrating parts of previous word. For example, the sentence "boonie bears last movie was" " is preprocessed by BPE: "bo @ @ on ie be @ @ ars last movie was" and the ni-gramLSTMM component considers similar grams [bo@ @ on @ @ ie be@@), J. [on@ ie be@@], J. [ie @e@@] J when representing @e@ word vector. Transfrranf equipped with the residual connection (model 5) is able to avoid losing original word pieces features, thus shows better performance on MSParS.
Model variantion. We conducted the experiments to check the impact of localness modeling be- tween BiLSTM (model 5) and the Sum function (model 6). On both datasets, the model using BiLSTM achieved better performance because the LSTM model is better than the Sum function in meaning representation. However, on the Atis dataset, PhraseTranssfrmee using Sum improved slightly (about 0.3 %) with the original Transformer. This result shows that local context is one of the important features for this dataset.
Table 2: Sentence-level accuracy using exact matching (EM) and logic matching (LM) on two datasets Atis and MSParS using BPE 6000 operations. The abbreviation Res. implies that we used residual connection in Equation 13
Atis (EM/LM
MSParS (EMAL)
Id. Model
gram sizes (n))
Dev
Test
Dev
Test
1. PhraseTrrans
00;0; 0: 0; 2; 2:
86.76 / 88.80
87.95 / 88.84
85.62 / 85.99
84.68 85.18
PhraseTraans
0; 0; 0; 3; 3:3
86.76 / 88.80
88.17 89.51
86.07 86.52
85.
85.72
Phrase'Tram
0;;0; 2;
86.76 / 88.19
89.06 / 89.96
85,53 85.99
85.04 85.39
4. Phrase PhraseTrran.
2;
87.17 / 89.21
89.51 / 90.40
85.88 / 86.24
85.08 85.47
5. Phrase Trans, Res.
(0;0;;;;;;;3;
87.58 / 89.61
88. 62.6 / 89.51
86.23 86.73
85.72 86.21
PhraseTrans.
2;;0;;
86.96 88.59
87,05 / 87.95
85.68 86.18
85.21 85.82
Table 3: Evaluation results using Logic Matching on all datasets. The reported results on Geo are mean and standard deviation values. The values marked (*) mean that the evaluation metric is denotation match that different from others using sentence-level accuracy. This table contains two parts, the upper part shows the results of previous works and bellow part present our results. Models 4, 5 refer the Id of model in Table 2
Geo
Atis
MSParS
Z&C (Zettlemoyer Collins
2007
86.1
84.6
X- WASP (Wong & Mooney 2207
86.6
FUBL (Kwnatkowskr el al. (01T))
88.6
82.8
TISP (Zhao & Huang 2015)
88.9
84.2
Seq2tree (Dong & Lapata 2016
87.1
84.6
Seq2xeq=Copy (Jia & Liane 22066]
89.3*
83.3
Coarse2Fine (Dong & Lapata
2018
88.2
87.7
DualLearning (Cao et al. 2019
89.1
Sert-Sketch (L. al.
84.47
Transformer (Ge ct 2019)
85.68
Transformer (ours)
86.8110.76
87.7
86.19
PhraseTrans. (Model 4)
87.9+-0.36
90.4
85.47
PhraseTrans. Res. (Model 5)
89.5
86.21
Other methods comparison We compare the performance of Transforraee with the original Transformer and other methods in previous works in Table 3 The learning curve in Figure 4 also shows that beat clearly Transformer on the Geo dataset on all checkpoints. On Atis dataset, PhraseTransformar is better than Transformer all settings of gram sizes (Table

---

Under review as a conference paper at ICLR 2021
Token-level Acc.
Token-level Acc.
97
98
BPE 6000
96
Trans. (ours)
BPE 8000
96
BPE 12000
Phrase
95
BPE 16000
Steps (x1000)
Steps (x1000)
3
4
5
7
8
9 10
Figure 3: The impact of BPE preprocess- ing performance of ThraseTransforme MSParS dev set.
Figure 4: Token-level accuracy (min, max and average) of PhraseTransforme and the original Transformer on Geo test set.
Our model achieves better results Atis, MSParS and competitive with previous results the Geo dataset. While our method does not use augmented datasets similarly to Jia & Liang (2016); Ge et (2019) the sketch information (Dong & Lapata 2018), these results show that model learns more effectively than the others.
PhraseTransffrmm Encoder layers Some recent works (Hao et al., 2019 Yang et al., 2018) show that the different combinations of the layers caturing local context can impact the performance of the model. Therefore, we conducted the experiments that drop the phrase mechanism on some top layers to explore this impact (Table 4 Comparing with the original Transformer, the PhraseTTrans former improved performance even only applying phrase mechanism on the first layer. Besides, Phrase Transforrre is more general when applying phrase mechanism all layers.
Table 4: Evaluation using Logic Matching for Transforree in difference layers on Atis dataset. Column Layers indicates the layers ap- plying g.g.
Table 5: Comparison of number param- eters (M=million) and training speed (to- kens per second, K=thousand) MSParS dataset.
Layers
#Param.
Dev
Test
Model
#Param.
Speed
[1]
44.5 M
89.21
88.17
Transformer (ours)
47.1 M
9,0 K
2]
44.7 M
88.80
89.06
PhraseTrans.
7.1
3
44.9 M
89.41
89.29
Phrase Reans.Res.
48.3 M
6,9 K
6
45.5 M
89.21
90.40
6
44.9 M
89.00
89.29
Computation time We compare the number of parameters and training speed between the Trans- former and ChraseTransformer on the largest dataset MSParS (Table 5). This experiment con- ducted on 1 GPU P100, 16Gb ram with batch size is 8192 tokens. The training speed of Phrase- Transformer model is about 76-79% the original Transformer. fact, although we used LSTM on Heads, the computation time is not dependent on the length of sentence because we can forward and backward all n u-grams of all sentences in minibatch at the same time. Therefore, the computation time is more dependent the gram size (in this case, the maximum gram size is 4). Besides, the number of parameters the Phrase Transformer slightly increased (about 2.5%) when compare with the original Transformer.
4.3.2 AWARENESS
Alignment We inspect the information learned in Phraseerrnntfrmrr in Attention layers (Fig- ure 5a more in Appendix A.2). We observe that Phrase ThraseTransformer could represent attention in- formation more clearly than Transformer. In both two models, the token ground_rranspon in LF is aligned correctly to phrase "ground transport" in the sentence (red aligmments). In PhraseTrans former, tokens city, from airport also correctly aligned the corresponding words "ap0",

---

Under review as a conference paper at ICLR 2021
"ci0" in the sentence (green and yellow alignments) because these word vectors probable to capture local context better than Transformer. Besides, all tokens decoded by Phrase Trausffrannfrmer paid the same attention other words that is not key information, such "is there", "into" "citi". These evidences is positive signals show that the self-awareness of Phrase'Tram better than Transformer.
"parree
-08
braundtraaapan
Ireust tranigett
10-
rardd
bichy
18ght
han spar
from ahoor
der EXIEU
gho burogee
(a) Decooerr Attention.
(b) Self-Attentin of PhraaeTTransforn Encoder.
Figure 5: Heatmap visualization Attention. Figure shows the difference of alignment Encoder- Decoder attention between the original Transformer (Teft) and Phrase Transforree (right). Consid- cring one row, the value in each column is corresponding to the rate of the attention of token in LF to the word in the sentence. Figure shows Self-Attention in 8 heads of the last Phrase Transformer Encoder layer. Two blue rectangles are zoomed-in separately head 1 (not use n. m-grrmL.STMM head 3 (use 2 gTT....MM..
Figure 5b (bigger version in Appendix A.I shows the difference between heads in Self-Attention Encoder of PranseTransformer. The self-attention in heads that do not use n-.rrmLSTMM is more incoherent than other heads. For example, in head I, almost words in query focus on "cil" and the other words are paid attention is key information words such as "da0", "arriv", "ri0" (the green rectangles). From head 8, the attention focuses on the separated clusters, which shows that model learned the dependencies of the phrases instead of the single words. On these heads, the attentions are usually between groups important words such as "flight" with "cil on", "daO nm0 dno arriv" with "nmo dn0" (the orange rectangles).
Meaning phrase In this experiment, we explore the natural language understanding capacity of our Transforrae We use Principal Component Analysis (PCA) method to visualize the sim- ilarity of phrases in Phrase Transformer in Figure by using hidden state of heads 7, 8 (the vector [g': 46} where of from Equation TTp. We also highlight 30 closest points (the distance using Co- sine distance) the particular phrase carrying key information such "round trip". " "from cil to cio". Besides, we also visualize the vector of words ( ((7): qs] where qi from Equation to show the lacked local context information of word vectors in the original Transformer in Appendix A.3
Considering two phrases "from cil to ciO" and "from ciO to cil" in Figure 6a the phrases closest to two phrases concentrate on blue and cyan clusters. These two clusters closest to each other but separate without overlapping. This feature helps the decoder decode different semantic components such as (from $0 ci0) (to $ 0 cil) and (from $0 cil) (to $0 ci0). Figure (b] shows that the phrase "from cil to cio" is represented by the similar vectors in various contexts. For example, this phrase in Atis data sentence 175 "show me nonstop flight from cil to ciO" has the same meaning in sentence 339 "a flight from cil to ciO arriv between ti0 and til" In Figure 6c there are many different phrases have the same meaning that the model finds out, such as "could have", "tell me again" ", "find me all" or the phrases closest to "list the" and "show all" in Figure/ba] These phrases do not consist of query information, which the robustness feature of human natural language, this is evidence that the model is capable of learning complicated characteristics of natural language.
Examples of improvement We analyze examples that our TrrassTraanfommer improved over the original Transformer (Table 6). The improvement can be grouped into three types of errors: (1) 46.2% the errors are caused by Transformer confusing the role of entities name such as "ci2" and
8

---

Under review as a conference paper at ICLR 2021
(109. di to
to
(339. 5) from
(232. 5) from
to
(220, 9) from c1 to
(221. 51 from (il to
frem
(250.
d1
(186. 5) from di to do
(176, 5) from cil to
5) from cil ci0
(100, from
(102,
(b) Phrases closest "from ciO
4
1356. 2) could have
1355. could have
1364. 2) tell ma agoin
find me
1377. pleas
[151. tell all
1261. 21 pleas list
1346. 2) pleas list all
1376. plees list all
pleas
(400. show me all
1330, 2) show me all
(a) All phrases
(c) Phrases closest to "show all"
Figure 6: Figure draws the representing vector phrases in Selft-Attention Layer using PCA on Atis test set. Figures b
zoomed-in view of the blue and red clusters. The labels are annotated for each point in two figures show the information of the phrase corresponding to point following the template (sentence id, pharse-position position) phraae__cntem
"ciO" (Row Table 6 (2) 27.3% missing semantic components such "(round $0)" (Row 2): (3) 27.3% wrong in predicate name of logic component (Row 3). In our opinion, almost the improvement of the PhraseTransfore Transformer when compared with Transformer based on the capacity of capturing local context information.
Table Examples that frequent incorrect predictions of Transformer, are improved PhraseTrans former the Atis test set.
Sentence Gold LF
what are the flight from cil to ci2 that stop in ciO lambda $0 e and flight $0 from S0 cil ) to $0 ci2 ( stop 50 ci0
Transformer
( lambda $0 and flight $0 from 50 cil ) to $0 ci0 stop S0 ci2 ))
Sentence
give me the cheapest round trip flight from ciO to cil around mn0 dnO
Gold LF
argmin 50 and flight $0 ) (month $0 mn0) ) (round_trr $0) ) fare $0)
Transformer
argmin $O and flight $0 ) (month $0 mn0 ) fare $0)
Sentence Gold LF Transformer
show me the airport servic by al0 ( lambda $0 ( and airport $0 ) (ervices a10 $0
lambda $0 and airport S0 airline $0 al0)
CONCLUSION
In this paper, we proposed novel model named PhraseTransforme that can improve the perfor- mance of the Transformer in semantic parsing tasks. We enhance Transformer Encoder to improve the representing ability of the detailed meaning of sentence based on learning the phrase depen- dencies. In the methods using Neural Network, this model obtains SOTA results on the Atis dataset and achieves competitive result with the SOTA other datasets. We also conducted experiments to compare with Transformer and show the improvement of self-altention in chitecture. future work, we would like to extract more information about this architecture about the relationship between words or phrases and how to inject prior knowledge improve it. We believe that this architecture can be widely applied in many problems using sequence to sequence models such neural machine translation and abstract text summarization.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ruisheng Cao, Su Zhu, Chen Liu, Jieyu Li, and Kai Yu. Semantic parsing with dual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp- 51-64, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1007, URL Cttthttps:: www. aclweb. org/ aatpaaooooooool-noaa 1007
Ruisheng Cao, Su Zhu, Chenyu Yang, Chen Liu, Rao Ma, Yanbin Zhao, Lu Chen, and Kai Yu. Unsupervised dual paraphrasing for two-stage semantic parsing. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pp. 6806-6817, Online, July 2020. Association for Computational Linguistics. doi: 10. aatp:////m222...mmmimaa 608. URL
https: wwwww/wwccooooom...oo www.waclweb.oo org/ anthology 2wwpyy/2220..o acl -main. 608
Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Slumicke-Smite David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. Expanding the scope of the atis task: The atis-3 corpus. In Proceedings of the Workshop on Human Language Technology, HLT '94, pp. 43-48, USA, 1994. Association for Computational Linguistics. ISBN 1558603573. doi: 10.3115/1075812.00777 1075812.1075887
URL https://dot http:///ooo.oro1.... 3015/1..111//
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Pupers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. (0.18653/v1/N19-14 1423. URL https: 77 www. ac wwwweb.w............ htt.::ww............eee.ee http//atthoogyy//// 1423
Li Dong and Mirella Lapata. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 33-43, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v 1/P16 1004. URLhttps:
www aclweb. org http:///nhoooogyy/oeeo 1004
Li Dong and Mirella Lapata. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 731-742, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: (0.18653/v1/P188 1068. URL(https: www. ac woccweeb.ro wwweeboorg//tttooo http://www......rr////ttoooooo/o
1068
Nan Duan. Overview of the nipee 2019 shared task: Open domain semantic parsing. In Jie Tang, Min Kan, Dongyan Zhao, Sujian Li, and Hongying Zan Nads.). Language Processing and Chinese Computing, pp. 81 811-817, Cham, 2019. Springer International Publishing. ISBN
978-3-030-33236-6
Donglai Ge, Junhui Li, and Muhua Zhu. A transformer-based semantic parser for nipce-2019 shared task 2. In CCF International Conference on Natural Language Processing and Chinese Comput- ing, pp. 772-781. Springer, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth Interna- tional Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256. Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL http: ptt.:/diiocceeeig.... m..r..r............. www.ppresssvvve wttp:///ggloloooo html
Omer Goldman, Veronica Latcinnik, Ehud Nave, Amir Globerson, and Jonathan Berant. Weakly supervised semantic parsing with abstract examples. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1809-1819, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/ P18- 1168. UtL/https://// www. ac web. org/ http//anhhooloooooeoo1e
Jie Hao, Xing Wang. Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu. Multi-granularis self- attention for neural machine translation. In Proceedings of the 2019 Conference on Empiri- cal Methody in Natural Language Processing and the 9th International Joint Conference on
10

---

Under review as a conference paper at ICLR 2021
Natural Language Processing (EMNLP-IJCNLP)) pp- 887-897, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653/11/11--000
URL
https://www.acc. hww.:wwwlww..... org/ www.://www.r...ahooooooo.o.ooo// aathholooy/DSs-1082 D1tpy/D19-1222
Gary G. Hendrix, Earl D. Sacerdoti, Daniel Sagalowicz, and Jonathan Slocum. Developing a natural language interface complex data. ACM Trans. Database Syst, 3(): June 1978. ISSN
0362-5915. doi: 10. 10.11155 00.114///20251332000
URL htl.http::: /oi http:///...o.//....1oo /t//:/....rr/11111133222200022
320253
Jonathan Herzig and Jonathan Berant. Don't tt paraphrase, detect! rapid and effective data collection for semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pp. 3810-3820, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19- 1394. URLhttps: www. aclweb.
org//anthology/D13
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8); URL
1735-1780, November 1997. ISSN 0899-7667.
doi: 10. 11116///eco.1997.9.....
https://doi. orooo.org/100.. wtt.oo//............... www...16ennn..o.. 1997. 9.88...8... 1735
Robin Jia and Percy Liang. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 12-22, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 18653/v1/P16-1000
www. web. wttpbb//rggaaanooggg./ooooo 6atp://1111110000 6-1002
Tomá's Kočiský, Gábor Melis, Edward Grefenstette, Chris Dyer, Wang Ling, Phil Blunsom, and Karl Moritz Hermann. Semantic parsing with semi-supervised sequential autoencoders. In Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp- 1088,1087, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/w/DD1-1111 1116. URL htRpstts:://ee
Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. Lexical generaliza- tion in CCG grammar induction for semantic parsing. In Proceedings of the 201 Conference on Empirical Methods in Natural Language Processing, pp. 1512-1523, Edinburgh, Scotland, UK., July 2011. Association for Computational Linguistics. URL https::///// / www. wewweee. org/
Zechang Li, Yuxuan Lai, Yuxi Xie, Yansong Feng, and Dongyan Zhao. A sketch-based system for semantic parsing. In CCF International Conference on Natural Language Processing and Chinese Computing, pp. 748-759, Springer, 2019.
Percy Liang, Michael Jordan, and Dan Klein. Learning dependency- based compositional semantics. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pp. 590-599, Portland, Oregon, USA, June 2011. Association for
Computational Linguistics. URL https::::: wwwp://wwwww/..o////////////ww awwweb..w...... org/
Xuan-Phi Nguyen, Shafiq Joty. Steven Hoi, and Richard Socher. Tree-structured attention with hierarchical accumulation. In International Conference on Learning Representations, 2020. URL
ht htttps://openreview wwwp.ewwww...eeccoooooet..ooow.p net http:eewwwee//ffouuueeeee forum? iic H
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/1//11--1116 URL https: www aclweb.
org/ hotg//nntholloo//P16111
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan. and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017. URL http::/ wwwpeerrssiisssooo.em wwtpipscccpppp...pec 1aa-888-atttttt attention --lu-----ee---ddd pdff
11

---

Under review as a conference paper at ICLR 2021
David Waltz and Brad Goodman. Planes: A data base question system. SIGART Bull.,
(61):24, February 1977. ISSN 0163-5719. doi: 00.1145/1045288.144558888 URL https:
7/doi. 00oi.org/100 www..1111//112.....11 1045288
Xing Wang. Zhaopeng Tu, Deyi Xiong, and Min Zhang. Translating phrases in neural machine trans- lation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 1421-1433 Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: (0.18653/v1/D17-111 D17-114 49
Btthhtts:: /top::://www...
www http://wwwwwwllll. awwweb. org/ antoologgtttogy////oooo
Yaushian Wang, Hung- Yi Lee, and Yun-Nung Chen. Tree transformer:
Integrating tree structures into self-attention. Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-HJCNLP), Pp. 1061-1070, Hong Kong, China, November 2019, Association for Com- putational Linguistics. doi: 20.188533v1/D199 1/D19-10998 URL https: wwtp://wwwww./..// http///wwwwwww.bb..oopo.moo/. aclweb. org/
Yuk Wah Wong and Raymond Mooney. Learning for semantic parsing with statistical machine translation. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pp. 439-446, New York City, USA, June 2006. Association for Computational Lin- guistics. URL https: http://www.eeb...rr/aaaaoooooooa
1056
Yuk Wah Wong and Raymond Mooney. Learning synchronous grammars for semantic parsing with lambda calculus. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistice, pp. 960-967, Prague, Czech Republic, June 2007. Association for Computational Linguisties. URL. htt.https:/// hwwp//wwaaa. ac www.org//nttoll.o/oo/o////ooo http:/thhhooolo0011111100
W. A. Woods. Progress in natural language understanding: An application lunar geology. In Proceedings of the June 4-8, 1973, National Computer Conference and Exposition, AFIPS '73. pp. 441 1-450, New York, NY, USA, 1973. Association for Computing Machinery. ISBN 9781450379168. doi: 10.1145/1499586. 1499695.
URL
org ort.//.11/5/1
1499586. 1499695
Wei Wu, Houfeng Wang, Tianyu Liu, and Shuming Ma. Phrase-level self-attention networks for universal sentence encoding. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pp. 3729-3738, Brussels. Belgium, October-Novemble 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18. 1408. URL https: // w/www.ao ac www.::ww............ www.oorgggggoooggoooo
1408
Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. Modeling localness for self-attention networks. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, pp. 4449-4458, Brussels, Belgium, October- November 2018 Association for Computational Linguistics. doi: 10.18653/v1/D18. 1475. URL http://www....aaooooooo///------
https:
Pengcheng Yin, Chunting Zhou, Junxian He, and Graham Neubig. StructVAE: Tree-structured latent variable models for semi-supervised semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 754-765, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/
BtRhhtts:: www. aclwesa org/ anthology/P18-1100
John M. Zelle and Raymond J. Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence- Volume 2, AAAI'9,, pp. 1050-1055. AAAI Press, 1996. ISBN 02625 1091X.
Luke Zettlemoyer and Michael Collins. Online learning of relaxed CCG grammars for parsing to logical form. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natu- ral Language Processing and Computational Natural Language Learning (EMNLP- CoNLLL, pp. 378-687, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https:: / www. a web. www..org/anchologg///oaaa//
12

---

Under review as a conference paper at ICLR 2021
Luke S. Zettlemoyer and Michael Collins. Learning to map sentences logical form: Structured classification with probabilistic categorial grammars. Proceedings of the Twenty-Firg Confer- Uncertainty in Artificial Intelligence, UAI'05, pp. 658 666, Arlington, Virginia, United States, 2005. AUAI Press. ISBN 0-9749039-1-4. URL htt.htt://// /ttetp///dl.. .acm ortp://tt//ottt..........
fmmmidd302033366 3020416
Kai Zhao and Liang Huang. Type-driven incremental semantic parsing with polymorphism. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1416-1421, Denver, Colorado, Saeeaaaaaaaaaoiioaaaooaae
May-June 2015. Association for Computational Linguistics. doi: 10.3115/w1/N15-166 1162.. URL
https:// wwt////www.. wwwwwwwwwwwwww...c.o/oomoo..
Amir Ziai. Compositional pre-training for neural semantic parsing. Proceedings of the 3rd In- lernational Conference Natural Language and Speech Processing, pp. 135-141, Trento, Italy, September 2019. Association for Computational Linguistics. URL https:: www. ac web..
org/anthology//119 7411
A
APPENDIX A
A.I
SELF-ATTENTION VISUALIZATION
i'd
like
round
trip-
flight
from
ci0
cil
dao
mn0
dn0
arriv
at
tio
r'd
like
ound
trip
flight
from
CID
to
on cil
daO
mn
dnD arriv
tio)
I'd
like
ouund
trip
flight
from
cio
to
cil
on
dap
mnD
dnD
arriv at tio
Figure 7: Heatmap visualization of Seff-Attention in 8 heads at last layer of Phrase Phrase Transformer Encoder. Heads 1 are ordered from left right. The gram sizes n [0, 0, 2, 3, 3,3,3,4 4). The highlighted rectangles in these heads are highly attended alignments. Two blue rectangles are zoomed-in separately of head (not use n gramLSTMM, head (use 2.gramLSTMM,,
13

---

Under review as a conference paper at ICLR 2021
A.2 RICODER-DECODREE ATTENTION VISUALIZATION
(
lambda
lambda
7x
7x
(
and
and
(
r-msoffim.film.aa director
r-mso:film. filmm..ffimm.. director
tom
tom
san@@
san@@
ders
ders
f-mso:fil.film story
r-mso:film.fin story
tom
tom
cruise
cruise
7x




which
movie
was
writtenn
by
tom
cruise
and
art
directed
by tom- san@@ ders
which
movie
was
written-
by
tom-
cruise
and
art
directed
by
tom-
(a) The sentence 6456 in MSParS test set.
bambda
lambda
Xx
7x
eait
extt
bas.bol..ppppoo....o.oo.. aver batting
masibaseball battiiggtnngtst
mascbaseball batting statist co.seeath.
maccassbblll batting statist co. ssusennn-
Ty
Ty-
19789
197.9
league
basebali
baseball
seaonh
mio basebell bott www.tiggttttttsscccccccoooo dag
moo baseball bott mggsmtisssss.ee
12
bbbr
201ge
miter
bacoball
whit
miler
basdball reasin
rik-
mindl - ang-
have
(b) The sentence 440 in MSParS test set.
Figure 8: Heatmap visualization Encoder-Decone Decoder Attention of Transformer (left) and Phrase- Transformer (right). Two sentences randomly chosen in the MSParS test set. Transforamer pays more attention than Transformer to words that are entities name the sentence. For example: words "tom cruise" and "tom san @ @ ders" in sentence 6456 "rick man @ @ ning" in sentence 440.
14

---

Under review as a conference paper at ICLR 2021
A.3 SIMILAR WORD VECTORS BY TRANSFORMER
In this experiment, we found that the representations of words in the original Transformer is often confusing without considering the local context. Considering cil to cio") and "cil" (in the context "from ciO to cil ") in Figure the words closest to these words concentrate on blue and cyan clusters. These clusters are overlapping while PhraseTransformer is separated clearly. Figure shows that the word "ci0" in the context "from cil to ciO" is confused with the "cil" " in the context "from ciO to cil" in many times. The similar problem showed on the Figure 9c
words "ci0" (in the context "from
(260. 8) from cio to (cil]
(298, 8) tare from d1 [to]
[224, 5) from c10 to [cil]
[284. 7) economi from [cil]
Wends
(262. 7)
[to]
Weeds eiseeet It "round
(102.
Werds elosest "the "letthee
[ci1]
Wentt riseectio "trnsspen" in lhout the grond transpont
Weods cissestice "ell "fom (D0 to cil"
disiesd "db "frm G3 ci
[163. 7) from ciO to [c11]
(104, 5) from ci0 to [cil]
(244, 5) round trip flight (fram)
Wents ricesctio to "wP in hahe me at
76
(131. 6) one way befor [00]
(1332 6) from ail to [c10]
from
[ci0]
(b) Words closest to "ci0" in context "from cil to ciO"
15
(129, which [afternoon]
[between]
(143, 61 from cil to (ci0)
(149. 5) from a0 [to]
(1110 5) from c10 to [ctL]
(111. 51 from <10 to [cil]
(328. 71 flight from [to]
[d]]]
transport [in]
(48. 3) the airport [in]
(46. 3) flight from cio [to]
(47. 31 list sto st1 [st2]
(44, 4) from cl0 to [c]]]
(a) All words
(c) Words closest to "cil in context "from ciO to cil"
Figure 9: Figure a draws the representing vector words in Selft-Attention Layer using PCA on Atis test set. Figures C zoomed-in view of the blue and cyan clusters. The labels are annotated for each point in two figures show the information of the word corresponding to point following the template (sentence id, word position) phrase context (considering word].
15


Reviews:
[{"review_id": "hmHor482DA2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends Transformer to integrate representations of ngrams for semantic parsing. The key idea is to split input sentences into ngrams of different orders and utilize LSTM to build their representations before feeding them to the layers inside a transformer. The experimental results show that this modification leads to marginal improvement on three benchmark datasets of semantic parsing.\n\nThe encoders of the previous neural semantic parsing methods only learn the dependencies between tokens but ignore the local context around the tokens. Therefore, to exploit the local context information, this work provides some main contributions: \n-- This work introduced a novel Transformer encoder to encode the n-grams in each utterance.  \n-- This work showed the effectiveness of this new model on three benchmark datasets. \n-- This work displayed the model capacity by visualizing the alignment between the source tokens and the target logical forms, and the similarity of the phrase representations.  \n\nThe method is evaluated on three datasets, Geo, Atis and MSParS with two evaluation metrics, Exact Matching and Logic Matching. Compared with Exact Matching, Logic Matching is able to compare the variants of the logical forms.\n\nStrengths:\n\n-- It is a good method to exploit the local context information with the Transformer architecture. And this method seems to be easy to implement.\n\n-- There is a thorough evaluation which displays the model performance, and visualizes the attention alignment and the similarity of the phrase representations. \n\nWeakness:\n\n-- The first thing I am concerned about is the novelty. Although in semantic parsing, there is no previous work that utilizes local context information with a Transformer, there are many similar architectures in machine translation. The proposed Transformer-based model is also totally applicable to machine translation scenarios, which makes it necessary to compare the model performance with the machine translation models that exploit local context as well.\n\n-- The second is that the performance is not significant enough. Although this work claims that the performance is superior to the baselines on two benchmark datasets, it should be noted that the other baselines report only the Exact Match accuracy while this work reports the Logic Match performance. Considering only the Exact Match comparison, the proposed method is only 0.4% and 0.02% higher than the baselines on Atis and MSParS, respectively, which is not significant at all. If using Logic Match as the main metric, it would be better to re-evaluate the baselines with Logic Match as well for fair comparison.\n\n--  The description of the model in Sec. 3 is not clear enough.\nHow does the model split an input sentence into ngrams? Do the adjunct ngrams have overlapped subwords/characters or not?\n\n-- I found this contribution too close to the following work.\nHao, Jie, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. \"Modeling recurrence for transformer.\" arXiv preprint arXiv:1904.03092 (2019).\n\n-- In addition:\n\n---- The citation of logic matching seems to be incorrect.\n\n---- The sentence “W is parameters, LayerNorm, FeedForward are the functions by proposed by … .” is misleading because LayerNorm and feed forward networks are not proposed by Vaswani et al.\n\n---- What is model 4 and model 5 in Table 3?\n\n---- Figure 5 (b) is hard to read\n\nMinor issues and improvement suggestions:\n\n-- Compare the proposed method with the baselines which are from machine translation fields which also exploit the local context.\n-- Improve the model performance. \n\nReasons to accept: \n-- Proposed an interesting Transformer-based encoder to exploit the local context information.\n\nReasons to reject:\n-- Novelty is not enough. The evaluation results can not show the superiority of this model.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UAgO0xMUWXg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\nThis paper takes on a challenging task, i.e., semantic parsing from natural language text.  The paper proposes an improved version of the transformer for the task of semantic parsing, called PhraseTransformer to overcome the limitations (i.e., failure to capture local sentence context effectively) of the transformer. It proposes to use LSTM networks and the Self-Attention mechanism of the original Transformer and provides experimental results for up to 4-grams representations of the sentences to better accommodate the local contexts and achieves state-of-the-art (SOTA) performance on ATIS dataset, and competitive performance on other datasets such as Geo and MSParS. \n \n##########################################################################\nReasons for score: \n \nOverall, I like the idea and I would like the paper to be accepted. The main reason for my acceptance is that in a natural language many values are actually phrases (e.g., one way) and it is very critical to capture the meaning of n-grams as one unit (i.e., one representation for the full n-gram) and consider their context locally to better understand the meaning in the context of the sentence, and effectively parse the natural language texts. \n \n##########################################################################Pros: \n \n1. The paper tackles a very important problem, i.e., parsing natural language text into a semantic frame. Natural language is complex and the semantic frame is easy to handle, which facilitates many down-stream NLP tasks. \n \n2. The proposed approach is also very interesting, i.e., captures the meaning of the phrases or n-grams in the context of the whole sentence.\n \n3. Reasonable experiments have been provided to prove the efficacy of the proposed approach. Moreover, it also outperforms the SOTA model on the ATIS dataset.\n \n##########################################################################\nCons: \n \n1. Although the paper compares with several baselines and SOTA models, an important model for comparison is skipped, i.e., SpanBERT: Improving Pre-training by Representing and Predicting Spans, recently published in ACL 2020 (July 2020 published more than two months ago). Still, I will not recommend rejecting this paper only because of this.\n \n2. The paper fails to explain in detail how n-grams are generated? And how these representations are put back to generate predictions at the word level. Since some n-gram would make sense to form a phrase (e.g., “one way” or “San Francisco”), but many would not be valid phrases (e.g., “a one”, “way ticket”, “ticket from” are not valid phrases in natural language text). Also, it is not trivial to know in advance what is the suitable value for “n” in the n-grams for a given sentence. Please explain this step: not sure but maybe a working example can explain it in a much better way. \n \n##########################################################################\nQuestions during the rebuttal period: \n \nPoint # 1 in the cons section is not very critical for my decision, it may help the authors to improve their paper.\n\nPoint # 2 is critical for me to understand the key idea. I would love to see a working example that shows a sentence, how n-grams are generated (also, how n is decided), and how final predictions are performed. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "RSQy37Iuia3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary of the paper*: One drawback of the transformer architecture is that they often fail to capture local interactions within the sentence. In this paper, the authors propose the PhraseTransformer architecture which incorporates Long Short-Term Memory (LSTM) into the Self-Attention mechanism of the original Transformer to capture more local context of phrases. Experimental results on three semantic parsing datasets show that the phrase level transformer is better at capturing the local information than the original transformer.\n\n*Strength of the paper*:  This paper proposes to use phrase-level information to capture the local dependencies of the transformer architecture and the experimental results show that semantic parsing could benefit from such local dependencies. The authors also conduct empirical experiments -- section 4.3.2 to show why their method is helpful. The paper is well-structured and easy to follow.\n\n*Weakness of the paper*: \n\n(1) The idea of using different-granularity representation including phrase-level representations of transformer architecture has been proposed before. For example, Hao, et al. 2019 study multi-granularity representations for self-attention in machine translation; Yang, et al. 2018 study the localness of the self-attention mechanism; Nguyen, et al. 2020 study the tree-structured representations of the self-attention networks. The authors should have done a detailed literature review along this line of works. \n\n(2) The proposed method is quite empirical and from Table 2 it is unclear how to set the n-gram size for different layers. A more detailed experiment demonstrating how different n-gram sizes in different layers affects the model would be quite helpful. \n\n(3) The improvement over the original transformer is marginal except on the Atis dataset, the claims would be more convincing if the authors could conduct similar experiments on other tasks. \n\n(4) For the error analysis part, I would like to see a more systematic analysis rather than two examples posted in the paper: is there a specific type of error being corrected by the phrase-level representation? In what scenario will the phrase-level representation help most?\n\n*Reason for score*: Overall, I vote for rejecting this paper. I like the idea of trying different-granularity representations for the transformer. However, such kind of ideas has been proposed before and this paper does not bring new insights into using such representations.  \n\n*Reference*: \n\nHao, Jie, et al. \"Multi-Granularity Self-Attention for Neural Machine Translation.\" arXiv preprint arXiv:1909.02222 (2019).\n\nYang, Baosong, et al. \"Modeling localness for self-attention networks.\" arXiv preprint arXiv:1810.10182 (2018).\n\nNguyen, Xuan-Phi, et al. \"Tree-Structured Attention with Hierarchical Accumulation.\" arXiv preprint arXiv:2002.08046 (2020).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7dO025WVpw3", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper modifies the self-attention mechanism in transformers to function at the phrase level, rather than at the token level, as a means to improve alignments between input phrases and logical form predicates for Semantic Parsing tasks. They achieve this by using LSTMs on the token representations to form n-gram representations in the attention head, and then performing attention on these n-gram representations. They call this modified transformer, a PhraseTransformer. They are able to improve over baseline transformers on 3 datasets.\n\n**Strengths and reasons to accept**\n\n1. The model improves upon existing neural models on three dataset, achieving SOTA on MSparS. \n2. The idea is adequately motivated and the model analyses appropriately corroborate the motivation.\n3. The model is described well. Other than the tiny size of the figures and grammatical errors, the model is quite easy to follow.\n\n**Weaknesses and possible reasons to reject**\n\n1. I'm concerned about repeated evaluations on the test set i.e. the model might be overfit to the test set. For example the numbers reported in Table 2, seem to be test set numbers, since they exactly match the numbers in Table 3. This suggests that n-gram sizes were fine-tuned on the test set, which is concerning.\n2. How does adding multiple LSTMs to the attention affect the runtime complexity and parallelization ability of the attention mechanism? I feel that the LSTMs add a lot of overhead to the attention mechanism, and might not scale to larger datasets. Can the LSTM be replaced with a simpler operation to achieve the same result? \n3. Although they outperform recent neural models, don't forget about Wang et al. 2014 (Morpho-syntactic Lexical Generalization for CCG Semantic Parsing) who are still the SOTA on ATIS with an accuracy of 91.3 \n4. The datasets used are quite old and performance on these datasets is quite saturated. Might be useful to evaluate on newer semantic parsing datasets such as TOP or other SQL based datasets such as WikiSQL or SPIDER.\n\n**Other issues:**\n\n1. The citation format is quite terrible in the paper. There are no separators whatsoever between the cited papers and the main text. \n2. Lots of grammatical errors. Very hard to understand in certain spots. \n3. Figure 2 is extremely tiny on print and even on screen.\n\n**Update after rebuttal - I would like to keep my score**\n1. Although the authors have provided dev set numbers, the fact that test set numbers were computed, is highly concerning. In fact, for ATIS, the best dev model is different from the best test model. The other models should never have been evaluated on the test set. \n2. Am still concerned about the datasets being old and saturated, and would love to see results on more recent datasets.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an attention mechanism that works at the phrase level for semantic parsing.
Reviewrs agree that the idea has been previously explored outside semantic parsing, that the gains should be shown on less saturated datasets, and that there are issues in the experimental design (observing test set results for many experiments). Thus, at this point I recommend that the paper is rejected.
Decision: Reject