Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper modifies the self-attention mechanism in transformers to function at the phrase level, rather than at the token level, as a means to improve alignments between input phrases and logical form predicates for Semantic Parsing tasks. They achieve this by using LSTMs on the token representations to form n-gram representations in the attention head, and then performing attention on these n-gram representations. They call this modified transformer, a PhraseTransformer. They are able to improve over baseline transformers on 3 datasets.

**Strengths and reasons to accept**

1. The model improves upon existing neural models on three dataset, achieving SOTA on MSparS. 
2. The idea is adequately motivated and the model analyses appropriately corroborate the motivation.
3. The model is described well. Other than the tiny size of the figures and grammatical errors, the model is quite easy to follow.

**Weaknesses and possible reasons to reject**

1. I'm concerned about repeated evaluations on the test set i.e. the model might be overfit to the test set. For example the numbers reported in Table 2, seem to be test set numbers, since they exactly match the numbers in Table 3. This suggests that n-gram sizes were fine-tuned on the test set, which is concerning.
2. How does adding multiple LSTMs to the attention affect the runtime complexity and parallelization ability of the attention mechanism? I feel that the LSTMs add a lot of overhead to the attention mechanism, and might not scale to larger datasets. Can the LSTM be replaced with a simpler operation to achieve the same result? 
3. Although they outperform recent neural models, don't forget about Wang et al. 2014 (Morpho-syntactic Lexical Generalization for CCG Semantic Parsing) who are still the SOTA on ATIS with an accuracy of 91.3 
4. The datasets used are quite old and performance on these datasets is quite saturated. Might be useful to evaluate on newer semantic parsing datasets such as TOP or other SQL based datasets such as WikiSQL or SPIDER.

**Other issues:**

1. The citation format is quite terrible in the paper. There are no separators whatsoever between the cited papers and the main text. 
2. Lots of grammatical errors. Very hard to understand in certain spots. 
3. Figure 2 is extremely tiny on print and even on screen.

**Update after rebuttal - I would like to keep my score**
1. Although the authors have provided dev set numbers, the fact that test set numbers were computed, is highly concerning. In fact, for ATIS, the best dev model is different from the best test model. The other models should never have been evaluated on the test set. 
2. Am still concerned about the datasets being old and saturated, and would love to see results on more recent datasets.
review_title: Good motivation, execution and results - but issues with model selection, scalability and impact on the community

final_score: rating: 3: Clear rejection
review_content: *Summary of the paper*: One drawback of the transformer architecture is that they often fail to capture local interactions within the sentence. In this paper, the authors propose the PhraseTransformer architecture which incorporates Long Short-Term Memory (LSTM) into the Self-Attention mechanism of the original Transformer to capture more local context of phrases. Experimental results on three semantic parsing datasets show that the phrase level transformer is better at capturing the local information than the original transformer.

*Strength of the paper*:  This paper proposes to use phrase-level information to capture the local dependencies of the transformer architecture and the experimental results show that semantic parsing could benefit from such local dependencies. The authors also conduct empirical experiments -- section 4.3.2 to show why their method is helpful. The paper is well-structured and easy to follow.

*Weakness of the paper*: 

(1) The idea of using different-granularity representation including phrase-level representations of transformer architecture has been proposed before. For example, Hao, et al. 2019 study multi-granularity representations for self-attention in machine translation; Yang, et al. 2018 study the localness of the self-attention mechanism; Nguyen, et al. 2020 study the tree-structured representations of the self-attention networks. The authors should have done a detailed literature review along this line of works. 

(2) The proposed method is quite empirical and from Table 2 it is unclear how to set the n-gram size for different layers. A more detailed experiment demonstrating how different n-gram sizes in different layers affects the model would be quite helpful. 

(3) The improvement over the original transformer is marginal except on the Atis dataset, the claims would be more convincing if the authors could conduct similar experiments on other tasks. 

(4) For the error analysis part, I would like to see a more systematic analysis rather than two examples posted in the paper: is there a specific type of error being corrected by the phrase-level representation? In what scenario will the phrase-level representation help most?

*Reason for score*: Overall, I vote for rejecting this paper. I like the idea of trying different-granularity representations for the transformer. However, such kind of ideas has been proposed before and this paper does not bring new insights into using such representations.  

*Reference*: 

Hao, Jie, et al. "Multi-Granularity Self-Attention for Neural Machine Translation." arXiv preprint arXiv:1909.02222 (2019).

Yang, Baosong, et al. "Modeling localness for self-attention networks." arXiv preprint arXiv:1810.10182 (2018).

Nguyen, Xuan-Phi, et al. "Tree-Structured Attention with Hierarchical Accumulation." arXiv preprint arXiv:2002.08046 (2020).

review_title: interesting directions but more work needs to be done

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################
Summary:
This paper takes on a challenging task, i.e., semantic parsing from natural language text.  The paper proposes an improved version of the transformer for the task of semantic parsing, called PhraseTransformer to overcome the limitations (i.e., failure to capture local sentence context effectively) of the transformer. It proposes to use LSTM networks and the Self-Attention mechanism of the original Transformer and provides experimental results for up to 4-grams representations of the sentences to better accommodate the local contexts and achieves state-of-the-art (SOTA) performance on ATIS dataset, and competitive performance on other datasets such as Geo and MSParS. 
 
##########################################################################
Reasons for score: 
 
Overall, I like the idea and I would like the paper to be accepted. The main reason for my acceptance is that in a natural language many values are actually phrases (e.g., one way) and it is very critical to capture the meaning of n-grams as one unit (i.e., one representation for the full n-gram) and consider their context locally to better understand the meaning in the context of the sentence, and effectively parse the natural language texts. 
 
##########################################################################Pros: 
 
1. The paper tackles a very important problem, i.e., parsing natural language text into a semantic frame. Natural language is complex and the semantic frame is easy to handle, which facilitates many down-stream NLP tasks. 
 
2. The proposed approach is also very interesting, i.e., captures the meaning of the phrases or n-grams in the context of the whole sentence.
 
3. Reasonable experiments have been provided to prove the efficacy of the proposed approach. Moreover, it also outperforms the SOTA model on the ATIS dataset.
 
##########################################################################
Cons: 
 
1. Although the paper compares with several baselines and SOTA models, an important model for comparison is skipped, i.e., SpanBERT: Improving Pre-training by Representing and Predicting Spans, recently published in ACL 2020 (July 2020 published more than two months ago). Still, I will not recommend rejecting this paper only because of this.
 
2. The paper fails to explain in detail how n-grams are generated? And how these representations are put back to generate predictions at the word level. Since some n-gram would make sense to form a phrase (e.g., “one way” or “San Francisco”), but many would not be valid phrases (e.g., “a one”, “way ticket”, “ticket from” are not valid phrases in natural language text). Also, it is not trivial to know in advance what is the suitable value for “n” in the n-grams for a given sentence. Please explain this step: not sure but maybe a working example can explain it in a much better way. 
 
##########################################################################
Questions during the rebuttal period: 
 
Point # 1 in the cons section is not very critical for my decision, it may help the authors to improve their paper.

Point # 2 is critical for me to understand the key idea. I would love to see a working example that shows a sentence, how n-grams are generated (also, how n is decided), and how final predictions are performed. 
review_title: This paper takes on a challenging task, i.e., semantic parsing from natural language text....

final_score: rating: 3: Clear rejection
review_content: This paper extends Transformer to integrate representations of ngrams for semantic parsing. The key idea is to split input sentences into ngrams of different orders and utilize LSTM to build their representations before feeding them to the layers inside a transformer. The experimental results show that this modification leads to marginal improvement on three benchmark datasets of semantic parsing.

The encoders of the previous neural semantic parsing methods only learn the dependencies between tokens but ignore the local context around the tokens. Therefore, to exploit the local context information, this work provides some main contributions: 
-- This work introduced a novel Transformer encoder to encode the n-grams in each utterance.  
-- This work showed the effectiveness of this new model on three benchmark datasets. 
-- This work displayed the model capacity by visualizing the alignment between the source tokens and the target logical forms, and the similarity of the phrase representations.  

The method is evaluated on three datasets, Geo, Atis and MSParS with two evaluation metrics, Exact Matching and Logic Matching. Compared with Exact Matching, Logic Matching is able to compare the variants of the logical forms.

Strengths:

-- It is a good method to exploit the local context information with the Transformer architecture. And this method seems to be easy to implement.

-- There is a thorough evaluation which displays the model performance, and visualizes the attention alignment and the similarity of the phrase representations. 

Weakness:

-- The first thing I am concerned about is the novelty. Although in semantic parsing, there is no previous work that utilizes local context information with a Transformer, there are many similar architectures in machine translation. The proposed Transformer-based model is also totally applicable to machine translation scenarios, which makes it necessary to compare the model performance with the machine translation models that exploit local context as well.

-- The second is that the performance is not significant enough. Although this work claims that the performance is superior to the baselines on two benchmark datasets, it should be noted that the other baselines report only the Exact Match accuracy while this work reports the Logic Match performance. Considering only the Exact Match comparison, the proposed method is only 0.4% and 0.02% higher than the baselines on Atis and MSParS, respectively, which is not significant at all. If using Logic Match as the main metric, it would be better to re-evaluate the baselines with Logic Match as well for fair comparison.

--  The description of the model in Sec. 3 is not clear enough.
How does the model split an input sentence into ngrams? Do the adjunct ngrams have overlapped subwords/characters or not?

-- I found this contribution too close to the following work.
Hao, Jie, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. "Modeling recurrence for transformer." arXiv preprint arXiv:1904.03092 (2019).

-- In addition:

---- The citation of logic matching seems to be incorrect.

---- The sentence “W is parameters, LayerNorm, FeedForward are the functions by proposed by … .” is misleading because LayerNorm and feed forward networks are not proposed by Vaswani et al.

---- What is model 4 and model 5 in Table 3?

---- Figure 5 (b) is hard to read

Minor issues and improvement suggestions:

-- Compare the proposed method with the baselines which are from machine translation fields which also exploit the local context.
-- Improve the model performance. 

Reasons to accept: 
-- Proposed an interesting Transformer-based encoder to exploit the local context information.

Reasons to reject:
-- Novelty is not enough. The evaluation results can not show the superiority of this model.


review_title: novelty is not enough

Meta-review: This paper proposes an attention mechanism that works at the phrase level for semantic parsing.
Reviewrs agree that the idea has been previously explored outside semantic parsing, that the gains should be shown on less saturated datasets, and that there are issues in the experimental design (observing test set results for many experiments). Thus, at this point I recommend that the paper is rejected.
Decision: Reject