Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Motivated by the fact that the attention mechanism in transformers is symmetric which might not be able to disambiguate different orders, this work proposes to use a subject vector (in addition to query, key states) for each attention head, and multiply it elementwise with the context vector for each head before merging the heads. Experiments on a mathematics dataset shows superior performance compared to the normal transformer. Qualitatively, the proposed model exhibits attentions that are more interpretable, and clustering by the subject vector gives some insights into how the model solved this problem.

Pros:
1. This work shows better performance than baseline transformer.
2. The clustering of the subject vectors gives some insights into model's behavior .

Cons:
1. In terms of experiments, the proposed approach adds a few million parameters to normal transformer (table 1), but in terms of interpolation it only improves 3% (extrapolation improves 0.5%) at 700k steps. The comparison would be fairer if the normal transformer can be given more parameters.
2. In terms of experiments, this approach is only evaluated on the mathematics dataset, but the argument for relational encoding is pretty general. It would be nice if experiments on other tasks are shown in addition to the math dataset.
3. In terms of motivation, the claim that there're ambiguities introduced by multiple layers of  regular attention needs to be supported by evidence. I think (which authors also pointed out) the feedforward network and non-linearties can disambiguate as well.
4. In terms of interpretablity, there's claim that the learned attention maps more interpretable than transformer. Can there be more quantitative measures? It appears to me that both are hard to interpret.

While this work shows superior performance on the mathematics dataset, I have a few concerns about the generalizability of this proposed architectural change to other problems, as well as the fairness of comparison to baseline. Therefore, I am inclined to reject this paper.

----updates after reading rebuttal----
Thanks for adding the new NMT experiment in Appendix A3. My concern is that the proposed TP-Transformer is not very effective on NMT. Therefore, I'm keeping my score.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors incorporated tensor-product representations within the Transformer.  By creating an attention mechanism called TP-Attention, they explicitly encode the relations between each Transformer cell and the other cells, whose values are retrieved by attention. By introducing tensor products, the proposed algorithm can empirically perform well for noncommutative operations with multiple arguments, such as division. The authors trained models with the proposed algorithm on the Mathematics Dataset and compared the performances with two baselines (simple LSTM and the original Transformer). At last, several model snapshots are provided to help interpret several key elements of the model: the learned roles, the attention maps, the TP-transformer columns and so on. 

Overall, the paper is well-written. The experimental results generally support the high-level intuition behind the introduction of tensor-product representation. I would recommend accepting this paper. 

Some quick questions:

1. It was claimed in the Conclusion section that the performance of the proposed algorithm beats the previously published state of the art by 8.24%. I guess the number comes from the 2nd and the last row of interpolation accuracy in Table 1. However, these two results are obviously trained for different numbers of iterations: The baseline algorithm was trained for 500k steps, while the proposed algorithm is trained for 1.7M steps. Is it a fair comparison? If the proposed algorithm is also trained for 500k steps, the improvement is around 2.3%. 

2. Why is the extrapolation accuracy results for TP-Transformer missing in Table 1? 


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper illustrates the TP-Transformer architecture on the challenging mathematics dataset. The TP-Transformer combines the transformer architecture with tensor-product representations. The experiments show a dramatic improvement of accuracies compared with SOTA models. Moreover, the paper also explains the reason why the TP-Transformer can learn the structural position and relation to other symbols with a detailed math proof.
 
Overall, this paper is nice as it makes a milestone for math problem solving from unique perspectives. To be specific, the paper makes the following contributions:

1. Demonstrate a novel architecture TP-Transformer in details;
2. Achieve a better accuracies in the challenging mathematics dataset than the SOTA transformer models;
3. Illustrate in fundamental math that why TP-Transformer can learn the structural position and relation, and solve the binding problems of stacked attention layers.
 
Here are a few minor questions that may further improve the paper:

1. The conclusion states that TP-Transformer beats the previously published SOTA by 8.24%. However, it does not match to the experiment results (see section 4).

2. In figure 5, there are 4 tasks in the bottom with accuracies lower than 0.5. It would be nice to provide more insights on this.
 
3. It would be interesting to see whether it transferable to the other downstream tasks (such as natural language understanding) besides the experiments on the challenging mathematics dataset.
review_title: Official Blind Review #2

Meta-review: This paper proposes a change in the attention mechanism of Transformers yielding the so-called "Tensor-Product Transformer" (TP-Transformer). The main idea is to capture filler-role relationships by incorporating a Hadamard product of each value vector representation (after attention) with a relation vector, for every attention head at every layer. The resulting model achieves SOTA on the Mathematics Dataset. Attention maps are shown in the analysis to give insights into how TP-Transformer is capable of solving the Mathematics Dataset's challenging problems. 

While the modified attention mechanism is interesting and the analysis is insightful (and improved with the addition of an experiment in NMT after the rebuttal), the reviewers expressed some concerns in the discussion stage:

1. The comparison to baseline is not fair (not to mention the 8.24% claim in conclusion). The proposed approach adds 5 million parameters to a normal transformer (table 1, 5M is a lot!), but in terms of interpolation, it only improves 3% (extrapolation improves 0.5%) at 700k steps. The rebuttal claimed that it is fair as long as the hidden size is comparable, but I don't think that's a fair argument. I suspect that increasing the feedforward hidden size (d_ff) of a normal transformer to match parameters (and add #training steps to match #train steps) might change the conclusion.
2. The new experiment on WMT further convinces me that the theoretical motivation does not hold in practice. Even with the added few million more parameters, it only improved BLEU by 0.05 (we usually consider >0.5 as significant or non-random). This might be because the feedforward and non-linearity can disambiguate as well. 

I also found the name TP-Transformer a bit misleading, since what is proposed and tested here is the Hadamard product (i.e. only the diagonal part of the tensor product). 

I recommend resubmitting an improved version of this paper with  stronger empirical evidence of outperformance of regular Transformers with comparable number of parameters.
Decision: Reject