Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: CONTRIBUTIONS:
C1. Simplicialization of attention. Interpreting standard attention weights of a head as the model’s estimate of the probability of an edge = 1-simplex linking the variables encoded by 2 blocks of the Transformer, representing that the blocks stand in a binary relation encoded in the head, a generalization to 2-simplexes is made: now attention also estimates the probability of a 2-simplex indicating that three blocks stand in an arity-3 relation. 
C2. 2-simplicial attention. The standard query-key matching function, the scalar (dot) product, is related to the area of the 2-simplex determined by the 2 vectors and the origin, and this is generalized to the (unsigned) scalar triple product <a,b,c>, analogously related to the volume of the 3-simplex determined by 3 vectors and the origin. This now serves as the matching function between a query and 2 keys. Each head in each block generates a value vector u and two key vectors k1, k2 and the weight of attention from block(i) (with query p(i)) to the ordered pair (block(j), block(k)), a(i,j,k), is a softmax over <p(i), k1(j), k2(k)>. Attention returns to block(i) a sum in which a(I,j,k) weights B(u(j)*u(k)), with B a learned linear map and * the tensor product. 
C3. Experimental results applying Transformers T1 (with standard 1-simplicial attention) and T2 (with new 2-simplicial attention) to modeling an agent in bridge BoxWorld, trained with deep RL. This game crucially involves 3-way entity interactions, as keys of 2 colors open a box yielding a key of a 3rd color. T2 learns significantly faster than T1 (in the sense that the 1-standard-deviation-neighborhood of the learning curve of T2 becomes better than that of T1, plotted against environmental steps: Fig.4, and also essentially so when plotted against time adjusted steps: Fig 5).
RATING: Accept
REASONS FOR RATING (SUMMARY). Generalizing attention from 2nd- to 3rd-order relations is an important upgrade, and the mathematical context in which this is derived is insightful and may lead to further progress in the development of Transformers capable of constructing still richer structures. The experiments yield clear evidence of the value of 3rd-order attention in the context of a game designed to highlight 3rd-order relations.
Strengths
The exposition is clear and situated in a rather sophisticated formal setting. The connection to Clifford algebras may yield further fruit, besides the scalar triple product that is crucial to the definition of 2-simplicial attention. Although I am not an expert in RL, the experiments reported seem sound and the results clear. I believe that the strength of the paper justifies its length of 8.5 pages: the exposition of the key ideas, for this reader, hits a sweet spot between overly concise and overly verbose, and the ideas call for the quantity of space devoted to them. The decisions of what material to place in the Appendix seem well made. Although I have not studied the entire (13-page) Appendix, what I have read is clear and enlightening, another major contribution of the paper.
Weaknesses
Future work testing the value of 3rd-order attention in tasks that are less clearly perfectly designed for it will substantially strengthen the case for it. But in my view it is right to start testing a new architecture by showing it can indeed do what it is designed to do; further tests showing that what it is designed to do is of general utility are a second step. In this case, the first step, including creating of the model itself (consuming 5 non-verbose pages), is substantial enough to warrant publication.
In my good-quality printout of the paper, I can’t see curves for best runs in Fig. 4.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes a transformer block with higher-order interactions. More precisely, instead of computing a dot product between a query vector and a key vector, 2-simplicial attention computes scalar triple product. Instead of computing a weighted average of value vectors, 2-simplicial attention computes the weighted average of tensor products of value vectors. The resulting architecture has improved representation power which is demonstrated using experiments on bridge BoxWorld environment.

The major gripe I have is a lack of discussion in the main paper of how precisely, 2-simplicial attention can help solve better bridge BoxWorld. I did notice some discussion of this in Appendices F, H and I. But I believe the reader would be better served by having such a discussion in the main body. More generally, what kind of tasks can 2-simplicial attention address better? Answering/discussing this question can go a long way in making the paper more valuable.

Is it possible to evaluate the improved expressivity of 2-simplicial attention on real-world datasets/tasks?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper is an extension of the Transformer Algorithm used to solve sequential problems such as in NLP and games (such as the BoxWorld environment from Zambaldi et al.), stating that the Transformer Algorithm is an inductive bias for learning structural representations.
The authors argue that, the use of simplicial inductive biases over the normal relational inductive bias (provided by the Transformer algorithm) may be a better way to achieve abstract learning; in fact they argue that the “relational inductive bias” used in the normal Transformer is just a 1-simplicial Transformer, thus the more complex topological space provided by the 2-simplicial Transformer should generate better results than the 1-simplicial Transformer.
While technically sound  it’s not obvious whether  this implementation is indeed better than the normal.

First of all the significance of the results are heavily determined by the fact that the authors just displayed the results of the proposed algorithm against the “normal Transformer”, using a modified version of the BoxWorld. Moreover  in the paper it is stated that, they tested the new algorithm with the original BoxWorld environment, but they omitted it as the proposed algorithm excelled over the old one in such environments. Also the relational induction algorithm used in this paper does not seem to be the same as the one used by Zambaldi et al.

Second, Im concerned about the explanation made where they try to justify that the use of the bridge-BoxWorld which is argued to provide a more complex logical structure because of the use of linear logic-formulas using more connectives. In other words, the question arises as to whether the authors thought that a more complex logical structure would i mply a more complex environment, thus justifying the use of the bridge-BoxWorld. 

Finally, it is worth reiterating that, although the use of simplices and simplicial complexes is a very interesting idea, especially to provide a mathematical explanation of how attention works in the Transformer, it’s not entirely obvious  whether the implementation of the 2-simplicity Transformer really presents an improvement over the original Transformer, in the learning of abstract representation. 
review_title: Official Blind Review #1

Meta-review: This paper extends the Transformer, implementing higher-dimensional attention generalizing the dot-product attention. The AC agrees that Reviewer3's comment that generalizing attention from 2nd- to 3rd-order relations is an important upgrade, that the mathematical context is insightful, and that this could lead to the further potential development. The readability of the paper still remains as an issue, and it needs to be address in the final version of the paper.
Decision: Accept (Poster)