Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors propose a binding-unbinding mechanism for translating natural language to formal language. The idea is good and novel. As far as I know, this is indeed the first work for handling this task using binding-unbinding mechanism. The experimental results also look promising in compared with the exsiting models. However, the designed specific neural network does not support the claimed binding-unbinding theory very well. Moerover, there seem to be some errors about the correctness of the theory (See the first point below).

Firstly, in the last paragraph of Section 2, the authors claim that the role matrix $R$ would be invertible such that there exists a matrix $U = R^{-1}$ such that the fillers would be recovered. However, $R$ is defined as a non-square matrix in the previous paragraph. How can a non-square matrix be invertible? 

Secondly, the design of the specific neural network cannot describe the theory behind proposed binding-unbinding mechanism properly. The authors try to interpret the design of the neural networks using the concepts in the proposed binding-unbinding theorybut are not convincible.  In Section 2, the basics of binding-unbinding are introduced and many mathematical properties are required to make the binding-unbinding work. However, all the parameters/variables in the neural networks are freely designated and are not correlated to each other, thus they cannot work together to meet the requirements in the binding-unbinding mechanism. According to my understanding, at least there should be some direct connections between the parameters in the encoder and decoder. For example, is there any restriction on the parameters in encoder and decoder respectively to reflect the property $UR=I$ as in Section 2.

Lastly, in the encoder part, the role and filler are learned in an unsupervised without any evidence. The input for the decoder is an "assumed" TPR, thus the only evidence from the objective function are cut-off by the assumed TPR. Given that there are no other connections between encoder and decoder, the design of the encoder cannot learn role and filler properly.

Other suggestions:
The natural language to formal language problem is named semantic parsing in natural language processing field. In semantic parsing problem, langugae to programatic language is a typical task. I would recommend include some references in semantic parsing.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes a sequence-to-sequence model for mapping word sequences to relation-argument-tuple sequences. The intermediate representation (output of the encoder) is a fixed-dimensional vector. Both encoder and decoder internally use a tensor product representation. The experimental results suggest that the tensor product representation is helpful for both the encoder and the decoder. The paper is interesting and the experimental results are positive, but in my opinion the exposition could use some substantial work. Fixing the most substantial flaws in the exposition would be sufficient to warrant an accept in my view.


Major comments:

I found the mix of levels of detail in the model specification in section 3 confusing. It would be extremely helpful to have a straightforward high-level mathematical description of the key parts of the encoder, mapping (which could be considered part of the encoder), and decoder in standard matrix-vector notation. While equations (7), (8), (9), (10), (11) and appendix A.2 go some way toward this, key high-level details seem to be missing, and I feel like the exposition would benefit from simply stating the matrix-vector operations that are performed in addition to describing their interpretation in terms of the semantics of the tensor product representation. Specific examples are noted below.

It would be helpful to be explicit about the very highest-level structure of the proposed model. If I understand correctly, it is a probabilistic sequence-to-sequence model mapping a word sequence to a probability distribution over relation-argument-tuple sequences. It uses an encoder-decoder architecture with a fixed-dimensional intermediate representation, and an autoregressive decoder using attention. Both the encoder and decoder are based on the tensor product representation described in section 2. Stating these simple facts explicitly would be extremely helpful.

Especially for the encoder, the learned representation is so general that there seems to be no guarantee that the learned roles and fillers are in any way related to the syntactical / semantic structure that motivates it in section 2. There doesn't seem to be any experimental investigation of the learned TPR in the encoder. If I understand correctly, the way encoder roles and fillers are computed and used is symmetric, meaning that the roles and fillers could be swapped while leaving the overall mapping from word sequences to relation-argument-tuple sequences unchanged. This suggests it is not possible to interpret the role and filler vectors in the encoder in an intuitive way.


Minor comments:

In section 2, "R is invertible" should strictly be "R has a left inverse".

In section 3.1.1, the claim that "we can hypothesize to approximately encode the grammatical role of the token and its lexical semantics" is pretty tenuous, especially given the apparent symmetry between learned roles and fillers in the encoder and given the lack of experimental investigation of the meaning of the learned encoder roles and fillers.

In section 3.1.2, my understanding is that the relation-argument tuple (R, A_1, A_2, A_3), say, is treated as a sequence of 3-tuples: (A_1, R, 1), (A_2, R, 2), (A_3, R, 3). Each of these 3-tuples is then embedded using learned embeddings (separate embeddings for argument, relation and position). If correct, it would be helpful to state this explicitly.

In section 3.1.2, it is stated that contracting a rank-3 tensor with a vector is equivalent to matrix-vector product, which is not the case.

In section 3.1.3, both high-level and low-level details of the MLP module are omitted. High-level, I presume that the matrix output by the encoder is reshaped to a large vector, the MLP is applied to this vector to produce another vector, then this is reshaped to a rank-3 tensor to input to the decoder. It would be helpful to state this. Low-level, the number of layers, depth and activation function of the MLP should be specified somewhere, at least in the appendix.

Did the authors consider using a bidirectional LSTM for the encoder? This might improve performance.

In section 3.1.2 and appendix A.2, why use the LSTM hidden state for subsequent processing rather than the LSTM output (which would be more conventional). The LSTM output is defined in appendix A.2 but appears not to be used for anything. Please clarify in the paper.

Did the authors consider passing the output of the reasoning MLP into every step of the tuple LSTM instead of just using it to initialize the hidden state?

It would be helpful to state the rank of the tensors H, B, etc in section 3.2.2.

In section 3.2.2, what does "are predicted by classifiers over the vectors..." mean? This seems quite imprecise. What is the form of the classifier? My best guess is that the vector a_i^t is passed through a small MLP with a final softmax layer which outputs a probability distribution over the 1-of-K representation of the argument. The main text says "more details are given in the appendix", but appendix A.2 just has "Classifier(a_1^t)". Please clarify in the paper.

What is the attention over in equation (9)? Attention needs at least two arguments, the query and the sequence being attended to. It seems that (9) only specifies one of these. It would also be helpful to be explicit about the form of attention used.

What is f_linear in (11)?

It seems unnecessarily confusing to switch notation for the arguments from A_1 in section 3.1.2 to a r g_1 in section 3.2.2, and similarly for the relations.

For the decoder tuple LSTM, how exactly is the previous relation-argument tuple (R, A_1, A_2, A_3), say, summarized? Are each of R, A_1, A_2 mapped to a vector, these vectors concatenated, then passed into the LSTM? Or is the positional decomposition into (A_1, R, 1), ... used? Please clarify in the paper.

Based on section 3.3, it seems that the model assumes that, in the decomposition of (R, A_1, A_2, A_3) into a sequence (A_1, R, 1), (A_2, R, 2), (A_3, R, 3) of 3-tuples at each decoder output step, the three 3-tuples are conditionally independent of each other and the three entries of each 3-tuple are conditionally independent of each other. Is this indeed assumed? If so, it would be helpful to state this explicitly. It seems like this is likely not true in practice.

Section 3.3 refers to "predicted tokens". Where are these predicted tokens in (9), (10) or (11)?

In section 3.3, it seems the loss at each decoder step is the log probability of the relation-argument tuple at that step. Thus, by the autoregressive property, the overall loss is the log probability of the sequence of relation-argument tuples. If so, it would be helpful to state both these facts explicitly.

Section 3 seems to be missing a section, which is how decoding is performed at inference time. For the output of the decoder at each step, is random sampling used, if so with a temperature, or is greedy decoding (selecting the most likely class, equivalent to a temperature of 0) used? Also, what is done if decoding outputs different R's for (A_1, R, 1), (A_2, R, 2), (A_3, R, 3)? The three R values here should be equal in order for this to represent a relation-argument tuple (R, A_1, A_2, A_3), but there is no guarantee the model will respect this constraint.

Unless I missed it (apologies if so), many experimental architecture details were omitted. For example, how many hidden cells were used for the LSTMs, etc, etc? These should at least be stated in the appendix.

It would be interesting to investigate how long input / output sequences need to be before the fixed-dimensional internal representation breaks down.

In section 4.1.1, it was not clear to me what "noisy examples" means. Does this mean that the dataset itself is flawed, meaning that the reference sequence of operations does not yield the reference answer? Please clarify in the paper.

In table 1, please state the total size of the fixed-dimensional intermediate representation for all systems. This seems crucial to ensure the systems can be meaningfully compared.

In figure 4, left figure, the semantic classes don't apper to be very convincingly clustered. (And it seems like K-means clustering could easily have selected a different clustering given a different random initialization.)

In appendix A.2, mathematical symbols are essentially meaningless without describing what they mean in words. Please explain the meaning of all the symbols that are not defined in terms of other symbols, e.g. w^t, T_{t-1}, ..., f_s m (is this softmax???), f_l i n e a r (what does this mean?), C o n t e x t, C l a s s i f i e r, etc, etc. C o n t e x t in particular doesn't even have a hint of a definition.

In (19) and (27), why would a temperature parameter be helpful? This can be absorbed as an overall multiplicative factor in the weight matrix of the previous linear layer. Is this temperature parameter learned during training (I presume so)? Please clarify in the paper.

Usually * is used for convolution, not simple multiplication (e.g. equation (17)).

Throughout the main body and appendix, there are lots of instances of poor spacing. For example, $f_{linear}$ should be written as something like $f_\text{linear}$ in latex to avoid it coming out as l i n e a r (which literally interpreted means l times i times n times e, etc). Please fix throughout.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper considers the challenging problem of learning to generate programs from natural language descriptions: the inputs are sequences of words describing a task and the outputs are corresponding programs solving the task. The proposed approach elegantly relies on tensor product representations. Inference with the proposed model is done in 3 steps: (1) encode the symbolic information present in the text data as a TPR, (ii) maps the input TPR to an output TPR encoding the symbolic relations of the output programs (here the authors use a simple MLP), and (iii) decode the output TPR into an actual program. The parameters of the models used in the 3 steps are learned jointly. For step (iii), the authors proposes a novel way of encoding an n-ary relation into a TPR which facilitates the recovery of the relation's arguments using unbinding operations: this is a neat trick (though I think it increases the number of parameters and may limit the expressiveness of the TPR, since reaching "full-rank" of the TPR will occur faster than with the encoding used in [Smolensky et al., 2016]). Experiments on two datasets demonstrate the validity of the approach.

The paper is very well written and easy to follow. The idea seems original and well executed but I think the experimental section could be improved. In particular, adding/reporting stronger baselines to the comparison would straighten the paper. I also feel some relevant literature may be missing from the related work. Nonetheless, I think it is a good paper which will be relevant to the community, I thus recommend acceptance.

* Comments / Questions *

- Section 3.1.1: if I understand correctly, the length of the sequence affects the rank of the TPR. Could that be a problem in practice? E.g., the capacity of the TPR could likely be saturated quickly for long sequences?

- Section 3.2.1: the filler vector f_t = Fu is computed as a convex combination of the learned filler vectors. Is it a design choice to choose a convex combination rather than taking the column corresponding to the argmax of the vector u? Or is it because otherwise the model can not be trained using the classical backprop approach? 

- The results of the Seq2Tree+Search model from (Bednarek et al. (2019)) is not reported in Table 2. Why? I believe it should be included (it is ok that it outperforms the proposed method. In addition you can maybe identify clear advantages of your method illustrating a trade-off, e.g., running time, end-to-end, scalability ...). 

- A more thorough ablation study could also improve the strength of the experiments. For example, do you know to which extent the attention model in the decoder is necessary to achieve good performances?

- I am not very familiar with the literature but it seems some relevant work may be missing from the review. In particular, I believe there are many papers tackling the problem of learning programs from input output examples or execution traces, e.g. "DeepCoder: Learning to Write Programs", "Neural Turing machines",  "Inferring algorithmic patterns with stack-augmented recurrent nets", "Inferring and Executing Programs for Visual Reasoning", "Learning to infer graphics programs from hand-drawn images"... This list is by no means meant to be exhaustive in any way, just to illustrate a large body of work that seems relevant to the present paper (even though I understand that those papers do not consider natural language description as inputs).

* Typos *

- Eq. (5) Should be r' instead of r_i' (?)

review_title: Official Blind Review #2

Meta-review: The paper proposed a new seq2seq method to implement natural language to formal language translation.  Fixed length Tensor Product Representations are used as the intermediate representation between encoder and decoder.  Experiments are conducted on MathQA and AlgoList datasets and show the effectiveness of the methods.  Intensive discussions happened between the authors and reviewers.  Despite of the various concerns raised by the reviewers, a main problem pointed by both reviewer#3 and reviewer#4 is that there is a gap between the  theory and the implementation in this paper.  The other reviewer (#2) likes the paper but is less confident and tend to agree with the other two reviewers.
Decision: Reject