Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors propose a neural network approach to variable unification and
reasoning by example as a way to mimic the human ability to identify invariant
patterns in examples and then apply them more generally in practice.
This general idea of identifying invariates and mapping new instances to
them is well motivated by the authors, citing work in philosophy of mind,
cognitive science, and developmental psychology.

The authors go on to propose MLP, CNN and Memory Network models
of unification for sequence, grid, and story reasoning tasks respectively.
Experiments on the sequence and grid datasets demonstrate the data efficiency
of this approach. MLP and CNN models with unification achieve near perfect
performance in fewer iterations (an order of magnitude fewer in the MLP case!)
 than their non-unification enabled counter parts.
Unification enabled models also demonstrate high performance in a reduced
training set setting (using only 50 training examples).
While this is encouraging, these are very simple toy tasks.

I also am in doubt as to whether the representation of these problems
causes some issues. In the sequence task, one question the models are
trying to solve is what symbol is the head or tail of the sequence.
Modeling variables over the sequence of symbols here is, in a sense, the
wrong object of study. The position of the symbols would need to be
represented, e.g.

a b c d
1 4 3 1

where I've represented positions as a-d, and the learned invariant about
head questions would be:

X:a b c d
Y:1 4 3 1

As is, by mapping symbols and not positions to variables, one cannot,
at the variable level distinguish between the two 1s in the sequence above.
My guess is that in practice the bi-GRU model that produces embedding
features of the symbols in sequence is implicitly representing head/tail
positioning.

Similar arguments could be made about the grid example.

I don't find the experiments/analysis on the bAbI dataset very convincing.
For instance, in the example given in Figure 4b (reproduced below)
is shown as an example of
temporal reasoning, where a symbol Z is mapped to the
word morning (a symbol distinguishing a time), and the question asked is
where was Bill before school.
If logical reasoning is being used to solve this question, surely the
symbol 'before' must also be represented as a variable. Its possible that
the model is instead learning a trick about mutual exclusivity, i.e. that
Y:school is the only location symbol not mentioned in question but this
could fail as a general strategy.


this Z:morning X:bill went to the Y:school
yesterday X:bill journeyed to the A:park
where was X:bill before the Y:school
A:park

Figure 4b

It would make for a much more interesting paper if the authors took
examples such as these and formed counter-factuals to probe the way
the models are answering the questions. E.g., transforming the question
in 4b to "where was X:bill today" or "where was X:bill after school."

Because the authors use soft unification, interpretability is difficult
to assess. Interpretability is crucial here because to claim that unification and reasoning by logical induction
is being used to solve tasks, it becomes important to show how the neural networks
make their decisions. Given the instances of extra variables and one to many
mappings on the bAbI dataset it seems very likely that the models are not
solving many tasks
using unification as it would be possible to learn to use the symbols directly
to learn to answer. As such, I think these issues are not addressed in the
paper sufficiently to warrant acceptance.


Minor Notes

- In definition 1, the definition of Variable is a little confusing because there are two different senses of the word in use. I understand them to be (1) Variable (X) in the logical template that is intended to be learned and used in problem solving, and
(2) variable (x) in the neural network model that is a soft asignment of
the Variable to a default symbol s. It would be nice if this distinction could
be noted or made clearer.

- In the definition 2, in the phrase "is the invariant example such as a tokenized story" it might be worth stating that the tokens are the symbols in S.


- My understanding is that each unique symbol in the invariate is a potential
variable. Does this mean there are no co-referent symbols in the invariate?
Would be helpful to state whether babi contains co-referent expressions
and how these might affect the model.


- It might be interesting to see how model architecture affects variable
learning. For example, does a CNN result in more sensible variable
assignments  than the mlp on a flattened representation of the grid problem?


- What is the strongly supervised case? These are token level annotations I think (at least for babi) but it might be good to specify in more detail what
they  are.

- The figure and explanation of the UMN are not very clear. From the figure
is does not seem that the variables interact with the memory at all. More
space could be devoted to this section.

Possibly Relevant Related Work

Brenden Lake. Compositional generalization through metasequence-to-sequence learning. NeurIPS 2019.






review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper presents a novel approach for learning invariants that can capture underlying patterns in the tasks through Unification Networks. This effectively allows the machine to learn the notion of `variable`, which is a symbol that can take on different values. 

Pros:
The authors evaluated and presented empirical results on four common benchmark datasets, showing superiority over plain baseline without unification. 
They further performed analysis on the learned invariants, and verified the sensibility. 
The paper overall is well written and structured.

Cons:
Despite its superiority over plain baseline, the paper does not provide thorough comparison with other state-of-the-art methods on reasoning related tasks.

Some of the technical details regarding the choice of hyperparameters are missing. For example:
In section 6, what’s the rationale of setting $t$ differently for bAbI solely?
In Equation 5, how is the sparsity regularization parameter $\tau$ chosen optimally for a particular task? A bit more discussion on these choices would be helpful. 

Overall, this paper presents a seemingly promising architecture capable of learning and using variables, with the caveat for lack of experiments and comparison with other state-of-the-art methods. 

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper explores a very interesting idea: can a model learn what variables are and how to use them? Unfortunately, the paper doesn't seem quite ready: the model description was very hard to follow and it's not clear the approach has found a compelling use case.

I read the paper carefully three times, and try as I might, I simply can't get my head around the entire architecture. The modeling section jumps straight into a series of definitions, without trying to build intuition or provide a worked example. There is an example in Figure 2, but it isn't really explained and I didn't find it helpful. Unification seems to be implemented as a form of attention (or self-attention) where the model can control the degree to which a symbol acts as variable. But the relationship between soft unification and attention isn't really spelled out -- what's the same, what's different? Ultimately it's not clear to me what the model is attending over during soft unification.

There are various other aspects of the paper that aren't clear:
- strong vs. weak supervision
- comparison models DMN and IMA are not introduced at all, and include no references
- the logical reasoning experiment is not clearly described
- there is only a cursory conclusion

I am not sure the model has found a compelling use case. On bAbi with weak supervision, the model is worse than the comparison models. It only slightly beats out memory networks with strong supervision. For logical reasoning, it's not clear what it is compared against or if the comparison is fair. The clearest win over standard networks is on the simple synthetic experiments.

Finally, the authors mention the paper has a cognitive science motivation, in that "Humans learn what variables are and how to use then at a young age" or that "symbolic thought with variables is learned...", taking a strong "nurture" stance on the origin of variables. But variables could very well be innate and simply early emerging. Any discussion of the origin of variables in the mind requires more nuance.

I am excited about this research direction, and it could ultimately be a very nice contribution as the work matures. I don't think the paper is ready in its current form.

review_title: Official Blind Review #3

Meta-review: The main concern raised by reviewers is the limited experiments, which are on simple tasks and missing some baselines to state-of-the-art methods. While the overall approach is interesting, the reviewers found the empirical evidence to be fairly unconvincing. 
Decision: Reject