Title: Neural Program Lattices
Authors: Chengtao Li
Daniel Tarlow
Alexander L. Gaunt
Marc Brockschmidt
Nate Kushman
Venue: ICLR 2017
Keywords: Deep learning
Semi-Supervised Learning

Abstract: We propose the Neural Program Lattice (NPL), a neural network that learns to perform complex tasks by composing low-level programs to express high-level programs. Our starting point is the recent work on Neural Programmer-Interpreters (NPI), which can only learn from strong supervision that contains the whole hierarchy of low-level and high-level programs. NPLs remove this limitation by providing the ability to learn from weak supervision consisting only of sequences of low-level operations. We demonstrate the capability of NPL to learn to perform long-hand addition and arrange blocks in a grid-world environment. Experiments show that it performs on par with NPI while using weak supervision in place of most of the strong supervision, thus indicating its ability to infer the high-level program structure from examples containing only the low-level operations.


Review discussion:
The stack in this work resembles the one proposed by Mikolov et al (https://arxiv.org/abs/1503.01007). Was there a particular requirement that led to the choice of this type of stack, versus other neural stack/queue models, e.g. https://arxiv.org/abs/1506.02516?
For each timestep, our lattice maintains a grid of different possible stacks, one for each number of elementary operations performed so far, $i$ and each possible stack depth, $l$.  Each grid node in the following timestep will contain a merged (weighted average) version of three different stacks from the previous timestep.  In the general case, these three different stacks will have no particular relationship to each other, and will not necessarily share any elements.  Our stack merging operation is a natural extension of the stack model from Mikolov et al.
However, the stack model from https://arxiv.org/abs/1506.02516 takes advantage of the fact that entities are added to the stack in a temporal order, which is not the case in our setting since at each timestep entities are pushed/popped from all nodes in the grid in parallel.
Thus we need the ability to take a weighted average of full stacks, and it seems less natural to do this with the representation of https://arxiv.org/abs/1506.02516
Would it be possible that you clarify the relation between your stack and the one from Joulin & Mikolov in the paper? Currently you're not making any connection with this work in your paper.
At test time, our current implementation is equivalent to the model of Reed and de Freitas since we use a greedy decoding process just as they do.  In this scenario there is no use of a differentiable stack, and thus no relation to Joulin & Mikolov.  At training time, our framework uses the same primitive operation used by Joulin & Mikolov of stack averaging weighted by probabilities.
However, we extend/modify their model in two important ways:
1. We use the stack to implement explicit function abstraction.
In contrast, the stack in Joulin & Mikolov is used simply as an additional memory beyond that in the internal RNN cell/hidden state without any constraints to encourage/enforce its use as a mechanism for function abstraction.
2. We trade off computation for accuracy by representing many possible stack states at each time step, one for each number of elementary operations performed so far, $i$ and each possible stack depth, $l$.  The resulting stack updates are more complicated and more computationally intensive, but also more accurate.
That said, we recognize these distinctions are not very clear in the current version of the paper, and we are in the process of restructuring the technical sections to make this comparison more explicit and easier to understand and will include this in our next update to the paper.
Thanks for the answer!
Let me ask a few more questions though:
"In this scenario there is no use of a differentiable stack, and thus no relation to Joulin & Mikolov"
--> I believe that Joulin & Mikolov also uses a greedy decoding schemes at test time, so it seems actually related.
"We use the stack to implement explicit function abstraction.
In contrast, the stack in Joulin & Mikolov is used simply as an additional memory beyond that in the internal RNN cell/hidden state without any constraints to encourage/enforce its use as a mechanism for function abstraction."
--> Sorry if I miss something, but:
- what do you mean by "function abstraction"? In the paper it seems that you re pushing on the stack E[g], that is the embedding of the program. In Joulin & Mikolov, they use multiple stacks, which means that they can push an embedding as well if required (each stack pushing one number, if you have 10 stacks it can push an embedding of size 10). That being said, they don't push explicitly embeddings on their stacks.
- "without any constraints to encourage/enforce its use as a mechanism for function abstraction": The value on the top of the stacks are used to update the hidden states, which then control other stacks. Doesn't that mean that it acts actually as a "mechanism for function abstraction"?
"We trade off computation for accuracy by representing many possible stack states at each time step, one for each number of elementary operations performed so far, $i$ and each possible stack depth, $l$.  The resulting stack updates are more complicated and more computationally intensive, but also more accurate."
--> I believe that Joulin et Mikolov also use multiple stacks that interact with each other, how is that different?
Thanks again for your questions.  I will try to answer them point-by-point in the following:
"I believe that Joulin & Mikolov also uses a greedy decoding schemes at test time, so it is actually related."
--> You are correct, my previous statement was not very precise.  All I meant was that at test time our model is essentially the same as the model of Reed & de Freitas. All three models are related in that they use RNNs with the ability to keep around additional information in a stack.
"""what do you mean by "function abstraction"? In the paper it seems that you re pushing on the stack E[g], that is the embedding of the program. In Joulin & Mikolov, they use multiple stacks, which means that they can push an embedding as well if required (each stack pushing one number, if you have 10 stacks it can push an embedding of size 10).
- "without any constraints to encourage/enforce its use as a mechanism for function abstraction": The value on the top of the stacks are used to update the hidden states, which then control other stacks. Doesn't that mean that it acts actually as a "mechanism for function abstraction"?"""
The Reed and de Freitas model on which we build enables/enforces explicit function abstraction in 2 main ways that the Joulin and Mikolov model does not:
1. The model enforces a separation of state between the caller function, and the callee function.
Specifically, when calling into a function, the LSTM hidden state is reset to zero, and the only information passed into the function must come through the learned program and argument embedding vectors; and when returning from a function, the LSTM state from before the call is restored.
Furthermore, the model enforces a coupling between calling into a function and returning from that function, such that the model always returns from every function that it calls into, and returns only once from the top/main function.
In contrast the stack mechanism in Joulin and Mikolov is very general and does not contain any mechanisms to encourage/enforce this separation.  This is similar to how function abstraction can be implemented on top of a simple stack, but there's much more to this abstraction than just the stack itself.  We believe the separation of state in the function abstraction is part of what enables the strong generalization exhibited by the model.
Additionally, this separation of state forces the program and argument embedding vectors to contain all the information necessary to execute the function, and thus only these embeddings needed to be update to learn new subroutines.
While we did not explore this in our experiments, Reed and de Freitas showed this in their original paper, and in future work we would like to explore this in the context of our training framework.
2.
When the model pushes/pops from the stack in order to call into/return from a function it *does not* perform an elementary  operation (i.e., produce observable output).   This enables the model to make a decision to call into a function at one timestep, and at the next timestep make a separate decision about whether to call into another function or directly perform an elementary operation.  This nested decision making can be arbitrarily deep (up to some constant set at training time).
In contrast, the training setup of the Joulin & Mikolov framework requires an output token to be generated at *every* timestep, and enables only one element to be pushed onto each stack in a given timestep.  This creates a strong coupling between the decision to call into a function, and the decision around the first step in that function, and makes it difficult for the model to call into a function which in turn immediately calls into another function.  Consider for example a calling program performing the operation "if cond then A() else B()".  In the Joulin & Mikolov setting, the model would not only need to learn to branch between different functions like this, but also the first output token produced by A and B. In tasks that require many recursive calls (such as the NanoCraft example), this becomes extremely hard.
"I believe that Joulin et Mikolov also use multiple stacks that interact with each other, how is that different?"
--> If one considers the set of possible program execution paths as a tree with an exponential number of leaves, then the goal of much of the neural program inference literature is to represent the intermediate and final states in that tree as accurately as possible to enable gradient descent to push the parameters towards following the set of paths that generate correct outputs. One extreme (albeit impractical) point in that space is to exactly represent all possible paths in that tree through exhaustive exploration. Joulin and Mikolov (along with most other work in this space) choose what is in some sense, the opposite extreme, which maintains a single state representation (i. e. a single value for the state of each stack) for each timestep (i. e. depth in the tree) and they generate that approximate state by probabalistically averaging together all states at that depth.
Our training framework chooses a different point in this space by maintaining at each timestep (i. e. tree depth) a separate state of the stacks for each call depth $l$ and number of elementary operations performed so far, $i$ (i. e. $L*I$ different states for each call depth). These states are generated by probabalistically averaging all paths which are at call depth $l$ and elementary operation $i$ at timestep $t$. This results in an increase in memory and computation time during training, but more accurate training. We have not tested this explicitly, but we believe this additional accuracy reduces the number of local minima, resulting in superior performance of the learned models.
Thanks for the clarification. It would be great if these differences would be discussed in the paper too.
We will definitely highlight these differences more clearly in the upcoming update to the technical sections.  Thanks again for your feedback.
In the introduction, the paper mentions an example of abstracting human tasks from Kinect skeletons.
However, it is difficult to easily imagine how the proposed approach can learn the model from such noisy observations possibly having multiple variations. We would like to see more clarifications/discussions on this.
Thank you for your question.  That paragraph in the introduction was only intended to motivate the general direction of learning from action sequences by claiming that it is quite natural to gather this type of data for real world problems.  We have not yet tested on data with noise so we did not intend to make any claims about that scenario.  We will make this distinction more clear in the writing.
That said, we don't have any reason to believe that the basic framework won't work with noisy data without much modification,  since even a standard RNN trained with maximum likelihood will correctly learn a distribution over action sequences given enough data.  We  would like to explore this scenario in our future work.
Neural Programmer-Interpreters (NPI) achieves greatly reduced sample complexity and better generalization than flat seq2seq models for program induction, but requires program traces at multiple levels of abstraction for training, which is a very strong form of supervision. One obvious way to improve this situation, addressed in this work, is to only train on the lowest-level traces, with a latent compositional program structure. This makes sense because the "raw" low-level traces can be cheaply gathered in many cases just by watching expert demonstrations, without being explicitly told the more temporally abstract structures.
This paper shows that a variant of NPI, named NPL, can achieve even better generalization performance with weaker supervision (mostly flat traces), and also extends the model to a new grid world task. Unfortunately, it still requires being told the overall program structure by being given a few *full* execution traces.
Still, I see this as important progress. It extends NPI in a quite nontrivial way by introducing a stack mechanism modeling the latent program call structure, which makes the training process much more closely match what the model does at test time. The results tell us that flat execution traces can take us almost all the way toward learning compositional programs from demonstrations - the hard part is of course learning to actually discover the subprogram structure.
The paper presents the Neural Program Lattice (NPL), extending the previous Neural Programmer-Interpreters (NPI). The main idea is to generalize stack manipulation of NPI by making it probabilistic. This allows the content of the stack to be stochastic than deterministic, and the paper describes the feed-forward steps of NPL's program inference similar to the NPI formulation. A new objective function is provided to train the model that maximizes the probability of NPL model correctly predicting operation sequences, from execution traces. We believe this is an important extension. The experimental results illustrate that the NPL is able to learn task executions in a clean setting with perfect observations.
The paper is clearly presented and its background literature (i.e., NPI) is well covered. We also believe the paper is presenting a conceptually/technically meaningful extension of NPI, which will be of interest to a broad audience. We are still a bit concerned whether the NPL would be directly applicable for noisy observations (e.g., human skeletons) in a continuous space with less explicit structure, so more discussions will be interesting.
First I would like to apologize for the late review.
This paper proposes an extension of the NPI model (Reed & de Freitas) by using an extension of the probabilistic stacks introduced in Mikolov et al.. This allows them to train their model with less supervision than Reed & de Freitas.
Overall the model is a nice extension of NPI.
While it requires less supervision than NPI, it still requires "sequences of elementary operations paired with environment observations, and [...] a couple of examples which include the full abstraction hierarchy". This may limit the scope of this work.
The paper claims that their "method is leverages stronger supervision in the form of elementary action sequences rather than just input-output examples (sic).  Such sequences are relatively easy to gather in many natural settings". It would be great if the authors clarify what they mean by "relatively easy to gather in many natural settings". They also claim that "the additional supervision improves the data efficiency and allow our technique to scale to more complicated problems".
However, this paper only addresses two toy problems which are neither "natural settings" nor of a large scale (or at least not larger than those addressed in the related literature, see Zaremba et al. for addition).
In the introduction, the author states that "Existing techniques, however, cannot be applied on data like this because it does not contain the abstraction hierarchy." What are the "existing techniques", they are referring to? This work only addresses the problem of long addition and puzzle solving in a block world. Afaik, Zaremba et al. has shown that with no supervision, it can solve the long addition problem and Sukhbaatar et al. ("Mazebase: A sandbox for learning from games") shows that a memory network can solve puzzles in a blockworld with little supervision.
In the conclusion,  the author states that "remarkably, NPL achieves state-of-the-art performances with much less supervision compared to existing models, making itself more applicable to real-world applications where full program traces are hard to get." However for all the experiments, they "include a small number of FULL samples" (FULL == "samples with full program traces"). Unfortunately even if this means that they need less FULL examples, they still need "full program traces", contradicting their final claim.
Moreover, as shown figure 7, their model does not use a "small number of FULL samples" but rather a significantly smaller amount of FULL examples than NPI, i.e., 16 vs 128.
"All experiments were run with 10 different random seeds": does the environment change as well between the runs, i.e. are the FULL examples different between the runs? If it is the case and since you select the best run (on a validation set), the NPL model does not consume 16 FULL examples but 160 FULL examples for nanoCraft.
Concerning the NanoCraft example, it would be good to have more details about how the examples are generated: how do you make sure that the train/val/test sets are different? How the rectangular shape are generated? If I consider all possible rectangles in a 6x6 grid, there are (6x6)x(6x6)/2 = 648 possibilities, thus taking 256 examples sum up to ~40% of the total number of rectangles. This does not even account for the fact that from an initial state, many rectangles can be made, making my estimate probably lower than the real coverage of examples.
Concerning the addition, it would interesting to show what an LSTM would do: Take a 2 layer LSTM that takes the 2 current digits as an input and produce the current output ( "123+45" would be input[0] = [3,5], input[1]=[2,4], input[2]=[1, 0] and output[0] = 8...). I would be curious to see how such baseline would work. It can be trained on input/output and it is barely different from a standard sequence model.
Also, would it be possible to compare with Zaremba et al.?
Finally, as discussed previously with the authors, it would be good if they discuss more in length the relation between their probabilistic stacks and Mikolov et al.. They have a lot of similarities and it is not addressed in the current version. It should be addressed in the section describing the approach. I believe the authors agreed on this and I will wait for the updated version.
Overall, it is a nice extension of Reed & de Freitas, but I'm a bit surprised by the lack of discussion about the rest of the literature (beside Reed & de Freitas, most previous work are only lightly discussed in the related work). This would have been fine if this paper would not suffer from a relatively weak experiment section that does not support the claims made in this work or show results that were not obtained by others before.
Missing references:
"Learning simple arithmetic procedures", Cottrell et al.
"Neural gpus learn algorithms", Kaiser & Sutskever
"Mazebase: A sandbox for learning from games", Sukhbaatar et al.
"Learning simple algorithms from examples", Zaremba et al.
Thank you for your detailed and thorough review.  We would like to address each of your concerns point-by-point:
"""" "a couple of examples which include the full abstraction hierarchy". This may limit the scope of this work." """
We agree that this is a limitation of our work, but still believe that our work represents a substantial step towards a solution which can induce the abstraction hierarchy without any supervision of the hierarchy itself.
"""  It would be great if the authors clarify what they mean by "relatively easy to gather in many natural settings".  """
This was simply meant as a backpointer to the motivation section in the introduction.  In the motivation section we gave the examples of using 3-d body tracking hardware to record a human performing real world tasks, and recording users mouse clicks and keystrokes as they perform virtual tasks in an online web application.
""" What are the "existing techniques", they are referring to? """
That claim should have referred directly to NPI, and we will update the writing to reflect this.
""" In the conclusion, the author states that "remarkably, NPL achieves state-of-the-art performances with much less supervision compared to existing models, making itself more applicable to real-world applications where full program traces are hard to get." However for all the experiments, they "include a small number of FULL samples" (FULL == "samples with full program traces"). Unfortunately even if this means that they need less FULL examples, they still need "full program traces", contradicting their final claim. """
This claim of less supervision should also have referred directly to NPI, and we will update the writing to reflect this.
""" does the environment change as well between the runs, i.e. are the FULL examples different between the runs? """
The set of FULL examples is the same for all runs.  We only change the random seed used for initializing the weights.
""" Concerning the NanoCraft example, it would be good to have more details about how the examples are generated: how do you make sure that the train/val/test sets are different? How the rectangular shape are generated? If I consider all possible rectangles in a 6x6 grid, there are (6x6)x(6x6)/2 = 648 possibilities, thus taking 256 examples sum up to ~40% of the total number of rectangles. This does not even account for the fact that from an initial state, many rectangles can be made, making my estimate probably lower than the real coverage of examples. """
We generate a large set of random samples, and throw out any duplicates.  We then shuffle this set, and pull the train/val/test sets from this large set.  The number of possible samples is much larger than your calculations because we randomly choose a subset of the blocks to be already in place in the environment.
Thus for each possible rectangle with total wall length $n$, there are $2^n-1$ possible samples/tasks.
""" Concerning the addition, it would interesting to show what an LSTM would do: Take a 2 layer LSTM that takes the 2 current digits as an input and produce the current output ( "123+45" would be input[0] = [3,5], input[1]=[2,4], input[2]=[1, 0] and output[0] = 8...). I would be curious to see how such baseline would work. It can be trained on input/output and it is barely different from a standard sequence model.
Also, would it be possible to compare with Zaremba et al.? """
The setup you describe is exactly the one we used to generate the set of baselines from figure 8 called BiSeq-*. We will attempt to compare directly with Zaremba et al. using their posted source code.
""" Finally, as discussed previously with the authors, it would be good if they discuss more in length the relation between their probabilistic stacks and Mikolov et al.. They have a lot of similarities and it is not addressed in the current version. It should be addressed in the section describing the approach. I believe the authors agreed on this and I will wait for the updated version. """
We are in the process of updating the technical section to make this relationship more clear.
""" Missing references…"""
Thanks for pointing these out, we will include these in the next version
Thanks again for your thorough review.  Per our response, we have looked into comparing directly with Zaremba et. al. using their posted source code.  Unfortunately, it appears that the posted code only supports the reinforcement learning setup, and not the ‘supervised’ setup which is analogous to the results in our paper.  It’s not clear what we would learn from comparing to the reinforcement learning setup, since it’s an inherently more challenging than our ‘supervised’ setup.
Further, the supervised model is similar to our BiSeq baseline and it appears from the code that the results in the Zaremba et al. paper rely on a much larger number of training samples than are used in our work.
So it's not clear to us how valuable it would be to compare to a modified version of their code.
We’ve significantly revised the draft to address the concerns voiced by the reviewers.  This includes a rewrite of the intro and technical sections to clarify our contributions and our relationship to past work, esp. that of Joulin and Mikolov.

Meta-review:
role: PC
data: comment: This paper demonstrates a novel (although somewhat obvious) extension to NPI, namely moving away from training exclusively on full traces (in order to model the nested calling of subprograms) to training, in part on low-level program traces. By exploiting a continuous stack in the vein of Das et al. 1992, Joulin and Mikolov 2015, or Grefenstette et al. 2015 (any of which could plausibly have been used here, by my reading, contrary to claims made by the authors in discussion), they can model nested calls with a stack-like structure which needs only partial supervision of a few full traces. The claim is that the resulting model is more sample efficient than the NPI of Reed and de Freitas.
 
 The reviews are mixed. R1 is grumpy about citations, which I see as grounds for rejection, but raises the point that the claims to data efficiency may be overblown since it is only shown that few full traces are needed to train the model, not that it can infer structure without full traces, which would be more impressive. Confusingly, they refer to continuous stacks as probabilistic, which is misleading as while there is (in all variants mentioned above) a possible probabilistic interpretation of non-{0, 1} push/pop operations, the updates to the stack state are deterministic, as are all other aspects of the network except test-time sampling of actions from the multivariate distribution induced by the softmax on output.
 
 R2 has not understood the paper or background material if they mistakenly believe that the stack-like structure and resulting model are probabilistic, as there is no sampling of actions on the stack, but only deterministic state updates. The superficiality of the review combined with the fairly crucial misunderstanding sadly means I cannot rely on it to make a recommendation.
 
 R3 is broadly sympathetic to the paper, while recognising that it still requires information about the program structure through partial FULL supervision in order to learn to manipulate the continuous stack (and thus nested function calls). It is disappointing that the authors did not reply to this point in this review. I acknowledge that they address this claim in part in a response to R1, but the reply is mostly that they will rewrite the formulation of their claims.
 
 Overall, the novelty of this paper lies in the integration of an existing flavour of differentiable data structure, variants of which have recently been presented at NIPS in 2015, into NPI in order to learn program structure. This would have made for an excellent paper if this augmentation had shown that, without partial supervision on the stack operations (thus training solely from low level traces) the model had been able to infer program structures. The need to provide some full traces is disappointing in this respect, although the claim about providing a more data-efficient training regime thanks to the data structure is plausible. 

Stil, overall, the PCs encourage the authors to address the remainig issues in the camera reasy version of their paper and to present this as a poster at the main conference.

Decision: Accept (Poster)