Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This looks like an interesting paper with an original proposal. The empirical results on synthetic tasks are also good. The main problem that I am having is with the proposed network, specifically equation 3. I do not see why it makes sense to consider an outer product of $n$ and $e$ as an argument to FWM. As is mentioned in the paper (in appendix A) it would make more sense (both conceptually and from the perspective of complexity) to concatenate those two inputs, or even consider two separate inputs for the associative memory module. The authors argue that in that case the memories would interfere with each other. This is true if a weak associative memory, like the one considered in this work is used. However, if the authors used a modern Hopfield network such an interference would not be a problem. Specifically, consider the situation when after applying the FWM weights to $n$ and $e$ the results are passed through a steep non-linear activation function, like in Ref [1] (see for instance formula 10). This would suppress the interference between the memories and provide a nice memory recovery. Additionally, with these “stronger” models of associative memory the key vectors do not have to be orthogonal.  

Experimental results look fine, however, I think the work would benefit from some comparisons with other proposals for fast changing weights models, for example Ref [2]. 

I am not sure I understand the last paragraph on page 2. It is very easy to convert Modern Hopfield Networks from the autoassociative to heteroassociative type. One just needs to introduce additional matrices for queries, keys and values, like it is done in Ref [3] when comparing Modern Hopfield Networks with attention. Also, when referring to Modern Hopfield Networks, the reference for the original work, Ref [1], is missing.  

A couple of presentational suggestions: 

1. Figure 1 seem to be inaccurate. In order to generate x_{t+1} one needs to take into account both the output of FWM and current state h_t. Only the first arrow is shown in figure 1.

2. After equation 4, what is W_n? Looks like a misprint - should it be W_q? Also in the second line after equation 4 there are some misprints in the formulas. 

I also have some questions: 

1. Typically most associative memory models converge to a fixed point if one runs them for a long time. It is not obvious to me if dynamical rules described by equations 1-3 converge to a fixed point after a sufficiently large number Nr of iterations. Do they converge to a fixed point or not? 

2. It looks to me that the results reported in table 2 indicate that LSTM without FWM have lower perplexity than LSTM with FWM on that task. At the same time, the authors seem to say in the text (second paragraph on page 8) the opposite. Could the authors please clarify this? 

I am willing to increase the scores for this submission if the questions/comments above are addressed. 

References: 

[1] Krotov and Hopfield, NeurIPS 2016. Dense associative memory for pattern recognition, arXiv:1606.01164.

[2] Ba, et al, NeurIPS 2016, Using fast weights to attend to the recent past, arXiv:1610.06258.  

[3] Ramsauer, et al., 2020. Hopfield networks is all you need, arXiv:2008.02217.
review_title: The work proposes to complement an LSTM with an additional associative memory model with fast changing weights. The proposed combination demonstrates good results on several ML tasks. 

final_score: rating: 7: Good paper, accept
review_content: The solution proposed is the combination of an RNN (LSTM) and Fast Weighted Memory (FWM). The LSTM produces a query to the memory used to retrieve information from the memory and be presented at the model output. It also controls the memory through fast weights that are updated through a Hebbian mechanism. The FWM is based on Tensor Product Representations (TPR). The FWM is differentiable and builds upon the work of TPR-RNN from Schlag and Schmidhuber and Metalearned Neural Memory (MNM) by Munkhdalai et al. In the experimental section, the authors propose a concatenated version of the bAbI dataset to test their model with language modeling and question answering. Further the model is trained on a meta-learning task over POMDPs on graphs, and on language modeling on the PennTree Bank dataset. They show that the LSTM-FWM model generalizes better than without memory and similar models and with smaller capacity.

======================================

Indeed, the FWM model is relevant to this community and involves current scientific discussion and challenges. The paper is clear and is enjoyable to read. Math derivations and experimental results seem sound. Nevertheless, there are some clarity issues with the PTB language modeling task.

======================================

Would appreciate if the authors can answer to the following questions:

How is the FWM (tensor $\mathbf{F}_t$) initialized? How does the initialization influence training and performance?

How is Nr selected?

What is the vocabulary size in catbAbI? Is the embedding layer learned or pre-trained?

“The experimental results in table 2 demonstrate a relative improvement over the AWD- LSTM baselines, which suggest the benefit of our FWM.” It is unclear what is the benefit in the PTB dataset. The results show that the LSTM model has slightly better perplexity (60.0 / 57.3) than the LSTM-FWM (61.39 / 59.37). Please, could you clarify the above note versus the numbers?

Does Figure 2 have missing details? The caption doesn’t seem to match the figure or it is unclear what authors are referring to.

Figure 3 can benefit from using a bigger font for the node and edge values.

======================================

I'm inclined to accepting this paper. I found the idea simple but yet effective, and tested correctly in the experimental sections. Would appreciate it if the authors can improve the clarity surrounding Figure 2, and explain the misleading comment regarding the PTB task.

======================================

Minor issues:

-Page 2: “An biologically” -> “A biologically”

-Page 2: “pattern is is different” -> “pattern is different”

-Page 5: Please correct with the missing number “suffered a TODO% drop”

-Page 5: “figure 4.1.1” -> “Figure 2”

-Page 6: “noteable” -> “notable”

==================================
UPDATE

Thank you for replying to my questions and clarifying in the document.

review_title: Official review #3

final_score: rating: 7: Good paper, accept
review_content: This paper presents a new method called Fast Weights Memory (FWM) to add an associative memory to an LSTM. 

Model:
* FWM updates its fast weights through a differentiable perceptron like update at every step of an input sequence. The slow weights of the LSTM are instead updated only during training using gradient descent.
* FWM is based on previous work: TPR (Tensor Product Representation). TPR is a mechanism that uses tensor products to generate unique representations of combination of components.
* For long sequences FWM also has specialized components that allow it to update deprecated associations.

FWM is related to:
* TPR-RNN: a sentence-level model for reasoning on text, achieving excellent results on bAbI.
* MNM (Metalearned Neural Memory): a word-level model which augments an LSTM with a FFNN as its memory, trained with a meta-learning objective.

The authors propose the new task "catbAbI", a variation of the existing task "bAbI".  catbAbI seems to be mostly just a concatenation of the stories, questions and answers in bAbI into a single textual sequence. It's unclear how much harder catbAbI is compare to bAbI in principle.

TPR-RNN and MNM are only trained for short sequences and so will have a hard time on catbAbI. The authors show that MNM in particular does poorly on the long sequences in catbAbI.

Results:
* good performance on catbAbI (language reasoning) -- but this is a new task, so no real baselines in other papers.
* good results meta-reinforcement-learning for POMDPs compared to LSTMs.
* good results on PTB language models, better than other published models, but not state of the art.

Limitations:
* FWM requires an order 3 tensor, which scales poorly in both time and space computational complexity. This limits this work to relatively small models.

Questions:
* catbAbi simply converts bAbI into a single sequence of tokens. Does this really increase the true difficulty of the task, or is it rather a way of artificially limiting the class of models used to solve the task to simple LM-like models? Is it possible to reconstruct bAbI from catbAbI with simple heuristics?
* Could you report results for FWM on bAbI? It’s pretty unclear at the moment how to compare the results on bAbI of FMW to the ones e.g. in cited “Metalearned Neural Memory” paper. Or at least results on a version of bAbI where predictions are run for each story separately, so that MNM is not as penalized for not being able to deal with long sequences of text.
* In figure 2, what does the color represent?

Typos:
* Page 2:
   * *A biologically more plausible
   * *stateful weights that can adapt
   * Most memory-augmented NNs are based *on content-based…
* Page 3: 
   * becomes a part of the *model's output.
   * Figure 1: A simplified illustration of *our proposed method
   * third-order tensor operations using *matrix multiplications
* Page 4:
   * Wq → Wn in equation (1)
* Page 5:
   * there is TODO left

review_title: Interesting method for fast memory, but the experiments are not totally convincing.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The authors present a working memory model composed of a recurrent neural network trained via gradient descent and an associative memory based on the approach taken by Ba et al. (2016) in "Using Fast Weights to Attend to the Recent Past".  The model consists of an LSTM to which takes the input and its own state from the previous step to produce an output (or new state) which is then passed to a fast weight memory (FWM) module.

The application of the fast weights are decomposed into two steps: read and write.  The write step composes the fast weight matrix update where new information is written into F given the LSTM hidden state and the fast weight matrix from the last step.  The read step consists of potentially several  recurrent "inference steps" over the FWM producing an output (e.g. a next step prediction or encoding). 

The authors evaluate the model over two separate datasets.  The first is a modified version of the bAbI dataset which concatenates separate bAbI stories together and can be trained and evaluated in either a language modelling (LM) mode or question-answering (QA) mode where knowledge about past facts must be utilized.

Strengths & Weaknesses:

The problem itself is well motivated since associative inference is useful in solving problems that require an accurate working memory.  Fast weight approaches allow us to learn to produce good state representations of the input sequence via slow weights (h_t) and where fast weights provide the associative mechanism to make important links across time. 

The authors propose a new model that combines a novel read-write mechanism that relies on a number of inference steps over the fast weights allowing a nice disentanglement of read/write operations taking advantage of the associative inference to both add new relevant associative information (v) while also filtering stale data (v_old). 

That said the overall form of the model doesn't seem fundamentally different from what is proposed by Ba et al. (2016) who also used fast weights as a way to attend over past hidden states in combination with a "slow" weighted RNN trained via gradient descent optimization albeit some of the details differ.

Further it would be helpful if the authors could clarify more around the rationale around why particular architectural choices were made.  For instance, why are two keys are generated in the write operation? 

Results for both catbAbI don't seem to exceed the performance of the TransformerXL when comparing perplexities in both QA and LM mode and don't exceed TrXL accuracy in LM mode.  However it is noted that the FWM model is in fact much smaller.  It may have been useful to investigate the gated transformer XL which is known to exhibit stronger stability for RL.  Figure 2 is nice though, is there any intuition why the reads vary among strong negative or positive activations as it seems to indicate?

As for the meta-RL problem it would have been nice to see comparisons to baselines other than an LSTM.  For instance, Ritter et al. (2020) in "Rapid Task Solving in Novel Environments" introduce a model that combines an episodic memory with self attention to meta-learn how to explore and exploit navigation to goals in connected graphs. 


Other points:

The labelling for the edges Figure 3 isn't really clear.
There's a missing reference in second to last paragraph on page 5: "... QA-mode suffered a TODO% drop in accuracy ..." 


Recommendation:

I don't think there's enough here to recommend acceptance.  For starters, I don't think there's quite enough in justification around the architectural choices of the model and exactly what distinguishes this from the model proposed by Ba et al. which also used fast weights in combination with a "slow" weighted RNN.  Next, the results are not strong enough and additional or stronger baselines would have helped paint a better picture of the potential benefits of this approach.  For the results in general, while I think that these results point in the possible direction of the utility of FWM I don't believe the paper in its current form demonstrate that FWM exceeds state of the art in the chosen domains in which it was evaluated.  That said, I believe this is a promising line of research and encourage the authors to try to address the issues raised.




review_title: Review of "LEARNING ASSOCIATIVE INFERENCE USING FAST WEIGHT MEMORY"

Meta-review: This paper proposed a way to combine LSTMs with Fast weights for associative inference.

While reviewers had concerns about comparison with Ba et al., and experimental results, the authors addressed all the concerns and convinced the reviewers. The revision strengthened the paper significantly. I recommend an accept.
Decision: Accept (Poster)