Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose Generative Pseudo-Inverse Memory (GPM), a family of generative models that offer read and write operations of constant time complexity. Encoding new memories and decoding data from memories are postulated as Bayesian updates for which an equivalent minimization problem is proposed. This minimization problem essentially amounts to solving a linear system of equations, which can be efficiently done via computing matrix pseudo inverses. The authors demonstrate the utility of GPM on a variety of applications such as image denoising, image generation, and storage retrieval. 
main_review: (i) The overall conceptualization presented here is neat. The authors propose an efficient implementation for their algorithm, demonstrate performance on a variety of tasks both quantitatively and qualitatively. 

(ii) The text, however, is a bit hard to follow at times and needs to be severely improved both clarity-wise and grammatically.
(a) I found the various usages of "prior memories" and "posterior memories" confusing. To me, it seemed to imply that probability densities were explicitly defined on the memories (i.e. Ms). But it turns out that the parameterization was actually on the read in/out weights (Ws) which implicitly defined distributions on the memories. Furthermore in the section "Memory inference" the authors switch to considering deterministic memories. In general, it would be much appreciated if the authors clarified if/when they are estimating true posterior densities.

(b) There are also terminologies presented in the main body of the paper (such as "temporary readout") that are barely re-used throughout the article. It might be better to avoid these (at least while presenting the core ideas) to improve readability. 

(c) In Equation 3, Z is presented as a linear function of M_0, but Z is in fact a function of X from Figure 1. Could the authors clarify this mismatch?

(d) The paper can benefit from a thorough grammatical check and proofreading. Examples include:
Pg 1 "This dynamics enables GPM..." --> These dynamics enable GPM...
Pg 2 "dependent of the episode" --> dependent on the episode
Pg 5 "Detail is given in Appendix J" --> Details are given in Appendix J

(iii) This study also warrants proper comparisons to modern variational approaches. The empirical evaluations presented in the study only reports metrics from classic approaches. Is there a reason why Table 1 is incomplete? Also, the functional benefits of the proposed approach versus variational methods aren't detailed explicitly. On a more general note, a dedicated literature review will greatly help.

(iv) One of the main claims of the paper is the reformulation of the Bayesian update as finding the least-squares solution to a linear system. This equivalence isn't explicitly proven though. It would be a worthy addition to the main text.

(v) Could the authors include any quantification of "fixed-point" behavior?
summary_of_the_review: This is a neat and well-thought-out idea. However, as I've expressed in the main review, some of the claims need further justification and the paper clarity needs to be improved. I am willing to update my score if the authors are able to provide a convincing response!
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new memory model following the research line of Kanerva Machine, DKM, and Kanerva++. The proposed model reformulates the Bayesian updates of memory and address as finding least-square solution to linear systems. This requires matrix inversion operations and the authors proposed to approximate it iteratively by using Ben-Cohen algorithm for pseudo-inverse matrix. It results in a memory read/write system that is rapid/accurate and can store large batch of data. The evaluation is performed thoroughly on various datasets including binarized MNIST, binarized Omniglot, Fashion MNIST, CIFAR10/100, CelebA. It shows the superiority of the proposed method in negative ELBO of test likelihood, denoising success rate and hamming error, generation, memory capacity, and run-time per iteration. 

main_review: Strength
- The general memory model (e.g., for episodic or working memory) is an important topic in achieving human-like general AI. 
- The paper is well-written and adequately clear (but can be improved).
- The experiment is performed thoroughly. 
- It shows clear advantages to the previous works.
- It also provides theoretic analysis on error bound.

Weakness
- The explanation can be improved for the readers who are not familiar with the previous works of KMs. For example, each line of Algo 1. can more explicitly explained. 
- The meanings of "temporary read-out" and "dynamic" weight were not clear in the beginning.
- In introduction, it says "Importantly, our model ... not only store and retrieve ... but also generate ..." This sounds like a feature of only the proposed model, but the previous works can also do this. 

Minor
- It may be beyond the scope of the paper, but showing the actual benefit of this memory in an RL agent will make the paper more complete.
summary_of_the_review: The paper tackles an important problem and is well and clearly written (but clarity can be improved). The proposed idea is valid and interesting. The experiment clearly shows the superiority of the proposed method in rapid memory read/wright, memory capacity, and de-noising, and generation, etc. The proposed method is also supported by a theoretic analysis on error bound.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a novel deep generative memory model called Generative Pseudo-inverse Memory (GPM) that extends Dynamic Kanerva Machines (DKM, [1]). These models are both deep generative models that maintain a hidden state similar to recurrent neural networks called memory and have ways of writing into and reading from this memory. A key limitation of DKMs is that the time complexity of updating the memory scales cubically in the memory dimensionality and linearly in the episode length. The authors observe that by slightly altering the optimization objective that GPM has to solve, under mild assumptions the memory update can be performed much more efficiently, allowing them to write entire datasets into the memory of the GPM with little computational cost.

The authors perform some experiments on some toy datasets to validate their method and obtain good results against related methods.

[1] Y. Wu, G. Wayne, K. Gregor, and T. Lillicrap. Learning attractor dynamics for
generative memory. NeurIPS 2018
main_review: ## Strengths
   - The proposed method seems to perform much better than competing methods on the toy experiments

## Weaknesses
   - In my opinion, the authors don't provide a clear enough review of the relevant literature. The only section with some detail on related work is section 2.2.1, and connections are only drawn to DKMs and the EM algorithm. Given that GPM is compared against several methods in Table 1, none of which are described in any depth besides KMs and DKMs it would be useful to provide some context for these methods.
   - Some of the comparisons in Table 1 seem a little strange. For example, every method in the "Improved decoders" section outperforms GPM in terms of negative ELBO on CIFAR-10, which is arguably a more relevant (though still toy) problem. Based on this, I would expect that these methods would outperform GPM on CIFAR-100 and CelebA too, however, these numbers were omitted from the table and instead GPM was proclaimed as the best. I think if the authors want to compare against these methods, then either they should include results for the other datasets as well, omit the comparison or not conclude GPM as the best on datasets on which performance metrics for the other methods are unavailable.
   - It is unclear to me what is being reported as negative ELBO in Table 1 for the memory models. Is it the quantity from Eq (1) or Eq (2)?
   - I find the results for VAEs, IWAEs and the "Richer priors" section in Table 1 superfluous, can the authors clarify why these were included?
   - My current understanding is that the "only" difference from DKMs is that GPM optimizes the objective in Eq (4) instead of Eq (3) and uses fast approximations to the pseudo-inverses of matrices involved in the memory update step. Given this, it is unclear to me how the large improvements reported in Table 1 from DKM to GPM can arise. Could the authors comment on this?
   - Apart from Figure 1, every figure needs improvement. Especially in Figure 4, without zooming heavily the axes are completely unreadable. In my opinion, Figure 3 and Figure 4b and 4c are trying to show too much and the result is cluttered. I think it would be more valuable to include fewer rows with larger images.

### Minor items
   - in the paragraph below Eq 1 the sentence contains a mistake, the KL in Eq is from $q(W)$ to $p(W)$ and not from $p(W)$ to $q(W)$
   - I believe that on a few occasions the authors use the word "quantize" to mean "quantify"

summary_of_the_review: I believe that the theoretical contributions of the paper with respect to DKMs is good if perhaps a bit incremental, however, I found the empirical results questionable.
If the authors can address my questions I will consider raising my score.
review_title: null

Meta-review: The authors present a new memory-augmented neural network that is related to the Kanerva machine of Wu et. al.  The reviewers considered the ideas in the paper novel and interesting, but were concerned about presentation issues and literature review.  The authors have improved both... however- authors: please even under limited space constraints, make more room for related work!  Clarifying your contribution in the context of the literature is critical for reader understanding, and neglecting this almost had your paper rejected out of hand.  

I am voting to accept
Decision: Accept (Poster)