Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The model consists of three parts: encoder, decod er, and RFTM, The encoder 
makes a nonlinear transform to the input data x in the latent space. The latent data goes through a re-representation (in the same dimensionality) by the RFTM. Then the data is reconstructed by the decoder. The anomaly of the test sample is measured by reconstruction error. In particular, the RFTM  uses limited prototypes to record and re-represent the latent representation, which makes suppression of overgeneralization feasible, and can be trained end-to-end without introducing any additional penalty terms on top of the original task loss function. The rest of the paper is used to show the functioning of the RFTM module with some comparative results and a lot of ablation studies.

strength_and_weaknesses: Strenghts: 
Unsupervised anomaly detection is a very interesting topic, more insidious than outlier detection.


Weaknesses:
The comparative results are obviously insufficient, since only Vanilla AE, MemAE and MNAD were used as comparison models. Instead, Trust-MAE, DAAD, [Ruff et al., 2021] should be considered (and already mentioned by the authors in the state of the art). Other approaches are

-- Deep Unsupervised Image Anomaly Detection: An Information Theoretic Framework 2020
-- FastFlow: Unsupervised Anomaly Detection and Localization via 2D Normalizing Flows
-- Self-Supervised Predictive Convolutional Attentive Block for Anomaly Detection
--Robust Subspace Recovery Layer for Unsupervised Anomaly Detection  2019
These approaches should also be present and discussed in the state of the art

Second, the study of figure 7 is not clear at all. The space constructed by the autoencoder and the two types of memory of the RFTM should have cavities (autoencoder) and less cavities (the second type of memory of the RFTM in particular). This is somewhat visible in the figures b and c. What is not clear is why the model needs to have a representation space that is dense enough that the anomalies are always forced to be embedded in the distribution of normal representations. Isn’t this a way to hide anomalies instead of highlighting them?

clarity,_quality,_novelty_and_reproducibility: Clarity: not clear why density would be needed in the space to avoid overgeneralization. Not clear the figures of Fig.7, and how they demonstrate the goodness of the proposed approach. 

Quality: Results are obtained wrt a limited number of state of the art approaches.

Novelty: the approach seems novel, but I cannot judge how much powerful given the limited comparisons

Originality: Cortex and hippocampus parallelism with the medicine should be more stressed since they are very attractive as justification of the approach.
summary_of_the_review: - state of the art is insufficien, and I have provided some of the SOTA approaches for UAD
- qualitative results in FIg.7 are not clear
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The work presents a memory network-based autoencoder method for unsupervised anomaly detection, in which random forgetting gates and top-k prototype selection functions are used to regularize the autoencoder networks to avoid overfitting. 
strength_and_weaknesses: Strengths.
- The studied problem - having small reconstruction errors in reconstructing anomalous samples - is a common problem in autoencoder-based AD approaches
- In the presented method, the key idea of enforcing less prototypes in the memory to alleviate the problem is plausible
- A set of empirical results on three commonly used datasets is used to justify the effectiveness of the proposed method. 

Weaknesses.
- Although the key idea is plausible, it shares similar insights to the previous memory network-based autoencoder approaches for anomaly detection, such as MNAD. The main objective there is to learn less yet compact prototypes to avoid the overfitting of the data. Although the specific way to achieve this objective is different from each other, the new prototype learning in this work does not make clear major contributions. The proposed method seems to simplify the memory learning, but its effectiveness is not clear (see comments below).
- The presented results are not convincing. 1) The reported results of the competing methods in the paper are significantly worse than the ones in the their original papers and some recent relevant papers [A,B]. 2) The performance of the presented method is far below that of the current SOTA models (e.g., see the results on the three datasets in [A,B]).
- There are a number of false/misleading claims, such as what does it mean by unlabeled normal samples in "Unsupervised anomaly detection (UAD) only needs to fit the unlabeled normal samples to learn the normal patterns" (why is it unlabeled if the samples are known to be normal), and "most of the experiments were not conducted based on the USAD setup" when reviewing memory network-based AD methods.
- It is unclear why the learned memory can be named as Hippocampus and Cortex memory.
- It claims that the method can work as a plug-and-play component, but not empirical results are given to support this claim.

References.
- A. "Anomaly Detection via Reverse Distillation from One-Class Embedding." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9737-9746. 2022.

- B.  "Deep one-class classification via interpolated gaussian descriptor." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 1, pp. 383-392. 2022.
clarity,_quality,_novelty_and_reproducibility: The presentation quality is below the average of ICLR given the issues identified above. The originality of the paper is also weak.
summary_of_the_review: The paper has major issues in main claims, empirical justification and clarity, and its technical novelty is marginal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper mainly explores the issue of “overgeneralization” in unsupervised anomaly detection and proposed a bio-inspired solution, termed “Random Forgetting Twin Memory (RFTM).” The new model did not change the fundamental structure of existing learning models, e.g., autoencoder, and work in a plug-and-play fashion. It achieves state-of-the-art performance on several public benchmarks.
strength_and_weaknesses: Strength:
* The new method is promising compared to other memory mechanism-based anomaly detection methods under the unsupervised semantic anomaly detection (USAD) setting.

Weakness:
* The major issue is about the experiment design that only USAD setting is considered. While this setting is challenging, other routine settings would be considered as well. This allows the new methods to be evaluated on a variety of data such as image/video datasets.
* How about the performance of the new method compared to non-reconstruction methods, e.g., [1], which also provides appealing performance under the USAD setting? 
* While it is fair to use the same encoder/decoder in the experiments, readers may be more interested in other methods, e.g., DAAD mentioned in the “introduction.”

[1] Jihoon ,et al; CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances

clarity,_quality,_novelty_and_reproducibility: The paper is well written. The provided ablation study, illustration, and insights are helpful, too.

The focus of this paper is to design an improved memory-based and bio-inspired learning module. There are similar methods in this lineup already, although the new one improves the performance on USDA benchmarks.

The paper may be reproducible as implementation and experiment details are both provided. 

summary_of_the_review: In brief, the new method developed an interesting bio-inspired RFTM model under the USAD setting to solve the issue of “overgeneralization.” The model details are well-elaborated. However, the experiments are inadequate as only the USAD setting is considered. Other routine settings and non-reconstruction methods should be considered and compared in a more comprehensive way.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper tackles anomaly detection using a memory-based autoencoder approach. The main problem with reconstruction-based AD is that both normal and anomalous examples are well reconstructed making the anomaly score weak. Memory-based AE approaches have not addressed this problem sufficiently. This paper proposed a fairly complex memory mechanism which claims to overcome those issues. It draws inspiration from human cognition (although the precise analogy is not made very clear). The method is compared to a handful of memory-based AE approaches on simple dataset, and claims to obtain better results. 
strength_and_weaknesses: The task of anomaly detection is obviously important. Reconstruction-based methods have been around for a long time, but getting them to SoTA performance would be very interesting. 

I am not positive towards this paper. It makes very strong claims but does not deliver. 
* It claims "We experimentally demonstrated the human brain’s similarity between RFTM and hippocampus-cortex structures". This is a really strong claims and I do not think there is sufficient evidence to support it.
* It claims "state-of-the-art experimental results" but in fact the experiments show it's ROCAUC on Cifar10 is around 50% (random performance). The comparison was only made against memory-based AE methods which are far from SoTA. Comparison against CSI (Tack et al.) or PANDA (Reiss et al.) would not support the above claims.

Furthermore, the approach is probably too complex for its level of performance. The design choice is not justified by a sufficient ablation study making it unclear which aspects are in fact helpful.
Finally, the complexity of the method made it unclear to me. With difficulty I think I managed to understand it, but it still seems arbitrary to me.
clarity,_quality,_novelty_and_reproducibility: Clarity: The language is clear, but the complexity of the method made it unclear to me although, with difficulty, I think I managed to understand it.

Quality: as mentioned above the key claims are unsupported, ablations and state-of-the-art comparisons are missing. I do not consider the quality as high.

Novelty: While I do not think this mechanism has been proposed before, I did not see new ideas which I could clearly "take home" from this paper.

Reproducibility: The authors claim this can be implemented using a single line of code, however it would have been helpful to see the code.
summary_of_the_review: The paper tackles an important problems but the method is not well motivated, unclear. The experimental results are also unsatisfactory.  I recommend rejection.

#######################################################

I have read the the two responses by the authors but would like to keep my rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper deals with unsupervised anomaly detection by proposing an autoencoder variant that is inspired by the cascade structure in the human hippocampus. The proposed autoencoder, which can be cast within a recent line of research about memory-augmented neural autoencoders, has been evaluated in a number of unsupervised anomaly detection tasks on simple image datasets (MNIST, FashionMNIST, CIFAR10).

The reviewers appreciated the potential direction of the work but highlighted a number of weaknesses in terms of presentation and experimental results. First, the presentation of the contribution suffers from over-claiming results in terms of being better than competitors and its biological plausibility. Second, this is linked to the limited experimental setting which (despite a large grid search of hyperparameters), focuses on simple datasets and provides as a justification of goodness and biological plausibility only accuracy metrics in a reconstruction task. Third, some notable competitors are missing (see reviewers' comments). 
The authors' rebuttal did not manage to convince the reviewers as the questions about experimental settings and claims did not receive satisfying answers nor the paper presentation was revised accordingly.

The paper is therefore rejected.

We encourage authors to revise the current version of the manuscript by i) refactoring presentation as to tune-down the biologically-inspired claims and terminology, ii) revising presentation  and ii) extending the experimental comparison as to include ii.a) the same setting as previously introduced memory-augmented autoencoders, ii.b) extensive ablation tests on all experimental settings and iii.c) a controlled (and even synthetic) experimental setting where to validate the claims about the need for the memory augmentation. 
justification_for_why_not_higher_score: It is clear the paper is not yet ready for publication. See discussion above.
Decision: Reject