Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method that models the visual difference between images as a topological transformation for images. Using this model and computing the topological distance by minimizing the distortions between imagers, one can find neighbors that are conceptually similar to the input image. Evaluation results show that this simple method can achieve strong performance when only very few samples are available and no pretraining is allowed.
main_review: Pros:
1. The proposed model is interesting and neat.
2. The empirical results look promising. With very limited data and no pretraining, the proposed method archives significant results on MNIST, EMNIST and Omniglot.

Cons:
1. The benchmark seems too toy-size. Even the most challenging Omniglot dataset has a clean background and somewhat simple visual appearance. In this sense, I am not convinced whether the method can still work well when the image becomes more complex. When more complicated distortion exists (e.g.), is the model still able to capture it through the transformation? I would expect at least some more complicated benchmark datasets to be evaluated, for example, CIFAR10/100.
2. When the task is more challenging (e.g. EMNIST), the proposed method is caught up quickly by the TextCaps. On more complex datasets, modern deep learning models may catch up earlier, and with acceptable data size, even reach much better results that the proposed method cannot obtain.
3. I am not very convinced about the setting. It has been studied that good pretraining can benefit the downstream tasks even in a few-shot setting [R1]. So, I am not sure when the proposed method can be applied. Can authors give me some examples in real world about training a classifier with very few training samples while no additional pretraining data is available?
3. The paper lacks a conclusion.

Reference:

R1: Kolesnikov, Alexander, et al. "Big transfer (bit): General visual representation learning." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer International Publishing, 2020.
summary_of_the_review: This paper is interesting and has some promising results. But I have concern about its application in the real world and its robustness in more complex scenarios, which I hope the authors can address. Therefore, I incline to reject this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: **Summary**

The authors propose a white-box model that uses a similarity score in a space that is constructed based on topological transformation. In other words, two data points are close to each other if you can transform them into each other. To achieve this, they first compute the topological distance with training data points while minimizing the distortions (this part was a little bit unclear to me. I would appreciate it if the authors provide more context or examples of this). They use an interesting idea of using gradient descent to find a sequence of transformation which is the main reason that contributes to having a white-box model. I found this part of the paper very interesting and novel. Finally, it is not possible to apply this method at the pixel level. So they propose using a chain of lattices to be able to apply their technique in higher abstractions. 



main_review: **Strong points**: The authors connect ideas in a beautiful way. I liked how they connected ideas from how the human brain works to train their model. Well-structured writing. The idea seems novel and interesting. We optimize steps to generate the target image from our trainset and then pick the one that is closest. I really liked the idea. I also liked making the model work as a white box by using their method. Also finding ways to apply their method in higher levels.

**Weak points**: Biggest drawback is lacking experiments on more challenging datasets, and unfortunately it is very important for me to see the results compared at least on very well-known datasets for few-shot learning.

**Review**
Currently, I gave a score of 5 to the paper.
The main reason for my review is that there are other methods that leverage other data in the domain like (model-agnostic meta-learning[1]) and tested on a few samples on the classes that are never seen during the meta-learning phase. I understand the value of figuring out a way to remove this requirement. For example, unsupervised meta-learning (CACTUs[2]) achieves very good results without needing labelled data. It seems to me that this method has replaced the need for data with a good amount of domain knowledge (We know that characters could convert to each other by transformations, but what about cats and dogs? Another example is identity recognition from faces. If we have two images of two different people. In one picture the person laughs and in the other one, the person is wearing sunglasses. Which two are getting closer to each other?) For example, in figure 1, we know that we can apply these transformations from "7" to "1". However, the experiments did not convince me that this is always true. One interesting experiment that could be added is to apply this method to CelebA[2] face recognition dataset. Another baseline could be to apply this method to the Mini-Imagenet[1] dataset. These are well-known datasets in few-shot learning and studied alongside Omniglot. Results on these could convince the reader about the effectiveness of the method. 

**Questions**

In section 2, under "Representative digital and smoothed images" it is mentioned that "This differs from
Gaussian blurring as we do not discretize kernels". Can you specify what you mean? In practice, we must somehow use a discretize kernel on the computer. Can you describe the process in more detail?‌ Do you cut the kernel at some point?

**Other comments to improve the paper**


Related work does not seem comprehensive to me. Only a few papers are cited and some of the statements are kind of misleading. For example, it is true that meta-learning needs data for a pre "meta-learn" step, but there is a nuanced difference. The meta-learning step is applied to different classes or tasks and then evaluated on a completely new task. As a result, I would try to expand this section.

[1] Finn C, Abbeel P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning 2017

[2] Hsu K, Levine S, Finn C. Unsupervised Learning via Meta-Learning. In International Conference on Learning Representations 2018


summary_of_the_review: As a result, even though I liked the ideas in the paper a lot, I am going to vote for rejection unless I can see the experiments added to the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper on hand tackles the problem of learning from very few samples, which is of high relevance for many machine learning problems. To this end, an approach to model transformation-based topological similarity is introduced, allowing for covering many kinds of invariants in image data. The approach is demonstrated for well-known benchmark datasets for different classification models, demonstrating that in this way just using a small number of samples competitive classification results can be obtained.
main_review: The paper on hand tackles the problem of data variety in a different way. As  the typical approach nowadays would be to use some kind of data augmentation, here vice-versa some kind of transformation are proposed to provide meaningful similarities. The overall idea is not new, however, the specific way how this is realized is. The overall approach make sense and is backed up by a clear mathematical definition. From this point of view the paper would be a valuable scientific contribution. On the downside, however, there are flaws in the description and interpretation of the results. Moreover, it is unclear if the proposed approach would also work for more realistic datasets: MNIST and EMNIST might be to simple. On the other hand, the Omniglot challenge seems to be a too complex tasks, that no existing approach can solve not even rudimentary. So using this dataset was probably not a good choice?


Further Comments:

(1)  Using partially blue text is hampering the reading flow.

(2) When describing the datasets in Sec.4 the correct abbreviation (given the real numbers) should be 60k and not 60K.

(3) To make the experimental setup more clear, all methods that are used for training should be mentioned explicitly.

(4) For readability, it might be better to split Fig.4 into two figures, one for each benchmark,

(5) Even though the results presented in Fig.4 are clear, a better description of the experimental setup and some kind of discussion would be necessarily required.

(6) For better understanding, it might be meaningful also to show examples for MNIST and EMNIST.

(7) To allow for a fair evaluation and to give better insights, it might be meaningful to use the same learning approaches for all datasets.

(8) The ocker background in Fig.6 does not look nice and removes the focus from the main content of the figure.

(9) The archetype generation does not fully fit to the rest of the paper and is not described and evaluated sufficiently. In this way, it would be better to describe the other experiments more detailed and to shift this part to a separate publication.

(10) In general, the paper is missing a conclusion and an interpretation of the results.

(11) The discussion on related work is rather short, which can also be seen from the very short bibliography. A more thorough discussion would be necessary.

(12) The bibliography needs to be seriously checked for consistency and correctness.
summary_of_the_review: Overall, the paper comes up with an interesting approach to learn from very few samples. The approach is plausible, backed up by theory and seems to work for simple tasks. However, it seems to be questionable if this would also work for more realistic datasets. Moreover, the experimental section would need to be overworked. In this way, the paper would be an interesting contribution, but there are still a few open points hampering a publication as it is.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel white-box model for one or few-shot learning, which tries to simulate the human recognition ability for “distort” objects. The authors use transformation-based topological similarity to build the model and propose an anchor system with gradient descent to train their models. Extensive experiments on standard character recognition benchmarks demonstrate the proposed method outperforms the classical machine learning methods with very limited training data, such as less than 20.
main_review: Strengths
1.	This approach is a white-box model and human-interpretable. 
2.	This paper is well written and easy to understand.
3.	The major idea is novel and interesting. This is a good attempt to explore a solution for the few-shot learning without pretraining or with very limited training data. Since most existing FSL models heavily rely on a large number of training data.

Weaknesses

1.	The proposed approach is based on topological similarity. It seems that it only is suitable for images with simple topological structure, such as the character images. Maybe it is hardly used to classifier complex nature images since we need more information for natural image classification, not only use topological structure. 
2.	The authors did not provide the experimental comparison with enough training data, such as the whole training set in MNIST. The reviewer wonders about the upper performance of this approach with enough data.
3.	The future applications of this approach may be limited, especially for the real-world tasks which require learning from a few labeled data, such as rare accident data in auto-drive and disease data for medical diagnose.

summary_of_the_review: The main idea is novel and interesting, and the experimental results on limited training data demonstrate the superior performance of the proposed method. However, it may be not easy to apply this approach to other more complex tasks. In summary, the reviewer thinks this paper is marginally above the acceptance threshold.
review_title: null

Meta-review: Meta Review of Learning from One and Only One Shot

The motivation of this work is to address the problem of learning from very few samples, which is of high relevance for many machine learning problems. The paper proposes an (interpretable) approach for one or few-shot learning, which tries to simulate the human recognition ability for “distort” objects. To achieve few-shot learning, they first model the topological distance with training data points while minimizing the distortions, to find neighbors that are conceptually similar to the input image. Their experimental results show that this simple method can achieve good performance when only very few samples are available and no pre-training is allowed.

All reviewers, including myself, agree that this paper is well motivated, nicely written, and appreciated how they connected ideas from neuro-psychology to their ML model, and the novelty is recognized. But there are issues raised by reviewers with the paper that prevent it from meeting the bar for me to recommend it for acceptance at ICLR 2022.

The main issues raised by all reviewers is that the proposed method is only experimentally verified on simple datasets such as MNIST, EMNIST and Omniglot (and to some extent, Quick, Draw!). The authors (to their credit) in the rebuttal noted that the narrative of the paper is to focus on abstract images, and the purpose is more from a scientific investigation perspective (rather than proposing an algorithm that is immediately useful for ML practitioners), and this is a fair point. However, I do believe the issue here is beyond the simple criticism of "it works on MNIST, how about ImageNet?" as I think some reviewers genuinely think there are fundamental aspects of the approach that might prevent it from scaling (as planned in future work, even by the authors in the last section). For instance, as gUCx noted:

1) The proposed approach is based on topological similarity. It seems that it only is suitable for images with simple topological structure, such as the character images. Maybe it is hardly used to classifier complex nature images since we need more information for natural image classification, not only use topological structure.

2) The authors did not provide the experimental comparison with enough training data, such as the whole training set in MNIST. The reviewer wonders about the upper performance of this approach with enough data.

I tend to agree with these points. Non-topological similarity can be displayed in abstract images / datasets. Even in "abstract" images, the paper should describe limitations of the approach, and whether it breaks down (like in the "abstract" Quick, Draw! dataset, there are different types of distinct "yoga" poses in the yoga class. And likewise, in the cat or pig class, there are animals with only the heads, and animals with the head and the full body). Conveniently, Quick, Draw! had not been used in any of their classification experiments [1], and only for a simple clustering example.

And for the other points, reporting the terminal MNIST performance will be useful, even if it doesn't look good, so the readers have an idea of the limitations of the approach, where it is good, where it is not, and what needs to be improved. I would love to see improvements (either in the writing or in the experiments) in future work where the paper can effectively convince the readers that the direction has the promise of being able to scale to "real" or "complex" images. (Perhaps even performing the approach on the output of a pre-trained self-supervised autoencoder on ImageNet, as a method to get "abstract" versions of real photos, like a parallel of the giraffe experiment, though this may distract from the narrative of no-pre-training). All in all, I don't want to discourage the authors as we are all excited about the direction of this work. I hope to see an updated version of this work published in a future venue, good luck!

[1] https://www.kaggle.com/c/quickdraw-doodle-recognition
Decision: Reject