Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new machine learning method for classification called Fuzzy Learning Machine. The paper draws from concepts from cognitive science to derive a method based on fuzzy similarity relations of examples on the input space. The training method learns a similarity function and selects a set of exemplars from each category used during the prediction phase to compute the similarity of new examples to the exemplars in each category and then assign it to the category with more similar examples.
strengths_and_weaknesses: Strengths:

The method proposed is interesting and brings up a number of novelty elements. The method seems to improve significantly in relation to existing classification methods on a large number of data sets.

Weaknesses:

The paper makes a lot of assertions about human cognition that are questionable. For instance:
- "In essence, the process of classification is the process of concept cognition"
- "Concept contains our knowledge about the world, and we use concept to understand and organize the world. Without it, there will be no human intelligence at all."
- "Similarity (...) plays a crucial role in the process of human classification"
- "Concept is represented based on similarity for children, which is also a basic choice for adults"

Also, sometimes it is difficult to understand if the paper makes assertions about its own definitions or about human cognition as in "the intrinsic property of concept is just the fuzziness rather than the randomness".

I do not see a problem in using assumptions based on cognitive science for building models. In fact, most models in AI do that somehow. However, care should be taken to not state these assumptions in the paper as settled truths. I rather see the paper provide in advance a list of theories, hypotheses, and assumptions considered along with references for them, and then describe the model proposed using them as a basis.

Finally, without details on how well the other methods used for comparison were adjusted, it is hard to know if the comparison is fair.


questions: Below I provide a list of questions and points that could be improved in my opinion:
1. Line 16: The term "Concept cognition" lacks reference and support in the literature. If this is a term coined in the paper, then first, it should be defined what is meant by "Concept" and "Cognition" in the views of the authors, since these terms are rather vague.
2. Line 18: Why classification and not categorization? The former may be rather too specific and we do not know enough about human intelligence to pinpoint it as the approach used by it. For instance, see Jacob, Elin K.. “Classification and Categorization: A Difference that Makes a Difference.” Libr. Trends 52 (2004): 515-540. More specifically: do humans need supervision to learn?
3. Line 28: It should be made clear that DNN may exceed the human level in very specific settings.
4. Lines 29-30 and 33: The way humans do classifications is also difficult to understand and explain. Therefore, I see a conflict in pursuing human-like classifiers and classifiers that are easy to understand and interpretable at the same time.
5. Line 30: I do not see how this can be direct and efficient because we first need to understand and explain how humans do classification and, although there exist theories, this is not settled in the literature.
6. Figure 1: Please, describe where the numbers used in the legend are derived from.
7. Line 74: Test data also need labels. Otherwise, the method can not be evaluated.
8. Definition 1: Observe that "phi(x) belongs to Y" does not define any mapping in particular. Is that the intended definition? If so, it does not resemble a classification problem, and any arbitrary mapping would be a solution.
9. Line 107: I do not see why an awkward situation arises from the definitions above.
10. Equation 1: no regularization terms are considered. Is overfitting a problem?
11. I see potential problems in the way the model chooses examples for the Ec set. Consider a class with high variability, such as "vehicles". In such a class, it is hard to select k exemplars that cover well the variability of the class (cars, trucks, planes, boats) and yet are similar among all of them.
12. Line 269: "On the contrary, the class labels predicted by NN-FLM are more in line with human cognition." it is hard to conclude this from the examples shown without adequate research with different subjects and in comparison with other methods.
13. Please, provide a description of the hyperparameter tuning method used for the methods. How well were they adjusted?

Typos and writing style:
1. Line 23: possess instead of process?
2. Line 26: humans instead of human?
3. Line 42: "A concept is" or "Concepts are" instead of "Concept is" ? 
4. Line 75: The question mark is not needed because the phrase is not a question.
5. Definition 1: In the notation used (X,Y, phi), phi seems rather an input than output or result. Consider something like phi = f(X,Y). (optional)
6. Line 89: why double parenthesis in ((xi, xj))?
7. Line 101: Concepts instead of Concept?
8. Line 141: "exemplar theory is about" instead of "exemplar theory is a kind of theory about".
9. Line 269: humans instead of human.

limitations: I do not see any limitations or potential negative social impact of this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In the paper "Fuzzy Learning Machine" the authors propose an approach to learn a classifier via a neural network forming a fuzzy equivalence relation. Deriving the approach from fuzzy set theory, the authors find their approach to perform particularly well across a number of datasets comparing the approach to various other classifiers.

strengths_and_weaknesses: ## Weaknesses
The idea of employing fuzzy set theory for classification tasks is not new at all and I am wondering what is now the methodological novelty of the approach. In general the idea of comparing instances / data points according to their similarity is the basic idea behind learners using kernel functions where the shape of a concept is specified via the respective kernel. However, there is also a relatively large corpus of literature on classifiers leveraging fuzzy set theory, even working exactly with neural networks and the idea of fuzzy equivalence relations. Still this related work is neither discussed nor cited in the paper. See for example the following references:
Acharya, U. Rajendra, et al. "Classification of heart rate data using artificial neural network and fuzzy equivalence relation." Pattern recognition 36.1 (2003): 61-68.
Moser, Bernhard. "On Representing and Generating Kernels by Fuzzy Equivalence Relations." Journal of machine learning research 7.12 (2006).
Meier, Andreas, and Nicolas Werro. "A fuzzy classification model for online customers." Informatica 31.2 (2007).
Senge, Robin, and Eyke Hüllermeier. "Top-down induction of fuzzy pattern trees." IEEE Transactions on Fuzzy Systems 19.2 (2010): 241-252.
Kuncheva, Ludmila. Fuzzy classifier design. Vol. 49. Springer Science & Business Media, 2000.
Sun, C-T., and J-S. Jang. "A neuro-fuzzy classifier and its applications." [Proceedings 1993] Second IEEE International Conference on Fuzzy Systems. IEEE, 1993.
Uebele, Volkmar, Shigeo Abe, and Ming-Shong Lan. "A neural-network-based fuzzy classifier." IEEE Transactions on Systems, Man, and Cybernetics 25.2 (1995): 353-361.
It is unclear to me how this part of the literature is widely ignored by the authors when they seem to come from that area.

Overall, the paper has a good structure but could benefit from proofreading. Especially, a vs an is a frequent problem in the text, e.g., "a input space", "a output space", "a FER". Then, "classifier", "concept" and "classification process"  are used without an article. Some parts also seem overly complicated to me. For example, consider the proof that a non-linear model is needed to tackle the derived problem where the instances are concatenated. I do not know whether yet another proof for the fact that an XOR problem cannot be tackled via a linear model is really needed. This could have been simplified. Furthermore, I find that the example given in Figure 1 is not very well chosen. The concepts cat and dog have crisp biological borders and a human not being able to distinguishing the two categories is rather due to epistemic uncertainty than fuzziness of the concept borders. Personally, I would also argue that non of the three cats is more or less representative of the category or concept "cat".

A claim that was made by the authors is that their approach indeed learns "concepts" instead of just assignments. However, there was no proof given in the paper that this is really the case. Especially, there is no presentation or demonstration of any particular concepts that were induced by fitting their model. I would even argue that from Figure 3 is rather becomes clear that it is learning not really any concepts as the FSR matrix shows more or less the same color for every cell not being on the main diagonal. If it was to learn real concepts I would also expect that a 0 would receive a lower membership score for the concept 1 than a 7 for example. A better overall performance is no proof for the claim that the method learns concepts.

Another branch of classification literature also tries to capture concepts for classification purposes: Analogy learning.
Bayoudh, Sabri, Laurent Miclet, and Arnaud Delhay. "Learning by Analogy: A Classification Rule for Binary and Nominal Data." IJCAI. 2007.

## Strengths
Since most people in the machine learning community will not be that much familiar with fuzzy set theory, I liked it very much that all fundamental definitions were provided by the authors in the paper or supplementary material to make it self sufficient.

According to the experiments the proposed method seems to perform very strong compared to a set of almost 200 classifiers. However, the way how the rankings were calculated is a little bit odd. Why are 65 learners sharing rank 1 with 100% accuracy receive a rank of 65? This will most likely also affect the average rank statistics compared for the ten classifiers later on. I would rather expect that performances with a tie receive the same higher rank, leaving free the next n-1 spots in the ranking.

questions: * In what regard does the proposed method really go beyond the already existing corpus of literature in fuzzy classifiers?
* Is the classification problem really solved with definition 2? To me, it seems like the information to which class at least a representative data point belongs to is missing. Hence, the problem is only solved if the y's of some representative class members are known.
* Is not it a problem that the model is not permutation invariant? At least it seems to me that it is not invariant to the order of inputs.

limitations: Limitations, except for runtime complexity to compute the FER matrix, are not really discussed. When does the approach fail and why does it fail?
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new learning machine for the general classification problem, which is one of the most important problems in ML/AI. The new learning machine is based on the concept cognition theory in cognitive science and fuzzy set theory in mathematics science. So its working mechanism is highly explainable and has a solid theoretical guarantee. Meanwhile, a large number of systematic experimental results demonstrate the superiority of the proposed method.
strengths_and_weaknesses: The manuscript focuses on the classification problem which is one of the most important problems in ML/AI.

The manuscript re-examines the classification from the perspective of concept cognition and reveals the essence of classification. And the manuscript provides a new view to interpret the structure of the classification problem by establishing the equivalence between binary classification problem and classification problem by employing equivalence relation in set theory. Furthermore, the manuscript realizes that fuzziness of concept is the main source of uncertainty in classification and then employs the fuzzy set theory to model this kind of uncertainty.

Based on the above conclusions, the classification problem is modeled as a fuzzy equivalence relation problem, which well preserves the nature and intrinsic fuzziness of the classification problem. What’s more, the manuscript designs a clever model and loss function to approximate the fuzzy equivalence relation effectively and efficiently.

Therefore, in this manuscript, the main proposals have the theoretical basis of cognitive science, and the key conclusions are proved mathematically. And extensive experiments (compared with 179 methods on 121 data sets) verify the rationality and superiority of the proposed method.

Overall, the manuscript is clearly written and well organized with good clarity. To enhance the readability and completeness, it is suggested that some contents in the appendix should be moved to the corresponding part of the main manuscript. For example, the analysis of the working mechanism of the existing classifiers should be moved to the Introduction of the main manuscript. However, in the current manuscript, these contents are placed in Appendix A.2.

questions: (1)	The title of the manuscript is FLM, but the performance of NN-FLM is shown in experiments. Please explain the difference between FLM and NN-FLM and why only the performance of NN-FLM is analyzed.

(2)	It appears that the optimization model of NN-FLM is a general multi-layer neural network plus a special loss function. Please explain what good properties of the loss function are able to guarantee the performance of NN-FLM.

(3)	According to the descriptions in Appendix A.1, there are at least four different concept representation theories. Why choose the exemplar theory to complete the concept representation in FLM? And in some machine learning methods, ‘prototype’ is also usually used for representing a class, such as ‘means’ in ‘k-means’, ‘modes’ in ‘k-modes’, and ‘prototype ’ in ‘prototype network’. Please explain the difference between ‘prototype’ in these methods and ‘prototype’ in ‘theory theory’ and why not use prototype theory to represent the concept?

limitations: N/A
review_title: null

Meta-review: metareview: The paper proposes an approach for the design of neural networks for classification based on fuzzy theory, and a specific implementation is presented and experimentally assessed. Arguments from cognition to justify the proposed approach are also used, although at the level of inspiration. The lack of reference to fuzzy systems based neural networks models in the relevant literature in the initial version of the paper has been solved in the revised version, and author's rebuttal seems to have clarified most of the issues raised by reviewers. The experimental assessment seems to be robust. Personally I find the jargon used in the paper a bit unfit for NeurIPS standards, however I do not think this should be a valid reason for rejecting a paper for which no serious drawback has emerged. In any case, I think it is good for NeurIPS to diversify the range of approaches and methodologies covered by the scientific program.
confidence: Less certain
Decision: Accept