Title: Classless Association using Neural Networks
Authors: Federico Raue
Sebastian Palacio
Andreas Dengel
Marcus Liwicki
Venue: ICLR 2017

Abstract: The goal of this paper is to train a model based on the relation between two instances that represent the same unknown class.  This scenario is inspired by the Symbol Grounding Problem and the association learning in infants.  We propose a novel model called Classless Association.  It has two parallel Multilayer Perceptrons (MLP) that uses one network as a target of the other network, and vice versa.  In addition, the presented model is trained based on an EM-approach, in which the output vectors are matched against a statistical distribution.  We generate four classless datasets based on MNIST, where the input is two different instances of the same digit.  In addition,  the digits have a uniform distribution.  Furthermore, our classless association model is evaluated against two scenarios: totally supervised and totally unsupervised.  In the first scenario, our model reaches a good performance in terms of accuracy and the classless constraint.  In the second scenario, our model reaches better results against two clustering algorithms.


TL;DR: Learning based on the relation between two instances of the same unknown class

Review discussion:
Interesting paper, but still i do not understand how to define the  "target distribution (E[z1 , . . . , zm ] ∼ φ ∈ Rc )" why it should be "uniform", should we know the prior of the different classes?
Thank you very much for your question.
Our model relies on the distribution of the classes  i.e., for a given dataset, the distribution should be known in advance.
For the paper, we made experiments assuming a uniform distribution.
However, it is perfectly possible to work with a different distribution (e.g., Gaussian, Zipf).
Choosing another distribution affects the way we define \phi (and should be adjusted accordingly).
Furthermore, we are thinking (as part of the future work) on different strategies where even the prior distribution is unknown.
The paper looks correct but still i am not convinced about the experimentation performed. Perhaps another experiment with more challenging data would be welcome. Honestly i don't find a clear motivation for this work however it could have some potential and it would be interested to be presented in conference.
We thank the anonymous reviewer for their comments and time.  As a general remark, we have improved the clarity of the paper, mainly in the motivation behind the model.
In addition, we have extended the experimental setup with three more classless datasets based on MNIST (Rotated-90 MNIST, Inverted MNIST, and Random Rotation MNIST).  Our findings still hold for these datasets where our model reaches better results than clustering algorithms and promising results in relation with the supervised scenario.
Finally, we have added a few extra examples as supplemental material.
Remark:  The previous version of the paper had a mistyping error about the MLP parameters.  We have reported the results of a model that has 200 neurons for each layer instead of two fully connected layers of 200 and 100 neurons, respectively.  We have already fixed this problem in the next version of our paper.  Note that the performance of both architectures is quite similar
* Thank you for pointing out that the motivation of the model is not clear.  We have improved the clarity of the paper.   Our model is motivated by the Symbol Grounding Problem and infants learning, mainly the binding between abstract concepts and the real world via sensory input signals and the association between the sensory streams via abstract concept.
Thus, an alternative training rule is required where the classes are unknown.  In this paper, we use a simple statistical constraint for learning the association between two streams of information.
The paper explores a new technique for classless association, a milder unsupervised learning where we do not know the class labels exactly, but we have a prior about the examples that belong to the same class. Authors proposed a two stream architecture with two neural networks, as streams process examples from the same class simultaneously. Both streams rely on the target (pseudo classes or cluster indices) of each other, and the outputs an intermediate representation z, which is forced to match with a statistical distribution (uniform in their case).
The model is trained with EM where the E step obtains the current statistical distribution given output vectors z, and M step updates the weights of the architecture given z and pseudo-classes. Experimental results on re-organized MNIST exhibits better performance compared to classical clustering algorithms (in terms of association accuracy and purity). The authors further provide comparison against a supervised method, where proposed architecture expectedly performs worse but with promising results.
The basic motivation of the architecture apparently relies on unlabeled data and agreement of the same pseudo-labels generated by two streams.
But the paper is hard to follow and the motivation for the proposed architecture itself, is hidden in details. What is trying to be achieved by matching distributions and using the pseudo-targets of the each other? Perhaps the statistical distribution of the classes is assumed to be uniform but how will it extend to other priors, or even the case where we do not assume that we know the prior? The current setup needs justifications.
What would be very interesting is to see two examples having the same class but one from MNIST, the other from Rotated-MNIST or Background-MNIST. Because it is hard to guess how different the examples in two streams.
At the end, I feel like the authors have found a very interesting approach for classless association which can be extended to lots of many-to-one problems. This is a good catch. I would like to see the idea in the future with some extensive experiments on large scale datasets and tasks.
But the current version lacks the theoretical motivations and convincing experiments. I would definitely recommend this paper to be presented in ICLR workshop.
Few more points:
Typo: Figure1.
second line in the caption "that" -> "than"
Necessity of Equation 2 is not clear
Batch size M is enormous compared to classical models, there is no explanation for this
Why uniform? should be clarified (of course it is the simplest prior to pick but just a few words about it would be good for completeness)
Typo: Page 6, second paragraph line 3: "that" -> "than"
We thank the anonymous reviewer for reading the paper and providing valuable feedback.  As a general remark, we have improved the clarity of the paper, mainly in the motivation behind the model.
In addition, we have extended the experimental setup with three more classless datasets based on MNIST (Rotated-90 MNIST, Inverted MNIST, and Random Rotation MNIST).  Our findings still hold for these datasets where our model reaches better results than clustering algorithms and promising results in relation with the supervised scenario.
Finally, we have added a few extra examples as supplemental material.
Remark:  The previous version of the paper had a mistyping error about the MLP parameters.  We have reported the results of a model that has 200 neurons for each layer instead of two fully connected layers of 200 and 100 neurons, respectively.  We have already fixed this problem in the next version of our paper.  Note that the performance of both architectures is quite similar
* We agree that the motivation of the model is not totally clear, and it is fixed in the next version of our paper.  Our constraint is inspired by the Symbol Grounding Problem and the association learning in infants.  The first part of our constraint is to learn without labeled data.
Thus, we have used the statistical distribution as an alternative mechanism for training MLPs.  The second part of our constraint forces us that different instances of the same unknown class must be classified with the same pseudo-class.
As a result, we have used one network as a target of the other network for converging to the same classification.
* We have proposed our model in an optimal scenario where the dataset is balanced, and the number of classes is known.  We believe that our model can handle unknown prior distributions changing the size of the output vector z \in R^d, where d is not the optimal number of ground-truth classes.  This can be seen as the number of clusters k in K-means.
In addition, the statistical distribution \phi can be modified as well.
* We have evaluated with different batch sizes (M) in our validation set, and the best result was presented in the paper.  Our assumptions are the training rule requires a big batch size for two reasons.
First, the more input samples, the closer to the statistical distribution.
Second, the model needs to learn slowly otherwise, all input samples would be concentrated in one.
The paper presents an alternative way of supervising the training of neural network without explicitly using labels when only link/not-link information is available between pairs of examples. A pair of network is trained each of which is used to supervise the other one.
The presentation of the paper is not very clear, the writing can be improved.
Some design choice are not explained: Why is the power function used in the E-step for approximating the distribution (section 2.1)? Why do the authors only consider a uniform distribution? I understand that using a different prior breaks the assumption that nothing is known about the classes.
However I do not see a practical situations where the proposed setting/work would be useful.
Also, there exist a large body of work in semi-supervised learning with co-training based on a similar idea.
Overall, I think this work should be clarified and improved to be a good fit for this venue.
We thank the anonymous reviewer for their comments and time.  As a general remark, we have improved the clarity of the paper, mainly the motivation behind the model.
In addition, we have extended the experimental setup with three more classless datasets based on MNIST (Rotated-90 MNIST, Inverted MNIST, and Random Rotation MNIST).  Our findings still hold for these extra datasets where our model reaches better results than clustering algorithms and promising results in relation with the supervised scenario.
Finally, we have added a few extra examples as supplemental material.
Remark:  The previous version of the paper had a mistyping error about the MLP parameters.  We have reported the results of a model that has 200 neurons for each layer instead of two fully connected layers of 200 and 100 neurons, respectively.  We have already fixed this problem in the next version of our paper.  Note that the performance of both architectures is quite similar
* We would like to mention that our model relies on sample pairs of different instances of the same unknown class (‘link information’).
However, the relation between two sample pairs is not available in our case (‘not-link information’).  We clarify this in the next version of our paper.
* Thank you for pointing out that some details of the model are not clear.  The output classification of the input samples is quite similar when the network is initialized.
For example, arg max_c z_i (i=1,...,m) might be classified as c=2.  The power function gives us the initial boost in order to separate the pseudo-classes since the first iterations.
Finally, we agree that a more extensive analysis is useful for the future.
* We agree that is not clear the motivation for using the uniform distribution and how to extend to different cases.  The new version of the paper includes the motivation.  We have selected an optimal case where the dataset is balanced (uniform distribution), and the number of classes is known.
Moreover, our model can be extended to more general cases where the input distribution and the number of classes are unknown.  One way is to change the size of the output vector z \in R^d, where d is not the optimal number of classes according to the ground-truth.  This step can be seen as deciding the number of clusters k in k-means.
* We are aware of semi-supervised learning with co-training[1,2,3].
However, the strict constraint of our challenge in Symbol Grounding is that the data is totally unlabeled. We agree that a small labeled dataset would improve the performance of our model, but we like to focus on the more challenging problem.
[1] Blum, Avrim, and Tom Mitchell. "Combining labeled and unlabeled data with co-training." Proceedings of the eleventh annual conference on Computational learning theory. ACM, 1998.
[2] Tur, Gokhan. "Co-adaptation: Adaptive co-training for semi-supervised learning." 2009 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2009.
[3] Sindhwani, Vikas, Partha Niyogi, and Mikhail Belkin. "A co-regularization approach to semi-supervised learning with multiple views." Proceedings of ICML workshop on learning with multiple views. 2005.
We have updated our paper.  The changes are
* We have improved the clarity and motivation of our model
* We have evaluated our model to three more classless datasets (Rotated-90 MNIST, Inverted MNIST, and Random Rotation MNIST).
* We have updated Figure 4 and 5 for showing some random output classification samples instead of the mean of all images.
* We have added two more examples and demo as supplemental material

Meta-review:
role: PC
data: comment: The paper explores neural-network learning on pairs of samples that are labeled as either similar or dissimilar. The proposed model appears to be different from standard siamese architectures, but it is poorly motivated. The experimental evaluation of the proposed model is very limited.

Decision: Reject