Online Unsupervised Learning of Visual Representations and Categories

Abstract: Real world learning scenarios involve a nonstationary distribution of classes with sequential dependencies among the samples, in contrast to the standard machine learning formulation of drawing samples independently from a fixed, typically uniform distribution. Furthermore, real world interactions demand learning on-the-fly from few or no class labels. In this work, we propose an unsupervised model that simultaneously performs online visual representation learning and few-shot learning of new categories without relying on any class labels. Our model is a prototype-based memory network with a control component that determines when to form a new class prototype. We formulate it as an online Gaussian mixture model, where components are created online with only a single new example, and assignments do not have to be balanced, which permits an approximation to natural imbalanced distributions from uncurated raw data. Learning includes a contrastive loss that encourages different views of the same image to be assigned to the same prototype. The result is a mechanism that forms categorical representations of objects in nonstationary environments. Experiments show that our method can learn from an online stream of visual input data and is significantly better at category recognition compared to state-of-the-art self-supervised learning methods.

Reviews and author responses:

The authors propose a method for unsupervised learning of instance-based clustering which is more robust to data imbalance and non-iid distributions than existing approaches, such as SwAW. In particular, they propose an Expectation Maximization algorithm that operates in temporal episodes (supposed to correspond to an agent moving through an environment).  At every step an observation (image with an overplayed instance segmentation mask) is first encoded with a CNN and the resulting vector is assigned to one of the existing clusters based on the distance in the feature space, or a new cluster is created (E-step). This is formulated in a probabilistic framework. The cluster prototypes are then updated accordingly (M-step). The loss, computed after a sequence of such steps, consists of a standard contrastive loss (encourages assigning similar images to the same clusters) + entropy (encouraging confident assignments) + a term encouraging the creation of new clusters. 

The method is mainly evaluated on RoamingRooms - a dataset with an agent moving through indoor environments with instance labels assigned to objects. The task is then to cluster different view points of the same object together. The main challenge is that the distribution of examples is non-iid, since in every episode mostly the same objects are seen. Additional evaluation on online variants of Omniglot and ImageNet is also provided.

When compared to offline and online contrastive learning algorithms the proposed approach demonstrates stronger performance especially with low batch size and non-iid distribution of examples. The proposed approach is novel to the best of my knowledge. It also seems sound. The experimental evaluation on the task of unsupervised, instance-based clustering in the online, non-iid setting clearly demonstrates the superiority of the proposed method compared to the baselines. However, I have two main concerns with this paper.

Firstly, although it is well written, the presentation is quite dense and many details have to be inferred from the context. I understand that it's hard to pack so much content into 9 pages, but I would argue that the authors could make significantly more effort to better explain the results. The content could be densified by putting some equations in the text, for example, and the whole Omniglot evaluation can be moved to the supplementary material, since it's not that important for the story.

In fact, the main issue with the presentation is that no ablation analysis is provided. There are several tables in the supplementary material which compare the values of the many hyper parameters in the method, but the accompanying text literally just takes 9 lines. This is not sufficient to understand what actually makes this relatively complex method work better than the baselines. After all, the goal of a research  paper in not to report strong numbers on a benchmark, but to provided some knowledge about the problem being studied. The ablation analysis needs to be moved to the main text and the discussion needs to be expanded to 1-2 pages with clear conclusions about the importance of various components of the approach.

Secondly, and more importantly, the claims in the introduction are not supported by the experimental results. The proposed approach learns a representation that clusters different views of the same instance in an online manner (character classes are also more akin to instances than semantic categories), and this is the task on which it is evaluated (aside from Table 3). But is this an actual task that many people care about? I would argue that it's not, and the authors agree with me, since all the claims are made in terms of semantic classification/categorization. But instance ids are not categories. In fact, its natural that a method trained with a contrastive learning objective will cluster different views of the same object together, but does it learn a semantic representation? Or is the learned representation at least useful for downstream semantic tasks? The only evidence for that provided in the paper are in Table 3 with experiments on an online version of ImageNet, but the explanation of these results is not sufficient and no qualitative examples of discovered clusters are provided. To address these issues, the authors should either demonstrate that their method is capable of discovering categories, or show that the learned representation is superior to the baselines on downstream semantic tasks (supervised image classification, object detection, semantic segmentation) or remove any claims on learning category information, and focus on the instance classification story, providing some justification for its importance. 

Finally, it would be interesting to see which batch size is sufficient for the baselines to reach the performance of the proposed approach both in iid and non-iid setting, extrapolating the curves in Figures 3 and 6. The proposed approach is original and clearly shows strong results for the task of unsupervised, online, non-iid instance classification. However, there are major issues with the presentation, and the unsupervised categorization claims made in the introduction are not supported by the experimental results. If the authors can address both concerns in the rebuttal, I'd be happy to recommend an acceptance.

 As promised, we have updated our manuscript (all changes are colored in blue in the PDF). To summarize, we made the following changes:
- Described the datasets and evaluation protocols in more details.
- Described the training details of SimCLR, SwAV and SimSiam.
- Added evaluation protocols of NN and linear readout on semantic object classes.
- Added results using SimSiam (in the updated results, we doubled the training iterations of using 10,240,000 frames (batch size x iteration), same as SimCLR and SwAV, the conclusion remained the same as our initial authors' response).
- Added results discussion.
- Added ablation analysis paragraphs in the Appendix.
- Added and discussed the references mentioned by the reviewers in the related work section.

Please kindly check our updated paper and see if it has addressed your concerns. Thank you!

Authors


 I thank the authors for their detailed response, which addresses my concerns to some extent, and I particularly appreciate the new experiments.

First, I need to apologize for that my detailed review is only about negative points and questions. There are positive points only in the summary of the review. Also, the detailed review has no particular structure and is rather a raw listing of my thoughts while reading the paper. I do acknowledge the interest in this new setting and the quality of this work, and this is reflected in my rating.

Concerning relation to prior work, I need to clarify that my objective is not to highlight issues of novelty, but rather to improve positioning and presentation. I very much understand e.g. that an older model is very similar in some respect but without learning the representation. Still, similarities should be pointed out and in some cases it would be in the benefit of the paper to formulate it on the basis of some prior work.

The protocol is still not entirely clear. The authors should be careful in their re-writing to be as formal as possible.

I am afraid it is not clear either what variance has to do with count of examples. Again, a more formal justification in the paper would help.

Now, I understand that the authors have committed to a major revision of their work, mostly in terms of writing. I trust they will do their best to deliver this revision for this ICLR, but I do need to say that it may be more proper that this revision goes through another review.

Taking into account also the other reviews and responses, I tend to keep my rating but I would be ok if the paper is rejected so that a revision is submitted to the next venue.

 I thank the authors for their detailed response. My comment regarding the semantic properties of the representation is addressed. It appears that the proposed approach is clearly superior to the state-of-the-art in one specific setting - self-supervised representation learning in an online, non-iid, small batch regime. This is not surprising, since, to the best of my knowledge, this is the only approach designed specifically for this setting. That said, this still makes it a valid contribution, since the setting is worth exploring, just that the impact of the work might be low. I also think that the justification of the setting as being more similar to how people learn and having implications for practical robot learning and understanding of human intelligence is a bit too far fetched. Again, this does not make the contribution invalid, but I don't expect the impact to be as high as the positioning suggests.

As to the ablation analysis, I appreciate the explanation of the conclusions, but this needs to be extended and integrated into the main paper. Overall, presentation is a major issue as noted by most of the reviewers. On that front, the paper clearly requires a major revision, though the authors seem committed to do it.

Overall, I think it would be ok to accept the paper as a poster, though it could be better for everyone, including the authors, to go through the standard major revision process - resubmission and a new set of reviews for the revised manuscript. If all the reviewers' concerns are properly addressed, the resulting manuscript can be a much more valuable contribution for the community than it is now.

 
> Secondly, and more importantly, the claims in the introduction are not supported by the experimental results. The proposed approach learns a representation that clusters different views of the same instance in an online manner (character classes are also more akin to instances than semantic categories), and this is the task on which it is evaluated (aside from Table 3). But is this an actual task that many people care about? I would argue that it's not, and the authors agree with me, since all the claims are made in terms of semantic classification/categorization. But instance ids are not categories. In fact, its natural that a method trained with a contrastive learning objective will cluster different views of the same object together, but does it learn a semantic representation? Or is the learned representation at least useful for downstream semantic tasks? The only evidence for that provided in the paper are in Table 3 with experiments on an online version of ImageNet, but the explanation of these results is not sufficient and no qualitative examples of discovered clusters are provided. To address these issues, the authors should either demonstrate that their method is capable of discovering categories, or show that the learned representation is superior to the baselines on downstream semantic tasks (supervised image classification, object detection, semantic segmentation) or remove any claims on learning category information, and focus on the instance classification story, providing some justification for its importance.

While we believe that instance classification is not a trivial problem and could be a problem that people care about (e.g. recognizing human faces and tracking object identities in a robotic environment), we agree with the reviewer that it would also be interesting to investigate semantic classes as well. In our general response, we included semantic class readout from model checkpoints. Note that we have not at all tuned our model on these tasks and it turns out that we are still able to perform the best on both linear readout and nearest neighbor classification. This suggests that the performance between instance classification and semantic classification correlates well.

As we reasoned in our general remarks, we believe instance classification is a prerequisite for semantic classification. If a model is trained to match different views of the same instance together, which is the case for nearly all popular self-supervised learning methods, but at test time it cannot even retrieve the same instance, then this means that the model isn’t even successful on the task it is trained on, let alone the more challenging task of recognizing general categories. The fact that SimCLR and other self-supervised methods could perform well with large batch IID settings in our experiments, but their performances quickly drop when trained with online sequences suggests that it is not that they are not suitable to the instance classification task, but the nature of the training data (large batch IID vs. online non-iid) influences their final performances.

> Finally, it would be interesting to see which batch size is sufficient for the baselines to reach the performance of the proposed approach both in iid and non-iid setting, extrapolating the curves in Figures 3 and 6.

Thanks for the suggestion. The largest batch size version requires 8 GPUs in parallel, which is the largest type of experiment we can run in our lab environment. We are currently looking into more distributed solutions to make the figures look more complete. In the semantic class readout experiment that we ran for the rebuttal, we can see that SimCLR-IID can already catch up the performance compared to our method.

 We thank the reviewer for the insightful comments, and we look forward to a deeper discussion with the reviewer! We note the reviewer has concerns about the instance classification task. We’d like to point out that the classes can be defined at multiple levels of granularity (e.g. classification among facial identities vs. classification between dogs and cats). Nonetheless, as we explained in the general response, to address this point we ran an additional experiment, a semantic class readout task. The results are included in the general response and we would like to point the reviewer to these additional results to see if they address the reviewer’s concerns.

> Firstly, although it is well written, the presentation is quite dense and many details have to be inferred from the context. I understand that it's hard to pack so much content into 9 pages, but I would argue that the authors could make significantly more effort to better explain the results. The content could be densified by putting some equations in the text, for example, and the whole Omniglot evaluation can be moved to the supplementary material, since it's not that important for the story.

Thank you for the suggestions on the paper re-organization. We are currently making revisions to the paper so that we can fit more experimental details within the 9-page limit.

> In fact, the main issue with the presentation is that no ablation analysis is provided. There are several tables in the supplementary material which compare the values of the many hyper parameters in the method, but the accompanying text literally just takes 9 lines. This is not sufficient to understand what actually makes this relatively complex method work better than the baselines. After all, the goal of a research paper in not to report strong numbers on a benchmark, but to provided some knowledge about the problem being studied. The ablation analysis needs to be moved to the main text and the discussion needs to be expanded to 1-2 pages with clear conclusions about the importance of various components of the approach.

The ablation studies are moved to the Appendix unfortunately due to space limitations. However, we would like to highlight our findings in these  ablation studies here.

Table-8: We investigate the effect of the size of the prototype memory, and whether the model would benefit from a larger memory. It turns out that as long as the size of the memory is larger than the length of the input sequence for each gradient update step, it can learn good representations and the size is not a major determining factor.

Table-9: We examine whether the memory forgetting parameter is important to the model. We found that the forgetting rate at 0.995 is the best. 0.999 (closer to no forgetting) results in worse performance.

Table-10: We investigated the effect of various values for the new cluster loss coefficient. The optimal value is between 0.5 and 1.0.

Table-11: The threshold parameter is found to be the best at 0.5. However, this could be correlated with how frequently the frames are sampled.

Table-12: We found that the soft distillation loss is beneficial and slightly improves the performance compared to hard distillation.

Table-13: The entropy loss we introduced leads to a significant improvement on the Omniglot dataset but not on the RoamingRooms dataset.

Table-14: We found that the mean of the Beta prior is the best at 0.5. It has more impact on the RoamingRooms dataset, and has less impact on the RoamingOmniglot dataset. This parameter could be influenced by the total number of clusters in each sequence.

Regarding the reviewer’s comments on “relatively complex”: we believe that the current model is the simplest we could design. We compose a deep neural network with an online clustering layer on top and all the hyperparameters introduced here are necessary for the model to work. That being said, we do realize that some of the writing is not clear, and we are currently revising the paper and trying to make it clearer and easier to understand.

 We thank the reviewer for the insightful comments, and we look forward to a deeper discussion with the reviewer!

> Part of the successes of deep learning comes from the well collected dataset, e.g., ImageNet10K. To study some area, someone should first to collect enough high quality data. However, this is different from the cognitive process of human being. This paper questions the standard machine learning formulation of drawing i.i.d. samples and claims that the real world scenario should be nonstationary. This is very interesting and encourging. However, my main concern is that should we train our models in an online stream scenario just because the real world scenario nonstationary? You see, the tranditional learning formulation has its assumption that all samples are drawn in an i.i.d. way. But it doesn't violate the eventual performance. It works quite well.

We thank the reviewer for raising this question. As we explained in the general remarks, the engineering merit of the work is two-fold: first, it enables us to deploy real world agents in unfamiliar environments without pre-collecting a dataset; and second, the proposed algorithm is more memory and compute efficient. Moreover, we found that our method can outperform IID learning methods on the instance classification task, and therefore we highlight it is beneficial to leverage such temporal correlation in the model formulation. 

> I could understand that the author tries to mimic the learning process of human kind, just like lifelong learning. But the authors are encouraged to concentrate the necessity of this behaviour.
Besides, the language, the equations, are all above the thresold of the acceptance.

The online and continual learning behaviors might become necessary if we would like AI to continually adapt to new environments and learning without waiting for collection of an offline dataset.

We are also interested in a more scientific understanding of the human learning process. Even if for whatever reason the future AI doesn’t need our algorithm to succeed, we can still gain some insight into our own visual learning process by constructing a learning algorithm that can operate in a similar environment with a similar set of learning constraints.

 > The networks used are too small, more inline with few-shot learning and unlike most work on either self=supervised representation learning or continual learning.

We chose to use smaller networks so that we can directly compare with supervised models in the prior literature. Following your suggestion, we are currently running a version with ResNet-50 and we will update our response once we get the results.

 > "Since none of [SimCLR, SwaV] are designed to output classes with a few examples, we use our online clustering procedure to readout from these learned representations:" This is unclear. As a result, the protocol of comparisons to offline methods it is unclear. A formal description is in order.

We would like to describe the evaluation protocol again here:

At test time, we will let the model see a sequence of new images and the task is to let the model predict the class ID for each image. In the unsupervised mode, the model will only be given the sequence of inputs, and it needs to make a prediction at each time step after seeing the current image. In the supervised mode, the model will be given the sequence of inputs and labels (up until the previous time step), and predict the class label for the current input. And note that in instance classification, the instances do not appear in the training set and the model needs to predict them as new class IDs.

Now with this evaluation protocol, how can we evaluate general representation learners such as SimCLR and SwAV under the protocol? In order to read out the class IDs from the representations, we leverage the greedy online clustering procedure described in our model: For each incoming example, we first look for the closest centroid of the existing clusters, if the similarity is greater than a certain threshold, then we assign it to the closest cluster and update the mean of the cluster. If the similarity is less than the threshold then we create a new cluster. At the end of the sequence, we obtain a clustering of the embeddings and each example now has a class ID prediction. Of course, this procedure requires a threshold parameter, and these baseline models do not have a built-in parameter. Therefore, we need to search for the optimal threshold for each of the models and report the clustering performance. For our model, the threshold parameter is not critical since the model is trained to output clusters at the threshold of 0.5. Therefore, the sweeping of the threshold parameter is for the benefit of other representation learning baselines.

In the general response above, we also included additional studies on semantic class readout and this resembles a more similar evaluation protocol compared standard self-supervised learning.

> The entire argument against methods assuming iid batch sampling is reliance on iid for contrastive learning (SimCLR) or prototypes (SwaV). How about self-supervised representation learning without prototypes (e.g. BYOL: Grill et al., 2020) and without negatives (e.g. SimSiam: Chen & He, 2021)? Whouldn't such simple alternatives be much easier to adapt to online? Wouldn't those online versions be more suitable as baselines to compare with?

Thank you for pointing this out. We agree with the reviewer that these are interesting experiments to run, and we ran additional results of SimSiam on the RoamingRooms dataset, detailed in our general response above. We have also verified our implementation of SimSiam on CIFAR. The short conclusion is that SimSiam does not deliver better performance and their scores stay low with increased batch size (or sequence length). We speculate that this might be because the contrastive training objective could still be more robust on non-ImageNet based benchmarks than these self-supervised objectives without negative samples; however this is just our speculation. Moreover, we also have a contrastive loss in our training objective and therefore we believe that the current comparison is not unfair.

> The following works should be discussed. Although I believe they are not directly comparable, one may still consider additional baselines, e.g. self-supervised offline pretraining followed by online learning (supervised or not):
Zhu et al. 2021, "Prototype Augmentation and Self-Supervision for Incremental Learning"
Gallardo et al. 2021, "Self-Supervised Training Enhances Online Continual Learning"
Zhang et al. 2021, "Self-Supervised Learning Aided Class-Incremental Lifelong Learning"
Cha et al. 2021, "Co2L: Contrastive Continual Learning

Thanks for the references. We will cite them in our next version.

> The list of hyperparameters (e.g. Tables 4-6) is daunting. One should at least separate the method-specific hyperparameters (e.g. $K$, $\rho$, $\alpha$) from standard hyperparameters (like backbone and learning rate). From Tables 8-14, there are at least 7 hyperparameters in the ablation. What are the default values of remaining hyperparameters when studying each one? Is this search optimal, what is the cost, and to what extent could same values be used across datasets? Merging Tables 4-6 into one would help compare.

We will consider your suggestions of reorganizing the tables in the Appendices.

 > What is really missing is citing and discussing Bottou and Bengio 1995, "Convergence Properties of the K-Means Algorithms," which defines an online k-means algorithm by means of gradient descent on an error function. One would expect a generalization to a Gaussian mixture basically by preserving the same formulation and adding mixing coefficients and variances per component (cluster), but this is not done. Bottou and Bengio have a similar definition of count of examples per cluster (Eq. (7)), but which has nothing to do with variance.

Thanks for the reference above and we will cite more papers of the literature in the 90s. Our connection between inverse variance and count is described in the previous paragraph. As we wrote in the general comments, one key difference between our work and the online clustering literature is that we jointly learn the representations together.

> Why use decay (hyperparameter $\rho$)? Well, there is the argument of nonstationary environments but how about forgetting? The entire problem of incremental and online learning is to learn without forgetting. Here, not only nothing is done to avoid forgetting; decay is actually explicit forgetting. Is this justified by the unsupervised setting? Or is there another reason?

Forgetting might be beneficial for an online and continual system since 1) the set of active prototypes may be changing over time and 2) the representations that produce the prototypes are also changing during learning as well. We investigated the value of the decay parameter in Table 9 in the Appendix, and we verified that some level of forgetting is beneficial for our model. The forgetting parameter is not essential at test time, since at test time we evaluate the model through a sequence of images and the memory is reset in between sequences.

> "Popping out" the prototype with the least weight is too simplistic, because it does not consider interaction or redundancy among clusters. An little populated, isolated cluster may be more important than a highly populated cluster with many neighboring/overlapping clusters. 

Thanks for the suggestion. We will consider it for future model improvement.

> Avrithis and Kalantidis 2012, "Approximate Gaussian Mixtures for Large Scale Vocabularies" develop an offline but dynamic (in terms of components) EM algorithm that removes mixture components based on pairwise interaction. 

Thanks for the references and we will cite it in the next version. There have been a number of ways to estimate the number of clusters offline in the past as well and we would like to highlight that our model performs both representation learning and online incremental clustering at the same time.

> Tao et al. 2020, "Few-Shot Class-Incremental Learning" also uses a graph over prototypes and has a similar online update of prototypes (Eq. (3)).

Thanks for the references and we will cite it in the next version. Since this work is based on few-shot, and therefore it needs class labels for storing the prototypes whereas our model can be label-free.

> In Eq. (13), it is not clear if target labels are soft, as is standard in distillation. They should be, since there is a temperature parameter, but (13) should have a sum over $k$ that is missing. Similarly, (14) should have a sum over $k$ if this expression is meant to be an entropy. 

We will follow your suggestion and add a sum over $k$ to make it clearer.

> Eq. (15) is too abstract and unclear.

We take the average probability of creating a new cluster and evaluate the log probability of it based on a Beta distribution.

> In computing AMI, is the number of classes known? What is $T$? Sweeping over hyperparameter $\alpha$ is not realistic, especially if is unnecessary (see above).

T is the total number of images in the sequence. The number of classes could influence the decision over choosing the optimal $\alpha$ during evaluation but the number of classes is not known during training. You can think of the sweeping over $\alpha$ as a part of the readout mechanism, just like in linear readout people assumed to have access to all class labels of the training set.

 We thank the reviewer for the detailed and in-depth comments. We are glad to know that the reviewer is an expert in the area of clustering and self-supervised learning. We are grateful for the list of references provided by the reviewer, and we look forward to a deeper discussion with the reviewer.

> The uniform prior distribution for a new cluster is an improper distribution over continuous space. In fact, the overall distribution is a Gaussian-uniform mixture (GUM) (Lathuilière et al. 2018, "DeepGUM: Learning Deep Robust Regression with a Gaussian-Uniform Mixture Model"). In the original GUM model, the prior probability of an inlier is learned like all mixing coefficients (weights), whereas here is a fixed hyperparameter and is yet another unnecessary threshold hyperparameter (the new cluster probability could be simply compared to existing cluster probabilities).

We thank the reviewer for providing the reference and we will cite it in our next version as a reference for the uniform prior. Like our’s, the work you cited still has a parameter (iteratively updated throughout training). However, it only detects outliers for more robust clustering but it does not create new clusters for each outlier in an online and incremental manner.

Regarding your comment on “unnecessary threshold”, we did compare the new cluster probability with existing cluster probability, but the probability is a continuous value which we need to convert to a hard decision of whether or not we create a new cluster at time step t. And this discrete decision can influence future decisions. For our model, we used standard threshold=0.5 during training, but for other baseline models, their distances are not calibrated with a probabilistic model, and thus we need to optimize the threshold parameter for the other models in order to compute a reasonable performance measure. Therefore, we respectfully disagree that this is “yet another unnecessary threshold hyperparameter”: DeepGUM does not use a threshold because they are not performing online incremental clustering.

> The RHS of (2) is incorrect. Softmax is mapping a vector to a vector, while here only a scalar is shown (the numerator of the LHS). One should map the whole vector through softmax, then take the $k$-th element.

We thank the reviewer for pointing it out. We will revise our math syntax to clarify the mapping.

> Eq. (5) is unclear unless one reads the Appendices. Overall, section 3.1 is written as a summary of sections A.1 and A.2, which is hard to follow. For example, $y_{t,k}$ and $u_{t,k}$ in (9,10) are not defined. One should provide a clear motivation for the model and all choices (which is missing), definitions all quantities (some of which are missing) and clear pointers to Appendices for all missing steps in the derivation. The formulation is overall very similar to (Ren et al., 2021), so the derivation is not new (e.g. compare (3) of Ren et al. with (5) of this work.) The connection is missing, both in section 3.1 and in A.1, A.2.

We are currently revising the paper to make it clearer, but $y_{t,k}$ and $u_{t,k}$ are in fact defined in the manuscript under Section 3 on page 3. We agree that the formulation is similar, but Ren et al. (2021) did not provide a probabilistic explanation, and it is not purely unsupervised. Moreover, the memory forgetting parameter is a new component in our design.

> What does inverse variance have to do with the count of examples per cluster? Why estimate variance since variance is assumed constant? This remains unclear even after reading section A.2. Shouldn't symbol $\sigma_{t,d}$ (e.g. in (7,8) be $\sigma_{t,k}$?

Note that the inverse variance represents the uncertainty in estimating the posterior cluster mean, and naturally it has an inverse relationship with the number of members in the clusters. As we have more observations for the cluster, we are more confident in estimating the mean of the cluster. $\sigma_{t,d}$ is the variance for the transition dynamics and it is different from $\sigma_{t,k}$ which is the variance for the posterior mean estimate of cluster $k$.

 **Table-B: Nearest neighbor semantic readout test accuracy (%).**

| Online seq len |    50 |   100 |   200 |   400 |
|----------------|------:|------:|------:|------:|
| Ours           | **48.37** |       |       |       |
| SimCLR         | 44.84 | 45.51 | 48.01 | 49.03 |
| SwAV           | 40.04 | 43.29 | 42.60 | 44.79 |
| SwAV+Q         | 43.63 | 43.72 | 45.11 | 45.29 |
| SimSiam        | 40.37 | 44.55 | 45.10 | 44.17 |
| IID batch size     |    50 |   100 |   200 |   400 |
| SimCLR (IID)        | **46.50** | **47.68** | **49.36** | **53.93** |
| SwAV (IID)          | 43.92 | 45.02 | 46.26 | 46.94 |
| SwAV+Q (IID)        | 43.21 | 44.84 | 46.41 | 46.22 |
| SimSiam (IID)       | 42.99 | 46.88 | 46.84 | 47.21 |

**Table-C: Linear readout test accuracy (%).**

| Online seq len |    50 |   100 |   200 |   400 |
|----------------|------:|------:|------:|------:|
| Ours           | **52.28** |       |       |       |
| SimCLR         | 48.83 | 48.11 | 51.01 | 53.68 |
| SwAV           | 45.77 | 47.78 | 47.67 | 49.66 |
| SwAV+Q         | 45.31 | 49.65 | 50.23 | 50.40 |
| SimSiam        | 44.67 | 47.74 | 45.38 | 45.97 |
| IID batch size    |    50 |   100 |   200 |   400 |
| SimCLR (IID)     | **53.79** | **54.85** | **56.03** | **60.05** |
| SwAV (IID)          | 50.12 | 53.61 | 53.25 | 54.83 |
| SwAV+Q (IID)        | 51.10 | 53.14 | 53.67 | 55.17 |
| SimSiam (IID)       | 49.11 | 49.14 | 50.16 | 48.49 |

 **Instance vs. semantic category:** Several reviewers have critiqued our focus on instance classification in RoamingRooms, given that the premise of our work is to study categorization. We have therefore added a new set of evaluations on RoamingRooms that use semantic categories (i.e., chair, table, picture) instead of instances. However, we note that instance categorization is an important and nontrivial task, e.g., face identification. Given the variation in viewing angle, position, lighting, and occlusion in RoamingRooms, recognizing a specific object as such is a challenging problem that occupied computer vision research for many years. Moreover, in our evaluation, instance classification needs to happen based on only a single view of an instance, given that all instances are novel.  As evidence in support of the challenge here, we showed in the paper that popular self-supervised algorithms do not perform well in individuating instances, let alone in performing unsupervised semantic classification.

In our new experiments with semantic categories, we took advantage of the fact that the Matterport / RoamingRooms dataset has 21 semantic categories including “picture”, “chair”, “lighting”, “cushion”, “table”. We use learned representations from each model for this 21-way semantic classification task. We used both a k-nearest neighbor (kNN) classifier and a linear classifier for readout. For kNN, the hyperparameter k is chosen between 1 and 39 for each model based on the validation set performance (empirically, larger k is preferred); for the linear classifier, we used the Adam optimizer with learning rate 1e-3 for 20 epochs of training. The results are presented in Table A below, together with existing results on AMI and AP for novel instance grouping. In addition, we also provide results using a supervised CNN on this classification problem, as a performance “upper bound”. Our OUPN model achieves the best performance. Because we ran this experiment only days ago, we did not tune any hyperparameters on this new metric and the evaluation is directly from model checkpoints. It is encouraging to see that better instance classification performance translates to better semantic classification. We also ran an evaluation with increasing batch size (or sequence length) as we did in the paper. We include the results in Tables B and  C below, and we will also update the manuscript with the figures. The only case where our method is inferior to SimCLR is when SimCLR is trained with large batches under IID setting on semantic classification readout. This is reasonable since semantic classification and IID large batch training is the setting SimCLR was originally developed for. Again, IID large batch training is not what we aim to solve in this paper, and we include the IID experiments in the paper simply to better understand the failure points of existing algorithms.

**Comparison with SimSiam (a self-supervised algorithm without negative samples):** One reviewer commented that some later self-supervised algorithms (such as BYOL or SimSiam) succeed without a contrastive loss function. Without the use of negative samples, these algorithms could potentially succeed without suffering from treating images from the same instances as negative samples in the online setup. We agree with the reviewer that this is a valid hypothesis and therefore, we included SimSiam in our comparison among methods. Results are shown in Tables A, B, and C below. We will also update our manuscript to incorporate results. We have verified our implementation of SimSiam on CIFAR-10 and it reaches comparable performance to SimCLR. However, SimSiam fails to deliver competitive performance in our experiments. Shown in Tables B and C, SimSiam is indeed less dependent on the batch size but its performance is poor and does not improve much with batch size. Furthermore, SimSiam performs significantly worse on the clustering based metrics as its representations could be more scattered.

**Table-A: Updated results on RoamingRooms with semantic readout + SimSiam. Newly added numbers in *italic*.**

|                 | AMI   | AP    | Acc (kNN, %) | Acc (linear, %) |
|-----------------|-------|-------|--------------|-----------------|
| Supervised CNN  | -     | -     |        *72.11* |           *71.93* |
| Online ProtoNet | 79.02 | 89.94 | -            | -               |
| Random CNN      | 28.25 | 11.68 |        *28.84* |           *26.73* |
| SimCLR          | 50.03 | 52.98 |        *44.84* |           *48.83* |
| SwAV            | 42.70 | 37.31 |        *40.04* |           *45.77* |
| SwAV+Q          | 48.31 | 50.40 |        *43.63* |           *45.31* |
| SimSiam         | *40.92* | *35.11* |        *40.37* |           *44.67* |
| Ours            | **78.16** | **84.86** |        **48.37** |           **52.28** |

 
We thank all reviewers for their valuable time and insightful comments. We are appreciative and respectful of the fact that many reviewers are experts on self-supervised learning. We make a few general remarks here and respond to individual comments to each review below.

Motivation of online stream representation learning: Some reviewers question the motivation of our research: “If large batch IID training already works well, why bother studying learning from an online stream?” Our responses: (1) We would like to enable unsupervised learning in a real-world agent without large scale data collection, which is essential for deploying intelligent agents in unfamiliar environments, since collecting large scale datasets will imply a huge setup cost and less flexibility to new environments. (2) As showcased in the paper, our method is more memory and computation efficient compared to large batch algorithms, which require maintaining a large queue buffer, using multiple GPUs in parallel, and storing the entire dataset in the agent’s hard disk. (3) Such a learning scenario more closely resembles naturalistic human/animal learning environments, and can give us insights on questions related to how human and animal brains can learn visual representations efficiently through a visual stream. 

With online stream learning as our main motivation, we wished to model realistic temporal correlations in naturalistic data sequences. Our experiments focus on the  RoamingRooms dataset, chosen because it is an approximation to what an indoor mobile robot might encounter, and contrasts sharply with the IID setting of most previous work in unsupervised learning (including video learning, which relies on IID mini-batch samples rather than sequencing through the video one frame at a time). SimCLR/SwAV/SimSiam do not perform well when data are non-IID, and still perform worse on instance classification metrics when the input sequence is shuffled to be IID.

**Focus of our paper (Representation Learning + Clustering):** We agree with reviewers that many clustering algorithms in the past are quite similar to our greedy online scheme, and we appreciate pointers to the literature which we’ll cite in our upcoming revision. However, we would like to clarify that our contribution is to combine representation learning and category clustering in a unified framework, quite unlike pre-deep-learning papers mentioned by reviewers that perform clustering in the input space. Our work is akin to other clustering-based self-supervised representation learning algorithms (e.g., DeepCluster/SeLa/SwAV). The fact that our method outperforms SwAV confirms that our clustering procedure is more suitable to the online learning environment, whereas SwAV’s clustering procedure based on Sinkhorn-Knopp better suits IID environments. Therefore, we believe it is unfair to evaluate our paper based on the clustering component only, without consideration of representation learning and the environmental structure.

One novel feature of our clustering scheme is the forgetting parameter $\rho$, which allows us to model nonstationary distributions and to efficiently update the clusters without recomputing them at each time step. Due to space limitations, we present evidence of the effectiveness of this parameter in Table 9 in the Appendix, showing that some decay is beneficial for our online unsupervised learning setting.

**Experiment setup:** We did a lousy job of describing our experimental setup and evaluation procedure, and the reviewers’ confusion is understandable. Our revision will hopefully be clearer, and we summarize it here. In RoamingRooms, the dataset is split into different home environments (60 training, 10 val, and 20 test). Each training iteration consists of a sequence of images from one of the homes. At test time, for the instance classification task, we ask the model to recognize novel objects in a new sequence of images in one of the test homes. The model’s memory is reset between test sequences. 

**Evaluation protocols:** Reviewers expressed concern about our online clustering-based evaluation in that it might potentially favor our algorithm against competitors, and the criteria are unclear (e.g., why a single cluster, as opposed to multiple clusters per class). To address concerns with our evaluation, we have added two additional evaluation metrics to our original two for RoamingRooms -- k-nearest neighbor and linear readout on the representations for semantic classes -- as the next section addresses.

 > adjusted MI. While I follow the argument for using a max_alpha in the adjusted mutual information, I’m not sure I agree with it. For one, this is not something one can do in practice; therefore it gives an over-optimistic view of the output. Second, it is not clear if this procedure favors the current algorithm. By knowing that you are going to optimize over the number of clusters, one can adjust one’s algorithm to take advantage of this. I do not claim to understand whether this procedure favors your algorithm, but in general, such a procedure is not “neutral” to two different algorithms.

The maximization procedure should be treated as part of the readout mechanism. Just like linear readout assumes access to all training labels, this readout only assumes to have an approximate knowledge of the total number of clusters. Note that the sole purpose of introducing max_alpha is to make all other algorithms compatible under this evaluation protocol, and our algorithm does not rely on it. Our algorithm can reach almost its peak performance on these benchmarks with alpha to be set at 0.50 (like a hard threshold function), since the model’s prediction on the probability of the input being a new class has been calibrated throughout training. However, other self-supervised algorithms are rather uncalibrated, and their performance peaks when alpha is close to 0.90. We would need to use the percentile of pairwise distances to set the alpha value.

> The function g() is introduced with no definition.

We apologize for the confusion and will define it in our next version of the manuscript. g() represents the model embedding function plus the online clustering procedure. The online clustering procedure is a greedy procedure. For each iteration, it searches for the closest prototype; if it fails with sigmoid greater than alpha, it will create a new prototype, and otherwise it will aggregate the current embedding to the cluster centroid.

> How is K the number of classes chosen?

K is a hyperparameter of the memory capacity. It is chosen to be at least as large as the sequence length so that it can store all the examples of the current sequence in the prototype memory. As shown in our ablation studies, the value of K has a very tiny impact on our final performance.

 
> SimCLR and other such “generic” representation-learning approaches are solving a significantly different problem: they are typically trying to learn representations which have a high degree of generalizability so that they can be the backbone for a system trained to recognize general categories, with very high variability. Here, since the goal is to learn to recognize the exact same object, it makes more sense to take a prototype style approach, since this is very similar to say, k-nearest neighbors.

Regardless of whether SimCLR may or may not have a higher degree of generalizability and recognize more general categories, the problem of instance classification seems to be a prerequisite for a more general categorization process. The training objective of SimCLR only encourages two different views of the same instance to be close, and in no way does it capture more general categories explicitly in its training objective. So when applying to an instance classification problem, why would SimCLR have an excuse to fail to group different views of the same instance together? While we agree that semantic class readout is an important aspect (and we have also reported numbers on semantic classes in our rebuttal), we argue that SimCLR still should be expected to perform well on an instance classification task, and such an evaluation shouldn’t be unfair for SimCLR, at least in theory. Our controlled experiment shows that SimCLR actually works well on large batch IID settings and does not perform as well in small batch online settings. This suggests that it is not that our evaluation protocol is unfair to SimCLR, but that the online non-IID environment is breaking it.

With regard to the clustering aspect of our evaluation, it is actually not uncommon to use a clustering metric for clustering-based self-supervised learning. In the DeepCluster paper by Caron et al. (2018), the authors directly used normalized mutual information (NMI) to compare the learned clusters with the groundtruth class labels. So if the reviewers have doubts of using clustering to measure SimCLR’s performance, then our comparison to SwAV should still be considered as fair.

Any type of readout procedure has the trade-off between model complexity and bias. The clustering type readout does have the bias of favoring all features of the same class being grouped together, but we don’t see an issue of having such a bias for an instance classification problem, since, in the end, all of these self-supervised algorithms are supposed to achieve some level of “instance-invariance” in their representations according to their training objectives. In other words, any successful self-supervised learners studied in our paper should produce the most similar embedding for the same instance that can be separated from other instances.

> What I’m trying to say is perhaps that SimCLR and other such methods are not the right baselines. A more reasonable baseline is one that just looks at distances to the training sets.

As explained before, distances to the training sets will not be applicable to our instance classification problem since the test instances do not appear in the training set. However, following the reviewer’s suggestion, we included kNN evaluation for semantic categorization (see our general response). However, nearest neighbor search should not be understood as a baseline but rather another readout protocol, since nearest neighbor search itself cannot provide us with learned representations to compare to. 

> “we make two changes on the model inference procedure defined above.” Why did the authors describe the changes to the model in the Experiments section? Isn’t the right place to discuss this in Section 3, the section where the model and methods are described? This makes it sound like a last second change….Or, perhaps like it was a difficult-to-justify method that would be difficult to explain. What is the motivation for putting it in experiments?

The two changes are relatively minor and we can explain our motivation here. First, we change the distance distribution from squared Euclidean to negative cosine. We observed in our experiments that squared Euclidean could lead to stability issues and negative cosine is more stable. So instead of a Gaussian mixture, it would be a von Mises mixture. Secondly, we remove the mixing coefficients since we found that removing it can lead to better performance. This is equivalent to having a strong prior that all mixtures have equal weight and is compatible with our model formulation.

 > I couldn't even tell if SimCLR was trained on the same data, or trained on some other data. If it is trained on the current data, could you give details about how this training is done?

We apologize that this is not explicitly written in the text but we train all models on the same data. SimCLR, SwAV and SimSiam use varying batch sizes (50, 100, 200, and 400). For online (non-IID) settings, the notion of batch size can be understood as “sequence length”. Learning rate is scaled based on batch size / 256 * 0.3 using the default LARS optimizer with cosine learning rate decay. We trained for a total of 5,120,000 examples. So the total number of training steps is 5,120,000 / batch size. We also trained for double the training steps and observed that the performance is similar. For our proposed model, we used the batch size of 50 and trained for a total of 80000 steps (4,000,000 examples), using the Adam optimizer and staircase learning rate decay. All data augmentation parameters are the same as our model, and the same as the original SimCLR paper, except that in RoamingRooms the minimum crop area is changed to 0.2 instead of the original default 0.08.

> Why not just do nearest neighbor retrieval using the learned representations, and see how that works?

As we previously explained in the responses, during test time, the model is given a new sequence of instances and needs to perform classification among these instances. Therefore, it is unlike regular nearest neighbor retrieval of the training set, since none of the test instances appear in the training set. We could, however, perform nearest neighbor search within the current test sequence or even the entire test set, with the caveat being: 1) nearest neighbor search does not directly evaluate the quality of classification but rather the smoothness of the representations with regard to the class label. An embedding can achieve perfect k-nearest neighbor score as long as (k+1) data points of the same class are bundled together in the embedding space, but they could still fail in the clustering-based evaluation; 2) nearest neighbor search requires class labels, so it still cannot replace our unsupervised readout evaluation, where models need to predict class ID without any class labels; 3) nearest neighbor search can only work when the number of instances belonging to the same class is equal or larger than two, but many of our test instances only show up once in a sequence. In our current evaluation, the representations of such a solo instance needs to be far away enough from other instances, but this could not be captured by a nearest neighbor evaluation protocol.

Although nearest neighbor search would not be the best metric for novel instance categorization, following the reviewer’s suggestion, we adopted kNN evaluation for our newly added results on semantic categorization. Please see our general response for details.


 We thank the reviewer for the detailed and in-depth comments. We are glad to know that the reviewer is an expert in the area of self-supervised learning, and apologize for any unclear writing that causes confusion. We hope that our response clarifies our work.

> I could not ascertain whether the bounding box of the object was given at test time, or only at training time. If it is provided at test time, this makes the task much much easier. Either way, it is a critical detail, and should be described clearly and unambiguously.

The bounding box of the object is given at test time, just like training time. However, we respectfully disagree that this makes the task “much much” easier, but it is rather a necessary setup. We remind the reviewer that unlike in ImageNet, where images mostly contain only a single object, in RoamingRooms, each frame contains multiple objects. We provide the bounding box in order to cue the model which object to attend to. Without such an additional channel, the task definition will be ambiguous. 

As the reviewer knows, outputting object localization is a related task but it does not necessarily need to lie inside the scope of this paper. We believe it is a reasonable initial step to first study the unsupervised learning of the classification problem, whereas the localization problem can be studied independently in the future.

Another potential misunderstanding, possibly due to our unclear writing, is whether the model is evaluated in environments it has never seen during training. To clarify, at test time, the model is given a new sequence of images, and none of the instances in the images have been present in the training set. The model needs to predict new class IDs for each image in the test sequence. Again, without knowing the bounding boxes during test time would make the task ambiguous, since there could be multiple objects in each image, and the model needs to predict distinct class IDs for the same image if the provided bounding boxes are different.

Lastly, we would like to emphasize that all models, including SimCLR, SwAV and SimSiam are trained on the same dataset with the same bounding box inputs (instead of using ImageNet checkpoints), so we have tried our best to make sure that the comparison is fair.

> How is the query image related to the “5000 images” which I assume you are searching? How close can they be in time? Could it actually be at the exact same time? If it is just a frame or two away from the current frame, we would expect the similarity of the same object across these close views in time to be very very similar. If this is the case, then I would expect to see other types of baselines that leverage this kind of information, such as simple nearest neighbor methods. In other words, if I give a query, just return the top-9 nearest neighbors using some pre-trained representation, centered on the provided bounding box. If such an approach is not reasonable, it is hard to tell from the current text. If it is reasonable, then it would be useful to compare against it to get a sense of what exactly is being learned.

Figure 4 is only for visualization purposes to showcase the quality of the embedding and hence it is based on a very different procedure compared to our evaluation. We regret that this is a misunderstanding and we are currently revising the text to make the testing procedure more explicit. We originally did not include full details due to space limitations and since it is also the same procedure as Ren et al. (2021), except without using any class labels. The test set contains approximately 1500 sequences, and each contains a maximum of 100 images. The task is to predict the instance class label for each image in each sequence (with or without labels). In unsupervised readout, the model directly predicts the label for each image. In supervised readout, the model has access to all labels up to time step t-1, and needs to predict the label for the t-th image. The class ID starts from zero for each sequence, and they do not overlap with the training set.

In Figure 4, we showcase the first 5000 images in the first 100 test sequences of length 50 and perform retrieval among these 5000 images. The query image is like a leave-one-out procedure. Although we agree that this is not a terribly difficult task, the suggested using “some pretrained representation” is not clear to us, since the whole paper aims to learn representations from scratch from online streams of data, and we have already compared our representations to SimCLR/SwAV (and SimSiam in our response. Using pre-trained representations from ImageNet violates our premise of studying the online learning of representation and categories and therefore should not be considered here.

This paper addresses the problem of doing unsupervised learning of visual representations, including clustering these representations to infer the existence of new categories. The paper focuses on more "un-curated" settings in which the data comes from samples of an environment that a real agent is passing through. This gives object distributions that are very different from typical, curated data sets, like ImageNet without labels. The authors compare their method with other popular self-supervised learning techniques like SimCLR. First of all, I am very enthusiastic about the general topic that is being addressed: unsupervised (or self-supervised) learning in more realistic environments--specifically those in which we may see a large number of examples of a small set of objects, and perhaps a few examples of some other classes. Thus, the topic is highly relevant and important at this time. 

Despite trying my best to understand the specifics of the experimental settings, I found it very difficult to tell exactly what was going on. I found several aspects of the paper difficult to follow. One major issue was trying to understand the exact setting of the RoamingRooms data set task. 

Referring to the experimental results from figure 4, here are a few comments:
	
- Only when I finally looked at the title of the figure in figure 4 was I able to discern some of the parameters of the experiments on the RoamingRoom data set such as the 5000 images that retrieval was done across.
	
- I could not ascertain whether the bounding box of the object was given at test time, or only at training time. If it is provided at test time, this makes the task much much easier. Either way, it is a critical detail, and should be described clearly and unambiguously.

- How is the query image related to the “5000 images” which I assume you are
searching? How close can they be in time? Could it actually be at the exact same time?  If it is just a frame or two away from the current frame, we would expect the similarity of the same object across these close views in time to be very very similar.  If this is the case, then I would expect to see other types of baselines that leverage this kind of information, such as simple nearest neighbor methods. In other words, if I give a query, just return
the top-9 nearest neighbors using some pre-trained representation, centered on 
the provided bounding box.  If such an approach is not reasonable, it is hard to
tell from the current text. If it is reasonable, then it would be useful to compare 
against it to get a sense of what exactly is being learned. 

- I couldn't even tell if SimCLR was trained on the same data, or trained on some other data. If it is trained on the current data, could you give details about how this training is done? It is possible that if I had read all of the referenced papers I would understand these details, but a reviewer should not have to wade back through papers to understand these details. 

- "we use our online clustering procedure to readout from these learned representations". Why not just do nearest neighbor retrieval using the learned representations, and see how that works?

SimCLR and other such “generic” representation-learning approaches are solving
a significantly different problem: they are typically trying to learn representations which
have a high degree of generalizability so that they can be the backbone for a system
trained to recognize general categories, with very high variability. Here, since the goal
is to learn to recognize the exact same object, it makes more sense to take a prototype
style approach, since this is very similar to say, k-nearest neighbors. 

What I’m trying to say is perhaps that SimCLR and other such methods are not the
right baselines. A more reasonable baseline is one that just looks at distances to the
training sets. 


The authors may feel that I have completely misunderstood the paper. That may be true, but it is not for lack of trying. I have worked with many of the cited self-supervised methods, and have published in self-supervised learning as well. I found a general lack of detail about the exact settings made it very hard to evaluate what was going on.


Other issues:

1) “we make two changes on the model inference procedure defined above.”
Why did the authors describe the changes to the model in the Experiments section?
Isn’t the right place to discuss this in Section 3, the section where the model and
methods are described? This makes it sound like a last second change….Or, perhaps
like it was a difficult-to-justify method that would be difficult to explain. What is the
motivation for putting it in experiments?

2) adjusted MI. While I follow the argument for using a max_alpha in the adjusted mutual
information, I’m not sure I agree with it.  For one, this is not something one can do in practice; therefore it gives an over-optimistic view of the output. Second, it is not clear if this procedure favors the current algorithm. By knowing that you are going to optimize over the number of clusters, one can adjust one’s algorithm to take advantage of this. I do not claim to understand whether this procedure favors your algorithm, but in general, such a procedure is not “neutral” to two different algorithms.

3) The function g() is introduced with no definition.

4) How is K the number of classes chosen?

 Despite being enthusiastic about this general area, I found it too difficult to follow the experiments. Not enough detail was given about:
- data sets used and how they were sampled to produce "training" and test data
- exactly what the experimental setting was. What problem, exactly, are these systems trying to solve? I think, but am not sure, that given many instances of objects with unique ids at "training time", they are simply trying to retrieve objects with the same ID as a query image at test time. However, not enough details are given about the details of this problem.
- Given the nature of the problem, it appears as though a variety of nearest-neighbor style baselines would be appropriate.



In my assessment about "confidence", I have said that I am fairly confident of my review. Ironically, I didn't understand the paper well, but this is the problem with the paper. It should be easier to understand what was done. Thus, I am confident in my assessment that the paper should have been more understandable. In particular, I am not someone coming from far outside the area; thus, the paper should have laid out the details more clearly so that someone with my background would understand it.

This work studies an online version of self-supervised representation learning. It uses Gaussian mixture  of constant isotropic variance as a prototype memory as well as a uniform distribution to handle new unknown classes. The overall distribution evolves using an online EM algorithm that supports adding and removing prototypes. Using this memory, representation learning follows standard unsupervised losses, primalrily a distillation loss over predicted prototype assignment probabilities. Why is the standard deviation $\sigma$ constant and not learned per component (cluster) of the mixture? Since $\sigma$ is a scalar (isotropic), this wouldn't add much complexity to the model and it would help in deciding when to add or remove prototypes, potentially getting rid of (or learning) hyperparameters $u_0$ and $\mu$.

The uniform prior distribution for a new cluster is an improper distribution over continuous space. In fact, the overall distribution is a Gaussian-uniform mixture (GUM) (Lathuilière et al. 2018, "DeepGUM: Learning Deep Robust Regression with a Gaussian-Uniform Mixture Model"). In the original GUM model, the prior probability of an inlier is learned like all mixing coefficients (weights), whereas here $z_0$ is a fixed hyperparameter and $\alpha$ is yet another unnecessary threshold hyperparameter (the new cluster probability could be simply compared to existing cluster probabilities).

The RHS of (2) is incorrect. Softmax is mapping a vector to a vector, while here only a scalar is shown (the numerator of the LHS). One should map the whole vector through softmax, then take the $k$-th element.

Eq. (5) is unclear unless one reads the Appendices. Overall, section 3.1 is written as a summary of sections A.1 and A.2, which is hard to follow. For example, $y_{t,k}$ and $u_{t,k}$ in (9,10) are not defined. One should provide a clear motivation for the model and all choices (which is missing), definitions all quantities (some of which are missing) and clear pointers to Appendices for all missing steps in the derivation. The formulation is overall very similar to (Ren et al., 2021), so the derivation is not new (e.g. compare (3) of Ren et al. with (5) of this work.) The connection is missing, both in section 3.1 and in A.1, A.2.

What does inverse variance have to do with the count of examples per cluster? Why estimate variance since variance is assumed constant? This remains unclear even after reading section A.2. Shouldn't symbol $\sigma_{t,d}$ (e.g. in (7,8) be $\sigma_{t,k}$?

What is really missing is citing and discussing Bottou and Bengio 1995, "Convergence Properties of the K-Means Algorithms," which defines an online $k$-means algorithm by means of gradient descent on an error function. One would expect a generalization to a Gaussian mixture basically by preserving the same formulation and adding mixing coefficients and variances per component (cluster), but this is not done. Bottou and Bengio have a similar definition of count of examples per cluster (Eq. (7)), but which has nothing to do with variance.

Why use decay (hyperparameter $\rho$)? Well, there is the argument of nonstationary environments but how about forgetting? The entire problem of incremental and online learning is to learn without forgetting. Here, not only nothing is done to avoid forgetting; decay is actually explicit forgetting. Is this justified by the unsupervised setting? Or is there another reason?

"Popping out" the prototype with the least weight is too simplistic, because it does not consider interaction or redundancy among clusters. An little populated, isolated cluster may be more important than a highly populated cluster with many neighboring/overlapping clusters. Avrithis and Kalantidis 2012, "Approximate Gaussian Mixtures for Large Scale Vocabularies" develop an offline but dynamic (in terms of components) EM algorithm that removes mixture components based on pairwise interaction. Tao et al. 2020, "Few-Shot Class-Incremental Learning" also uses a graph over prototypes and has a similar online update of prototypes (Eq. (3)).

In Eq. (13), it is not clear if target labels are soft, as is standard in distillation. They should be, since there is a temperature parameter, but (13) should have a sum over $k$ that is missing. Similarly, (14) should have a sum over $k$ if this expression is meant to be an entropy. Eq. (15) is too abstract and unclear.

In computing AMI, is the number of classes known? What is $T$? Sweeping over hyperparameter $\alpha$ is not realistic, especially if $\alpha$ is unnecessary (see above).

"Since none of [SimCLR, SwaV] are designed to output classes with a few examples, we use our online clustering procedure to readout from these learned representations:" This is unclear. As a result, the protocol of comparisons to offline methods it is unclear. A formal description is in order.

The entire argument against methods assuming iid batch sampling is reliance on iid for contrastive learning (SimCLR) or prototypes (SwaV). How about self-supervised representation learning without prototypes (e.g. BYOL: Grill et al., 2020) and without negatives (e.g. SimSiam: Chen & He, 2021)? Whouldn't such simple alternatives be much easier to adapt to online? Wouldn't those online versions be more suitable as baselines to compare with?

The following works should be discussed. Although I believe they are not directly comparable, one may still consider additional baselines, e.g. self-supervised offline pretraining followed by online learning (supervised or not):
1. Zhu et al. 2021, "Prototype Augmentation and Self-Supervision for Incremental Learning"
2. Gallardo et al. 2021, "Self-Supervised Training Enhances Online Continual Learning"
3. Zhang et al. 2021, "Self-Supervised Learning Aided Class-Incremental Lifelong Learning"
4. Cha et al. 2021, "Co$^2$L: Contrastive Continual Learning"

The list of hyperparameters (e.g. Tables 4-6) is daunting. One should at least separate the method-specific hyperparameters (e.g. $K, \rho, \alpha$) from standard hyperparameters (like backbone and learning rate). From Tables 8-14, there are at least 7 hyperparameters in the ablation. What are the default values of remaining hyperparameters when studying each one? Is this search optimal, what is the cost, and to what extent could same values be used across datasets? Merging Tables 4-6 into one would help compare.

The networks used are too small, more inline with few-shot learning and unlike most work on either self=supervised representation learning or continual learning.
 Online self-supervised learning is a very interesting and realistic setting. The online EM algorithm that maintains a set of prototypes makes a lot of sense.

There are a number of papers that are very related, most notably Bottou and Bengio 1995 (online $k$-means) and Lathuilière et al. 2018 (Gaussian-uniform mixture). Not only one would expect them to be cited and discussed, but rather the method formulation should be based on the formulation of this prior work. This would improve positioning of this work, make the formulation more elegant/better motivated and dispense the need for certain hyperparameters. The related work section needs improvement in general.

Writing is good in general, but the formulation in unclear at several points, relying on Appendices. Pointers to prior work are also missing in the formulation, primarily (Ren et al., 2021).

The protocol of comparisons to non-online methods is unclear. The choice of competitors is not well justified: there are simpler methods that are easier to adapt to online. There may be more baselines to consider e.g. on pretraining. There are too many hyperparameters. The networks used are too small.

This paper claims that in real world the data distribution is nonstationary, which is different from the current standard machine learning formulation. To solve this problem, the authors design an online unsupervised learning algorithm with Gaussian mixture model and EM algorithm. The experiment shows that the method can learn the online stream of visual input data. Part of the successes of deep learning comes from the well collected dataset, e.g., ImageNet10K. To study some area, someone should first to collect enough high quality data. However, this is different from the cognitive process of human being. This paper questions the standard machine learning formulation of drawing i.i.d. samples and claims that the real world scenario should be nonstationary. This is very interesting and encourging.

However, my main concern is that should we train our models in an online stream scenario just because the real world scenario nonstationary? You see, the tranditional learning formulation has its assumption that all samples are drawn in an i.i.d. way. But it doesn't violate the eventual performance. It works quite well.

I could understand that the author tries to mimic the learning process of human kind, just like lifelong learning. But the authors are encouraged to concentrate the necessity of this behaviour.

Besides, the language, the equations, are all above the thresold of the acceptance. In this paper, the author assumes a new scenario of nonstationary distribution. To solve the unsupervised and online setting, they design a novel online unsupervised prototypical networks (OUPN). By introducting the Gaussian mixture model and EM, the method could add or drop the cluster in an online way.

The idea is interesting and encouring, but the necessity is not well claimed. The difference between this method and life-long learning / incremental learning is encouraged. Also, the hyper-parameter of maximum number of K clusters should pay much more attention.

Meta-review: This paper tackles a small-batch online unsupervised learning problem, specifically proposing an online unsupervised prototypical network architecture that leverages an online mixture-based clustering algorithm and corresponding EM algorithm. Special features are added to deal specifically with the non-stationary distributions that are induced. Results are shown on more realistic streams of data, namely from the RoamingRooms dataset, and compared to existing self-supervised learning algorithms including ones based on clustering principles e.g. SWaV. 

Overall, the reviewers were positive about the problem setting and method, but had some concerns about hyper-parameters (hYzM, cvrN, LjvY) and motivation for the specific setting where the method excels compared to other methods not designed for such a setting (hYzM, cvrN), i.e. small-batch setting, where it is not clear where the line should be drawn in terms of batch size and memory requirements with respect to performance differences between the proposed approach and existing self-supervised methods. Importantly, all reviewers had significant confusions about all aspects of the work ranging from low-level details of the proposed method to the empirical setting and evaluation (including for competing methods). After a long discussion, the authors provided a large amount of details about their work, which the reviewers and AC highly appreciate. However, in the end incorporating all of the feedback requires a major revision of the entire paper. Even the reviewers that were more on the positive side (cvrN and LjvY) mentioned it would be extremely beneficial for this paper to be significantly revised and go through another review. Since so many aspects were confusing, it is not clear to the AC that the underlying method, technical contributions, and other aspects of the works had a sufficient chance to be evaluated fairly, given that much of the review period was spent on clearing up such confusion. 

In summary, while the paper is definitely promising and tackles an important area for the community, it requires a major revision and should go through the review process when it is more clearly presented. As a result, I recommend rejection at this point, since it is not ready for publication in its current form.

Decision: Reject