Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: A major concern about the paper is related to the unsupported claims and contribution throughout the paper. For example, the way the training copes with distribution shift or alleviate forgetting is not clear or elaborated on. Beyond the abstract and before the empirical validation no theory or justification is provided to substantiate this claim. The idea of the paper and the motivation are very interesting. The experiments look convincing. Writing and presentation are a good start point for improving the paper. 

review_title: The paper proposes a SGD based method to learn GMM in non-stationary and high dimensional setting. The paper is tackling an interesting problem however the contributions of the paper is not clearly supported.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a technique to training GMM using SGD instead of (s)EM. The major contributions are:
1. a proposal for numerically stable GMM training by SGD
this is achieved with max-component log-likelihood approximation and an annealed process to smooth the SGD.
2. an automatic annealing procedure that ensures SGD convergence from a wide range of initial conditions without prior knowledge of the data (e.g., no k-means initialization) which is especially beneficial for high-dimensional data
Section 4.4 has shown that such annealing hyper-parameter can control re-learning and retention process, which is good.
However, section 4.1 has shown that with or without random initialization does not impact the perf too much, then why not just use k-means initialization? is it very costly? I believe k-means initialization is also a randomized process.

However, there are some issues:
1. When comparing SGD vs sEM, two strong assumptions are made: 
1) SGD annealing has converged and 
2) GMM responsibilities are sharply peaked so that a single component has responsibility of around 1
It basically requires the data does not have a lot of noise, where each point can be assigned with to a label with a dominating probability, so what happens if the data has some noise, and how will such solution reacts for different level of noise?
A second question is: is there any theory or bounds to support the convergence assumption? all experiments do SGD for 2 or 3 epochs, how will the loss be like after 3 epochs?
2. No actual examples or discussions are given in terms of numerical instabilities when data dimension is high
3. For section 4.3, the streaming scenario is not clear, is mini-batch size constant? random? if fed data of one sample, is the SGD stilling running for 2 epochs? or halting and wait?
review_title: Add annealing feature so that SGD can solve max-component log-likelihood approximation of GMM, but still need more discussion in depth

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a new approach to train GMMs using SGD under a variety of settings (streaming, concept drift, etc) addressing the issues of catastrophic forgetting, problem of parameter initialization and numerical instability.

These are all important and interesting challenges that can help advance the training of latent variable models in general. However I feel that paper is not presented with sufficient clarity to pass the ICLR bar and the exposition could be greatly improved. It is hard for me to grasp the key findings or takeaways in the paper wrt to other existing baseline methods.

I also have some specific comments:

- The contributions sub-section in 1.3 are vague to read, instead of simply saying - "a novel method", "an automatic annealing procedure" it would be useful to explicitly state what is novel / automatic etc - this would make it easier for the reader to understand the novelty/exact technical contributions made by the authors

- In section 3.2, I am curious how the max-component approximation lower bound compares to the other lower bounds used in EM (e.g. Jensen derived lower bound or evidence lower bound in Variational Inference)? To me the presented lower bound seems like inefficient and loose (e.g. if the values are close in magnitude, then intuitively the gap between the sum and max is going to be very large). Maybe I am mistaken in understanding this bound, but it would need some more explanation and it should be contrasted with the vanilla lower bounds used in EM.

- In section 3.3, the authors mention updating a subset of components to break the symmetry. I am curious how a baseline of simply perturbing the GMM parameters randomly would perform since that would also help break the symmetry?

- I liked the idea of using annealing to avoid local optima and this seems to be one of the key contributions of this work in my opinion. One of my main questions here is: how did the authors measure the value addition of annealing? Did they compare the final solutions obtained by the proposed approach with a baseline (without any annealing) that uses random initialization to deal with local optima?

- Notations in the paper could be improved further to make it more readable. For e.g. in Algorithm 1, the iteration steps are missing in the updates (would be good to use t=1, 2, ..T and use them in the update equations since it is not clear which iterate time steps the parameters on the RHS belong to). Also, maybe I missed this while reading but prec_clipping(..) doesn't seem to be defined near the algorithm section. 

- I found some vague statements in the Empirical section which raises lot of follow up questions. Would suggest making the description and analysis of the results more precise. e.g. In the beginning of section 4, authors say: ".. repeated 10 times with identical parameters.." -> what does identical mean? what was varied?




review_title: Paper investigates some key research questions in training mixture models (GMMs) but lacks clarity

final_score: rating: 5: Marginally below acceptance threshold
review_content: It is clear that efficiectly training Gaussian Mixture Models with SGD on non-stationary, high-dimensional streaming data is very important for practical applications. The aim of this paper is quite good.  In fact, this paper proposes an annealing mechanism for the SGD algorithm and makes the experiments to compare the proposed training algorithm with sEM algorithm on several real-world datasets.  However, I have following major concerns: 
(1). The proposed training scheme does not require data-driven parameter initialization (e.g., k-means) , but the data-driven parameter initialization can improve the efficiency. So, I cannot consider this is an advantage of the proposed scheme. 
(2).  The  proposed annealing scheme is straightforward，and there is no deep analysis on its performance.
(3).  According to the experimental results, I cannot find out that the proposed training scheme is remarkably better that sEM. By the annealing procedure,  the clustering results should be improved much better.
(4).  The authors claim that the proposed training scheme is  good for non-stationary, high-dimensional streaming data, but there are not much analytic results with the dimenionality and non-stationary streaming data.  The forgetting  results by controlling the annealing parameter is too rough. 
review_title: This paper proposes an annealing stochastic gradient descent (SGD) approach for efficiently training Gaussian Mixture Models  on non-stationary, high-dimensional streaming data. Although it is well organized and the idea is acceptable and the traning process is clearly described, I still concern its originality and effectiveness. Moreover, its functions for non-stationary, high-dimensional streaming data are not analyzed and tested deeply.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presented a stochastic gradient descent approach to learn a non-stationary high-dimensional Gaussian mixture model from online data. The authors identified 3 challenges - local optima, numerical instability, and catastrophic forgetting, and proposed to address these challenges respective with adaptive annealing, exponential-free approximation, and adaptive SGD learning rate. The proposed approach is demonstrated with several vision/non-vision tasks.

Overall, I feel that the paper is slightly below the borderline. There lacks some theoretical analysis of the proposed ideas, an approach to identify the number of mixture components, and an argument as to why GMM is preferred instead of other representation learning techniques.

Pros:

+ Interesting combination of new research trends (continual learning) and old models (GMM).

Cons:

- Lack of an approach to identify the number of mixture components
- Lack of theoretical justification about the max-component approximation and soft max-component approximation.
- Lack of demonstration on why catastrophic forgetting is avoided and how nonstationary data affects this and other algorithms in experiments.

So the following improvements could improve my ratings: 

- An empirical analysis of how the proposed approach avoids catastrophic forgetting with nonstationary data, and a theoretical analysis/comparison between existing approaches, such as regularization, memory replay, and network morning.
- A mathematical justification about the soft max-component approximation: how it is related the classical GMM, to what extent it is an approximation.
- A theoretical or emprical approach to adapt the number of mixture components, for example, through introducing a prior. 

review_title: online gaussian mixture model learning with sgd and smoothed max-component log-likelihood

Meta-review: This paper proposes training Gaussian mixture models using SGD, creating an algorithm appropriate for streaming data. However, we feel that the current manuscript does not sufficiently support the proposed method, and lacks insight into its workings. The reviewers believed the method lacked justification (while the authors claim to have added theoretical justification to the revised manuscript, I did not see any such new theory), and were not convinced that the method offered a significant improvement on existing methods. 
Decision: Reject