Title: Online Bayesian Transfer Learning for Sequential Data Modeling
Authors: Priyank Jaini
Zhitang Chen
Pablo Carbajal
Edith Law
Laura Middleton
Kayla Regan
Mike Schaekermann
George Trimponias
James Tung
Pascal Poupart
Venue: ICLR 2017
Keywords: Unsupervised Learning
Transfer Learning
Applications

Abstract: We consider the problem of inferring a sequence of hidden states associated with a sequence of observations produced by an individual within a population.  Instead of learning a single sequence model for the population (which does not account for variations within the population), we learn a set of basis sequence models based on different individuals.  The sequence of hidden states for a new individual is inferred in an online fashion by estimating a distribution over the basis models that best explain the sequence of observations of this new individual.  We explain how to do this in the context of hidden Markov models with Gaussian mixture models that are learned based on streaming data by online Bayesian moment matching.  The resulting transfer learning technique is demonstrated with three real-word applications: activity recognition based on smartphone sensors, sleep classification based on electroencephalography data and the prediction of the direction of future packet flows between a pair of servers in telecommunication networks. 


Review discussion:
Added comparison between online transfer learning technique and EM algorithm.
Can you provide some comparisons with the other Bayesian inference methods?
You should at least discuss the comparison of MAP-EM, Variational Bayes, and MCMC approaches.
These are applied to GMM/HMM and also have some online extensions.
Is the EM used in the paper variational EM, MAP EM, or ML EM?
This also has to be clarified.
Thank you for your question. The implementation of EM for which we reported results in the paper is ML EM. We will clarify this in the paper.
Note that the main goal of the paper is to propose transfer learning as opposed to the use of a single average model.  In proposing a Bayesian transfer learning technique, we agree that several approaches are possible, including MCMC, Variational Bayes, EM and Bayesian Moment matching (BMM).  We went ahead with BMM since Farheen Omar (PhD thesis Waterloo 2015) did an extensive comparison and obtained better results than Variational Bayes and EM.  BMM was also shown to work better than other techniques in the following papers (Abdullah Rashwan et al., AISTATS 2016, Wei-Shou Hsu et al. NIPS 2016, Priyank Jaini et al. PGM 2016).
In short BMM is naturally online and therefore does not require mini-batches.
In contrast, MCMC is difficult to run in an online fashion, Variational Bayes can run in an online fashion by creating mini-batches and decreasing the learning rate, however the size of the mini-batches and the decay procedure for the learning rate require some fiddling.  In general, the use of mini-batches always leads to some information loss since data in previous mini-batches is not accessible.  BMM does not suffer from this type of information loss and there are no such parameters to fiddle with.   We will add some text to that effect in the paper.
Since MCMC, EM, VB and BMM are all in the same category of Bayesian algorithms that can be used for transfer learning, we thought it would be more interesting to do a comparison to a different class of models such as RNNs.
Hence, we just completed an empirical comparison to RNNs and added this comparison to the paper.
In your introduction you say what you do, but you do not outline what you consider to be the fundamental contributions, in terms of what you introduce conceptually that goes beyond what people have done before. It may be a good idea, but it seems very close to many other ideas for transfer. What is your main claim?
Thank you for the question about novelty.  In the related work section, we clearly pointed out that the novelty is in the design of an *online* transfer learning technique in contrast to previous *offline* transfer learning techniques.  For sequential problems like activity recognition, sleep stage classification and network traffic prediction, there is a need to obtain a technique that can start making predictions for new individuals as soon as the sequence starts.
Hence we proposed the first online transfer learning technique (as far as we know).
In the second paragraph of the paper we clearly outlined the 3 contributions of the paper.  At the algorithmic level, our first contribution is the extension of the BMM algorithm to HMMs with Gaussian mixture emissions.  As described in the background section, BMM was previously proposed in the context of topic modelling.  The extension to HMMs with mixture of Gaussian emissions is new and non-trivial.  We described how to do this in the context of source domains.  The second contribution is the online unsupervised application of BMM to target domains, which enables transfer learning.   Here the novelty is the fact that we can do transfer learning in an online fashion in comparison to previous offline transfer learning techniques mentioned in the related work section.  The third contribution highlighted in the introduction consists of the three real world experiments.  This demonstrates the general applicability of the approach.
This paper looks at transfer learning on sequence.
First individual Bayesian moment matched algorithms are used on each individual.
Then for a test setting, the probabilities for the new domain are a similarity weighted set of probabilities from the training setting. The similarity weighting is given by the (approximate) posterior probability of the observation (either complete or so far, depending on whether an online scheme is used) for each domain.
A few comments on the model.
First there is a discrepancy between train and test: in the test domain there is an assumption in some sort of relationship between individuals, but at training all individuals are treated independently. This contrasts with models that try to analyze between-subject and within-subject variation inherently in the training data. A discussion of these points would be valuable. Does this assume the data about individuals is extensive, so such sharing is not necessary?
The Bayesian posterior on lambda and pi provides a means of model averaging.
But weighted averaging of models is very different from weighted averaging of transition probabilities. Given this discrepancy, it would have been good to have a bit more discussion about this choice. Clearly it is pragmatic, but what do you lose and what do you gain? Does this fit into the moment matching interpretation?
Of course parameterized sharing is common in multitask settings, but more could be said about this.
Experiments: This seems to be the real point of the paper: the authors have an actual problem to solve and developed this as a method to do this.
Yet from the title it feels like the authors felt they had to bill it as a methodological paper for submission to ICLR. Personally I think it is unfortunate that this was a perceived need. A demonstration of what can be transferred in a domain like this is as important as how it is done. The experiments are on a valuable real world problem that people widely care about. This is the real strength of this paper, and a focus on this demonstrative aspect, and a corresponding conclusion would strengthen the paper.
Thank you for your feedback and comments.
Comment 1 : "A few comments on the model ....
so such sharing is not necessary?"
Perhaps there is a misunderstanding.
When we train a separate model for each source individual, we are simply trying to capture the characteristics of each individual.  At test time, all those models are "shared" or taken into account to obtain a better prediction for a new target individual.  The fact that we are not sharing models when training with the source individuals does not mean that we are not taking into account subject variability.  To the contrary, this helps to take into account subject variability since each individual is represented in a different model.
In contrast, when training a monolithic model for the entire population, subject variability is much harder to retain since the model tends to capture an "average" individual and therefore does not perform as well on future target individuals.
Comment 2 : "The Bayesian posterior on lambda and pi ....
but more could be said about this."
Our approach is in between pure model averaging and pure parameter averaging.  Model averaging would mean that we take a weighted combination of the HMMs of each source individual while parameter averaging would mean that we take a weighted combination of each parameter for each individual.
Instead, we do something in between where we take a weighted combination of the transition models of each source individual and we also take a separate weighted combination of the emission models of each source individual.  Ideally, we would like to do pure model averaging since the idea is to find the most relevant source individuals and use them to make predictions for the target individual.   The problem is that a weighted combination of HMMs does not yield an HMM.
However we can take a weighted combination of conditional distributions to obtain a new conditional distribution.  Since the transition and emission models in an HMM are conditional distributions, we can take separate weighted combinations of transition and emission models of the source individuals to obtain a new transition model and a new emission model.
Comment 3 : Experiments and positioning of the paper
Thank you for the positive comments regarding the experiments.  Note that this paper makes contributions both at the methodological and experimental level.  The reality is that we had to design a new approach to obtain an online transfer learning technique (in contrast to previous transfer learning techniques that are offline).  The introduction clearly states the 3 contributions of the paper (as outlined in our response to a question below).
I'd like the authors to comment on two questions:
(1) What are the the parameter storage and test time computation complexities of this approach, and how does it scale to "big" training data (i.e., lots of source domains) and large label spaces (perhaps unlikely in most applications)?
(2) What do you think the ICLR community's primary interest in this work would be (beyond "we can beat RNNs in a low data, transfer sequential learning setting")? Why did you decide to submit it to ICLR rather than, e.g., AISTATS or UAI, which prima facie seem like better fits?
Thank you for the questions. We have added a reply for both the questions in our comment to the full review above.
I'm bumping up my score to a 7 to acknowledge that the authors responded satisfactorily to reviewer feedback, and to indicate that I think that the updated manuscript is a strong paper and that I do not object to its acceptance to ICLR.
However, I also would not fight for its acceptance. I still think that it is a better fit for a venue with an explicit interest in more traditional Bayesian latent variable models (ICML, UAI, NIPS).
This manuscript describes a novel Bayesian approach to transfer learning focused on online sequence modeling settings where he primary concern is less distribution drift in an ongoing sequence and more variability between individual sequences (since each sequence can be thought of as defining its own conditional distribution over subsequent states). They provide the example of human gait classification, where each individual's gait may differ from others even while performing the same activity. In this setting we typically train the gait model on gait sequences from a set of "source" individuals but then apply it to previously unseen people.
The core model is an HMM, which they give a full Bayesian treatment; the central problem this introduces is that each new observation that arrives introduces an additional product term to the posterior that is itself a product over M components (clusters or hidden states). This rapidly becomes intractable. This paper applies Bayesian moment matching (BMM) to this problem, in which the posterior is approximated via projection onto a more tractable distribution that is adjusted to match some moments of the posterior. The experimental results are quite promising
Strengths:
- The problem setting is very appropriately framed as online sequence prediction via Bayesian transfer learning. There is almost certainly individual variability between the training sequences (for which explanatory variables not be available in the inputs). The Bayesian approach gives a natural approach to performing transfer and handling low data regimes (common in all experiments).
- The Bayesian formulation creates a computational challenge (the posterior becomes intractable). The proposed BMM approximation is both reasonable and relatively novel (particularly given the popularity of MCMC and variational methods).
- With the caveat that I am not well-versed in recent work on Bayesian moment matching, a cursory literature search suggests this is a novel application of BMM. A lot of the related BMM work is roughly contemporaneous with this work, and none of it seems concerned with online transfer learning.
- The overall results look quite strong: in activity recognition and flow prediction, the transfer learning approach is in general superior to the one-size-fits-all HMM, even when trained using BMM (which in turn is generally superior to the EM-based HMM). The proposed approach appears to beat the LSTM across all tasks (including sleep stage classification), possibly due to the lack of training data.
Weaknesses:
- The description of the LSTM training and architecture search is vague (and in one instance, contradictory), strongly implying that it was not fully tuned and may be an artificially weak baseline.
While it is plausible that the proposed approach might excel given the small data sets used in the experiments, there is not sufficient evidence and detail to support this claim. The authors should provide more detail about architecture search, hyperparameter tuning, and most important, attempts at regularization, given the limited training data.
In particular, the authors should experiment with a sufficiently rich set of settings for # hidden layers, # hidden units, weight decay, and dropout.
- Given the emphasis on transfer learning and the use of a Bayesian framework, the decision to train source models independently is a little odd. Why not perform some kind of joint training?
- The authors provide no analysis, analytical or empirical, of the proposed framework's (storage and computational) complexity, especially at prediction time. At the top of page 10, they mention using only one EEG channel in order to "reduce complexity and processing time." This is an ominous hint that the proposed framework may not scale practically.
- Additionally, I have a meta-concern about this paper's fit for ICLR. "Representation learning" -- the general theme of ICLR -- is not featured prominently in this work. Given the competitive nature of ICLR, we should consider seriously whether this paper is of interest to the wider ICLR community or whether it might be a better fit for a meeting such as AISTATS, ICML, or UAI.
Comments:
- The plots in Figures 1-3 are difficult to interpret. Putting patients along the X-axis is unintuitive since their order is arbitrary. Why not just make scatter plots of one vs. the other model's accuracy. The shape of the scatter should hopefully make it clear if there is a general trend.
- In Experimental Setup, the authors make conflicting claims about the LSTM architecture. They first state the single hidden layer has number of cells equal to the number of inputs. They then say that the number of LSTM units is finetuned based on empirical performance.
- While generally well-written, the paper has several obscenely long paragraphs. The single paragraph "Experimental Setup" section, for example, takes up more than half of page 8. These should be broken up into shorter paragraphs to make them easier to read. A good rule of thumb is that no paragraph should take up more than ~1/6 of a page.
In general, I like this work, but the vague details around the LSTM training raise serious red flags about their experimental results, at least the comparison vs. LSTMs, and I have concerns about how well it meets ICLR's CFP.
That said, my policy for interactive review is to carefully consider author responses with an open mind, so I will serious consider changing my score, if warranted.
We thank the reviewer for the positive comments and detailed feedback. It has helped us to make the manuscript better. We address the concerns of the reviewer below. The question on the RNN architecture will be answered by one of the co-authors who designed the experiment.
- "Given the emphasis on transfer learning and the use of a Bayesian framework, the decision to train source models independently is a little odd. Why not perform some kind of joint training?"
In principle, joint training of the source models and their weighted combination for prediction with the target individual could be done.
However, note that we are in an online setting where there is a need to make predictions as data from the target individual arrives.
Hence, it would be prohibitive to do this type of joint training.  This is why we train the source models offline and then train the weighted combination of those models online as the data from the target individual arrives.
- "The authors provide no analysis, analytical or empirical, of the proposed framework's (storage and computational) complexity, especially at prediction time."
We have now added an analytical analysis of the proposed framework’s computational complexity for both the training phase and the testing phase (which includes learning step and prediction step). The computational complexity analysis section has been added at the end of section 4.2 for the source domain learning phase and at the end of section 4.3 (page 8 beginning) for the target domain phase which includes the update step for the weights of the basis model and prediction.
- "At the top of page 10, they mention using only one EEG channel in order to "reduce complexity and processing time." This is an ominous hint that the proposed framework may not scale practically."
We apologize for the lack of clarity here. The phrase "reduce complexity and processing time." was with respect to the feature extraction and manual labelling step. For each patient, a sequence of 8 hours of sleep data was obtained which is then manually scored by a single registered PSG technologist.
Further, the preprocessing step involves feature extraction from the raw data which is very expensive if multiple channels are considered. This was what we were referring to when we said that we only consider one EEG channel in order to reduce complexity and processing time. It was not a comment in the context of the online transfer learning algorithm’s performance. We have now revised the paper to clarify that the reduction in complexity and processing time were for feature extraction and manual labeling.
- "Additionally, I have a meta-concern about this paper's fit for ICLR …. for a meeting such as AISTATS, ICML, or UAI."
The call for papers is about representation learning.  In this paper we show how to learn a representation that consists of a set of basis models.
Then as data from the target individual arrives, the system further learns a weighted combination of those representations in order to make online predictions.  If the concern is that the paper does not propose a new neural network representation, the call for paper clearly says that “learning representation” should be viewed broadly.
=== Minor Comments ===
- “The plots in Figures 1-3 are difficult to interpret. Putting patients along the X-axis is unintuitive since their order is arbitrary. Why not just make scatter plots of one vs. the other model's accuracy. The shape of the scatter should hopefully make it clear if there is a general trend.”
Thank you for the suggestion. We agree that the current plots were unintuitive to a certain extent. We chose to plot it in that manner to retain a comparison for each patient. We have now updated the paper and included the scatter plots for all the algorithms. The current analysis has now been relegated to the Appendix section in case a reader wants more details.
- “In Experimental Setup, the authors make conflicting claims about the LSTM architecture. They first state the single hidden layer has number of cells equal to the number of inputs. They then say that the number of LSTM units is fine-tuned based on empirical performance.”
There seems to be a misunderstanding about the number of hidden units in the RNNs.  We did not write that "a single hidden layer has [a] number of cells equal to the number of inputs".
Instead we wrote "we used architectures as many input nodes as the number of attributes".  Here the attributes as simply the features/inputs.
Hence the number of LSTM cells is independent of the number of input units.  As described in the paper we perform grid search to optimize over the various hyper-parameters including the number of LSTM cells.
- “While generally well-written, the paper has several obscenely long paragraphs …. more than ~1/6 of a page.”
Thank you for the suggestion. We agree that some paragraphs were very long negatively impacting the easy readability of the paper. We have now addressed this in the recent updated version of the paper we uploaded today.
- “In general, I like this work, but t….That said, my policy for interactive review is to carefully consider author responses with an open mind, so I will serious consider changing my score, if warranted.”
We would be happy to address any other concerns the reviewer may have. We also hope we were able to address all the concerns of the reviewer satisfactorily.
LSTM optimization:  There seems to be a misunderstanding about the goal of the paper.  The paper is not about "beating RNNs in a low data, transfer sequential learning setting".  This is a rather narrow view.  The paper is about online transfer learning and the need to take into account subject variability.  We demonstrate that our approach works better than baselines that do not do transfer learning.  It happens that one of the baselines is an RNN, but we also included BMM-HMM and EM-HMM as other baselines that are beaten.  Note also that among the baselines, the RNN performed better than BMM-HMM and EM-HMM for sleep stage classification and network flow direction prediction.  We just uploaded a new version of the paper where we explain in more detail how we optimized the hyperparameters of the RNN.
However, hyperparameter optimization does not address subject variability and does not perform transfer learning.  This is why RNNs (and the other baselines) did not perform as well as our transfer learning technique.  We believe that transfer learning is possible with RNNs and suggested that as future work in the conclusion section.
Note also that hyperparameter optimization is suitable in offline techniques when one is interested in obtaining the best possible results for one very specific application.  I work in an industrial research lab (Huawei Noah's Ark Lab) where we are designing algorithms that need to work well across a range of applications (as demonstrated in the paper).
So it doesn't make sense to optimize the parameters for one application only.
Furthermore, in online settings, there is no opportunity to perform an offline search over the hyperparameters.
There seems to be a misunderstanding about the number of hidden units in the RNNs.  We did not write that "a single hidden layer has [a] number of cells equal to the number of inputs".
Instead we wrote "we used architectures as many input nodes as the number of attributes".  Here the attributes as simply the features/inputs.
Hence the number of LSTM cells is independent of the number of input units.  As described in the paper we perform grid search to optimize over the various hyperparameters including the number of LSTM cells.
-"The description of the LSTM training and architecture search is vague (and in one instance, contradictory), strongly implying that it was not fully tuned and may be an artificially weak baseline."
We perform grid search to select the best hyper-parameters for each setting. For the training method, we either use Nesterov's accelerated gradient descent with learning rates [0.001,0.01,0.1,0.2] and momentum values [0,0.2,0.4,0.6,0.8,0.9], or rmsprop having epsilon=10^{-4} and decay factor 0.9 (standard values)  with learning rates [0.00005,0.0001,0.0002,0.001] and momentum values [0,0.2,0.4,0.6,0.8,0.9]. The weight decay takes values from [0.001,0.01,0.1], whereas the number of LSTM units in the hidden layer takes the possible values [2,4,6,9,12].
We experimented with various architectures before we ended up with the aforementioned values;
in particular, architectures with a single hidden layer consistently performed better than multiple layers, possibly because our datasets are not very complex. We train the network by backpropagation through time (bptt) truncated to 20 time steps.
We have also added these details in the paper in the Experiments section on page 8.
Just wanted to call this to the attention of the meta-reviewer.
This paper proposes an online inference algorithm by using online Bayesian moment matching for HMM-GMM. The method uses transfer learning by utilizing individual sequence estimators to predict a target sequence based on a weighted combination of individual HMM-GMM. Online Bayesian moment matching has a benefit of updating HMM-GMM parameters frame-by-frame, and fits to this problem. The authors compare the proposed method with the other sequential modeling methods including RNN and EM, and show the effectiveness of the proposed method. The paper is well written overall.
Comments:
1) Could you provide the average performance in table? It is difficult to compare the performance only with individual performance.
Also, it seems that the EM performance is sometimes good
2) I’m curious how initialization and hyper-parameter settings affect the final performance. If you provide some information about it, that is great.
3) It would be better to provide a figure of describing the transfer-learning-based proposed methods, since this is a unique and a little bit complicated setup.
Thank you for your comments and valuable feedback. Please take a look at our replies to each of your questions below :
1a) "Could you provide the average performance in table? It is difficult to compare the performance only with individual performance."
There seems to be a misunderstanding with respect to the comment on the average performance in table. As we mentioned in the paper, in the experiments section, for each individual, each algorithm was run 10 times and the values reported in the paper are the average accuracy achieved by each algorithm over those 10 runs.
Further, all the results are statistically significant under the Wilcoxon signed rank test with p-value < 0.05.
Therefore, the values reported in the tables and figures are the average performance of each algorithm
The main aim of the paper was to propose an online transfer learning technique. To measure the performance of an algorithm for transfer learning technique we need to train on a population and then transfer that learning to make predictions on a new individual. In this paper, we did experiments for three domains where we analyzed performance using leave one out method.
Therefore, we report the performance of the algorithms on each individual of a population since that is the natural setting.
1b) "Also, it seems that the EM performance is sometimes good "
We do not believe that EM performed well as compared to the online transfer learning method.
In fact, over the 3 experiment settings ranging over 170 individuals (19 for activity recognition, 142 for sleep stage classification and 9 for network traffic prediction), EM outperformed the online transfer learning technique just *once* (source 2 for network traffic prediction).
While, it's performance has been significantly worse in many cases in comparison to online transfer learning technique.
"2) I’m curious how initialization and hyper-parameter settings affect the final performance. If you provide some information about it, that is great."
For the Bayesian Moment Matching (BMM) algorithm and its variant for sequential learning, the only hyper-parameters that need initialization is the prior.  We choose the hyper-parameters for the prior randomly right now.
But, we can have a more data driven approach for the prior as well.
However, the performance is robust to a good degree for a reasonably chosen prior. Previous works as mentioned in the related section on BMM demonstrate this with experiments.
Further, the other parameters that needs to specified by the user are the number of hidden states and the number of components for the emission distribution.
"3) It would be better to provide a figure of describing the transfer-learning-based proposed methods, since this is a unique and a little bit complicated setup."
Thank you for this suggestion. In the revised draft we uploaded on Dec 28, we have included a schematic for the architecture of the proposed online transfer learning technique.
We hope we have satisfactorily addressed your concerns and would be happy to clarify any further questions.

Meta-review:
role: PC
data: comment: Though the method does not seem to really break new ground in transfer learning (see Reviewer 1), the reviewers do not question the validity of the approach. The online aspect of the approach as well as an application of Bayesian moment matching to HMMs with GM emissions seem novel. Though the topic may seem a bit peripheral within ICLR, I agree that it can be considered as a representation learning method and the divergence from the mainstream (within ICLR) may be as much a pro as a con. Also, on the positive side, the applications are interesting and real, and the methods seem well suited to the task. The paper is well written.
 
 + well-written
 + technically solid
 + interesting application
 
 - innovation is moderate 
 
 +/- not a typical ICLR paper

Decision: Accept (Poster)