Title: ONLINE BAYESIAN TRANSFER LEARNING FOR SEQUENTIAL DATA MODELING

Abstract: We consider the problem of inferring a sequence of hidden states associated with a sequence of observations produced by an individual within a population. Instead of learning a single sequence model for the population (which does not account for variations within the population), we learn a set of basis sequence models based on different individuals. The sequence of hidden states for a new individual is inferred in an online fashion by estimating a distribution over the basis models that best explain the sequence of observations of this new individual. We explain how to do this in the context of hidden Markov models with Gaussian mixture models that are learned based on streaming data by online Bayesian moment matching. The resulting transfer learning technique is demonstrated with three real-word applications: activity recognition based on smartphone sensors, sleep classification based on electroencephalography data and the prediction of the direction of future packet flows between a pair of servers in telecommunication networks.

1. INTRODUCTION:

In several application domains, data instances are produced by a population of individuals that exhibit a variety of different characteristics. For instance, in activity recognition, different individuals might walk or run with different gait patterns. Similarly, in sleep studies, different individuals might exhibit different patterns for the same sleep stages. In telecommunication networks, software applications might generate packet flows between two servers according to different patterns. In such scenarios, it is tempting to treat the population as a homogeneous source of data and to learn a single average model for the population. However, this average model will perform poorly in recognition tasks for individuals that differ significantly from the average. Hence, there is a need for transfer learning techniques that take into account the variations between individuals within a population.

We consider the problem of inferring a sequence of hidden states based on a sequence of observations produced by an individual within a population. Our first contribution is an online Bayesian moment matching technique to estimate the parameters of a hidden Markov model (HMM) with observation distributions represented by Gaussian mixture models (GMMs). This approach allows us to learn separate basis models for different individuals based on streaming data. The second contribution is an unsupervised online technique that infers a probability distribution over the basis models that best explain the sequence of observations of a new individual. The classification of hidden states can then be refined in an online fashion based on the individuals that most resemble the new individual. Furthermore, since the basis models are fixed at classification time and we only learn the weight of each model, good classification accuracy can be obtained more quickly as the stream of observations of the new individual are processed. The third contribution of this work is the demonstration of this approach across different real-world applications, which include activity recognition, sleep classification and the prediction of packet flow direction in telecommunication networks.

The paper is organized as follows. Section 2 reviews some related work on transfer learning. Section 3 provides some background regarding hidden Markov models Bayesian Moment Matching algorithm Gaussian mixture models. Section 4 describes the proposed online transfer learning technique. Section 5 illustrates the transfer learning technique in three real-world tasks: activity recognition, sleep stage classification and flow direction prediction. Finally, Section 6 concludes the paper and discusses directions for future work.

2. RELATED WORK:

There is a large literature on transfer learning (Pan & Yang, 2010; Taylor & Stone, 2009; Shao et al., 2015; Cook et al., 2013) . Depending on the problem, the input features, the output labels or the distribution over the features and the labels may be different for the source and target domains. In this work, we assume that the same input features are measured and the same output labels are inferred in the source and target domains. The main problem that we consider is subject variability within a population of individuals, which means that different individuals exhibit different distributions over the features and the labels. The problem of subject variability has been studied in several papers. Chieu et al. (2006) describe how to augment conditional random fields with a subject hidden variable to obtain a mixture of conditional random fields that can naturally infer a distribution over the closest subjects in a training population when inferring the activities of a new individual based on physiological data. Rashidi & Cook (2009) proposed a data mining technique with a similarity measure to facilitate the transfer of activity recognition across different people. Chattopadhyay et al. (2011) describe a similarity measure with an intrinsic manifold that preserve the topology of surface electromyography (SEMG) while mitigating distributional differences among individuals. Zhao et al. (2011) proposed a transfer learning technique that starts by training a decision tree to recognize the activities of a user based on smartphone accelerometry. The decision tree is gradually adjusted to a new user by a clustering technique that successively re-weights the training data based on the unlabeled data of the new individual. These approaches mitigate subject variability by various offline transfer learning techniques. In contrast, we propose an online transfer learning technique since the applications that we consider exhibit sequences of observations that arrive in a streaming fashion and therefore require an online technique that can infer the hidden state of each observation as it arrives.

In the next section, we describe an online transfer learning technique for hidden Markov models with Gaussian mixture models. The approach learns different transition and emission models for each individual in the training population. Those models are then treated as basis models to speed up the online learning process for new individuals. More specifically, a weighted combination of the basis models is learned for each new individual. This idea is related to boosting techniques for transfer learning (Dai et al., 2007; Yao & Doretto, 2010; Al-Stouhi & Reddy, 2011 ) that estimate a weighted combination of base classifiers. However, note that we focus on sequence modeling problems where the classes of consecutive data points are correlated while transfer learning by boosting assumes that the data points are identically and independently distributed.

3. BACKGROUND:

In this section, we give a brief overview of hidden Markov models (HMMs) and review the Bayesian moment matching (BMM) algorithm in detail with an example. We will use both HMMs and BMM subsequently in our transfer learning algorithm described in Section 4.

3.1. HIDDEN MARKOV MODELS:

In a hidden Markov model (HMM), each observation X t is associated with a hidden state Y t . The Markov property states that the current state depends only on the previous state. HMMs have been widely used in domains involving sequential data like speech recognition, activity recognition, natural language processing etc. An HMM is represented by two distributions

• Transition distribution: The transition distribution models the change in the value of the hidden state over time. The distribution over the current state Y t given that the previous state is Y t-1 = j is denoted by θ j = Pr(Y t |Y t-1 = j) where θ j = {θ 1j , ..., θ N j }, N is the total number of states and θ ij = Pr(Y t = i|Y t-1 = j). • Emission distribution: The emission distribution models the effect of the hidden state on the observation X t at any given time t and is given by Pr(X t |Y t ). In this work, we model the emission distribution as a mixture of Gaussians with M components, i.e., Pr(X

t |Y t = j) = M i=1 w i N X t ; µ j i , Σ j i

In this paper, we will first estimate the parameters of the transition and emission distributions by Bayesian learning from a set of source domains (individuals). Subsequently, we will use these distributions as basis functions when estimating the transition and emission distributions of a target domain in which we wish to predict the hidden state for each observation. Parameter learning of an HMM using Bayesian learning is done by calculating the posterior over the parameters given a prior distribution.

Pr Θ, Φ, Y t = j|X t , Y t-1 = i ∝ Emission distribution Pr(X t |Y t = j) Transition Probability Pr(Y t = j|Y t-1 = i) Prior for t -1 Pr(Θ, Φ, Y t-1 = i|X 1:t-1 )

∀j ∈ {1, 2, ..., N } where Θ and Φ parametrize the transition and emission distributions respectively.

3.2. BAYESIAN MOMENT MATCHING ALGORITHM:

The Bayesian moment matching (BMM) algorithm for Gaussian Mixture Models was proposed by Jaini & Poupart (2016) ; Jaini et al. (2016) . Exact Bayesian learning of mixture models based on streaming data is intractable because the number of terms in the posterior after observing each observation increases exponentially. BMM circumvents this issue by projecting the distribution of the exact posterior P on a tractable family of distributions P by matching a set of sufficient moments.

In this section, we give a brief overview of the BMM algorithm with an example.

Note that Variational Bayes (VB) and Markov Chain Monte Carlo (MCMC) techniques can also be used for approximate Bayesian learning as an alternative to BMM. However, MCMC is difficult to run in an online fashion. A recent comparison by Omar Omar (2016) showed that BMM achieves better results than online Vational Bayes (oVB) Sato (2001) and Stochastic Variational Inference (SVI) Wang et al. (2011) in the context of topic modeling. BMM was also shown to work better than other online techniques in several papers Rashwan et al. (2016) ; Hsu & Poupart (2016) ; Jaini et al. (2016) . This is due to the fact that BMM is naturally online and therefore does not require minibatches. In contrast, in order to run in an online fashion Variational Bayes requires mini-batches and a decreasing learning rate, however the size of the mini-batches and the decay procedure for the learning rate require some fine tuning. In general, the use of mini-batches always leads to some information loss since data in previous mini-batches is not accessible. BMM does not suffer from this type of information loss and there is no batch size nor learning rate to fine tune. Hence, we will adapt BMM to transfer learning in this work.

Let X 1:n be a set of d-dimensional i.i.d observations following

Pr(X|Θ) = M i=1 w i N x; µ i , Λ -1 where Θ = {(w 1 , µ 1 , Λ -1 1 ), (w 2 , µ 2 , Λ -1 2 ), ...(w M , µ M , Λ -1 M

)} and M is known. We choose the prior as a product of a Dirichlet distribution over the weights w and M Normal-Wishart distributions corresponding to the parameters (µ, Λ -1 ) of each Gaussian component. Such a prior forms a conjugate probability pair of the likelihood and is hence desirable. Concretely, P 0 (Θ) = Dir(w|α)

M i=1 N W(µ i , Λ i |δ i , κ i , W i , ν i ) where w = (w 1 , w 2 , ..., w M ), α = (α 1 , α 2 , ..., α M ), W is a symmetric positive definite matrix, κ > 0 is real, δ ∈ R d and ν > d -1 is real.

The posterior P 1 (Θ|X 1 ) after observing the first data point X 1 is given by

P 1 (Θ|X 1 ) ∝ P 0 (Θ) Pr(X 1 |Θ) ∝ Dir(w|α) M i=1 N W(µ i , Λ i |δ i , κ i , W i , ν i ) M j=1 w j N X 1 ; µ j , Λ -1 j

Since a Normal-Wishart distribution is a conjugate prior for a Normal distribution with unknown mean and precision matrix, N W(µ i ,

Λ i |δ i , κ i , W i , ν i )N X 1 ; µ i , Λ -1 i = cN W(µ i , Λ i | δi , κi , Ŵi , νi )

where c is some constant. Similarly, w j Dir(w|α 1 , α 2 , .., α j , .., α M ) = kDir(w 1 , w 2 , ..., w M |α 1 , α 2 , ..α j .., α M ) where k is some constant. Therefore, P 1 (Θ|X 1 ) is

P 1 (Θ|X 1 ) = 1 Z M j=1 c j Dir(w| αj )N W(µ j , Λ j | δj , κj , Ŵj , νj ) M i =j N W(µ i , Λ i |δ i , κ i , W i , ν i )

where αj = (α 1 , α 2 , .., αj , .., α M ) and Z is the normalization constant. The equation above suggests that the posterior is a mixture of product of distributions where each product component in the summation has the same form as that of the family of distributions of the prior P 0 (Θ). It is evident that the terms in the posterior grow by a factor of M for each iteration, which is problematic.

The Bayesian moment matching algorithm approximates this mixture P 1 (Θ) with a single product of Dirichlet and Normal-Wishart distributions P1 (Θ) by matching all the sufficient moments of P 1 with P1 which belongs to the same family of distributions as the prior:

P1 (Θ) = Dir(w|α 1 ) M i=1 N W(µ i , Λ i |δ 1 i , κ 1 i , W 1 i , ν 1 i )

We evaluate the parameters

α 1 , δ 1 i , κ 1 i , W 1 i , ν 1 i ∀i ∈ {1, 2, .

., M } by matching a set of sufficient moments of P1 (Θ) with P 1 (Θ). The set of sufficient moments in this case is S = {µ j , µ j µ T j , Λ j , Λ 2 j km , w j , w 2 j } ∀j ∈ 1, 2, ..., M where Λ 2 j km is the (k, m) th element of the matrix Λ j . The expressions for sufficient moments are given by E[g] = Θ gP 1 (Θ)d(Θ). The parameters of P1 can be computed from the following set of equations

E[w i ] = α i j α j ; E[w 2 i ] = (α i )(α i + 1) j α j 1 + j α j E[Λ] = νW; V ar(Λ ij ) = ν(W 2 ij + W ii W jj ) E[µ] = δ; E[(µ -δ)(µ -δ) T ] = κ + 1 κ(ν -d -1) W -1

Using this set of equations, the exact posterior P 1 (Θ) can be approximated with P1 (Θ). This posterior will then be the prior for the next iteration and we keep following the steps above iteratively to finally have a distribution Pn (Θ) after observing a stream of data X 1:n . The estimate is

Θ = E[ Pn (Θ)].

The exact calculations for the Bayesian Moment Matching algorithm are given in appendix A.

4. TRANSFER LEARNING USING BMM:

In this section, we first motivate the need for an online transfer learning algorithm for sequential data modeling and then explain in detail the different steps of the algorithm. The complete algorithm is given in Alg. (1).

4.1. MOTIVATION:

Several applications produce data instances from a population of individuals that exhibit a variety of different traits. For example, for the task of activity recognition, different individuals will have different gait patterns despite the fact that they are performing the same activity (e.g., walking, running, standing, etc.). Therefore, it is problematic to make predictions in such domains by considering the population to be homogeneous; however, every population will have individuals resembling each other in some characteristics. This suggests that we can use individuals in a population to make predictions about similar individuals by identifying those individuals who closely resemble each other. However, identifying individuals with similar traits is not straightforward. Alternatively, weights can be assigned to each individual in a population based on a target individual (individual on whom predictions are to be made). All those individuals who resemble closely the target individual will receive higher weights than those with dissimilar traits. Subsequently, predictions about the behavior of the target individual will be based mostly on the observed behavior of the similar individuals.

Our transfer learning algorithm addresses precisely these issues. It has three main steps -first, it learns a model (transition and emission distributions) for each source domain (or individual in a population) that best explains the observations of that source domain. Next, given a target domain (or target individual), it identifies those individuals that closely resemble the target individual by estimating a basis weight associated to each source domain. A higher weight for a source domain implies that the corresponding individual resembles more closely the target individual. Finally, it predicts the hidden states for each observation in the target domain by using the models learned in the source domain and the basis weights that are given to each transition and emission distribution of the source domains. We now explain each step of the algorithm in detail below.

4.2. SOURCE DOMAIN -TRAINING:

The first step is to learn a model for each source domain in the training data. Suppose that we have labeled sequence data for K different source domains. Let Y k t = hidden state label at time step t for source domain k X k t = feature vector at time step t for source domain k

Let the sequence of observations be given by X k

1:T = {X k 1 , X k 2 , ..., X k T } and the hidden states be {Y k 1 , Y k 2 , ..., Y k T } where Y k t ∈ {1, 2, .., N } ∀t. Furthermore, let us define Θ k ij = Pr(Y k t = i|Y k t-1 = j) i.e

. the transition probability from state i to state j

We denote the transition matrix for the k th source domain with Θ k . Let the emission distribution be modeled by a mixture of Gaussian with M components. This implies

Pr(X k t |Y k t = j) = M m=1 w k jm N (X k t |µ k jm , Σ k jm ) ∀j ∈ {1, 2, .., N }

Our aim is to learn the parameters characterizing the transition and the emission distribution for each source domain. More precisely, if

Φ k = {φ k 1 , φ k 2 , ..., φ k N } where φ k i = {(w k i1 , µ k i1 , Σ k i1 ), ...., (w k i M , µ k i M , Σ k i M )}

then we want to learn the parameters Θ k for the transition distribution and Φ k for the emission distribution for each source domain k ∈ {1, 2, ..., K}. Since, we use a hidden Markov model, the update equation at each time step for a source domain k is

P r Θ, Φ, Y k t = j|X k t , Y k t-1 = i ∝ Emission distribution Pr(X k t |Y k t = j) Transition Probability Pr(Y k t = j|Y k t-1 = i) Prior for t -1 Pr(Θ k , Φ k , Y k t-1 = i|X k 1:t-1 ) ∀j ∈ {1, 2, ..., N } (1)

The prior over (Θ k , Φ k ) is given by

Pr(Θ k , Φ k ) = N i=1 Dir(θ k i |α k i ) N j=1 Dir(w k j ; β k j ) M u=1 N W(µ k ju , Λ k ju ; δ k ju , κ k ju , W k ju , v k ju )

(2) After substituting the relevant terms in Eq (1), we get

Pr Θ, Φ, Y k t = j|X k t , Y k t-1 = i ∝ M m=1 w k jm N (X k t |µ k jm , Σ k jm )θ k ji N i=1 Dir(θ k i |α k i ) N j=1 Dir(w k j ; β k j ) M u=1 N W(µ k ju , Λ k ju ; δ k ju , κ k ju , W k ju , v k ju ) ∀j ∈ {1, 2, ..., N } (3) Further, Λ k ju = (Σ k ju ) -1 .

The prior in Eq (2) can be understood as having the following components

• Transition Distribution : Each column of the N × N transition matrix specifies the probability of making a transition from that column index to another state given by the row index. We define a Dirichlet distribution as a prior over each column of the transition matrix. Hence,

N i=1 Dir(θ k i |α k i ) is the prior over Θ k . • Emission Distribution : Dir(w k j ; β k j ) M u=1 N W(µ k ju , Λ k ju ; δ k ju , κ k ju , W k ju , v k ju )

defines a prior over a mixture of Gaussians for hidden state j with M components where the Dirichlet distribution is the prior over the mixture weights and the Normal-Wishart distribution is the prior over the mean and precision matrix of the mixture components. We take a product over j to obtain a prior over all emission distributions.

The posterior distribution (Eq (3)) after each observation is a mixture of products of distributions where each component has the same form as the prior distribution since Pr(X k t |Y k t = j) is a mixture of Gaussians. Therefore, the number of terms in the posterior increases exponentially if we perform exact Bayesian learning. To circumvent this, we use BMM for Gaussian Mixture Models as described in (Jaini et al., 2016; Jaini & Poupart, 2016) 3. The complete calculations for learning in the source domain are given in appendix B.

The main computation in the learning and updating routine is the calculation of the sufficient set of moments using the Bayesian posterior given in Eq. ( 9) in appendix B. Let M be the number of components in the mixture model for emission distributions, N the number of hidden states and d the number of features in the data. The computational complexity for updating the parameters in the source domain learning step for each iteration is O(M 2 N 2 ) for each scalar parameters and is O(M 2 N 2 d 3 ) for the parameters of the distribution over the precision matrix because that involves a matrix multiplication step.

4.3. TARGET DOMAIN -PREDICTION:

The goal is to predict the hidden states for a target individual (or domain) as we observe a sequence of observations. In the previous step, we learned the transition and emission distributions individually for K different sources. These sources can be thought of as individuals in a population. The transition and emission distributions learned from the individual sources form a basis for the transition and emission distributions of the target domain. Specifically, let the transition distribution for the k th source be denoted by g(Θ k ) and emission distribution be denoted by f (Φ k j ) for a certain hidden state j. Then, the transition and emission distributions for the target domain is a weighted combination given by

EQUATION

We first need to compute the basis weights λ = (λ 1 , λ 2 , ...., λ K ) and π = (π 1 , π 2 , ...., π K ). We estimate (λ, π) in an unsupervised manner using BMM. We define a Dirichlet prior over λ and π, i.e. Pr(λ, π) = Dir(λ; γ)Dir(π; ν). The posterior after observing a new data point is

EQUATION

where f (Φ k j )g(Θ m ji ) are known from the source domains,

π k Dir(π; ν) = a k Dir(π; ν), λ m Dir(λ; γ) = b m Dir(λ; γ) and C(i, j, k, m) = a k b m f (Φ k j )g(Θ m ji ).

The exact calculations are given in Appendix C. We approximate the posterior in Eq (8) by projecting it onto a tractable family of distributions with the same set of sufficient moments as the posterior using the Bayesian Moment Matching approach. Finally, the estimate of (λ, π) is the expected value of the final posterior. This completes the learning stage.

The transition and emission distributions for the target domain are the weighted combination of transition and emission distributions learned in the source domain respectively. The advantage of this linear combination is to account for heterogeneity in the data. The learning step in the target domain will ensure that only those source domains that resemble closely the target domain are given higher weights. This helps to bias the predictions according to the closest basis models when the population is not homogeneous.

Predictions can be made in two different manners

• Online -initialize the prior over λ and π to be uniform. As each new data point is observed in a sequence, a prediction is made based on the mean of the current posterior over λ and π and subsequently the posterior is updated based on Eq (8).

• Offline -compute the posterior of λ and π based on Eq (8) by using the entire sequence of observations of the target individual. Once, the posterior is computed, predict the hidden states for each observation in the sequence based on the mean estimates of the posterior.

In Fig. 1 , we show the schematic for the proposed online transfer learning algorithm. The figure shows the learning phase for each source domain where the emission and transition distributions are learned using Bayesian Moment Matching technique. After learning in the source domain, we learn the weights of the basis models in the target domain for each new observation and make predictions in an online manner. Published as a conference paper at ICLR 2017

The target domain step involves two routines :

• Update step -In this step, the hyper-parameters (γ, ν) over the weights (λ, π) are updated.

The main computation in this step is the calculation of the set of sufficient moments from the updated Bayesian posterior given in Eq. ( 8). Hence, the computational complexity of the update step in the target domain for each observation is O(K 2 N 2 ) where K is the number of source domains and N is the number of hidden states.

• Prediction step -In the prediction step, a hidden label is assigned to the observation based on the model obtained from the update step. The main computation is calculation of the likelihood of each hidden state for the observation. The computational complexity of the prediction step is hence O(M KN ) where M is the number of components in the mixture model, K is the total number of source domains and N is the number of hidden states.

Algorithm for k = 1 : K do 9:

Let f(Θ, Φ) be a family of probability distributions with parameters γ 10:

Initialize a prior P k 0 (Θ, Φ) from f over transition and emission parameters respectively 11:

for n = 1 : D k do D k : size of data for k th source domain 12:

Compute P n (Θ, Φ) from P n-1 (Θ, Φ) using Eq. 3 13:

Using BMM approximate P n with Pn (Θ, Φ) = f(Θ, Φ|γ) Compute P n (Θ, Φ) from P n-1 (Θ, Φ) using Eq. 8 23:

Using BMM approximate P n with Pn (λ, π) = g(λ, π)

24::

Predict : Ŷn = argmax j P r λ, π, Y n = j|X n using Eq (8)

25:

Return : λ = E λ [ Pn (λ, π)]

26:

Return : π = E π [ Pn (λ, π)]

27:

Return : prediction Ŷn

5. EXPERIMENTS AND RESULTS:

This section describes experiments on three tasks from different domains -activity recognition, sleep cycle prediction among healthy individuals and patients suffering from Parkinson's disease and packet flow prediction in telecommunication networks.

EXPERIMENTAL SETUP:

For each task, we compare our online transfer learning algorithm to EM (trained by maximum likelihood) and a baseline algorithm (that uses Bayesian moment matching) that both learn a single HMM with mixtures of Gaussians as emissions by treating the population as homogeneous. Furthermore, we conduct experiments using recurrent neural networks (RNNs) due to their popularity in sequence learning.

The baseline algorithm uses Bayesian Moment Matching to learn the parameters of the HMM. Concretely, we have data collected from several individuals (or sources) in a population for each task.

For transfer learning, we train an HMM with mixture of Gaussian emission distributions for each source (or individual) except the target individual. For the target individual, we estimate a posterior over the basis weights in an online and unsupervised fashion and make online predictions about the hidden states. We compare the performance of our transfer learning algorithm against the EM and baseline algorithms that treat the population as homogeneous, i.e., we train an HMM by combining the data from all the sources except the target individual. Then, using this model, we make online predictions about the hidden states of the target individual.

We report the results based on leave-one-out cross validation where the data of a different individual is left out in each round. For each task, we treat every individual as a target individual once. For a fair comparison, the HMM model learned for both the baseline algorithm and the EM algorithm has the same number of components as the HMM model learned by the online transfer learning algorithm.

Regarding RNNs, we used architectures with as many input nodes as the number of attributes, one hidden layer consisting of long short term memory (LSTM) units (Hochreiter & Schmidhuber, 1997) and one softmax output layer with as many nodes as the number of classes. We use the categorical cross-entropy loss as the cost function. We select LSTM units instead of sigmoid or hyperbolic tangent units due to their popularity and success in sequence learning (Sutskever et al., 2014) .

We perform grid search to select the best hyper-parameters for each setting. For the training method, we either use Nesterov's accelerated gradient descent (Nesterov, 1983; Sutskever et al., 2013) with learning rates [0.001,0.01,0.1,0.2] and momentum values [0,0.2,0.4,0.6,0.8,0.9], or rmsprop (Tieleman & Hinton, 2012) having ε = 10 -4 and decay factor 0.9 (standard values) with learning rates [0.00005,0.0001,0.0002,0.001] and momentum values [0,0.2,0.4,0.6,0.8,0.9]. The weight decay takes values from [0.001,0.01,0.1], whereas the number of LSTM units in the hidden layer takes the possible values [2, 4, 6, 9, 12] .

We experimented with various architectures before we ended up with the aforementioned values; in particular, architectures with a single hidden layer consistently performed better than multiple layers, possibly because our datasets are not very complex. We train the network by backpropagation through time (bptt) truncated to 20 time steps (Williams & Peng, 1990) . The RNNs are trained for a maximum number of 150 epochs, or until convergence is reached. Our implementation is based on the Theano library (Theano Development Team, 2016) in Python.

For each task, we run experiments 10 times with each individual taken as target and the rest acting as source domains for training. We report the average percentage accuracy and use the Wilcoxon signed rank test (Wilcoxon, 1950) to compute a p-value and report statistical significance when the p-value is less than 0.05. In the following sections, we discuss the results for each task in detail.

ACTIVITY RECOGNITION:

As part of an on-going study to promote physical activity, we collected smartphone data with 19 participants and tested our transfer learning algorithm to recognize 5 different kinds of activities: sitting, standing, walking, running and in-a-moving-vehicle. While APIs already exist to automatically recognize walking, running and in-a-moving-vehicle by Android and Apple smartphones, sitting and standing are not available in the standard APIs. Furthermore, our long term goal is to obtain robust recognition algorithms for older adults and individuals with perturbed gait (e.g., due to a stroke). Labeled data was obtained by instructing the 19 participants to walk at varying speeds for 4 min, run for 2 min, stand for 2 min, sit for 2 min and ride a moving vehicle to a destination of their choice. The data collected was segmented in epocs of 1 second where 48 features (means and standard deviations of the 3D accelerometry in each epoch) were computed by the smartphone.

The online transfer learning algorithm learned an HMM over 18 individuals which acted as basis models for prediction on the 19 th individual. In this manner, we ran experiments for each individual 10 times to get a statistical measure of the results. It demonstrates that the transfer learning algorithm performed better than the baseline on 15 individuals and in other cases its accuracy was close to the baseline. Furthermore, it is also worth noting that in most cases, the confusion in the algorithm's prediction was between the following pairs of classes: In a Moving Vehicle-Standing and In a Moving Vehicle-Sitting. This is expected because in most cases the person was either standing/sitting in a bus or sitting in a car. Table (1) also demonstrates the superior performance of online transfer learning algorithm as compared to the EM algorithm. Finally, note the poor performance of RNNs despite the fact that we fine-tuned the architecture to get the best results. RNNs are in theory very expressive. However, they are also notoriously difficult to train and fine-tune due to their non-convexity and vanishing/exploding gradient issues that arise in backpropagation through time. Indeed, in several cases they even underperform all other methods.

SLEEP STAGE CLASSIFICATION:

Sleep disruption can lead to various health issues. Understanding and analyzing sleep patterns, therefore, has great potential to significantly improve the quality of life for both patients and healthy individuals. In both clinical and research settings, the standard tool for quantifying sleep architecture and physiology is polysomnography (PSG), which is the measurement of electroencephalography (EEG), electrooculography (EOG), electromyography (EMG), electrocardiography (ECG), and respiratory function of an individual during sleep. The analysis of sleep architecture is of relevance for the diagnosis of several neurological disorders, e.g., Parkinson's disease (Peeraully et al., 2012) , because neurological anomalies often also reflect in variations of a patient's sleep patterns.

Typically, PSG data is divided into 30-second epochs and classified into 5 stages of sleep -wake (W), rapid eye movement sleep (REM) or one of 3 non-REM sleep stages (N1, N2, and N3)based on the visual identification of specific signal features on the EEG, EOG, and EMG channels. Epochs that cannot be distinctly sorted into one of the 5 stages are labeled as Unknown. While it is a valuable clinical and research tool, visual classification of EEG data remains time consuming, requiring up to 2 hours for a highly trained technologist to classify all the epochs within a typical 7hour PSG recording. Beyond that, inter-scorer agreement rates remain low around 80 (Rosenberg & Van Hout, 2013) . High annotation costs and low inter-scorer agreement rates have motivated efforts to develop fully automated approaches for sleep stage classification (Anderer et al., 2005; Jensen et al., 2010; Mal, 2013; Punjabi et al., 2015) . However, many of these methods result in generic cross-patient classifiers that fail to reach levels of accuracy and reliability high enough to be adopted in real-world medical settings.

The polysomnograms (PSGs) we used for our evaluation were obtained at a clinical neurophysiology laboratory in Toronto (name anonymized) according to the American Academy of Sleep Medicine guidelines using a Grael HD PSG amplifier (Compumedics, Victoria, Australia). We selected recordings from 142 patients obtained between 2009 and 2015. Out of these 142 recordings, 91 were from healthy subjects and 51 were from patients with Parkinson's disease.

Each recording was manually scored by a single registered PSG technologist. Recordings were first segmented into fixed-sized windows of 30 second epochs. To reduce complexity and processing time in the feature extraction and manual labeling step, we only retained EEG channel C4-A1, which is deemed especially important for sleep stage classification (Sil, 2007) . Channel selection and segmentation resulted in a ground truth data set where each instance was represented by a singlechannel time series of 7680 floating point numbers corresponding to 30 seconds of C4-A1, sampled at 256 Hz. A vector of 26 scalar features was extracted from each epoch. Bao et al. (2011) and Motamedi-Fakhr et al. (2014) give a detailed listing and explanation of all 26 features.

The online transfer learning algorithm learned an HMM over 50 individuals chosen at random which acted as basis models for prediction on the target individual. We did not use all 140 individuals for the basis models because it resulted in sources getting sparse weights diluting the effect of heterogeneity. We completed the experiments for each individual 10 times in this manner to get a statistical measure of the results. All the results are statistically significant under the Wilcoxon signed rank test with p-value < 0.05. More detailed results for comparison of the online transfer learning technique with the three baseline algorithms is given in appendix (D).

FLOW DIRECTION PREDICTION:

Accurate prediction of future traffic plays an important role in proactive network control. Proactive network control means that if we know the future traffic (including directions and traffic volume), then we have more time to find a better policy for the network routing, priority scheduling as well as rate control in order to maximize network throughput while minimizing transmission delay, packet loss rate, etc.

Better understanding the behavior of TCP connections in certain applications can provide important input to automatic application type detection, especially in those scenarios where network traffic is encrypted and DPI (Deep Packet Inspection) is nearly impossible. Different applications can be distinguished by the distinct behavior of their TCP connections, which are well described by the corresponding HMMs.

We performed our experiments with a publicly available dataset of real traffic from academic buildings. The dataset consists of packet traces with TCP flows. For our experiments, we only consider three packet sizes and flow size as the features. The hidden labels are the source of generation of the packet, i.e., Server or Client. We divided the dataset into 9 domains with each domain consisting of a number of observation sequences. For the online transfer learning algorithm, we learned an HMM for each of 8 sources that acted as basis models for prediction on the 9 th source. We compared the performance of the online transfer learning algorithm with EM and the baseline algorithm which treat the data as homogeneous. Table 2 reports the average (of 10 experimental runs) percentage accuracy for each source. The online transfer learning algorithm performs better than both the baseline and the EM algorithm. The results are statistically significant under the Wilcoxon signed rank test with p-value < 0.05. Furthermore, we compare our method to RNNs. It turns out that for the task of traffic direction prediction, RNNs can actually perform well, unlike for instance the activity recognition dataset. The better performance this time may be due to the simpler structure of the data that consists of a single attribute and a binary class. This is in sharp contrast to the activity recognition dataset whose instances contain 48 attributes and can belong to 5 classes, and is thus harder to train. 

6. CONCLUSION:

In many applications, data is produced by a population of individuals that exhibit a certain degree of variability. Traditionally, machine learning techniques ignore this variability and train a single model under the assumption that the population is homogeneous. While several offline transfer learning techniques have already been proposed to account for population heterogeneity, this work describes the first online transfer learning technique (to our knowledge) that incrementally determines which source models best explain a streaming sequence of observations while predicting the corresponding hidden states. We achieved this by adapting the online Bayesian moment matching algorithm originally developed for mixture models to hidden Markov models. Experimental results confirm the effectiveness of the approach in three real-world applications: activity recognition, sleep stage recognition and flow direction prediction.

In the future, this work could be extended in several directions. Since it is not always clear how many basis models should be used and that the observation sequences of target individuals can necessarily be explained by a weighted combination of basis models, it would be interesting to explore techniques that can automatically determine a good number of basis models and that can generate new basis models on the fly when existing ones are insufficient. Furthermore, since recurrent neural networks (RNNs) have been shown to outperform HMMs with GMM emission distributions in some applications such as speech recognition (Graves et al., 2013) , it would be interesting to generalize our online transfer learning technique to RNNs.

A NORMAL-WISHART AND DIRICHLET DISTRIBUTION DIRICHLET DISTRIBUTION:

The Dirichlet distribution is a family of multivariate continuous probability distributions over the interval [0,1]. It is the conjugate prior probability distribution for the multinomial distribution. We next show how the combining happens for a Dirichlet as has been highlighted in (3). where

αi = α i if i = m α i + 1 if i = m NORMAL WISHART PRIOR

The Normal-Wishart distribution is a conjugate prior of a multivariate Gaussian distribution with unknown mean and precision matrix (Degroot, 1970) . It is the combination of a Wishart distribution over the precision matrix and Gaussian distribution over the mean given the precision matrix.

Let µ be a d-dimensional vector and Λ be a symmetric positive definite d × d matrix of random variables respectively. Then, a Normal-Wishart distribution over (µ, Λ) given parameters (µ 0 , κ, W, ν) is such that µ ∼ N d µ; µ 0 , (κΛ) -1 where κ > 0 is real, µ 0 ∈ R d and Λ has a Wishart distribution given as Λ ∼ W(Λ; W, ν) where W ∈ R d×d is a positive definite matrix and ν > d -1 is real. The marginal distribution of µ is a multivariate t-distribution i.e µ|Λ ∼ t ν-d+1 µ; µ 0 , 

MOMENT MATCHING:

In this section we show the system of equations using which the parameters of a product of Dirichlet and Normal-Wishart distribution can be estimated once the set of sufficient moments are known. The set of sufficient moments in this case is S = {µ j , µ j µ T j , Λ j , Λ 2 j km , w j , w 2 j } | ∀j ∈ 1, 2, ..., M } where Λ 2 j km , is the (k, m) th element of the matrix Λ j .The expressions for the sufficient moments are :

E[w i ] = α i j α j ; E[w 2 i ] = (α i )(α i + 1) j α j 1 + j α j E[Λ] = νW; V ar(Λ ij ) = ν(W 2 ij + W ii W jj ) E[µ] = δ; E[(µ -δ)(µ -δ) T ] = κ + 1 κ(ν -d -1) W -1

C TARGET DOMAIN LEARNING USING BMM:

The prior over the weights is Pr(λ, π) = Dir(λ; γ)Dir(π; ν)

where γ and ν are the hyper-parameters for the Dirichlet distribution. The posterior after each observation is P r λ,π, Y t = j|X t ∝ Pr(X t |Y t = j) 

where Z = i,j,k,m C(i, j, k, m) is the normalization constant, K is the number of source domains and N is the number of hidden classes. Now, we can use the Bayesian Moment Matching algorithm to approximate Eq (8) as a product of two Dirichlets, in the same form as the prior. This posterior will then act as the prior for the next time step. Finally, the values of the weights will be the expected value of each Dirichlet. Let us next see how the combining happens for a Dirichlet. 

where

γi = γ i if i = m γ i + 1 if i = m

Therefore C(i,j,k,m) in Eq (8) is

C(i, j, k, m) = γ m i γ i π k i π i M u=1 N (µ k ju , Σ k ju )θ m ij (15)

Next, we outline the moment matching step. The set of sufficient moments is given by 

S = {λ i , λ 2 i , π i , π 2 i | ∀i

Similarly, the second moment can be evaluated as

:



Figures and Tables:

Figure 1: Transfer Learning architecture

= E Φ [ Pn (Θ, Φ)] 16: Return : emission and transition distributions for each source Target Domain -learning basis weights for each source domain & prediction 17: Input: unlabeled sequence data 18: procedure PREDICTTARGETDOMAIN(data, sourceDistributions) 19: Let g(λ, π) = Dir(λ; γ)Dir(π; ν) be a family of probability distributions 20: Initialize a prior P 0 (λ, π) from g with equal weights to each source distribution 21: for n = 1 : D do D : size of data for target domain 22:

Fig. (2) shows the scatter plots of accuracy for our online transfer learning technique and the three baseline algorithms -BMM, EM (maximum likelihood) and RNNs -which treat the data as homogeneous for the sleep stage classification dataset. For each plot, a point above the dotted line indicates higher accuracy of online transfer learning technique as compared to the corresponding baseline algorithm for the target patient. The plots show consistent superior performance of our online transfer learning technique as compared to both baseline algorithms -BMM and EM for all target patients. The online transfer learning technique also performs better on a majority of patients (102 out of 142) as compared to an optimized RNN.

Figure 2: Performance comparison of online transfer learning algorithm with three different baseline algorithms -BMM, EM (max. likelihood) and RNNs on Sleep Stage Classification data using scatter plots of accuracy.

Wishart distribution multiplies with a Gaussian with the same mean and precision matrix to give a new Normal-Wishart distribution.N d y; µ, (κΛ) -1 N W(µ, Λ; µ 0 , κ, W, ν) = cN W(µ, Λ; µ * 0 , κ * , W * , ν * ) where µ * 0 = κµ 0 + y κ + 1 κ * = 1 + κ ν * = ν + 1 W * = W + κ κ + 1 (µ 0 -y)(µ 0 -y) T

t = j|Y t-1 = i) Pr(λ, π, Y t-1 ) , k, m) Dir(π; ν)Dir(λ; γ)

, j, k, m)

1 Online Transfer Learning by Bayesian Moment Matching

Table content: <table/>

Average percentage accuracy of prediction for activity recognition on 19 different individuals. The best results among the Baseline, the EM algorithm, RNN and Transfer Learning algorithm are highlighted in bold font. ↑(or ↓) indicates that Transfer Learning has significantly better (or worse) accuracy than the the best algorithm among the baseline, EM and RNN under the Wilcoxon signed rank test with p-value < 0.05.

Table content: <table><tr><td colspan="2">TARGET DOMAIN BASELINE EM</td><td colspan="2">RNN TRANSFER LEARNING</td></tr><tr><td>PERSON 1</td><td colspan="2">91.29 83.57 71.15</td><td>88.36 ↓</td></tr><tr><td>PERSON 2</td><td colspan="2">81.37 79.87 79.58</td><td>87.65↑</td></tr><tr><td>PERSON 3</td><td colspan="2">74.68 75.91 69.56</td><td>93.15↑</td></tr><tr><td>PERSON 4</td><td colspan="2">73.39 68.29 74.25</td><td>84.70↑</td></tr><tr><td>PERSON 5</td><td colspan="2">95.94 89.59 95.36</td><td>99.75↑</td></tr><tr><td>PERSON 6</td><td colspan="2">73.98 69.77 61.71</td><td>96.43↑</td></tr><tr><td>PERSON 7</td><td colspan="2">57.62 55.15 69.22</td><td>70.75↑</td></tr><tr><td>PERSON 8</td><td colspan="2">91.72 86.05 74.49</td><td>97.80↑</td></tr><tr><td>PERSON 9</td><td colspan="2">81.19 78.88 78.72</td><td>88.75↑</td></tr><tr><td>PERSON 10</td><td colspan="2">99.12 93.60 92.00</td><td>97.35↓</td></tr><tr><td>PERSON 11</td><td colspan="2">76.59 74.67 84.75</td><td>88.75↑</td></tr><tr><td>PERSON 12</td><td colspan="2">55.36 59.71 53.63</td><td>95.05↑</td></tr><tr><td>PERSON 13</td><td colspan="2">79.66 73.46 65.54</td><td>97.60↑</td></tr><tr><td>PERSON 14</td><td colspan="2">92.06 89.11 63.59</td><td>93.12↑</td></tr><tr><td>PERSON 15</td><td colspan="2">79.25 72.24 91.08</td><td>94.20↑</td></tr><tr><td>PERSON 16</td><td colspan="2">84.08 79.23 74.74</td><td>83.51↓</td></tr><tr><td>PERSON 17</td><td colspan="2">93.95 91.03 81.25</td><td>97.60↑</td></tr><tr><td>PERSON 18</td><td colspan="2">82.84 74.88 79.45</td><td>87.20↑</td></tr><tr><td>PERSON 19</td><td colspan="2">95.97 89.06 95.88</td><td>95.06↓</td></tr></table>

) compares the average percentage accuracy of prediction for activity recognition with 19 different individuals.

Table content: <table/>

Average percentage accuracy of prediction for flow direction prediction for 9 different domains. The best results among the Baseline, the EM algorithm, RNN and the Transfer Learning algorithm are highlighted in bold font. ↑(or ↓) indicates that transfer learning has significantly better (or worse) accuracy than the best technique among the baseline algorithm, EM and RNN under Wilcoxon signed rank test with pvalue < 0.05.

Table content: <table><tr><td colspan="3">TARGET DOMAIN BASELINE EM</td><td colspan="2">RNN TRANSFER LEARNING</td></tr><tr><td>SOURCE 1</td><td>72.00</td><td colspan="2">54.90 80.00</td><td>71.02 ↓</td></tr><tr><td>SOURCE 2</td><td>85.33</td><td colspan="2">89.10 65.30</td><td>86.50↓</td></tr><tr><td>SOURCE 3</td><td>80.33</td><td colspan="2">81.90 86.50</td><td>83.33↑</td></tr><tr><td>SOURCE 4</td><td>86.50</td><td colspan="2">75.80 86.60</td><td>87.17↑</td></tr><tr><td>SOURCE 5</td><td>87.33</td><td colspan="2">82.80 81.70</td><td>86.00↓</td></tr><tr><td>SOURCE 6</td><td>93.33</td><td colspan="2">78.20 88.90</td><td>93.50↑</td></tr><tr><td>SOURCE 7</td><td>95.17</td><td colspan="2">90.70 93.50</td><td>95.33↑</td></tr><tr><td>SOURCE 8</td><td>89.83</td><td colspan="2">91.14 91.00</td><td>91.63↑</td></tr><tr><td>SOURCE 9</td><td>76.67</td><td colspan="2">75.68 81.98</td><td>78.83↑</td></tr></table>

Peer review (iclr2017):
I*m bumping up my score to a 7 to acknowledge that the authors responded satisfactorily to reviewer feedback, and to indicate that I think that the updated manuscript is a strong paper and that I do not object to its acceptance to ICLR. However, I also would not fight for its acceptance. I still think that it is a better fit for a venue with an explicit interest in more traditional Bayesian latent variable models (ICML, UAI, NIPS). ----- This manuscript describes a novel Bayesian approach to transfer learning focused on online sequence modeling settings where he primary concern is less distribution drift in an ongoing sequence and more variability between individual sequences (since each sequence can be thought of as defining its own conditional distribution over subsequent states). They provide the example of human gait classification, where each individual*s gait may differ from others even while performing the same activity. In this setting we typically train the gait model on gait sequences from a set of *source* individuals but then apply it to previously unseen people. The core model is an HMM, which they give a full Bayesian treatment; the central problem this introduces is that each new observation that arrives introduces an additional product term to the posterior that is itself a product over M components (clusters or hidden states). This rapidly becomes intractable. This paper applies Bayesian moment matching (BMM) to this problem, in which the posterior is approximated via projection onto a more tractable distribution that is adjusted to match some moments of the posterior. The experimental results are quite promising Strengths: - The problem setting is very appropriately framed as online sequence prediction via Bayesian transfer learning. There is almost certainly individual variability between the training sequences (for which explanatory variables not be available in the inputs). The Bayesian approach gives a natural approach to performing transfer and handling low data regimes (common in all experiments). - The Bayesian formulation creates a computational challenge (the posterior becomes intractable). The proposed BMM approximation is both reasonable and relatively novel (particularly given the popularity of MCMC and variational methods). - With the caveat that I am not well-versed in recent work on Bayesian moment matching, a cursory literature search suggests this is a novel application of BMM. A lot of the related BMM work is roughly contemporaneous with this work, and none of it seems concerned with online transfer learning. - The overall results look quite strong: in activity recognition and flow prediction, the transfer learning approach is in general superior to the one-size-fits-all HMM, even when trained using BMM (which in turn is generally superior to the EM-based HMM). The proposed approach appears to beat the LSTM across all tasks (including sleep stage classification), possibly due to the lack of training data. Weaknesses: - The description of the LSTM training and architecture search is vague (and in one instance, contradictory), strongly implying that it was not fully tuned and may be an artificially weak baseline. While it is plausible that the proposed approach might excel given the small data sets used in the experiments, there is not sufficient evidence and detail to support this claim. The authors should provide more detail about architecture search, hyperparameter tuning, and most important, attempts at regularization, given the limited training data. In particular, the authors should experiment with a sufficiently rich set of settings for # hidden layers, # hidden units, weight decay, and dropout. - Given the emphasis on transfer learning and the use of a Bayesian framework, the decision to train source models independently is a little odd. Why not perform some kind of joint training? - The authors provide no analysis, analytical or empirical, of the proposed framework*s (storage and computational) complexity, especially at prediction time. At the top of page 10, they mention using only one EEG channel in order to *reduce complexity and processing time.* This is an ominous hint that the proposed framework may not scale practically. - Additionally, I have a meta-concern about this paper*s fit for ICLR. *Representation learning* -- the general theme of ICLR -- is not featured prominently in this work. Given the competitive nature of ICLR, we should consider seriously whether this paper is of interest to the wider ICLR community or whether it might be a better fit for a meeting such as AISTATS, ICML, or UAI. Comments: - The plots in Figures 1-3 are difficult to interpret. Putting patients along the X-axis is unintuitive since their order is arbitrary. Why not just make scatter plots of one vs. the other model*s accuracy. The shape of the scatter should hopefully make it clear if there is a general trend. - In Experimental Setup, the authors make conflicting claims about the LSTM architecture. They first state the single hidden layer has number of cells equal to the number of inputs. They then say that the number of LSTM units is finetuned based on empirical performance. - While generally well-written, the paper has several obscenely long paragraphs. The single paragraph *Experimental Setup* section, for example, takes up more than half of page 8. These should be broken up into shorter paragraphs to make them easier to read. A good rule of thumb is that no paragraph should take up more than ~1/6 of a page. In general, I like this work, but the vague details around the LSTM training raise serious red flags about their experimental results, at least the comparison vs. LSTMs, and I have concerns about how well it meets ICLR*s CFP. That said, my policy for interactive review is to carefully consider author responses with an open mind, so I will serious consider changing my score, if warranted.