Recurrent Real-valued Neural Autoregressive Density Estimator for Online Density Estimation and Classification of Streaming Data
Authors: Tianyu Li
Bogdan Mazoure
Guillaume Rabusseau
Venue: Submitted to ICLR 2023 (2023)
Keywords: density estimation
online learning
streaming data
classification

Abstract: In contrast with the traditional offline learning, where complete data accessibility is assumed, many modern applications involve processing data in a streaming fashion. This online learning setting raises various challenges, including concept drift, hardware memory constraints, etc. In this paper, we propose the Recurrent Real-valued Neural Autoregressive Density Estimator (RRNADE), a flexible density-based model for online classification and density estimation. RRNADE combines a neural Gaussian mixture density module with a recurrent module. This combination allows RRNADE to exploit possible sequential correlations in the streaming task, which are often ignored in the classical streaming setting where each input is assumed to be independent from the previous ones. We showcase the ability of RRNADE to adapt to concept drifts on synthetic density estimation tasks. We also apply RRNADE to online classification tasks on both real world and synthetic datasets and compare it with multiple density based as well as nondensity based online classification methods. In almost all of these tasks, RRNADE outperforms the other methods. Lastly, we conduct an ablation study demonstrating the complementary benefits of the density and the recurrent modules.

Under review as a conference paper at ICLR 2023
RECURRENT REAL-VALUEE NEURAL AUTOREGRES- SIVE DENSITY ESTIMATOR FOR ONLINE DENSITY Es- TIMATION AND CLASSIFICATION OF STREAMING DATA
Anonymous authors Paper under double-bling blind review
ABSTRACT
In contrast with the traditional offline learning, where complete data accessibil- ity is assumed, many modern applications involve processing data in streaming fashion. This online learning setting raises various challenges, including concept drift, hardware memory constraints, etc. In this paper, propose the Recur- rent Real-valued Neural Autoregressive Density Estimator (RRNADE), a flexible density-based -bsed model for online classification and density estimation. RRNADE combines a neural Gaussian mixture density module with a recurrent module. This combination allows RRNADE exploit possible sequential correlations in the streaming task, which are often ignored in the classical streaming setting where each input is assumed be independent from the previous ones. We showcase the ability of RRNADE to adapt to concept drifts on synthetic density estimation tasks. We also apply RRNADE to online classification tasks on both real world and synthetic datasets and compare it with multiple density based as well as non- density based online classification methods. In almost all these tasks, RRNADE outperforms the other methods. Lastly, we conduct an ablation study demonstrat- ing the complementary benefits of the density and the recurrent modules.
INTRODUCTION
Many tasks in classic supervised machine learning, such regression and classification, involve processing batched data in an offline fashion: the data, often coming as input-output pairs, stored first and then used to learn predictive model for future unseen data. However, many modern applications favor the form where the model update and predict while receiving new data entries. This form is often referred to learning from data streams. he problem of learning from data streams is closely related to the problem continual or incremental learning (Losing et al., 2018 Zenke et al. 2017 Lopez-Pazz & Ranzato 2017) which have recently received increasing interest in the machine learning community
There are three major issues when learning from data streams: memory constrains, concept drifts as well as temporal correlations. The sheer amount of data many modern applications process daily makes infeasible to store all data and perform offline update of the model (Nacem et al.) 2022). In addition, certain data sources do not allow the indefinite hold of the data due to potential privacy regulations (Forti 202T) Therefore, when learning data streams, it often assumed that the model only has access to the recent history. Furthermore, concept drifts and temporal correlations are also common challenges when learning from data streams. Under the offline setting, data is often assumed to have the i.i.d. assumption, i.e. each data entry is independently drawn from the identical distribution. However, under the streaming data setting, the independent assumption can be violated, causing temporal correlations in the data, while the violation of the identical assumption can lead to concept drifts problem. These issues often invalidate the model learned from historical data, resulting in further deterioration of its performance.
Density estimation is one of the core tasks in the field of unsupervised learning, branching out to many applications such classification and clustering. Under the offline setting, Real-valued Neural Autoregressive Density Estimator (RNADE) leverages a neural network parameterized Gaussian mixture model to estimate the density function of real-valued vectors. It is then curious if extending

---

Under review as a conference paper at ICLR 2023
RNADE to its online form would be possible, namely, the model needs to be updated data arrives and we only have a limited amount of history stored in memory. In this paper, we show that the answer is in the positive. Concretely, our contributions are follows:
1. We propose the Recurrent Real-valued Neural Autoregressive Density Estima- tor (RRNADE), a versatile density estimator for online learning of data streams.
2. Moreover, we propose an RRNADE based Bayes classifier for online classification of streaming data.
Our model uses a recurrent module to maintain a set of sufficient statistics for the future and capture the potential temporal properties of the data. In addition, also uses a neural networks parame- terized Gaussian mixture model as the density module to compute the conditional density function of the current input given the previous data. We theoretically show that RRNADE is strictly more expressive than Gaussian hidden Markov models Bilmes et al. (1998). We present empirical results demonstrating the ability of RRNADE to adapt to concept drifts and approximating density func- tions with sequential relations. Moreover, we conduct extensive experiments on various benchmarks of online classification and show that RRNADE outperforms all the compared methods on almost every dataset. In addition, we further demonstrate the importance of both the recurrent module and the density module in the ablation study.
Related Works For online density estimation on streaming data, many the existing works focus on the adoption of the kernel density estimation (KDE) method (Procopiuc & Procopiuc [2005] Heinz & Seeger 2008 Kristan et al. 2011 (Boedihardjo et al., 2008). These estimators often relies on maintaining and updating (though merging) specific number of kernels while incorporating new instances, while in different fashions. In addition to these methods, KDE-Track (Qahtan et al. 2016) leverages an adaptive resampling strategy to deal with concept drifts and improve the estimation accuracy of the XDE-based methods. Another recent method, adaptive local online kernel density estimator (ALoKDE) (Chen al. 2021), leverages statistical test for concept drift detection to adapt fast to the concept drift. All these methods can be modified to a classification method via a Bayes classifier.
For online classification on streaming data, there are a number of methods that are direct adaptations of the original offline version to its online case. For example, the online SVM (OSVM) (Li & Yu 2015), the adaptive random forest (ARF) (Gomes et al.) 2017a), can be categorised to this type of methods. In addition, (Liang al.) 2006; Cauwenberghss & Poggio 2000; Lu et al. 2014) also belong to this class of methods. Other methods like (Bifet & Gavalda 2007 Bifet et al. 2013) leverage an adaptive window size of the past, (Losing et al. 2016) takes advantage of the short-term and long-term memories, while (Gomes et al. [017b] Polikar et al.] (001) use ensemble method to further improve the results. Another large class of online classification method is the prototype- based classifiers, such as incremental learning vector quantization (ILVQ) (Losing et al., 2015), generalized LVQ (Sato & Yamada 1995), robust soft LVQ (Heusinger et al. 2019). and the sparse prototype online kernel density estimator (SPOK) (Coclho & Barreto 2022)
BACKGROUND
In this section we will background knowledge including the real-valued neural autoregressive den- sity estimator (RNADE), recurrent models. We will also introduce the formulation of the online density estimation and classification tasks.
Real-valued neural autoregressive density estimator (RNADE) The real-valued neural autore- gressive density estimator (RNADE) (Uria et al.] (013)) is a generalization of the original neural autoregressive density estimator (NADE) (Uria al. (0166) to continuous variables. The core idea of RNADE is to estimate the joint density using the chain rule and approximate each conditional density via neural networks, i.e. p(x1,... ... _n)) = I p(rs(x=<)) with p( [x<=)) = Par (a; (0:), where Fzz denotes all attributes preceding v, E IR in fixed ordering PM is a mixture of
m Gaussians with parameters 0, = {G, E R" pe E R", E R. Moreover, we have:
"Later we will also use the notation T. where b N. to denote Betl,""., " Rd
2

---

Under review as a conference paper at ICLR 2023
(o.(0))))) Σーー 1 (x, o where 33 denotes the jth element of B,, same for μ and of and ([r o3) denotes the Gaussian density with mean μ and standard deviation of evalu- ated at 24. Note that G.. H4, o, are functions of z<s. These functions are often chosen to be various forms of neural networks. In the classic setting, RNADE with m mixing components and & hidden states has the following update rules:
h; = h + T 3. = softmax (V; -
(I)
0. hi + by: a: cxp(V; + b,),
(2)
where :: V and V are m x k matrices, by by and by are vectors of size m, and w is size n vector. The softmax function (Bridle [190) ensures the mixing weights B are positive and sum to one and the exponential ensures the variances are positive. RNADE is trained to minimize the negative log likelihood: L(x1 "........pp 0, = -i log(p>s (1,(p via gradient descent.
Recurrent models Recurrent neural networks (RNN) are class of neural networks designed to handle sequential data. An RNN takes as input a sequence (of arbitrary length) of elements from an input space X' and outputs an element in the output space y. In most applications, X is a vector space, typically IR.. RNNs process sequential data by reading one input at time and updating a real-valued vector referred to as the hidden state. Let g : R IRd + I2* be the transition function between the hidden states at time step t - 1: h E Re and at time step t. Formally, we have the following definition for RNNs:
Let X and Y' be the input and output space, respectively. A recurrent model with k hidden states is defined by a tuple R = (g. E, hg) where g : X' x I + R$ is the recurrent function, E: R& +Y is the output function and he E RR is the initial state. A recurrent model R computes function In : x + + defined by the (recurrent) relation:
Definition 1.
[R(01122-- ....2-..0000 v..) x..) = E(hn) where h; g( 11- for 1 St S and œ1, Ø2,,.... Fn X.
The difference of many kinds of recurrent models often resides in the transition functions. For ex- ample, in long short-term memory networks (LSTMss) (Hochreiter & Schmidhuber 1997). through introducing memory cells and gating mechanisms in the transition functions, it avoids the gradient vanishing/explodingg problem that vanilla RNNs often struggle with. In addition, gated recurrent unit networks (GRUs) (Chung et al. 2014) simplify the transition functions of LSTMs and significantly reduce the number of parameters of the model. However, most of these variants focus on the additive relation between the input vector and the hidden state, rarely exploring the multiplicative relation between these vectors. Second order RNNs (2RNN) (Lee et al.) 1986) include both the second order (multiplicative) and the first order relations (additive) in the transition functions.
Online density estimation and classification for streaming data In this paper, we focus on den- sity estimation and classification for streaming data. Formally, for the density estimation task, let S = { En, } be sequential data stream governed by some distribution fz(-), where Ie ~ ft and the subscript denotes the timestamp of the data entry, and fe : Rd + R denotes the distribution at time step f. We are then interested in finding an accurate approximation of the density function f at each time step. For a C classes classification task, let y/ E (1,... ... , C} be the label time& and SL = { 31). v1).... ..... (æ yn) .n).... } be a sequence of input label pairs. Moreover, the label y is drawn from the distribution while x; is drawn from the distribution f i.c. o, f(" where 1 (x:) p(æ, (va.) denotes the input distribution of class y, time step t. In this setting, we are interested in predicting the correct label at each time step t. As we are approaching these tasks in the streaming setting, it is infeasible to store all the data one have seen so far. Therefore, for both of these tasks, we constrain ourselves to only have access to a short window of data at each time step, e.g. .t- Ft. where is the window size and controlled to be a relatively small number.
One of the most challenging problems in learning data streams is the concept drift. Intuitively, a concept drift happens when the underlining distribution ft changes overtime. This change can happen abruptly, when the data changes significantly and occasionally (Iwashita & Papal 2018). This distribution shift can also occur gradually, when the data values slowly but constantly changes over time. In addition, for classification tasks, concept drifts For example, in video frames classification, the goal is to classify different objects that the current frame contains. In this case, shift in the camera angle will result in concept drift in the label's
occur not only in f. but also in q--
3

---

Under review as a conference paper at ICLR 2023
Parl
M
M pianed Kiun)
softmmx
softmax
(xtes 1+3)
(Doua (xp
(Xinea (Kt
Xa-
Kn+3
log
Xn-1-3
Kn-3-8
Xn=
hg
@ (x)-)))))
((x,))
(
The Recurrent Real-valued Neural Autoregressive Density Estimator for online density estimation (left) and online classification (right), where the window size is and the likelihood length p = Blue, orange and green boxes denote input data, functions and outputs, respectively.
Figure 1:
distribution, iee qt. This shift, naturally can occur abruptly (snap movement of the camera), or gradually (steady movement of the camera), resulting in different types of concept drifts.
METHODOLOGY
In this section, we introduce the Recurrent Real-valued Neural Autoregressive Density Estimator (RRNADE): a versatile model for density estimation and classification stream data.
3.1 RECURRENT REAL-VALUED NEURAL AUTOREGRESSIVE DENSITY ESTIMATOR (RRNADE) FOR ONLINE DENSITY ESTIMATION
is natural wonder if it is possible use RNADE for online density estimation. There two major issues for this adaptation: 1) the original RNADE model has one set of parameters per feature, which would lead infinite amount of parameters to estimate for indefinite length of the stream, and 2) the offline stochastic gradient descent routine needs to be adjusted to its online setting.
To tackle the first problem. instead of approximating the conditionals p(æ; (<<)) via the classic RNADE treatment (see Eq. IP we use recurrent model At = {g. &, ho) to model the conditionals: p(xx fn( Iz In contrast with RNADE, using a recurrent model allows us to make the state update function independent of the time step, enabling RRNADE to generalize to sequences of arbitrary lengths. Inspired by RNADE, we constrain the output of the recurrent model to be a mixture of Gaussians with diagonal covariance matrices. We now formally introduce the Recurrent Ralued Neural Autoregressive Density Estimator (RRNADE) model:
Definition 2. A Recurrent Rellev Neural Autoregressive Density Estimator (RRNADE) with l states and components is a tuple R (g, o, ho, where ho E I& is the initial state, g R Rd I& is the recurrent module, & IR Rd + R is the density module and Rd aRd the input encoder. RRNADE computes a function fR (Red)* - R sequence .1,-.. Fn. fR computes in the following fashion:
Given
h; = 1h: (xx)))
3 = softmax 8 h1-1 b²)
(3)
M, = vd h B"
Σ, -expV"
h + B²))
(4)
E(hi-1,0,)) 2.)= = Σ B/N(x, M diag Σ()), fR(I1, ",,, = E(hn-1,,
(5)
1=1
where Da E Rkxmxd Vo Rkxmxxd B Rmxd B Ramxd V E Bmak 6 m M (Mi)..: Rd E. (Ex),, E Rd, diag(E.) denotes the diagonal matrix having the components of ES on the diagonal and V o h denotes the mode-1 product defined by (V h) E, Visahh
'(Rd)* denotes the set of all possible sequences of arbitrary length constructed with d dimensional real- valued vector each time step.

---

Under review as a conference paper at ICLR 2023
As outlined by Equation 5 RRNADE models the conditional density of the current input data, given the history, as a Gaussian mixture. For simplicity, in the rest of the paper. we let d' = d and approximate each conditional via mixture of Gaussian with diagonal covariance matrix. This can be changed to a full covariance matrix, should the corresponding assumption (positive semi- definite) of the matrix is satisfied. Note this simplification does not affect the expressiveness of the model, as a Gaussian mixture with a diagonal covariance matrix is also an universal approximator for densities and can approximate a GMM with full covariance matrix (Benesty et al. 2008), given enough components. Furthermore, although we give specific forms of B., M, and Σ, in Definition]] in practice, one can use any differentiable function of h; to compute G., M, and Σ, so long as B. is positive and sums to one and Σ; is positive definitive. For ease of notation, we will denote by R(h, E") an RRNADE with ho = h and ç = E when necessary.
We show in the following theorem that RRNADE is strictly more expressive than Gaussian HMMs, which are well known for sequential modeling (Bilmes et al.) 1998).
Theorem 3.1. Given a Gaussian HMM with k states, there exists a k states k mixtures RRNADE R with full covariance matrices such that the density function over all possible trajectories sampled by
can be computed by R; p0(11 1.1...20 xnn = =II== (R(IZG)) ) for any trajectory I1 .1....2. Ln. Moreover, there exists an RRNADE R' such that no Gaussian HMM can compute its density.
For the second problem, by design, RRNADE approximates a conditional density function of the current input given the history. In the offline setting, one can learn such RRNADE via gradient ascent of the sequences likelihood, i.e, maximizing the likelihood function: Loffline p(z11.'.. ) _ g.) In the online setting, we assume that we have access to the past data entries from window of size I, i.e., Pt-d, Dt. at each time step f. One solution would be to maximize the likelihood over the entire window, i.e. L( 2x) -1 FR (1))-.))
.un) =
However, this procedure falls short when concept drift occurs. For example, if the data has abrupt drifts (infrequent), then choosing could be a dilemma: to capture and adapt to this concept drift fast, is preferable to be small to provide a significant gradient update, which, however, prevents the model from learning temporal dependencies in the data. To address this issue, we propose to have second hyperparameter p < 7, referred to as the likelihood length, to control the number of past observations that are taken into account in the loss function. That is, RRNADE takes all I past entries as input but the parameters are trained by maximizing the likelihood of only the past p observations. Therefore, not only can we capture temporal dependencies up to the window size but the choice of p enables us to adjust how fast RRNADE adapt to potential concept drifts in the data. Formally, we maximize the following likelihood function:
(6) The core idea of this procedure is to obtain and maintain a set of sufficient statisties for the future. At time i = 1, the model starts with the hidden state hg. After updating the parameters by maximizing
Lontine (#r-1+1,--- .xt)
the likelihood -i 1 (R(h,) (1.)). the internal state of RRNADE is then updated to h1 = g(o, 21). At the next time step, t = 1+1, the first observation x1 is discarded but h1 still represents sufficient statistics of all past observations, including 11. The parameters are then optimized to maximize the likelihood Th - (10 (.)) This process then iterates over the future time steps, carrying the sufficient statistics forward (i.e., at future time step I, even though only the past I observations are stored in memory, h captures sufficient statistics of all past observations).
The training procedure is detailed in Algorithm I and graphical illustration is presented on the left side of Figure [ We use L and L to denote the likelihood function and the log likelihood function, respectively. In our experiments, we select 1 and p using validation over the first n data points. For practical online learning, this can be done by training several RRNADE models in parallel and select the one with the best overall performance after training and predicting on the validation sequence.
3.2 RRNADE FOR ONLINE CLASSIFICATION
One straightforward application of approximating densities is online classification. Recall that RRNADE approximates the conditional density of the current input given the history, i.e., fr (#)()1.) ~ p(0r| ).). For the online classification problem, we are interested in the condi- tional probability of the current label given the history, iee p(y (x1 xz). Using Bayes rule:
p(yal [2) ... -xx) x p(ye)p(a, (et, vi) ~ pl
5

---

Under review as a conference paper at ICLR 2023
Algorithm RRNADE for Online Density Estimation and Classification
1: INPUT: Input data stream S {=1.-- .n..... for density estimation, SL { y1), ... (In: yn).... ... for C' classes classification; window size I; likelihood length p; randomly initialized RRNADE: Ri); for classification, extra C 1 density modules
2:
for -,,++1, .R... do
3:
Compute the log likelihood for each input (and for each class):
4:
Cay ((te)l(((t--))) = log( (og(, u.ee) (z(r)1a))) for j = -- -
5:
if' running density estimation task then Compute the sum of the log likelihood over the past p time steps:
6:
Contine (#1-1415""" 300
Σ
i=t-p+1
7:
Perform gradient ascent update to R(h F1). w.r.t Lontine Obtain the conditional density estimation time step t:
8:
p( ac <t) = (1(th-))((;) (.))
9:
else if running classification task then Compute the predicted class distribution: Po =c Ee, exp(C))
exp(C))
10:
11:
Obtain the predicted label at current time step: Út = arg max, ( =) E1),
12:
Perform gradient descent update to Rh,,-1, E1) C2, .. EC w.r.t. the categorical cross
entropy:
" 2===-pt1 CCE(P
13:
end if
14:
Update h to h via the transition function g of R(h:-i):
h--141 -((4
15: end for
One approach would be to train C different RRNADE models, one for each class: fRo (2)).))))))) ~ p(æ, &ee,, c), where R denotes the RRNADE model for class c. However, as we are learning online, each gradient update is often of high variance. This approach introduces too many model parameters, which will further increase the model's variance, resulting in a suboptimal performance. To reduce the number of parameters, we propose to share the recurrent module of all RRNADE models, while keeping the density module specific to each class. RRNADE's prediction at time f is thus given by
J+ = arg maxip((s = ()
For the choice of the prior distribution p(y:), we recommend using uniform distribution extra effort needs to be taken to mitigate the shift in the true prior distribution caused by concept drifts in the data. We defer the study of estimating proper prior for data streams with concept drifts to the future work. We present RRNADE for online classification in Algorithm T and graphical illustration of the model is presented on the right side of Figure 1]
4 EXPERIMENTS
In this section, we present empirical results on RRNADE for online density estimation and classi- fication. For density estimation, we experiment with synthetic data to evaluate RRNADE's ability of adapting to concept drifts and to verify Theorem 3.1 For classification, we conduct experiments on both synthetic as well as real world streaming data and compare with multiple density based and non-density based online classification methods. Finally, showcase the significance both the density module and the recurrent module of the RRNADE. We experimented with three different variants of the RRNADE model. By using LSTM, GRU and
show ablation study further

---

Under review as a conference paper at ICLR 2023
Abrupt Gaussian
Grodual Dauasian
States
pouto
Trinttps 200
Tirade:: 200
Thmacee
Timanip
Trratto
Figure 2: Top row (left to right): Log likelihood of (la) Gaussian with abrupt drift. (lb) Gaussian
with gradual drift. (1c) Gaussian HMM. Bottom row: Snapshots of learned density at corresponding time steps of Gaussian with abrupt drift, red curve represent the ground truth density function.
KDE
RRNADE -2RNN
RRNADE -LSTM
RRNADE
AUC
ALoKDE
oKDE
odKDE
LAIM
-rrck
-GRU
Sea
83.11
78.74
75.13
77.17
81.29
(3.42 (1.9)
88.88 (2.3)
Hyperplane
83.44
82.36
82.36
75.56
82.8
93.12 (2.9)
93.14
94.32 (3.1)
Mixed drift
90.12
88. 16
70.43
55.65
88.48
98.13 (1.2)
97.66(1.1)
96.11 (1.9)
Transient chessboard
79.6
77.4
77.08
77.18
77.98
89.16 (3.6)
86.39
91.34 (3.2)
Weather
76.81
68.43
66.26
73.47
76.
85.23 (6.4)
86.67 (3.3)
84.14 (2.2)
Flletricity
52.51
51.57
42.45
43.25
44.06
92.14 (4.5)
90.68 (5.3)
91.01 (6.7)
Cover type
97.31
57.86
96.67
93.04
96.04
98.01 (1.1)
93. (1.8)
90.79 (2.7)
Poker hand
91.01
88.36
82.92
82.19
91.01
94.39 (2.8)
96.24 (3.4)
93.15 (2.9)
Rialto
92.67
70.55
87.71
83.49
82.37
99.11 (0.9)
98.13 (1.5)
94.35 (0.8)
Table 1: Averaged online AUC score of RRNADE over 5 seeds (standard deviation in the brackets), compared with (Chen et al. 2021
2RNN for the recurrent module, we have RRNADE- ARNADE-GRU and RRNADE-2RNN, respectively. In all experiments, we use Adam optimizer (Kingma & Ba 2015).
4.1 DENSITY ESTIMATION
To evaluate the performance on density estimation, we first conduct experiments on learning drifting Gaussian to examine RRNADE's ability of adapting to concept drifts. We generate samples from a shifting Gaussian with random initial mean and variance 1. Recall there are two major types of concept drifts, abrupt and gradual drift. For the abrupt drift, the mean is increased by 2 every 100 time steps, while for the gradual drift, the mean is increased by 0.01 every time step. For the model hyperparameters of RRNADE, we set the number of components to be 10, both the window size and the likelihood size is set to 1, number of hidden states of the recurrent module is set to and we use one layer fully connected neural networks with 5 neurons to be the input encoder ő.
In Figure 2 (1a) and (Ib), we show the log likelihood of Gaussian density function with abrupt and gradual drift on its mean. From the figures, we can that all three different variants of RRNADE are able to learn the density function and adapt to both of these drifts. The black boxes in (Ia) indicates the first two abrupt drifts (time step 100 and 200) of the Gaussian distribution, where visible drops of model performance are observed. Moreover, the adaptation speed increases w.r.t. the time step. The bottom row of Figure 2 shows the learned Gaussain at various time steps. We observe that, at time steps 210 the mean of the mixture model has not been correctly adjusted after 10 time steps of adaptation, while at 410, the model has already adapted to the drift occurred at time step 400.
The results are displayed in Figure 2
To verify Theorem 3.1 and show RRNADE can approximate density functions of data streams with sequential features. We generate 1,000 examples from a random Gaussian HMM of 3 states. In this experiment, we set the hyperparameters to be the same as in the above experiment except = p = 5. In Figure (Ic), (1c), we show the learning curves w.r.t. log likelihood on all three variants. Here we can see all three variants are able to approximate the density function that the HMM emits at each time step. Note there is also a gradual concept drift with the HMM data, the density function at each time is a mixture of a set of Gaussians, where the mixing weights are the state distributions that the HMM maintains.
7

---

Under review as a conference paper at ICLR 2023
RRNADE
RRNADE
RRNADE
Accuracy
ISVM
LASVM
ORF
IIQQQ
LPP
IFLM
SGD
NB
-2RNN
-LSTM
-GRU
Electricity
N/A
N/A
0.699
0.725
0.675
0.548
0.846
0.632
0.864 (0.02)
0.878 (0.01)
0.858 (0.01)
Inter RBF
N/A
N/A
0.459
0.768
0.294
0.295
0.443
0.299
0.901 (0.02)
0,922 (0.02)
0.924 (0.03)
Moving RBF
NIA
N/A
0.456
0.766
0.18
0.159
0.406
0.172
0.744 (0.01)
0.739 (0.02)
0.772 (0.01)
Cover Type
N/A
N/A
0.896
0.883
0.397
0.513
0.946
0.546
0.960 (0.01)
0,999 (0.02)
0.951 (0.02)
Border
0.985
0.976
0.94
0.947
0,884
0,88
0.375
0.944
0.955 (0.05)
0.957 (0.02)
0.943 (0.03)
Overlap
0.817
0,788
0.782
0.811
0.727
0.748
0.679
0.675
0.754 (0.01)
0.734(0000)
0.770 (0.01)
Outdoor
0.864
0.823
0.342
0.826
0.685
0.733
0.18
0.65
0.958 (0.03)
0.963 (0.02)
0.942 (0.02)
COIL
0.754
0.663
0.666
0.791
0,587
0.631
0.096
0.702
0.851 (0.03)
0.859 (0.01)
0.832 (0.02)
Table 2: Averaged online accuracy of RRNADE over 5 seeds with standard deviations, compared
with (Losing et al. 2018)
RRNADE -2RNN
RRNADE
RRNADE
Accuracy
L++.NSE
DACC
LVGB
KNNs
KNNwa
SAM
SPOK
-LSTM
-GRU
CoverType
0,850
0.899
0.909
0,958
0.932
0.952
0.883
0.960 (0.01)
0.929 (0.02)
0.951 (0.02)
Electricity
0,728
0.831
0.832
0.713
0.739
0.825
0.742
0.864 (0.02)
0.878 (0.01)
0.858 (0.01)
Outdoor
0.422
0.644
0.601
0.86
0.837
0.888
0.810
0.958 (0.03)
0.963 (0.02)
0.942 (0.02)
Poker Hand
0.779
0.790
0.864
0.829
0.721
0.816
0.731
0.847 (0.02)
0.851 (0.02)
0.840 (0.01)
Rialto 0,596
0.711
0.604
0.772
0.750
0.814
0,618
0.887 (0.05)
0.936 (0.04)
0.915 (0.04)
Weather
0.771
0.732
0.781
0.785
0.769
0.783
0.741
0.786 (0.00)
0.790 (0.01)
0.783 (0.01)
Table 3: Averaged online accuracy of RRNADE over seeds with standard deviations, compared with (Coelho & Barreto 2022)
4.2 CLASSIFICATION
In this subsection, we present experiment results for online classification on various classic bench- marks of online classification and compare RRNADE to both density based as well as non density based methods. The results of these methods are obtained from the corresponding papers. To keep the fairness, we conduct data preprocessing and evaluation procedure the same way as mentioned in each of these papers. We validate on the first 1,000 examples or first 10% of the data (whichever is smaller) to select the number of mixture components, number of hidden states, window size I as well as likelihood length p. while 7 is set to be no larger than 10. The input encoder is still a one layer fully connected neural networks with number of neurons being one of the hyperparameters as well. This validation routine is consistent with all three papers we compare with.
First, we compare with several density based classifiers listed in (Chen et al. 2021). Note the "Cover type", "Poker hand", "Transient chesshoard", "Rialto" as well as "Mixed drift" are originally multi- class data streams. Following the same procedure as in (Chen al. (02I), we generate their binary versions by extracting the largest two classes from each data stream, respectively. The AUC scores on various datasets are presented in Table TT From this table, we can see all RRNADE variants consistently outperforming the other methods. In addition, in many datasets, e.g. "Electricity", "Hyperplane" etc., we outperform the best compared method by a significant margin.
Second, we compare with multiple non-density based classification methods in both (Coelho & Barrrtoo 2022, Losing et al. 2018). The running average of the classification accuracy is reported respectively. In Table N/A indicates that the corresponding result is not reported in the original paper. We can see that in almost every datasets, we achieve competitive results, if not significantly better. For synthetic datasets, in "Inter RBF". various Gaussians are replacing each other every 3000 samples, representing an abrupt concept drift, while the dataset "Moving RBF" is constructed such that Gaussian distributions with random initial positions, weights and standard deviations are moved with constant speed, representing a gradual concept drift. Here can see, on both of these datasets, outperform other methods, further showcasing RRNADE's ability to adapt to concept drift. For real world data, "Weather", "Flectricity", "Outdoor", "COIL." as well as "Rialto" are all data streams with sequential dependencies. Here we can also see that on these datasets, RRNADE outperforms other compared methods.
in Table and Table 2
4.3 ABLATION STUDY
In this ablation study, we are looking into the significance of RRNADE's two components, ie.. the density module & and the recurrent module g. We compare RRNADE with three baselines: RRNADE without recurrent module (NR), RRNADE without density module (ND), as well as RRNADE without both recurrent and density module (NRND). For NR, replace the recurrent
8

---

Under review as a conference paper at ICLR 2023
Accuracy
RRNADE
2RNN
RRNADE
LSTM
RRNADE
GRU
NR
NRND
LSIM
ND
2RNN
GRU
ND
Border
(396
0.96 (802)
0:34 (013)
0:3010055)
0.30
1.51 [002)
8.50 (2:2)
(131 (0:01)
Overlap
0.75 (8101)
0.73 (0.0))
0.77 (8.01)
0.75 (0.05)
0.6 (0.12)
0.64 (0.01)
8.65 (0.02)
0.66 (0.02)
Inter RBF
0.92 0.90:0002)
0.92 (0.83)
0.87 (0.02)
(0.05
0.57 (0.09)
0.66 (0:08)
0.79 (0.04)
Moving RBF
0,74 (0.01)
0,74 (0.02)
0.77 (0.81)
0.51 (0.02)
0.42 (0.01)
0.59 (0/03)
0.44 (0,02)
0.58 (0,04)
MRN tpar
Ouadoor
0.95 (0.03)
0.96 (0.92)
0.94 (0.02)
0.98 (0.01)
0.31 (0.02)
0.42 (0.02)
0.45 (0.03)
0.39 (0.02))
Wenther
0.79 (0.00)
0.79 (0.01)
0.75 (0.01)
0.78 (0.11)
0.78 (0.01)
0.79 (0.01)
0.78 (2:01)
0.79 (0.30
Rialo
(.89:
0.94 (0.04)
0.92 (0.04)
0.82 (0.6))
0.73 (0.01)
0.92 (0.04)
0.53 (0.05)
0.93 (0.03)
Figure 3: Comparison of RRRNDE with three baselines NR, ND, NRND
module of RRNADE by a two layered fully connected neural networks, which only takes the data at the current and previous time step as the input. For ND, we replace the density module of RRNADE by a two layered fully connected neural networks, while for NRND, a two layered neural network used to map from the current step input to its label. To correspond to the three variants of RRNADE, ND also has three different recurrent units, namely GRU, LSTM, and 2RNN. A graphical illustration of the baseline models can be seen in Figure 4 in the Appendix. The hyperparameters are selected using validation in the same fashion as we mentioned before. We present the results in Table 5
From the table, we can see that on all examined datasets, RRNADE outperforms all baselines. In some datasets, c.g. "Moving RBF" and "Inter RBF", only by using both of the modules, i.c. the density module and the recurrent module g. can the model reach to the best performance. However, on some other datasets, e.g. "Rialto", the missing of the recurrent module alone is detrimental, while for datasets like "Overlap", the density module is of great importance. On one hand, recurrent module captures the temporal relations in the data, which in some cases are the key to predicting the correct label. In the top right figure of Figure 3 we show the learning curves of the three variants of RRNADE and NR on "Rialto" " dataset. We can see clearly that NR converges slower RRNADE. This due to the fact that the label has a specific
and to a worse solution compared ordering, which, if not using the recurrent module, is hard to capture. On the other hand, the use of density module provides a proper inductive bias in some cases, which accelerate the convergence and improve the final solution. For example, both "Inter RBF" and "Moving RBF", as explained earlier, are generated from Gaussians with concept drifts. Therefore we can see that they overall have much better performance with the density module. In addition, we train the NRND model on "overlap" dataset in an offline fashion using gradient descent with batch size I without random permutation of the data. Note under this training routine, the first epoch is equivalent to learning under streaming setting. In bottom right figure of Figure [3 we show the learning curves of such trained NRND model of the first 10 epochs (the increasing color gradient corresponds to the increase in the number of epochs) as well as of NR and RRNADE-GRU. As the number of epochs increases, NRND slowly reaches to the performance of NR. However, in the first epoch, i.c. the online setting, we can see although all the methods start at close positions, the ones with density module improve much faster.
5 CONCLUSION
In this paper, we propose the Recurrent Real-valued Neural Autoregressive Density Estima- tor (RRNADE), an extension of the classic RNADE model to its online setting. The core idea of RRNADE is to maintain set of sufficient statistics for the future via recurrent function (recur- rent module) and approximate the conditional density function using mixture of Gaussian (density module), parameterized by neural networks. We show that theoretically, RRNADE is strictly more expressive than the Gaussian hidden Markov model, which is a classic model for learning sequential data. We then propose learning algorithms for using RRNADE on online density estimation and classification of data streams, respectively. For the empirical studies, we conduct experiments on synthetic data showing RRNADE is able to learn the density function parameterized by a Gaussian HMM and RRNADE is efficient in adapting to concept drifts. For the classification tasks, we com- pare RRNADE with various methods on multiple synthetic and real world datasets. We show that RRNADE outperforms all the methods on almost every dataset. In the ablation study, we further showease the importance of both the recurrent module and the density module, where the recurrent module helps capture the sequential dependencies of the data stream, while the density module helps with the online optimization. For the future work, recall that we use uniform distribution as the prior for RRNADE classification tasks. In the future, would like to investigate more adaptive way of estimating the prior. In addition, since our model is a density model, it would be interesting to investigate the possibility of online clustering.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jacob Benesty, M Mohan Sondhi, Yiteng Huang, et al. Springer handbook of speech processing, volume 1. Springer, 2008.
Albert Bifet and Ricard Gavalda. Learning from time-changing data with adaptive windowing. In Proceedings of the 2007 SIAM international conference data mining, pp. 443-448 SIAM, 2007.
Albert Bifet, Bernhard Pfahringer, Jesse Read, and Geoff Holmes. Efficient data stream classifica- tion via probabilistic adaptive windows. In Proceedings of the 28th anmual ACM symposium on applied computing, pp. 801-806, 2013.
Jeff A Bilmes et al. A gentle tutorial of the em algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. International Computer Science Institute, (510))126, 1998.
Arnold Boedihardjo. Chang-Tien Lu, and Feng Chen. A framework for estimating complex prob- ability density structures in data streams. In Proceedings of the 17th ACM conference on Infor- mation and knowledge management, pp. 619-628, 2008.
John Bridle. Probabilistic interpretation of feedforward classification network outputs, with rela- tionships to statistical pattern recognition. In Neurocomputing, pp. 227-236. Springer, 1990.
Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. Advances in neural information processing systems, 13, 2000.
Zhong Chen, Zhide Fang, Victor Sheng. Jiabin Zhao, Wei Fan, Andrea Edwards, and Kun Zhang. Adaptive robust local online density estimation for streaming data. International Journal of Ma- chine Learning and Cybernetics, 12(6):1803-1824 2021.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 1412:: 2014.
David N Coelho and Guilherme A Barreto. A sparse online approach for streaming data classifica- tion via prototype-based kernel models. Neural Processing Letters, pp. 1-28, 2022.
Mirko Forti. The deployment of artificial intelligence tools in the health sector: privacy concerns
and regulatory answers within the gdpr. Eur. J. Legal Stud., 13:29, 2021.
Heitor M Gomes, Albert Bifet, Jesse Read, Jean Paul Barddal, Fabrício Enembreck, Bernhard Pfharinger, Geoff Holmes, and Talel Abdessalem. Adaptive random forests for evolving data stream classification. Machine Learning, 106(9):1469-145) 1495, 2017a.
Heitor Murilo Gomes, Jean Paul Barddal, Fabrício Enembreck, and Albert Bifet. A survey on ensemble learning for data stream classification. ACM Computing Surveys (CSUR), 50(2):1-36, 2017b.
Christoph Heinz and Bernhard Seeger. Cluster kernels: Resource-aware kemel density estimators over streaming data. IEEE Transactions on Knowledge and Data Engineering, 20(7):880-893, 2008.
Moritz Heusinger, Christoph Raab, and Frank-Michael Schleif. Passive concept drift handling via momentum based robust soft learning vector quantization. In International Workshop on Self- Organizing Maps, pp. 200-209. Springer, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Adriana Sayuri Iwashita and Joao Paulo Papa. An overview concept drift learning. IEEE access, 7:1532-1547, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations (ICLR), 2015.
10

---

Under review as a conference paper at ICLR 2023
Matej Kristan, Alevs Leonardis, and Danijel Skoucaj. Multivariate online kernel density estimation with gaussian kernels. Pattern Recognition, 44(10-11):2630-66666 2011.
YC Lee, Gary Doolen, HH Chen, GZ Sun, Tom Maxwell, HY Lee, and C Lee Giles. Machine learning using a higher order correlation network. Physica D: Nonlinear Phenomena, 22(1-3): 276-306, 1986.
Xiaoou Li and Wen Yu. Data stream classification for structural health monitoring via on-line support vector machines. In 2015 IEEE forst international conference on big data computing service and applications, pp. 400-405. IEEE, 2015.
NangYing Liang, Guang-Bing Bin Huang, Paramasivan Saratchandran, and Narasimhan Sundararajan. A fast and accurate online sequential learning algorithm for feedforward networks. IEEE Transac- fions neural networks, (116::1411-422) 2006.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Viktor Losing, Barbara Hammer, and Heiko Wersing. Interactive online learning for obstacle classi- fication on a mobile robot. In 2015 international joint conference on neural networks (ijenn), pp- 1-8. IEEE, 2015.
Viktor Losing, Barbara Hammer, and Heiko Wersing, Knn classifier with self adjusting memory for heterogeneous concept drift. In 2016 IEEE 16th international conference on data mining (ICDM), pp. 291-300. IEFR, 2016.
Viktor Losing, Barbara Hammer, and Heiko Wersing. Incremental on-line learning: A review and comparison of state of the art algorithms. Neurocomputing, 275:1261-1274, 2018.
Yanyun Lu, Khaled Boukharouba, Jacques Boonært, Anthony Fleury, and Stéphane Lecoeuche. Ap- plication of an incremental svm algorithm for on-line human recognition from video surveillance 2014.
using texture and color features. Neurocomputiig, 126:
Muhammad Naeem, Tauseef Jamal, Jorge Diaz-Martinez, Shariq Aziz Butt, Nicolo Montesano, Muhammad Imran Tariq, Emiro De-la Hoz-Franco, and Ethel LeLLa-Hoz-Valdingg Trends and future perspective challenges in big data. In Advances in intelligent data analysis and applica- tions, pp. 309-322 Springer, 2022.
Robi Polikar, Lalita Upda, Satish S Upda, and Vasant Honavar. Learn++: An incremental learning algorithm for supervised neural networks. IEEE transactions on systems, man, and cybernetics, part C (applications and reviews), 31(4):497-508, 2001.
Cecilia M Procopiuc and Octavian Procopiuc. Density estimation for spatial data streams. In Inter- national Symposium on Spatial and Temporal Databases, pp. 109-126. Springer, 2005.
Abdulhakim Qahtan, Suojin Wang, and Xiangliang Zhang. Kde-track: An efficient dynamic density estimator for data streams. IEEE Transactions on Knowledge and Data Engineering, 29(3):642- 655, 2016
Atsushi Sato and Keiji Yamada. Generalized learning vector quantization. Advances in neural information processing systems, 8, 1995.
Benigno Uria, lain Murray, and Hugo Larochelle. Rnade: The real-valued neural autoregressive density-estimator. Advances in Neural Information Processing Systems, 26, 2013.
Benigno Uria, Mare-Alexandre Côté, Karol Gregor, lain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. The Journal of Machine Learning Research, 17(1):7184- 7220, 2016.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995. PMLR, 2017.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 PROOF OF THEOREM
Theorem A.1. Given Gaussian HMM with l states 1 (,,,,O', where O R x Rd R is the Gaussian emission function, μt E R is the initial state distribution and T E [0, is the transition matrix, there exists a l states i components RRNADE R with full covariance matrices such that the density function over all possible trajectories sampled by b be computed by R:
y (x)) ... tn) = I JR( (at-n) ) for any trajectory 21 ... Lnn Moreover, there exists an RRNADE R' such that no Gaussian HMM can compule its density.
Proof. For the Gaussian HMM 77, given an observation sequences T1 .... Enn its density under 1) is: ..... ... = O(m .Z (0mm T x2) O(m TT"-1, ),
where ( x) = Σi=1 h,NVa p1, Σ) for some mean vector μ and covariance matrix Σ. Let
R = (ho. 9.0.9.C. G.C. be an RRNADE, more specifically, let he m, gh,, = A o h 2 ((x)
A.,.: T for E [k). Φ(x)
T E.
K and & O. Note it reasonable to let & = O since
as long
let B3 hg T B ho. ,4 1 and Σ; E, following equations 4 then
for any h E &R=,, x E IR, we have E(h, x) = Oh, x). Then under this parameterization, we have
A 2 (x,,) T. Then the RRNADE computes the following function:
.. x..) = E((AA 1 to 2 x(x1)) (A 22 o(x2)) ... (A (A ((x;,1)), x;))
E(ho T x;) = O(m TT x.))
Therefore, we have:
po ...) = 0(mm 0n )0(m T O(m Tn-1 xnn)
=fR(=1)fR(011 .. f2(x1,"". ... ,x,) II fR (xgn) 1=1
For the proof of the second half of the theorem, consider a shifting Gaussian HMM, where the mean vector of the Gaussian emission is function of the time steps, i.e., μ (1). where i = 1, .... For simplicity, assume the shifting Gaussian HMM is for one dimensional sequences and has one components. In addition, let (i) i and assume the variance is 1. Then the emission function can be written as Oz(x) = (altt, 1). Then the density of a sequence ø1. En under this shifting Gaussian HMM 17s is:
ndh (21.-.....) .2,) (zx)))) O0...)....)))))))
We show that this density cannot be computed by a Gaussian HMM of finite states. If pd can be computed by a Gaussian HMM, then for the mean vector μ there exists an initial weight vector m, a transition matrix T satisfying the following linear system:
m 1 m Tp
:
m TTm-1 -
=
This linear system is, however, overdetermined, as μ is a vector of finite size, while there are infi- nite linearly independent equations satisfy. Therefore, Gaussian HMM of finite states cannot compute the density function of a shifting Gaussian HMM.
1 1
We now show such density can be computed by a RRNADE. Let hT = [1, 1]. and A, =
0 1
M, = (h4-1, 1)), Mxx) T = [0.5, 0.5). Σ; = 1. Then we have: fx(z1,". .x:) = =E((A + h 22 Φ()1)) ( ((zz))))))) ... (A 02 ((z--))))
=((h, T xx)))) = E[[1, = (z.)i.))
12

---

Under review as a conference paper at ICLR 2023
Therefore:
(I ... (.))
= ,,,
n
= ( .. [n)) II fm (m)n)
==1
Therefore, for the given shifting Gaussian HMM density, can be computed by RRNADE, but cannot be computed by a Gaussian HMM with finite states.
A.2 THREE BASELINE MODELS FOR ABLATION STUDY
For NR, we replace the recurrent module of RRNADE by a two layered fully connected neural networks, which only takes the data at the current and previous time step the input. For ND, replace the density module of RRNADE by a two layered fully connected neural networks.
pn+1
Pa
softmax
softmax
P
(xn+1 X1)
[ log
(Xn+1 (1I,nn)
softmax
FC
Xn+1
Xn+1
hn-1
g
g
FC
(xxn)
((Xn-1+1)
(x
(xxx)
((x,)
Figure 4: The three baseline models NR (left), ND (middle), NRND (right).
13


Reviews:
[{"review_id": "vSL2fFcIqc2", "reviewer": "Reviewer_BvnL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of density estimation and streaming data classification in the online learning setting. It proposes a new algorithm -- Recurrent Real-valued Neural Autoregressive Density Estimator (RRNADE), which combines recurrent module and existing work (RNADE). The paper provides some theoretical results and detailed experiment results to show the effectiveness of the proposed algorithm. ", "summary_of_the_review": "Considering the lack of innovation, the tricky experimental results, and some typos, I tend to give a rejection.", "strength_and_weaknesses": "## Strength\nThe paper extends a previous work, RNADE, to the online learning setting. The algorithm is clearly stated and easy to understand. The experiment includes many baseline algorithms to make a detailed comparison. \n\n## Weakness\nThe novelty of the proposed algorithm is limited. The proposed algorithm is very similar to an existing work, RNADE. Specifically, based on Eqn. (1)~(5), the main difference between RRNADE and RNADE is the usage of the recurrent unit. Additionally, RNN is a popular tool in the online learning literature. I don't think combining the two technologies is innovative enough to meet ICLR's standards.\n\nThe experiment results are somewhat tricky. Specifically, the performance is highly related to the RNN type, which leads to some tricky results. For example, in line \"Cover type\" of Table 1, the usage of \"2RNN\" seems to be necessary, while \"GRU\" appears to be necessary sometimes (line \"Moving RBF\" in Table 2). Doing so is unfair to other baseline algorithms because their structures are fixed rather than carefully tuned by the authors. Meanwhile, the details of how the baseline algorithms are implemented should be written in detail. Another question is how we choose the type of RNN in reality. \n\nThe writing needs to be polished. After reading the first paragraph in the introduction, I feel like I am reading an early-stage draft. For example, the \"he\" should be \"The\", and the punctuation is missing in the last sentence. The current version is apparently not ready to be published on ICLR. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "mngU42S26J", "reviewer": "Reviewer_ybf6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for learning data probability density functions in an online manner. The proposed method introduces a recurrent module into a neural Gaussian mixture density model. The method takes into account temporal dependencies by allowing the output depend on the past observations within a fixed window. It also allows us to tune how adaptive the model is by specifying the window over which the likelihood is maximized. The authors show that the proposed model is strictly more expressive than Gaussian HMMs. The experiments on the classification application show that the proposed method often outperforms existing methods.", "summary_of_the_review": "The idea of introducing a flexible recurrent module to RNADE seems interesting, and the resulting proposed method is effective according to the experiments. However, the paper is not clear enough and does not present the contribution in a convincing way. I think the paper falls a little below the acceptance bar for those reasons.", "strength_and_weaknesses": "# Strengths\n- The results for the experiments are impressive.\n- The authors point out an important weakness of the existing model, RNADE, on which the proposed model is built: it only allows an additive update to the hidden state. The proposed method improves on this point.\n- They prove that the proposed model is strictly more expressive than Gaussian HMMs.\n\n# Weaknesses\n- The proposed method has several hyper-parameters that are not systematically selected in the experiments.\n- The proposed method introduces an additional complexity the existing model by introducing a flexible recurrent module.\n- The main contribution of the work is not clear to me from the paper. It seems RNADE can be almost directly applicable to the online setup. The difference I can see from RNADE is removing the limitation of the recurrent module being additive and allow it to be any function (the first eq. of Eqs. (3)). But the paper does not provide enough evidence showing that the modification is really worth the complication.\n- The problem setup is not mathematically defined. It is not clear what distribution drift or what evaluation metric the paper considers. Without defining those, I don't think we can even discuss the learnability issue. They use terms such as \"abrupt concept drift\" or \"smooth drift\", but they are not clearly defined.\n- The paper does not explain how and how much the update (line (7) and (12) of Algorithm 1) is performed. For example, the learning rate should matter in online learning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kWluV794YE7", "reviewer": "Reviewer_qnzA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of online density estimation and classification of streaming data. A new model named RRNADE (Recurrent Real-valued Neural Autoregressive Density Estimator) is proposed for problem-solving. In RRNADE , a recurrent module is used to maintain a set of sufficient statistics for the future state, and a neural net-based mixture of Gaussian module is used to approximate the conditional density function. The authors also prove theoretically that RRNADE is strictly more expressive than the Gaussian hidden Markov model. Experimental studies on both synthetic and real-world datasets are conducted to show the effectiveness of RRNADE.", "summary_of_the_review": "The paper does tackle an important problem for stream data modeling. However, the technical contribution is not strong enough to support acceptance. So it is unclear to me whether the contribution of this paper is significant. The paper should also consider more datasets and baselines. Thus, overall I can not recommend acceptance.", "strength_and_weaknesses": "Strengths:\n+ The motivation to use a recurrent module and a mixture of Gaussian (parameterized by NNs) is interesting;\n+ The theoretical result verifies, in theory, the potential merits of RRNADE\n\nWeaknesses:\n+ The technical contribution is generally limited since RNADE and recurrent models (RNN as a representative) are existing works. Authors do not show significant novelty in framework/algorithmic development \n+ Theorem 3.1 is not easy to understand. What is the assumption behind the theory? After presenting mathematically the theorem, further remarks and/or discussion on its meaning and usefulness are needed. This may help readers follow the key point behind the theoretical result.\n+ It is not clear to what extent and how the proposed RRNADE outperform RNADE. More discussion and empirical studies on clarifying this issue should be beneficial to enhance the contribution.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "hCnkWBX-00K", "reviewer": "Reviewer_fRiQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of density estimation and classification on streaming data. The key component of the proposed method is RRNADE, a recurrent extension of NADE. NADE models the density of each feature given previous features, i.e., $p(x^i|x^{<i})$. Similarly, the authors model the density of the current sample given previous samples, i.e., p(x_t|x_{<t}) for the streaming data. They introduce RNN to capture statistics of earlier samples allowing for variable length history. This method maintains a memory of a few past samples. To account for the drift in data distribution, the model is updated with the likelihood of recent samples rather than the entire history that may be available in memory.\n\nRRNADE is extended to predict class labels by learning class conditional density models, and prediction is made with the Bayes rule. \n\nThe authors demonstrate the superiority of their method theoretically by showing that RRNADE is more powerful than gaussian HMM and empirically by evaluating on a synthetic dataset for density estimation and several real-world binary classification tasks.\n", "summary_of_the_review": "Overall, the proposed method is interesting and relevant. However, many experimental details are missing. Moreover, some important neural network-based baselines are missing. Therefore, I am leaning towards rejecting a paper and would suggest revising the paper. I am willing to increase the score if the authors address the missing baselines and update the manuscript with experimental details.", "strength_and_weaknesses": "Strengths:\n- This paper addresses an important problem of learning from streaming data, and the proposed method is shown to outperform several baselines. \n- The method is easier to understand and simple to follow. However, some parts are unclear, and the writing could be much better (see weakness)\n\nWeaknesses: \n- A description of baselines needs to be included; only pointers to previous papers that use these baselines are provided. Authors should make the papers as self-contained as possible. \n- Another problem is that most baselines use non-deep learning methods. Authors do not compare/cite deep learning methods for online learning, such as [1,2]. How does your method compare to these?\n- The complexity of tasks is unclear. Mainly authors have not provided any description (e.g., # features, # samples, what is being predicted, etc.) of the datasets, which makes it harder to judge the complexity of tasks. \n- Theoretically, RRNADE can solve multi-class classification problems. However, it is only empirically evaluated on binary classification tasks. Would we see similar empirical improvements on multi-class problems?\n\n\n[1] Online Deep Learning: Learning Deep Neural Networks on the Fly, Sahoo, et al., IJCAI 2018 https://openreview.net/forum?id=SyWKcXMOWH \n[2] Adaptive online incremental learning for evolving data streams, Zhang, et al., Applied Soft Computing 105 (2021) https://arxiv.org/abs/2201.01633\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "egfznYapprW", "reviewer": "Reviewer_7rkR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper looks at the task of online density estimation and classification. This is motivated by the problem of memory constraints, concept drifts, and temporal correlations. To this end, they propose Recurrent Real-valued Neural Autoregressive Density Estimator (RRNADE) which is an extension of the previous work RNADE with a recurrent neural network component. This new model is theoretically more expressive than Gaussian HMM and empiracally better than selected baselines.", "summary_of_the_review": "Overall, I vote for reject. The task is well motivated and the model is clearly presented. My major concerns are about clarifications on the related work and the experiences (see cons). Hopefully the authors can address my concern in the rebuttal period.", "strength_and_weaknesses": "Pros:\n\n1. The task of online classification and density estimation is important and well motivated.\n2. The method is simple, clear, and brings some practical improvements. The model section is easy to follow.\n\nCons:\n\n- Related works: I feel that the related work section could be more detailed. The authors mention continual and increamental leanring as related problems but these are not discussed in the related work. It might also be interesting to discuss the related works around other density estimation methods (e.g. Normalizing Flows). Further the concept of drift could also be more detailed. E.g. (Generalized Out-of-Distribution Detection, Yang et al) makes the distinction between sensory/covariate and semantic shifts. Action suggestion: Enrich the related work wth some discussion on continual learning, density estimation methods, and shift types.\n- Mathematical notation: I was confused with the not bold x notation in formula (1), (2) vs the bold x notation in defintion 1. What are the difference between these two notations ? I was also confused with the argument of f_R in the definition 1 which looks like a multiplication of all x_1, x_2, … and x_n. Should it be f_R(x_1, x_2, x_n). Action suggestion: Fix and clarify the mathematical notations.\n- Experiences:\n    - The importance of the hyper-parameters l and p is not clear. It would be interesting to have one experiment where these hyper-parameter are changing. It allows practionners to know what is the impact of these introduced hyper-parameters. Action suggestion: Make an experiment when l and p vary.\n    - I had the feeling that the test in Fig. 2 text was very small and not easy to read withou important zooming. Action suggestion: Make the text bigger.\n    - The experiments on density estimation focus on only on synthetic toy datasets. While these experiments are interesting since they provide the ground-truth, I feel that it is important to test also on real-datasets. Further, it would also be important to compare to other methods. Currently results only show RRNADE results on Fig. 2. Action suggestion: add real-dataset and other baselines in the density estimation experiments.\n    - For completeness, it would be nice to have details on the used datasets in the paper or appendix. Further, these datasets look fairly small datasets and require only very simple input encoder. E.g. RNADE paper look also at image and maybe speech datasets. Action suggestion: consider larger datasets with more complex input encoder.\n    - Since RRNADE is an extension of RNADE, it would be interesting to have some comparison of the two methods in the experiments to show that RRNADE fixes practical limits of RNADE (e.g. number of parameters). Action suggestion: Make a comparison of RRNADE and RNADE.\n    - “The results of these methods are obtained from the corresponding papers.”: It sounds that the the baselines results are simply copy-pasted from the previous papers. I feel that this is generally a bad practice. For a precised evaluation, methods should be implemented using the same framework and use the same evaluation pipeline. In partciular, it is imposible to know how ISVM and LASVM perform compare to RRNADE on four datasets. Further, the baselines do not have error bars. Action suggestion: Ideally, run each baseline on each datasets and get error bars. If this is too difficult in the available time, I would expect the repoduction of at least the two strongest baselines and complete the N/A missing numbers.\n\nOthers:\n\n- Typo: “**T**he problem” first paragraph on p.1\n- Typo: “constrain**t**s” second paragraph on p.1\n- Cite RNADE paper  when introducing it on p.1\n\nI am happy to improve my score if a majority of the above points are addresses (e.g. with the action suggestions).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Lfx3dfBktVS", "reviewer": "Reviewer_jzmZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a proposal to enhance real-valued neural autoregressive density estimator (RNADE) by utilizing RNN-based mechanisms to be applicable over a concept-drifting data stream for density estimation and classification tasks. Using RNNs for estimating mean and variance, and constraining the number of parameters, the authors tweek the RNADE method for online settings. Empirical results over benchmark stream datasets show that the proposed approach performs best in majority of cases, both for density estimation and classification tasks, even under synthetically generated concept drifts.", "summary_of_the_review": "Overall, though the proposed approach seemingly works well on typical benchmark datasets, the quality of evaluation needs to dramatically improve, particularly with the use of better datasets (since a NN is being used), and better competitive approach should be used for comparison. Furthermore, the paper is missing key details such as cold-start problem, drift detection etc.", "strength_and_weaknesses": "Strengths:\n- Paper describes the proposed approach well conceptually. The detailed update equations clearly illustrates the contribution.\n- Empirical results show good performance of the proposed method on benchmark stream datasets typically used in data stream papers.\n\nWeaknesses:\n- The proposed approach seem simple compared to the existing RNADE method. It would be better to discuss problems with data stream when using RNADE with a bit more deeper insight rather than stating the 2 reasons provided. Moreover, the reason of number of parameter size going to infinity in RNADE seems like a stretch.\n- Empirical evaluation is not strong enough to show efficacy of the proposed approach. It would be better if more complex datasets are used apart from the benchmark datasets, particularly using image or text datasets as mentioned in the paper's motivation. Furthermore, the competitive approaches used for classification does not seem to be systematically chosen. For example, in the classification scenario, there are multiple non-density based data stream classifiers showing better results on the same dataset as reported in Table 2 and 3 such as Lv, Yanxia, et al. \"A classifier using online bagging ensemble method for big data stream learning.\" Tsinghua science and technology 24.4 (2019): 379-388. It would be better to compare with better approaches than just the 2 papers cited in the table.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject