SYSTEMS AND METHODS FOR ONLINE TIME SERIES FORCASTING

Embodiments provide a framework combining fast and slow learning Networks (referred to as “FSNet”) to train deep neural forecasters on the fly for online time-series fore-casting. FSNet is built on a deep neural network backbone (slow learner) with two complementary components to facilitate fast adaptation to both new and recurrent concepts. To this end, FSNet employs a per-layer adapter to monitor each layer's contribution to the forecasting loss via its partial derivative. The adapter transforms each layer's weight and feature at each step based on its recent gradient, allowing a finegrain per-layer fast adaptation to optimize the current loss. In addition, FSNet employs a second and complementary associative memory component to store important, recurring patterns observed during training. The adapter interacts with the memory to store, update, and retrieve the previous transformations, facilitating fast learning of such patterns.

TECHNICAL FIELD

The embodiments relate generally to machine learning systems, and more specifically to online time series forecasting.

BACKGROUND

Deep neural network models have been widely used in time series forecasting. For example, learning models may be used to forecast time series data such as continuous market data over a period of time in the future, weather data, and/or the like. Existing deep models adopt batch-learning for time series forecasting tasks. Such models often randomly sample look-back and forecast windows during training and freeze the model during evaluation, breaking the time varying (non-stationary) nature of time series.

Therefore, there is a need for an efficient and adaptive deep learning framework for online time forecasting.

DETAILED DESCRIPTION

As used herein, the term “network” may comprise any hardware or software-based framework that includes any artificial intelligence network or system, neural network or system and/or any training or learning models implemented thereon or therewith.

As used herein, the term “module” may comprise hardware or software-based framework that performs one or more functions. In some embodiments, the module may be implemented on one or more neural networks.

A time series is a set of values that correspond to a parameter of interest at different points in time. Examples of the parameter can include prices of stocks, temperature measurements, and the like. Time series forecasting is the process of determining a future datapoint or a set of future datapoints beyond the set of values in the time series. Time series forecasting of dynamic data via deep learning remains challenging.

Embodiments provide a framework combining fast and slow learning Networks (referred to as “FSNet”) to train deep neural forecasters on the fly for online time-series fore-casting. FSNet is built on a deep neural network backbone (slow learner) with two complementary components to facilitate fast adaptation to both new and recurrent concepts. To this end, FSNet employs a per-layer adapter to monitor each layer's contribution to the forecasting loss via its partial derivative. The adapter transforms each layer's weight and feature at each step based on its recent gradient, allowing a finegrain per-layer fast adaptation to optimize the current loss. In addition, FSNet employs a second and complementary associative memory component to store important, recurring patterns observed during training. The adapter interacts with the memory to store, update, and retrieve the previous transformations, facilitating fast learning of such patterns.

In this way, the FSNet framework can adapt to the fast-changing and the long-recurring patterns in time series. Specifically, in FSNet, the deep neural network plays the role of neocortex while the adapter and its memory play act as a hippocampus component.

FSNet Framework Overview

FIG.1is a simplified diagram illustrating an example structure of the FSNet framework100for forecasting a time series, according to embodiments described herein.

The FSNet framework100comprises a plurality of convolution blocks104a-nconnected to a regressor105. The FSNet framework100may receive time series data102, denoted by χ=(x1, . . . , xT)∈as a times series of T observations each having n dimensions, from an input interface such as a memory or a network adapter. In some embodiments, the time series data102may be data in a look back window of length e starting at time i: χi,e=(xi, . . . , xi+e). The model100may use a look back window based on the availability of memory in such as GPU memory based on the size of the time series data or based on the seasonality of the data and the like. The model100may generate an online forecast106predicting the next H-steps of the times series based on the input time series data102, e.g., fω(χi,H)=(xi+e+1. . . xi+e+H), where w denotes the parameter of the forecasting model. Here, a pair of lookback window and forecast window data are considered as a training sample. For multiple step forecasting (H>1), a linear regressor105is employed to forecast all H steps in the horizons simultaneously.

In one embodiment, FSNet framework100may include a temporal convolutional neural network (TCN) backbone having L layers (e.g., the blocks1-L104a-n) with parameters θ={θl}l=1L. The TCN backbone104a-nmay implement a deep learning algorithm (that learns slowly online and is a deep neural network) which receives an input such as a time series data102, assigns importance (learnable weights and biases) to various aspects/objects in the time series data102and differentiates various aspects/objects in the time series data102from the other aspects/objects in the time series data102. The TCN backbone104a-nmay extract a time-series feature representation from the time series data102.

Based on the TCN backbone104a-n, the FSNet framework100further includes two complementary components: a per-layer adapter ϕl(shown at315inFIG.3) for each TCN layer104a-nand a per-layer associate memory Ml(shown at318inFIG.3) for each TCN layer104an. Thus, the total trainable parameters for the framework is ω={θl,ϕl} and the total associate memory is M={Ml}l=1, . . . , L.

FIG.2is a simplified diagram illustrating an example structure of a TCN layer (block)104aof the FSNet framework described inFIG.1, according to embodiments described herein. At each TCN layer (block), e.g.,104a, the block input202may be processed by a number of dilated convolution layers204,2-6, and the convoluted output is added to the original block input202to generate block output208. It is noted that while two dilated convolution layers204and206are shown inFIG.2for illustrative purpose only, any other number of dilated convolution layers may be used in a TCN block.

In one embodiment, each TCN block104amay rely on its adapter315and associative memory318to quickly adapt to the changes in time series data102or learn more efficiently with limited data. Each block or layer104a-104nmay adapt independently rather than restricting the adaptation to the depth of the network, i.e., gradient descent over the depth of the network104a-n. The partial derivative ∇θlfor each layer104a-ncharacterizes the contribution of the convolutional layer θl104a-nto the forecasting loss. The ∇θlmay be used to update the l-th layer θlIn some embodiments, a gradient associated with each convolutional layer may be computed based on a partial derivative ∇θl. Such gradient may be further smoothed out using the exponential moving (EMA) average within the dilated convolution204or206as described in relation toFIG.3.

Therefore, each convolution filter stack is accompanied by an adapter and an associative memory. At each layer, the adapter receives the gradient EMA and interacts with the memory and convolution filter accordingly, as further illustrated in relation toFIG.3.

FIG.3is a simplified diagram illustrating an example structure of the dilated convolution layer204(or206) in the TCN layer (block)104adescribed inFIG.2, according to embodiments described herein. The dilated convolution layer204may comprise convolution filters310, a per-layer adapter315, a per-layer memory318. Input202to the dilated convolution layer204may be fed to the convolution filters310, which in turn computes the exponential moving average (EMA)313of the TCN backbone's gradients. Specifically, because a gradient of a single sample can highly fluctuate and introduce noises to the adaptation parameters, EMA is used to smooth out online training's noises by:

where gltdenotes the gradient of the l-th layer at time t and ĝldenotes the EMA gradient. In this way, the fast adapter315may receive the EMA gradient ĝlas input and maps it to the adaptation coefficients ul, as shown at316.

In some embodiments, the fast adapter315may use the element-wise transformation as the adaptation process due to its efficiency for continual learning. The resulting adaptation parameter ul316may include two components: (i) a weight adaptation parameter αl; and (ii) a feature adaptation parameter βl, concatenated together as ul=[αl;βl]. In some embodiments, the fast adapter315may absorb the bias transformation parameter into αlfor brevity.

In one embodiment, the adaptation for a layer θlmay involves a weight adaptation and a feature adaptation, as shown at319. First, the weight adaptation parameter αlacts on the corresponding weight of the backbone network via an element-wise multiplication as

wherein, θ is a stack of l features maps of C channels and length L, θldenotes the adapted weight, tile (αl) denotes that the weight adaptor is applied per-channel on all filters via a tile function, and ⊙ denotes the elementwise multiplication.

Similarly, a feature adaptation component βlof the gradient, wherein the feature adaptation parameter changes the convolutional layer feature map based on an element-wise multiplication between the feature adaptation component and the first convolutional layer feature map. For example, the feature adaptation βlalso interacts with the output feature map hlto generate the output322as

In this way, the convolutional layer θlmay be updated based on the weight adaption component αland the feature adaptation component βl.

In some embodiments, the gradient may be directly mapped to the per-element adaptation parameter and this may result in a very high dimensional mapping.

In some embodiments, a chunking operation, denoted as Ω(⋅;ϕl), may be implemented to split the gradient into equal size chunks and then maps each chunk to an element of the adaptation parameter. Specifically, the chunking operation may be implemented as (1) flattening the gradient EMA of a corresponding block of the TCN model120into a vector; (2) splitting the gradient vector into d chunks; (3) mapping each chunk to a hidden representation; and (4) mapping each hidden representation to a coordinate of the target adaptation parameter u. For example, by using a vectorizing operation (vec (⋅)) that flattens a tensor into a vector, a splitting operation (e,B) splitting a vector e into B segments, each has size dim (e)/B, the backbone's layer EMA gradient313of the TCN backbone to an adaptation coefficient u∈via the chunking process as:

where the Wϕ(1)and Wϕ(2)are the first and second weight matrix of the adapter. In this way, the adaptation may be applied per-channel, which greatly reduces the memory overhead, offers compression and generalization.

In summary, letdenotes the convolution operation, at step t, the FSNet adapter may use a fast adaptation procedure for the l-th layer is summarized as:

In one embodiment, in time series, old patterns may reappear in the future, and it is beneficial to recall similar knowledge in the past to facilitate learning further. While storing the original data can alleviate this problem, it might not be applicable in many domains due to privacy concerns. Therefore, an associative memory318may be implemented to store the adaptation coefficients of repeating events encountered during learning. While the adapter315can handle fast recent changes over a short time scale, recurrent patterns are stored in the memory318and then retrieved when they reappear in the future. For this purpose, each adapter315is equipped with an associate memory318, denoted by∈where d denotes the dimensionality of ul, and N denotes the number of elements. The associate memory318only sparsely interacts with the adapter to store, retrieve, and update such important events.

Specifically, as interacting with the memory318at every step can be expensive and susceptible to noises, memory interaction may be triggered only when a substantial change in the representation is detected. Interference between the current and past representations can be characterized in terms of a dot product between the gradients. Therefore, a cosine similarity between the recent and longer term gradients may be computed and monitored to trigger the memory interaction when their interference fails below a threshold, which could indicate the pattern has changed significantly. To this end, in addition to computing the gradient EMA ĝl(313), a second gradient EMA ĝl′ with a smaller coefficient γ′<γ is computed and their cosine similarity to trigger the memory interaction as:

where τ>0 is a hyper-parameter determining the significant degree of interference. Moreover, r may be set to a relatively high value (e.g., 0.7) so that the memory only remembers significant changing patterns, which could be important and may reappear. For example, example EMA hyperparameter may be set as: adapter's EMA coefficient γ=0.9, gradient EMA for triggering the memory interaction γ′=0.3, memory triggering threshold τ=0.75.

In one embodiment, when the current adaptation parameter may not capture the whole event, which could span over a few samples, memory read and write operations may be performed using the adaptation parameter's EMA (with coefficient γ′) to fully capture the current pattern. The EMA of ulis calculated in the same manner as ĝl. When a memory interaction is triggered, the adapter queries and retrieves the most similar transformations in the past via an attention read operation, which is a weighted sum over the memory items:

where r(k)[i] denotes the i-th element of rl(k)and[i] denotes the i-th row of. As the memory could store conflicting patterns, sparse attention is applied by retrieving the top-k most relevant memory items, e.g., k=2. The retrieved adaptation parameter characterizes old experiences in adapting to the current pattern in the past and can improve learning at the present time by weighted summing with the current parameters as

where the same threshold value r can be used to determine the sparse memory interaction and the weighted sum of the adaptation parameter. Then a write operation is performed to update and accumulate the knowledge stored in:

where ⊗ denotes the outer-product operator, which allows to efficiently write the new knowledge to the most relevant locations indicated by rl(k). The memory is then normalized to avoid its values scaling exponentially.

In one embodiment, the FSNet framework described in relation toFIGS.1-3is suitable for the task-free, online continual learning scenario because there is no need to detect when tasks switch explicitly. Instead, the task boundaries definition can be relaxed to allow the model to improve its learning on current samples continuously.

Computing Environment

FIG.4is a simplified diagram of a computing device that implements the FSNet framework, according to some embodiments described herein. As shown inFIG.4, computing device400includes a processor410coupled to memory420. Operation of computing device400is controlled by processor410. And although computing device400is shown with only one processor410, it is understood that processor410may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) and/or the like in computing device400. Computing device400may be implemented as a stand-alone subsystem, as a board added to a computing device, and/or as a virtual machine.

Memory420may be used to store software executed by computing device400and/or one or more data structures used during operation of computing device400. Memory420may include one or more types of machine readable media. Some common forms of machine readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.

Processor410and/or memory420may be arranged in any suitable physical arrangement. In some embodiments, processor410and/or memory420may be implemented on a same board, in a same package (e.g., system-in-package), on a same chip (e.g., system-on-chip), and/or the like. In some embodiments, processor410and/or memory420may include distributed, virtualized, and/or containerized computing resources. Consistent with such embodiments, processor410and/or memory420may be located in one or more data centers and/or cloud computing facilities.

In some examples, memory420may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor410) may cause the one or more processors to perform the methods described in further detail herein. For example, as shown, memory420includes instructions for an online time series forecasting module430that may be used to implement and/or emulate the systems and models, and/or to implement any of the methods described further herein. In some examples, the online time series forecasting module430, may receive an input440, e.g., such as a time-series data in a lookback window, via a data interface415. The data interface415may be any of a user interface that receives uploaded time series data, or a communication interface that may receive or retrieve a previously stored sample of lookback window and forecasting window from the database. The times series forecasting module430may generate an output450, such as a forecast to the input440.

In some embodiments, the time series forecasting module430may further include a series of TCN blocks431a-n(similar to104a-nshown inFIG.1) and a regressor432(similar to105shown inFIG.1). In one implementation, the time series forecasting module430and its submodules431-432may be implemented via software, hardware and/or a combination thereof.

Example Workflows

FIG.5is a simplified pseudo code segment for a fast and slow learning network implemented at the FSNet framework described inFIGS.1-3, according to embodiments described here. For example, for the stack of L layers (e.g.,104a-ninFIG.1), forward computation may be performed to compute the adaptation parameter comprising the weight adaptation component αland the feature adaptation component βlat each layer. Memory read and write operation may be performed via the chunking process and the adaptation parameter may be updated by a weighted sum of the current and past adaptation parameters.

Next, the weight adaptation and feature adaptation may be performed according to Eq. (5). After updating the adaptation parameters through forward computation over L layers, forecast data can be generated via the regressor (e.g.,105inFIG.1). The forecast data is then compared with the ground-truth future data from the training sample to compute the forecast loss, which is then used to update the stack of L layers via backpropagation. The regressor may also be updated via stochastic gradient descent (SGD). The adaptation parameters and EMA adaptation parameters are then updated backwardly.

FIG.6is a simplified logic flow diagram illustrating an example process600corresponding to the pseudo code algorithm inFIG.5, according to embodiments described herein. One or more of the processes of method600may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the processes. In some embodiments, method600corresponds to the operation of the FSNet framework100(FIG.1) for forecasting time series data at future timestamps in a dynamic system.

At step602, a time series dataset that includes a plurality of datapoints corresponding to a plurality of timestamps within a lookback time window (e.g.,102inFIG.1) may be received via a data interface (e.g.,415inFIG.4).

At step604, a convolutional layer (e.g., block104ainFIGS.1-2) from a stack of convolutional layers (e.g., Blocks104a-ninFIG.1) may compute a first gradient based on exponential moving average of gradients corresponding to the respective convolutional layer, e.g., according to Eq. (1).

At step606, a first adaptation parameter u corresponding to the convolutional layer may be determined by mapping portions of the first gradient to elements of the first adaptation parameter. For example, the first adaptation parameter comprises a first weight adaptation component αland a first feature adaptation component βl.

At step608, for at least one convolutional layer in a temporal convolutional neural network, a layer forecasting loss indicative of a loss contribution of the respective convolutional layer to an overall forecasting loss according to the plurality of datapoints may be optionally determined, based on the plurality of datapoints. For example, the layer forecasting loss may be computed via the partial derivative ∇θl.

At step610, the at least one convolutional layer may be optionally updated based on the layer forecasting loss. In this way, each layer may be monitored and modified independently to learn the current loss by learning through the layer forecasting loss.

At step612, a cosine similarity between the first gradient of the updated convolutional layer and a longer-term gradient associated with the at least one first convolutional layer may be computed, e.g., according to Eq. (6).

At step614, when the cosine similarity is greater than a pre-predefined threshold, method600proceeds to step616to perform a chunking process for memory read and write. Specifically, at step616, a current adaptation parameter is retrieved from an indexed memory (e.g.,318inFIG.3) corresponding to the convolutional layer. At step618, content stored at the indexed memory (e.g.,318inFIG.3) is updated based on the current adaptation parameter and the first adaptation parameter. At step620, the first adaptation parameter is updated by taking a weighted average with the retrieved current adaptation parameter.

At step622, an adapted layer parameter {tilde over (θ)}lis computed based on the first weight adaptation component αland a layer parameter θlcorresponding to the first layer, e.g., according to Eq. (5).

At step624, a feature map hlof the first convolutional layer is generated with the first feature adaptation component βl. For example, the first feature map is a convolution of the adapted layer parameter and a previous adapted feature map from a preceding layer. At step626, an adapted feature map {tilde over (h)}lis computed based on the first feature adaptation component βland a first feature map hlof the first convolutional layer.

At step628, a regressor (e.g.,105inFIG.1) may generate time series forecast data corresponding to a future time window based on a final feature map output from the stack of convolutional layers corresponding to the time series data within the lookback time window.

At step630, a forecast loss may be computed based on the generated time series forecast data and ground-truth data corresponding to the future time window.

The stack of convolutional layers and the regressor may then be updated based on the forecast loss via backpropagation. At step632, the regressor may be updated via stochastic gradient descent. At step634, the gradient and the adaptation parameter of each layer of the stack may then be updated backwardly.

Example Performance

Data experiments have been carried out to verify following hypotheses: (i) FSNet facilitates faster adaptation to both new and recurring concepts compared to existing strategies; (ii) FSNet achieves faster and better convergence than other methods; and (iii) modeling the partial derivative is the key ingredients for fast adaptation.

Specifically, a wide range of time series forecasting datasets have been used in the data experiments: (i) ETT1(Zhou et al., Informer: Beyond efficient transformer for long sequence time-series forecasting, in Proceedings of AAAI, 2021) records the target value of “oil temperature” and 6 power load features over a period of two years. The ETTh2and ETTm1benchmarks are used, where the observations are recorded hourly and in 15-minutes intervals respectively. (ii) ECL (Electricty Consuming Load)2dataset collects the electricity consumption of321clients from 2012 to 2014. (iii) Traffic3dataset records the road occupancy rates at San Francisco Bay area freeways. (iv) Weather4dataset records11climate features from nearly 1,600 locations in the U.S. in an hour intervals from 2010 to 2013.

In addition, two synthetic datasets are constructed to explicitly test the model's ability to deal with new and recurring concept drifts. A task may be synthesized by sampling 1,000 samples from a first-order autoregressive process with coefficient φ ARφ(1), where different tasks correspond to different p values. The first synthetic data, S-Abrupt contains abrupt, and recurrent concepts where the samples abruptly switch from one AR process to another by the following order: AR0.1(1), AR0.4(1), AR0.6(1), AR0.1(1), AR0.3(1), AR0.6(1). The second data, S-Gradual contains gradual, incremental shifts, where the shift starts at the last 20% of each task. In this scenario, the last 20% samples of a task is an averaged from two AR process with the order as above.

At implementation, data is split into warm-up and online training phases by the ratio of 25:75 and consider the TCN backbone for experiments, except the Informer baseline. Optimization details in Zhang et al., Informer: Beyond efficient transformer for long sequence time-series forecasting, in Proceedings of AAAI, 2021, by optimizing the 12 (MSE) loss with the AdamW optimizer. Both the epoch and batch size are set to one to follow the online learning setting. A fair comparison is implemented by making sure that all baselines use the same total memory budget as FSNet, which includes three-times the network sizes: one working model and two EMA of its gradient. Thus, for ER, MIR, and DER++, an episodic memory to store previous samples to meet this budget. For the remaining baselines, the backbone size can be increased instead. Lastly, in the warm-up phase, the mean and standard deviation are calculated to normalize online training samples and perform hyper-parameter cross-validation. For all benchmarks, the look-back window length is set to be 60 and the forecast horizon of H=1. The model's ability to forecast longer horizons is tested by varying H∈{1, 24, 48}.

A suite of training from both continual learning and time series forecasting are adopted for comparison. First, the OnlineTCN strategy that simply trains continuously (described in Zinkevich, Online convex programming and generalized infinitesimal gradient ascent, in Proceedings of the 20th international conference on machine learning (icml-03), pages 928-936,461, 2003. Second, the Experience Replay ER strategy (described in Lin, Self-improving reactive agents based on reinforcement learning, planning and teaching, Machine learning, 8(3-4):293-321, 1992) where a buffer is employed to store previous data and interleave old samples during the learning of newer ones. Three recent advanced variants of ER. First, TFCL (Aljundi et al., Task-free continual learning, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11254-11263,325, 2019) introduces a task-boundaries detection mechanism and a knowledge consolidation strategy by regularizing the networks' outputs. Second, MIR (Aljundi et al., Online continual learning with maximal interfered retrieval. Advances in Neural Information Processing Systems, 32:11849-11860, 2019) replace the random sampling in ER by selecting samples that cause the most forgetting. Lastly, DER++(Buzzega et al., Dark experience for general continual learning: a strong, simple baseline, in 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020) augments the standard ER with a knowledge distillation strategy (described in Hinton et al., Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015). ER and its variants are strong baselines in the online setting since they enjoy the benefits of training on mini-batches, which greatly reduce noises from singe samples and offer faster, better convergence (see Bottou et al., Online learning and stochastic approximations, Online learning in neural networks, 17(9):142, 1998). While the aforementioned baselines use a TCN backbone, Informer, the time series forecasting method based on the transformer architecture (Vaswani et al., Attention is all you need. Advances in neural information processing systems, 30, 2017) is also included.

First, the Online Gradient Descent (OGD) (described in Zinkevich, Online convex and generalized infinitesimal gradient ascent, in proceedings of the 20thinternational conference on machine learning, pp. 928-936, 2003) strategy that simply trains continuously. OGD (L), a large variant of OGD with twice the TCN's filters per layer is also included, resulting in a roughly twice number of parameters5. Another baseline includes Experiment Replay (described in Chaudhry et al., On tiny episodic memories in continual learning, arXiv preprint arXiv:1902.10486, 2019) strategy where a buffer is employed to store previous data and interleave old samples during the learning of newer ones. Another baseline includes DER++(Buzzega et al., Dark experience for general continual learning: a strong, simple baseline, in proceedings of 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020) which further adds a knowledge distillation (described in Hinton et al., Distilling the knowledge in a neural network, arXiv preprint arXiv:1503.02531, 2015) loss to ER. ER and DER++ are strong baselines in the online setting since they enjoy the benefits of training on mini-batches, which greatly re-duce noises from singe samples and offers faster, better convergence.

FIG.7reports cumulative mean-squared errors (MSE) and mean-absolute errors (MAE) at the end of training. It is observed that ER and DER++ are strong competitors and can significantly im-prove over the OGD strategies. However, such methods still cannot work well under multiple task switches (S-Abrupt). Moreover, no clear task boundaries (S-Gradual) presents an even more challenging problem and increases most models' errors. On the other hand, FSNet shows promising results on all datasets and outperforms most competing baselines across different forecasting horizons. Moreover, the improvements are significant on the synthetic benchmarks, indicating that LSFNet can quickly adapt to the non-stationary environment and recall previous knowledge, even without clear task boundaries.

FIG.8reports the convergent behaviors on the considered methods. The results show the benefits of ER by offering faster convergence during learning compared to OGD. However, it is important to note that storing the original data may not apply in many domains. On S-Abrupt, most baselines demonstrate the inability to quickly recover from concept drifts, indicated by the increasing error curves. It is also observed that promising results of FSNet on most datasets, with significant improve-ments over the baselines on the ETT, WTH, and S-Abrupt datasets. The ECL dataset is more challenging with missing values (Li et al., 2019) and large magnitude varying within and across dimensions, which may require calculating a better data normalization. While FSNet achieved encouraging results on ECL, handling the above challenges can further improve its performance. Overall, the results shed light on the challenges of online time series forecasting and demonstrate promising results of FSNet.

The model's prediction quality on the S-Abrupt is visualized as shown inFIG.8, as it is a univariate time series. The remaining real-world datasets are multivariate are challenging to visualize. Particularly, the model's forecasting at two-time points is plotted: at t=900 and the end of learning, t=5900 inFIG.9. With the limited samples per task and the presence of multiple concept drifts, the standard online optimization collapsed to a naive solution of predicting random noises around zero. However, FSNet can successfully capture the time series' patterns and provide better predictions.

This description and the accompanying drawings that illustrate inventive aspects, embodiments, implementations, or applications should not be taken as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure the embodiments of this disclosure. Like numbers in two or more figures represent the same or similar elements.