Incremental learning algorithms and applications

Incremental learning refers to learning from streaming data, which arrive over time, with limited memory resources and, ideally, without sacrificing model accuracy. This setting fits different application scenarios where lifelong learning is relevant, e.g. due to changing environments, and it offers an elegant scheme for big data processing by means of its sequential treatment. In this contribution, we formalise the concept of incremental learning, we discuss particular challenges which arise in this setting, and we give an overview about popular approaches, its theoretical foundations, and applications which emerged in the last years. 1 What is incremental learning? Machine learning methods offer particularly powerful technologies to infer structural information from given digital data; still, the majority of current applications restrict to the classical batch setting: data are given prior to training, hence meta-parameter optimisation and model selection can be based on the full data set, and training can rely on the assumption that the data and its underlying structure are static. Lifelong learning, in contrast, refers to the situation of continuous model adaptation based on a constantly arriving data stream [38, 149]. This setting is present whenever systems act autonomously such as in autonomous robotics or driving [156, 5, 112, 65]. Further, online learning becomes necessary in any interactive scenario where training examples are provided based on human feedback over time [134]. Finally, many digital data sets, albeit static, can become so big that they are de facto dealt with as a data stream, i.e. one incremental pass over the full data set [116]. Incremental learning investigates how to learn in such a streaming setting. It comes in various forms in the literature, and the use of the term is not always consistent. Therefore, first, we give a meaning to the terms online learning, incremental learning, and concept drift, giving particular attention to the supervised learning paradigm. 1.1 Online learning methods In supervised learning, data D = ((~x1, y1), (~x2, y2), (~x3, y3), . . . , (~xm, ym)) are available with input signals ~xi and outputs yi. The task is to infer a model M ≈ p(y|~x) from such data. Machine learning algorithms are often trained in a batch mode, i.e., they use all examples (~xi, yi) at the same time, irrespective of their (temporal) order, to perform, e.g., a model optimisation step. ∗This research/work was supported by the Cluster of Excellence Cognitive Interaction Technology ’CITEC’ (EXC 277) at Bielefeld University, which is funded by the German Research Foundation (DFG). Challenge 1: Online model parameter adaptation. In many application examples, data D are not available priorly, but examples arrive over time, and the task is to infer a reliable model Mt after every time step based on the example (~xt, yt) and the previous model Mt−1 only. This is realised by online learning approaches, which use training samples one by one, without knowing their number in advance, to optimise their internal cost function. There is a continuum of possibilities here, ranging from fully online approaches that adapt their internal model immediately upon processing of a single sample, over socalled mini-batch techniques that accumulate a small number of samples, to batch learning approaches, which store all samples internally. Online learning is easily achieved by stochastic optimisation techniques such as online back-propagation, but there are also extensions of the support vector machine (SVM) [164]. Prototype-based models such as vector quantisation, radial basis function networks (RBF), supervised learning vector quantisation, and self-organising maps (SOM) all naturally realise online learning schemes since they rely on a (approximate) stochastic gradient technique [140, 115, 15, 83]. Second order numeric optimisation methods and advanced optimisation schemes can be extended as well, such as variational Bayes, convex optimization, second order perceptron learning based on higher order statistics in primal or dual space, and online realisations of the quasi-Newton Broyden-Fletcher-GoldfarbShanno technique [114, 117, 49, 125, 62]. Stochastic optimization schemes can be developed also for non-decomposable cost function, [80]. Further, lazy learners such as k-nearest neighbour (k-NN) methods lend itself to online scenarios by their design [140]. Interestingly, online learning has already very early been accompanied by their exact mathematical investigations [162]. 1.2 Incremental learning methods Incremental learning refers to online learning strategies which work with limited memory resources. This rules out approaches which essentially work in batch mode for the inference of Mt by storing all examples up to time step t in memory; rather, incremental learning has to rely on a compact representation of the already observed signals, such as an efficient statistics of the data, an alternative compact memory model, or an implicit data representation in terms of the model parameters itself. At the same time, it has to provide accurate results for all relevant settings, despite its limited memory resources. Challenge 2: Concept drift. Incremental learning shares quite a number of challenges with online learning, with memory limitations adding quite a few extras. One prominent problem consists in the fact that, when the temporal structure of data samples is taken into account, one can observe changes in data statistics that occur over time, i.e. samples (~xi, yi) are not i.i.d. Changes in the data distribution over time are commonly referred to as concept drift [88, 157, 33, 126]. Different types of concept drift can be distinguished: changes in the input distribution p(~x) only, referred to as virtual concept drift or covariate shift, or changes in the underlying functionality itself p(y|~x), referred to as real concept drift. Further, concept drift can be gradual or abrupt. In the latter case one often uses the term concept shift. The term local concept drift characterises changes of the data statistics only in a specific region of data space [157]. A prominent example is the addition of a new, visually dissimilar object class to a classification problem. Real concept drift is problematic since it leads to conflicts in the classification, for example when a new but visually similar class appears in the data: this will in any event have an impact on classification performance until the model can be re-adapted accordingly. Challenge 3: The stability-plasticity dilemma. In particular for noisy environments or concept drift, a second challenge consists in the question when and how to adapt the current model. A quick update enables a rapid adaptation according to new information, but old information is forgotten equally quickly. On the other hand, adaption can be performed slowly, in which case old information is retained longer but the reactivity of the system is decreased. The dilemma behind this trade-off is usually denoted the stability-plasticity dilemma, which is a well-known constraint for artificial as well as biological learning systems [113]. Incremental learning techniques, which adapt learned models to concept drift only in those regions of the data space where concept drift actually occurs, offer a partial remedy to this problem. Many online learning methods alone, albeit dealing with limited resources, are not able to solve this dilemma since they exhibit a so-called catastrophic forgetting behaviour [108, 132, 45, 44, 103] even when the new data statistics do not invalidate the old ones. One approach to deal with the stability-plasticity dilemma consists in the enhancement of the learning rules by explicit meta-strategies, when and how to learn. This is at the core of popular incremental models such as ART networks [77, 56], or meta-strategies to deal with concept drift such as the just-in-time classifier JIT , or hybrid online/offline methods [120, 43]. One major ingredient of such strategies consist in a confidence estimation of the actual model prediction, such as statistical tests, efficient surrogates, or some notion of selfevaluation [8, 43, 78]. Such techniques can be enhanced to complex incremental schemes for interactive learning or learning scaffolding [130, 84]. Challenge 4: Adaptive model complexity and meta-parameters. For incremental learning, model complexity must be variable, since it is impossible to estimate the model complexity in advance if the data are unknown. Depending on the occurrence of concept drift events, an increased model complexity might become necessary. On the other hand, the overall model complexity is usually bounded from above by the limitation of the available resources. This requires the intelligent reallocation of ressources whenever this limit is reached. Quite a number of approaches propose intelligent adaptation methods for the model complexity such as incremental architectures [166], self-adjustment of the number of basic units in extreme learning machines  or prototype-based models [77, 144, 98], incremental base function selection for a sufficiently powerful data representation , or self-adjusting cluster numbers in unsupervised learning [79]. Such strategies can be put into the more general context of self-evolving systems, see e.g. [92] for an overview. An incremental model complexity is not only mandatory whenever concept drift is observed, hence a possibly changing model complexity is present, but it can also dramatically speed-up learning in batch scenarios, since it makes often tedious model selection superfluous. In batch learning, not only the model complexity, but also essential metaparameters such as learning rate, degree of error, regularisation constants, etc. are determined prior to training. Often, time consuming cross-validation is used in batch learning, whereby first promising result how to automate the process