0Explainable Lifelong Streaming Learning

CHUKIONG LOO
ckloo.um@um.edu.my

www.um.edu.my

Faculty of Computer Science & Information
Technology

Outlines• Continual Learning Overview• Introduction to Lifelong Learning• Lifelong Learning Setup• Explainable Lifelong Streaming Learning• Future Works

1

2

3

Introduction to‘Continual Learning’
literature• ‘Continual learning’ vs. ‘lifelong learning’• Often used interchangeably• Popularity of ‘continual learning’ more recent

We use these terms as follows:- Continual learning (narrow) how to deal with non-stationarity in training data
- Lifelong learning(broad) an agent learning throughout its lifetime

45Continual Learning
Challenges inPractical Applications• A robot acquiring new skills in
different environment, adapting to
new situations, learning new tasks

6Continual Learning
Challenges inPractical Applications• A self-driving car adapting todifferent environments (from acountry road to a highway to a city)

7

Continual Learning
Challenges inPractical Applications• Conversational agents adapting
to different users, situations, tasks

8Continual Learning
Challenges inPractical Applications• Medical applications: adapting to
new patients, new hospitalconditions

9

Continual Learning
Challenges inPractical Applications• Multi-game environments(e.g. OpenAI gym)

Static datasets: Controlled

10Static datasets: large scale

11

12

Challenge: Data Stream
50GB/s streaming data.
~30240 TB of data after
only aweek.

Impossible to re-train themini-spot brain fromscratch and to adapt fast.
https://www.bostondynamics.com/products/spot

Continual Learning…..

13

Human-like learning
Small data, n → 1

Supervised learning
Big data, n → ∞14Relation to other fields• One task
• Data available
at the same time

• Multiple tasks• Data arriveincrementally• Goal: all tasks

Nonstationary• Multiple tasks• Data arriveincrementally• Goal: last task

• Multiple tasks• Data available atthe same time
• Goal: all tasks• One task
• Data arriveincrementally

Source: De Lange et al. (2021, TPAMI)

The canonical continual learning example:
Split MNIST•••MNIST dataset is split into multiple parts/episodes/tasks(*) that must be learned
sequentially
After all tasks have been learned, the model should be good at all tasks
Typically, no or only a small amount of data from past tasks can be stored

Important problem: catastrophic forgettingWhen learning a new task, deep neural networks tend to rapidly forget past tasks

15

Going beyond Split MNIST•

Splitting up existing image datasets:• CIFAR-10• CIFAR-100• (Tiny)ImageNet• ….

•

Datasets specific for continual learning• CORe-50• Stream-51• The CLEAR Benchmark•

Beyond Classification• Continual reinforcement learning• Continual object detection• Continual semantic segmentation
• ….

16

Instance

CORe50: different types of continual
learning
Class

17

Source: Lomonaco & Maltoni (2017, CoRL)

Three Continual Learning Scenarios: intuitively 18

Task-based vs. Task-free Continual Learning

19Baselines: finetuning (lower target) & joint
training (upper target)

20

None: Network sequentiallytrained on each task in thestandard way (lower target)
Joint: Network trained on alltasks at the same time(upper target)

Code for these experiments: https://github.com/GMvandeVen/continual-learning

Continual Learning Categorization

21

Continual Learning Strategies

22Regularization• In continual learning, regularization typically means adding a penalty term to
the loss function to encourage the model to stay close to a previous version of
itself.
• Often, the version relative to which changes are penalized is a copy of the model
stored after finishing training on the last task
• Two forms of regularization:

23

Parameter Regularization• Parameters important for past tasks are encouraged not
to change too much when learning a new task
• Can often be interpreted as sequential approximate
Bayesian inference on the network’s parameters• Representative methods:• Elastic Weight Consolidation [EWC] (Kirkpartrick et al., 2017
PNAS)
• Synaptic Intelligence [SI] (Zenke et al., 2017 ICML)

24

Functional Regularization

25• The input-output mapping learned previously is
encouraged not to change too much at a particular
set of inputs (the ‘anchor points’)• Also referred to as knowledge distillation• Representative methods:• Learning without Forgetting [LwF] (Li & Hoiem, 2017
TPAMI)• Functional Regularization Of Memorable Past [FROMP](Pan et al., 2020 NeurIPS)

Memory buffer size (FROMP): 100 examples per class

Code for these experiments: https://github.com/GMvandeVen/continual-learningReplay26• Current training data is complemented with data
representative of past observations• The replayed data can be sampled from a memory
buffer or a generative model• Representative methods:• Experience Replay [ER] (Chaudhry et al., 2019 arXiv)• Deep Generative Replay [DGR] (Shin et al., 2017 NeurIPS)

Memory buffer size (FROMP): 100 examples per class

Code for these experiments: https://github.com/GMvandeVen/continual-learning

Context-specific components

27• Parts of the network are only used for specific tasks• Commonly used example: multi-headed output
layer• Requires knowledge of task identity at test time

• Context-dependent Gating [XdG] (Masse et al., 2018
PNAS)
• Separate Networks [SepN]

Memory buffer size (FROMP): 100 examples per class

Code for these experiments: https://github.com/GMvandeVen/continual-learningTemplate-based classification

28• A ‘template’ is learned for each class, and classification is
performed based on which template is most suitable forsample to be classified• Examples of templates are prototypes or generative models• Allows comparing classes ‘at test time’, rather than during
training• Representative methods• Incremental Classifier and Representation Learning [iCaRL] (Rebuffi etal., 2017 CVPR)• Generative Classifier [GenC] (van de Ven et al., 2021 CVPR-W)

Memory buffer size (FROMP): 100 examples per class

Code for these experiments: https://github.com/GMvandeVen/continual-learning

Overview29

Source: van de Ven et al. (2022, Nature Machine Intelligence)

30

Hybrid Models

Why Hybrid?
• Each CL algorithm has advantages anddisadvantages and works best within
specific scenarios• Such approaches are often orthogonal withrespect to each other• Biological learning systems seems to apply
several approaches for learning continually• Hybrid approaches are underexplored andmay potentially find better EffectivenessEfficiency trade-offs

31

AR1 and (Negative) Generative Replay?
Key Aspects• Generative Replay is often difficult to scale(quality and diversity), what about generative
latent replay?
• Sharing weights between the discriminator andthe generator is possible• Incremental training of the generator in the loop
• Negative replay: use generated patterns as
negative examples only

Generative Negative Replay for Continual Learning, Graffiti et al, 2021.

32

Summary and Next Steps

33• Hybrid approaches are more complex and moredifficult to parametrize in general but they canprovide Effectiveness-Efficiency trade-offs.
• Such approaches are still not well investigated but
offer a nice path for future research explorations.
• They are often among the winning approaches incontinual learning challenges?
?? ?

• More flexible and tunable algorithms (and possible
self-adjusting hybrid approaches) may be quite
interesting to investigate.
CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions, Lomonaco et al, 2020.

Research Journey (2012 – 2023)
Algorithm
•
•
•

•
•
•

•
•
•

Application

Kernel Bayesian ART/ARTMAP/Associative
MemoryTopological Kernel Bayesian ART/ARTMAP
Deep Kernel Bayesian
ART/ARTMAP/Associative Memory

Cognitive Robotics

Kernel Bayesian ART/ARTMAP/Associative
MemoryOnline Recurrent Kernel-based Extreme
Learning/Reservoir Machine
Genetic Ensemble Fuzzy Extreme Learning
MachineConnected Health
Analytics – Diagnosis &Prognosis

Topological Kernel Bayesian ART/ARTMAP
Biologically Inspired Vision
Deep Kernel Bayesian
ART/ARTMAP/Associative Memory

Topological
2D/SLAM/surface
reconstruction34

ART-based Triple Memory Model (2019-2022)
Multi-Channel Art-based Triplet Memory (2022)
Multi-Modal ART-based TripleMemory + Recurrent Kernel Machine
Multi-Channel Art-based Dual-Memory (2020)
Multi-Modal ART-based DualMemory + Recurrent Kernel Machine
Enhanced Episodic Memory ART (EEM-ART) 2019
Multi-modal ART-based Episodic Memory

35

WH Chin, N Kubota, CK Loo, (2022) An Episodic-Procedural Semantic Memory
Model for Continuous Topological Sensorimotor Map Building, Cognitive
Robotics and Adaptive Behavior, Interopen publisher.

Wei Hong Chin, Chu Kiong Loo, Stefan Wermter, (2020), Multichannel
Recurrent Kernel Machines for Robot Episodic-Semantic Map Building, 1st
SMILES (Sensorimotor Interaction, Language and Embodiment of Symbols)
workshop, ICDL 2020
Wei Hong Chin, Yuichiro Toda, Naoyuki Kubota, Chu Kiong Loo, Manjeevan
Seera (2019) Episodic Memory Multimodal Learning for Robot Sensorimotor
Map Building and Navigation. IEEE Trans. Cogn. Dev. Syst. 11(2): 210-220

Multi-Channel BART 2016
Topological Bayesian ART + Multi-Modality

Wei Hong Chin, Chu Kiong Loo, Manjeevan Seera, Naoyuki Kubota, Yuichiro
Toda (2016) Multi-channel Bayesian Adaptive Resonance Associate Memoryfor on-line topological map building. Appl. Soft Comput. 38: 269-280

Topological Gaussian ART (TGA) 2016

Wei Hong Chin, Chu Kiong Loo, Yuichiro Toda, Naoyuki Kubota (2016) An
Odometry-Free Approach for Simultaneous Localization and Online Hybrid
Map Building. Frontiers Robotics AI 3: 68

Topological Gaussian ART + Fuzzy
Motion PlanningResearch Journey (2012 – 2023)
Continual learning (narrow)
how to deal with non-stationarity
in training data
Lifelong learning - (broad)an agent learning throughout its
lifetime
Continual / Lifelong
Learning

Challenges

PeriodInternational Collaboration(Continual Learning)
Data arrive incrementally• Concept drift• Data stream learning

2012 – 2022

(Lifelong Learning)
Multiple tasks

•
•
•
•

•
•
•
•

Concept drift
Data stream learning
Catastrophic Forgetting
Task/Domain/Class
Incremental

2023 - now

Tokyo Metropolitan University
Osaka Metropolitan University
University of Hamburg
King Mongkut Institute ofTechnology Latkrabang• Dalian Maritime University• Hohai University• Murdoch University

36Research Journey (2012 – 2023)
Continual learning (narrow)
how to deal with non-stationarity
in training data
Lifelong learning - (broad)an agent learning throughout its
lifetime
Continual / Lifelong
Learning

Challenges

PeriodInternational Collaboration(Continual Learning)
Data arrive incrementally• Concept drift• Data stream learning

2012 – 2022

(Lifelong Learning)
Multiple tasks

•
•
•
•

•
•
•
•

Concept drift
Data stream learning
Catastrophic Forgetting
Task/Domain/Class
Incremental

2023 - now

Tokyo Metropolitan University
Osaka Metropolitan University
University of Hamburg
King Mongkut Institute ofTechnology Latkrabang• Dalian Maritime University• Hohai University• Murdoch University

37

38

What is Lifelong Learning?
• The ability to update a learner one
sample ata time on an evolving data stream• Closely resembles the evolving natural world(e.g., temporal correlations)• Inputs are typically not independent
identically distributed (non-i.i.d.)• Challenges

and• Catastrophic forgetting of previous knowledge• Enabling backward and forward knowledgetransferWhy Study Embedded Lifelong Learning?
• Less compute means agents learn on-device• No cloud computing in space or in internet
deprived locations• Customized for each user without sending
personal information through the web• On-device learning for AR/VR, smart toys, robots,phones, and more

39

How Could Lifelong Learning Benefit AI?
1. Agents could learn and adapt in real-time• Data naturally evolves over time
• Immediate inferences about new data could be made• On-device learning could reduce privacy concerns

2. Transfer of knowledge among similartasks could make learning more efficient3. Could result in more efficient learningalgorithms

40

Lifelong Learning Considerations
1. Learn and recall immediately
2. Learn data in any order without catastrophicallyforgetting• Learning one class at a time induces the most forgetting• Learning from an i.i.d. data stream induces the least
forgetting• Must test both capabilities and those in between

3. Transfer of learned representations for
efficiency
4. Learn using limited memory and computeoverhead
5. Scale to large-scale, high-dimensional problems

41

The Episodes-to-Concepts (EpCon) Model
Initial Encoding• Attention Biasing
Concept Formation• Pattern completion• Memory-based prediction error• Pattern separation• Pattern integration

M.L. Mack et al. / Neuroscience Letters 680 (2018) 31–38• Concept coding• Affected attention biasing

42

43

Lifelong Learning Setup

Incremental Batch Learning
1. Dataset is broken into several
batches (chunks)• Includes task-based learning

2. At each time step, the learner…

• Receives a batch of data from one ormore classes• Loops over the batch until learned• Is evaluated at the end of training the
batch

44

Incremental Batch Learning
Advantages• Recently demonstrated much success• Makes learning easier since large batches
are closer• to independent & identically distributed(i.i.d.)Caveats• Slow (i.e., data accumulation, looping,delayed evaluation)• Not reminiscent of how humans and animals learn

Relevant for applications where:• More memory and compute are available• Immediate updates and inferences are not needed• Batch processing could be advantageous45

Streaming Learning
1. Instances often have temporalcorrelations and are non-i.i.d. (videos)
2. At each time step, the learner…

• Receives one new sample• Learns the sample and then is evaluated

3. The learner is only allowed one loop
through the entire dataset

46

Streaming Learning
Advantages• Closer to how humans/animals learn• Better suited for real-time applications

Caveats• Performance may not match offline batchprocessing• For supervised learning, must have labels at
each time-step

Relevant for applications where:• Memory and compute are limited• Immediate updates and inferences areneeded47

Pre-Training Before Lifelong
Learning
Many continual learning systems perform
an offline pre-training phase
Offline pre-training:• Train on the first N classes / examples / first mega-batch of
the dataset in an offline way to initialize the deep neuralnetwork• Then perform lifelong learning afterwards

1048

Lifelong Representation Learning
Every time new data is observed, update both the classifier and feature
representations
Could include a pre-training phase prior to lifelong learning

Image: https://mila.quebec/en/article/la-maml-look-ahead-meta-learning-for-continual-learning/49

Pre-Trained Features vs. Lifelong
Representation Learning

50

Pre-Training Phase (without lifelong representation
learning):• Downstream tasks are similar to pre-training tasks• Limited compute is available for downstream tasks• Very little downstream task data is available

Lifelong Representation Learning:••••

Downstream tasks change significantly over timeTask-specific features are needed
Requires more compute to update features
Another consideration is feature update types:• Supervised, Unsupervised/Self-supervised

Image: https://www.v7labs.com/blog/self-supervised-learning-guidePractical Hardware Considerations• Beyond real-time processing, successful lifelong learners could makelearning algorithms more efficient thus reducing energy and power
consumption• Better for the environment

Desislavov et al., 2021

51

Lifelong Learning for Embedded Devices• Embedded Device: a device that is purpose built for its application• Typically resource constrained in terms of compute, storage, and power52

Embedded Lifelong Learning Considerations• Embedded lifelong learning poses unique challenges
due to real-world and hardware constraints• Limited memory and compute• Natural world is imbalanced/long-tailed with need forlow-shot learning• Scalability to infinite data streams• Natural world is temporally correlated (videos)
We expect our agent to be performant regardless ofthese factors

53

Embedded Lifelong Learning Challenges
Applications may be memory or compute limited:• Some lifelong learning approaches may not be applicable• There has not been a significant amount of research on imbalanced
and low-shot lifelong learning• Lifelong learning approaches may not scale to millions of classes or
examplesMore research is needed to apply lifelong learning mechanisms
to real-world applications like embedded devices

17

54

55

Underexplored Capabilities forLifelong Learning• Better pre-trained features•

Better self-supervised methods and on larger/more diverse datasets• Techniques to improve low-shot learning• Learning from very few instances can yield speed improvements• Techniques to improve robustness to additional domain shifts
Open-world learning:• Identify unknown inputs and then incrementally learn themScheirer et al., 2013

56

Explainable Lifelong Streaming
Learning (ExLL)• The Episodes-to-Concepts (EpCon) Model• Empirical Data Analysis (EDA)

The Episodes-to-Concepts (EpCon) Model
Initial Encoding• Attention Biasing
Concept Formation• Pattern completion• Memory-based prediction error• Pattern separation• Pattern integration

M.L. Mack et al. / Neuroscience Letters 680 (2018) 31–38

Concept coding• Affected attention biasing

57

58

Empirical Data
Analysis (EDA)

Data Clouds

59

60

Data Clouds vs Clusters
Features

Clusters

Data Clouds

Boundaries

Defined as hyper-ellipsoids

Voronoi tessellation

Centre/Prototype

DefinedExtracted post factum

Distance between a data
point

Centre/Mean

Focal point

Membership function

Approximation of an ideal
Reflect the real data
distribution, assumed a priori distribution

Having a data clouds, we can extract rules

Rulei : IF (x ~ Ci) THEN (Outputi)

Empirical Data Analysis (EDA)
Probability theory, statistics number of restrictive assumptions which usually do nothold in reality• Pre defined smooth, “convenient to use” types of distribution;• Infinite amount of observations/data points;• Independence between data points (so called iid (Independent and IdenticallyDistributed data)• EDA, Entirely based on the empirical observations of discrete data points andtheir mutual position forming a unique pattern in the data space.
• An effective combination of the frequency and the space distance61

EDA – Cumulative Proximity
Cumulative proximity is a measure indicating the degree of closeness/similarity of aparticular data point to all other existing data points:

62

EDA – Basic measure,Standardized Eccentricity - represents the association of the data pointwith the tail of the distribution and the property of being an outlier/anomaly

ε is very convenient to represent well known Chebyshev inequality. It turns
into a simple check if>10 or not for n=3 because:

63

64

EDA – Basic measure,
Data density is inversely proportional to the standardized
eccentricity.

It can be proven that for Euclidean and Mahalanobis
distances D gets the form of Cauchy function:
RDE–Cauchy type, Angelov(‘02)

The density in the data space is a key characteristic ofanomalies and model structure (focal points for local submodels).

65

EDA – Basic measure,
Multi-modal (discrete global) typicality

Can be used as pdf, but is derived entirely
from data with no prior assumptions

Evolving Intelligence based on Density
Learning concept from experience (extract knowledge from data streams)• A data sample with high density (D) is eligible to be a focal point of a
Cloud/local sub model• A data sample that lies in an area of data space not covered by other local
sub-models is also eligible to form a new local sub model• Avoid overlap and information redundancy in forming new local sub models• Remove old clouds and low support utility ones

66

EDA + Episodes-to-Concepts (EpCon) Model
Empirical Data Analysis

Episodes-to-Concepts (EpCon)
Model

A data sample with high density (D) iseligible to be a focal point of a
Cloud/local sub model

Initial Encoding• Attention Biasing

A data sample that lies in an area ofdata space not covered by other local
sub-models is also eligible to form a
new local sub model

Concept Formation• Pattern completion• Memory-based prediction error• Pattern separation

Avoid overlap and information
redundancy in forming new local sub
models• Memory-based prediction error• Pattern integration

Remove old clouds and low support
utility ones

• Pruning67

68

Explainable Lifelong Streaming
Learning (ExLL)Explainable Lifelong Learning Considerations
1. Learn and recall immediately
2. Learn data in any order without catastrophicallyforgetting
3. Transfer of learned representations for
efficiency
4. Learn using limited memory and computeoverhead
5. Scale to large-scale, high-dimensional problems
6. Explain model decisions at the intermediate and final
stages of the decision-making process

69

ExLL -Training

Normalization
Convert to cosine
dissimilarity forhigh-dimensional
data processing

Global Metaparameter Update

Local Metaparameter Update

70

ExLL -Training

Normalization

Global Metaparameter Update

Local Metaparameter Update

Recursive Density
Estimation
Attention Biasing

Convert to cosine
dissimilarity forhigh-dimensional
data processing

71

ExLL -Training

Normalization

Global Metaparameter Update

Local Metaparameter Update

Recursive Density
Estimation

Prototype Identification

72

Prototype Formation
Pattern separation

Pattern Completion

Attention Biasing

Convert to cosine
dissimilarity forhigh-dimensional
data processing

Pattern Integration

Memory-based error
Prediction

Else

Topology Formation
Voronoi Tessellation

ExLL -Training

73

Explainable Rules
Extraction
Prototype Rule

Concept Rule

Normalization

Global Metaparameter Update

Local Metaparameter Update

Recursive Density
Estimation

Prototype Identification

Prototype Formation
Pattern separation

Pattern Completion

Attention Biasing

Convert to cosine
dissimilarity forhigh-dimensional
data processing

Pattern Integration

Memory-based error
Prediction

Else

Topology Formation
Voronoi Tessellation

ExLL - Inference

Shrinkage regularization

Prototype-based Inference

Megacloud-based Inference

Global Pair-wise Fusion

74

ExLL – Rule Extraction75Datasets and Orderings• OpenLORIS:• Videos of 40 object classes each with 1 to 9
object instances (121 instances)• Each object instance collected under 4 domains
with 9 difficulties• Orderings: Instance and Low-Shot Instances• Places-365:• Scene classification of 365 classes• We study the original version with over 1.8M
images• We also study a long-tailed version• Orderings: iid and class iid• F-SIOL-310• Static images of 22 household items.
• Total 310 object instances and 620 static images• Using class-iid data ordering, study 5-shot &10-shot learning scenarios• Evaluating a model’s ability to learn from few
training samples.

76

77

Comparison Methods
Online Continual Learning:• Fine-Tune• Multi-Class Perceptron• Streaming One-vs-Rest (SOvR)• Nearest Class Mean (NCM)• Gaussian Naïve Bayes• Streaming Linear Discriminant
Analysis (SLDA)• Replay• Store 2 examples or 20 examples per
class

Mobile CNNs:• MobileNet-v3 (Small)• MobileNet-v3 (Large)• EfficientNet-B0• EfficientNet-B1• RestNet-18

78

NetScore for Lifelong Learning• a(M) measures the final accuracy of a model• p(M) is the total number of parameters• c(M) is the number of minutes needed for the experiment• Scores methods across three axes: classification efficacy, memory, and computers = 20 and α = 2 to prioritize classification accuracy, and β = γ = 0.25 to moderate the large values ofp(M) and c(M) [73].
• Higher NetScores indicate better performance.

79

Instance ordering trains learners on all object instances while low-shot instanceordering trains learners on one object instance from each object class.
OpenLORIS
models displayed lower accuracy when using low-shot
How well do methods • All
instance ordering. Perceptron and Fine-Tune showed a muchbigger drop, shows poor generalization to out-of-domain inputs
generalize from few• Naive Bayes, SOvR, and NCM were less accurate than Perceptron
instances?
and Fine-Tune for the full instance ordering, but outperformed
them for the low-shot condition.

• The ExLL models showed the best balance between the twoordering methods, while ExLL-F outperformed all other models
for both orderings.

80

Table II presents the best accuracy scores of online
continual learning models across different CNN backbones
when trained using instance ordering.

Table III presents the performance of the models acrossdifferent CNN architectures for low-shot instance ordering.

OpenLORIS

How does CNN choiceaffect performance?

• Performance is best with EfficientNets andworst with ResNet-18 using instance ordering• EffcientNet yields better performance andrequires fewer computational resources• For low-shot instance ordering, the EfficientNet
backbone CNNs again outperformed the otherbackbone CNNs.

Places-365 for two data ordering methods iid and class-iid.

81

Places-LT for two data ordering methods iid and class-iid, Places-LT tests how well models
perform with severe imbalance

Places-365 & Place-LT

How robust are• We compute the harmonic mean across orderings, emphasizing theimportance of performing well on both orderings
methods to scale and • Bestmethods: ExLL-F->ExLL-P ->ExLL-M->SLDA
imbalance?
• ExLL-M was the least affected by dataset imbalance while

prototype-based inference and pairwise fusion showed a 7.4%
and 12.1% loss in performance respectively when trained with
Places-LT.

F-SIOL-310 was selected to observe how the online continual learning methods perform in low-shot 82
continuous learning applications.

How well are methods
in low-shot continual
learning?

F-SIOL-310• For the 5-shot scenario, ExLL-F is slightly
outperformed by SLDA (0.884 vs. 0.889
respectively).
• On the other hand, for the 10-shot scenario,
ExLL-F significantly outperformed the nextbest methods, ExLL-M and SLDA (0.958 vs.
0.936 and 0.931 respectively).

ExLL – Prototype Analysis

83

ExLL – Rule Extraction

84

ExLL – Topology Visualization
Visualizing the topology of learned prototypes acquired
from the FSIOL-310 dataset.

85

Visualizing the topology of learned prototypes acquired
from the Places-365 dataset, for the first 10 classes.

86

Results Summary• We studied the robustness of online continual
learners across various axes useful for
embedded learning:• Omega: classification efficacy, compute
memory• Video: ability to learn from temporally
correlated videos• Low-Shot: ability to generalize from few
inputs• Scale: ability to scale to large-scale data
• Imbal: ability to perform well on
imbalanced data
• Main Takeaways:• ExLL-F (0.91) showed the best overall
performance. Replay 20pc (0.88) and SLDA
(0.88) outperformed the second-best ExLL
model, ExLL-P (0.84). The worst-performing
ExLL model, ExLL-M (0.78) is also
outperformed by NCM (0.81).
• ExLL is less efficient with respect tocomputation and memory requirements.

Future Research Directions• Self-supervised pre-trained features• Currently do not work well for mobile CNNs
• Additional pre-training datasets• Use of multi-layer features• Online learners that update feature representations• More compute time and concept drift• Techniques to improve low-shot learning• Additional tasks and efficiency improvements

87

88

“…it is not the strongest
that survives; but…theone that is able best to
adapt…to the changing
environment….”“Once you stop
learning, you startdying.”

L.C. Megginson, re “On
the Origin of Species”

Albert Einstein
https://www.izlesene.com/iz/memcn3342

Thank you