# Aggregated f-average Neural Network applied to Few-Shot Class Incremental Learning

Mathieu Vu, Emilie Chouzenoux, Ismail Ben Ayed, Jean-Christophe Pesquet

► To cite this version:

Mathieu Vu, Emilie Chouzenoux, Ismail Ben Ayed, Jean-Christophe Pesquet. Aggregated f-average Neural Network applied to Few-Shot Class Incremental Learning. Signal Processing, 2025. hal-04874006

HAL Id: hal-04874006

https://inria.hal.science/hal-04874006v1

Submitted on 8 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Aggregated *f*-average Neural Network applied to
Few-Shot Class Incremental Learning

Mathieu Vu<sup>a,1,*</sup>, Émilie Chouzenoux<sup>a,1</sup>, Ismail Ben Ayed<sup>b</sup>, Jean-Christophe Pesquet<sup>a</sup>

<sup>a</sup>OPIS - CVN, Inria Saclay, CentraleSupélec, Université Paris Saclay, 3 rue Joliot Curie, Gif-sur-Yvette, 91190, Île-de-France, France

<sup>b</sup>LIVIA - École de Technologie Supérieure, 1100 rue Notre Dame Ouest, Montréal, H3C 1K3, Québec, Canada

**Abstract**

Ensemble learning leverages multiple models (i.e., weak learners) on a com-
mon machine learning task to enhance prediction performance. Basic ensem-
bling approaches average weak learners outputs, while more sophisticated
ones stack a machine learning model in between the weak learners outputs
and the final prediction. This work merges both aforementioned frameworks.
We introduce an *aggregated f-averages* (AFA) shallow neural network which
models and combines different types of averages to perform an optimal ag-
gregation of the weak learners predictions. We emphasise its interpretable
architecture and simple training strategy and illustrate its good performance
on the problem of few-shot class incremental learning.

*Keywords:* ensemble learning, estimator aggregation, few-shot learning, incremental learning

1. Introduction

Deep Convolutional Neural Networks (CNNs) have become the state-of-
the-art reference in most computer vision problems. They rely on curated
datasets, for which quality and quantity have a critical impact on the model

<footnote>* mathieu.vu@inria.fr</footnote>

<footnote n="1">¹M. Vu and E. Chouzenoux acknowledge support from the European Research Council under Starting Grant MAJORIS ERC-2019-STG-850925.</footnote>
performances. For instance, a challenging scenario arises when classifying im-
ages belonging to new (unseen) classes, with only a few annotations. More
specifically, in supervised classification, models are only able to classify cat-
egories that were seen during training phase. If unseen classes are added, it
is not straightforward to handle them without retraining the whole model.
Incremental learning aims at designing solutions for this situation. How-
ever, when upcoming classes include only a few training samples, incremental
learning methods do not perform well. Few-Shot Class Incremental Learning
(FSCIL), a challenging scenario introduced recently in [1], focuses on design-
ing methods that can deal with both incremental and few-shot settings. On
the one hand, a small training set may induce overfitting, hindering clas-
sification performances on the base classes. This phenomenon is known as
*catastrophic forgetting*. On the other hand, the focus on avoiding catastrophic
forgetting may have a counterproductive effect as it impedes the learning of
novel classes.

To address FSCIL, we propose to consider it as a combination of suc-
cessive few-shot learning problems. More precisely, we learn a dedicated
model for each novel set of class of a new session. These models (called weak
classifiers in ensemble learning) are, in our case, simple nearest-neighbour
classifiers based on class prototypes. Since they are specialised on their own
set of classes, they may not perform well on classes that were not included
in their training set (whether they were seen before or after). This is why
we train a final ensembling model, which performs predictions on all classes
by integrating the weak classifiers. To perform the ensembling, we introduce
a novel method called *Aggregated f-averages*. Our ensembling method is a
supervised neural network model with a specific architecture, specific activa-
tion functions and specific weight constraints, enabling to model and combine
various types of averages. It uses a few parameters while being interpretable.
This yields a strong FSCIL baseline, which integrates simple prototype-based
classifiers and performs competitively in comparisons to specialised and con-
voluted state-of-the-art FSCIL techniques on several datasets. Moreover, we
show experimentally the substantial effect of our *f*-averages ensembling in
comparisons to standard ensembling methods. Also, we closely examine how
FSCIL methods are evaluated, and argue that the average accuracy over all
classes may not be appropriate in FSCIL as the problem is an instance of
imbalanced classification. For instance, a model trained only on the base
classes, while ignoring subsequent new classes, has a rather steady mean ac-
curacy over all the classes when the number of base classes is greater than the
total number of new classes. Instead, we advocate the use of the F1 score,
which is commonly used in imbalanced classification problems.

## 2. Related Work

2.1. *The standard few-shot classification setting*

Deep learning models have reached human-level performance in many
tasks thanks to large scale datasets. However, their generalisation abilities
are challenged in settings where only a few training samples are available for
new, unseen classes. Numerous research articles have attempted to tackle this
few-shot classification problem, including for example [2, 3, 4, 5, 6, 7, 8, 9, 10].
The few-shot classification setting consists of two stages: first, in the base
stage, a model is trained with a substantial amount of labelled data on a
set of base classes; then comes the generalisation stage where only a small
number of labelled training samples are available for the novel classes.

In the standard few-shot setting, the literature abounds of convoluted
methods based on meta-learning, which consists of "learning to learn" [10, 9,
11, 6, 12]. The base stage is used to perform a series of *episodic training* that
will simulate the generalisation stage. For example, prototypical networks [9]
meta-learns how to represent each class with an embedding prototype and
to maximise the log-probability of test samples. Similarly, MAML [10] is a
method producing a meta-learner that helps the model to better fine-tune on
novel classes during generalisation stage. However, recent studies have shown
that standard training strategies coupled with transfer learning outperforms
most meta-learning methods [3, 8, 4]. For instance, SimpleShot [4] performs
a basic nearest-neighbour classifier based on class prototypes, outperforming
most meta-learning methods.

2.2. *Few-shot Class Incremental Learning*

Few-shot Class Incremental Learning (FSCIL), recently introduced in [1],
attempts to address the poor performances of CIL methods [13, 14] when
only a few training samples are available for the new classes. A variety of
specialised and, in most cases, convoluted methods (e.g., based on meta-
learning) were recently developed for FSCIL. For instance, TOPIC [1] origi-
nally implements a neural gas network, which preserves the topology of the
feature space when learning new classes, thereby mitigating the forgetting of
previously learnt classes. IDVLQ-C's [15] classification head is built using
reference vectors that are prototypes optimised through a custom loss, avoid-
ing overlap. State-of-the-art methods achieve best results using even more
elaborate classification heads combined with meta-learning-inspired training
schemes. For example, SPPR [16] generates incremental episode during each
iteration of base training. Furthermore, new-class samples are leveraged in
a self-promoted prototype refinement mechanism that compute their rela-
tions with respect to previous training samples in a latent space. Similarly,
CEC [17] meta-learns a Graph Attention Network (GAT) [18] in a pseudo-
incremental learning stage. This external module is able to adapt new clas-
sification heads given context from previous sessions and new-class samples.
C-FSCIL [19] also meta-learns a module called Explicit Memory (EM) and
embedding networks that form a content-based attention mechanism. This
takes advantage of prototypes stored in the EM similarly to episodic memory
introduced by CIL methods.

The work in [20] argued against relying solely on the average accuracy
over all the classes as evaluation metric, due to the class-imbalanced nature
of the FSCIL problem. This issue is further exacerbated in earlier sessions,
with only a few new classes compared to the many classes of the base sessions.
Instead, two metrics are used: average accuracy on base classes and average
accuracy on new classes. Using these two metrics enable to disentangle the
discovery of new classes and the prevention of catastrophic forgetting.

2.3. Ensemble learning

Ensemble learning is a set of methods which leverage an ensemble of
models (also called weak learners) instead of relying on a single learner to
perform a certain task. While ensembling is obviously more demanding in
terms of computing power, it can achieve better accuracy and generalisation,
improve overall stability and reduce prediction variance and bias. Two main
phases are identified in the process of building an ensemble model: weak
learners training and output fusion [21]. The former focuses on producing an
ensemble of diverse models, which is a crucial principle in ensemble learning.
For example, bootstrap aggregating (also known as bagging) [22] is a method
that trains each model on a different subset of the training data to produce
diverse weak learners. The latter, output fusion, consists of gathering outputs
from every weak learner of the ensemble and on combining them in order to
produce the final prediction.

Two categories of methods can be distinguished to perform output fusion
in ensemble learning [21]. The most direct and naive method to produce
the unified prediction is to average weak learners outputs or, in the case of
classification, to use a majority voting scheme. Different types of averages
could be used (e.g., arithmetic, geometric, harmonic, etc), or weights that
could be included to further refine results. Those weights can be set using
various kinds of criteria like, for example, based on weak learners isolated
performances [23]. The second category makes use of meta-learners that are
an additional model, which is responsible for taking advantage of the weak
learners. Mixture of experts is a variant of meta-learners, where a gating
network does not combine weak learners output but rather select the weak
learner that is most suited to produce the correct prediction given a certain
input [24]. A more straightforward output fusion based on meta-learners is
the stacking of the additional learning model. Taking weak learners output
as input, it learns the best combination to assemble the unified prediction
[25].

## 2.4. *Our contributions*

As mentioned earlier, state-of-the-art FSCIL methods follow the method-
ological trend that have dominated the standard few-shot setting by relying
on complex meta-learning strategies. We propose here a simpler approach,
where the ensemble learning paradigm is used, with the aim to optimally fuse
a set of nearest-neighbour classifiers based on class prototypes. Given the
configuration of the FSCIL setting, training data is characteristically split
into subsets whose classes are exclusive. Using those subsets to train weak
learners naturally provides the desired diversity. Furthermore, we introduce
an original ensembling method, which focuses on the output-fusion part of en-
semble learning. We show that, in the FSCIL setting, our ensembling model
significantly outperforms the output-fusion strategies commonly used in the
ensemble learning literature, while being competitive with state-of-the-art
FSCIL methods.

Our ensembling method enables to model various types of generalised
weighted averages, and to combine and/or select them optimally, in a su-
pervised manner. Moreover, our specific architecture models explicitly the
balancing weights between the different types of averages, and hence yields
an interpretable ensembling mechanism. In order to train our ensemble learn-
ing model in the context of FSCIL, we adopt an episodic memory to learn
the balance between base and new classes. Initial training images are not
stored. Instead, only their projection in the feature space is needed, which
is memory efficient.
Our experimental analysis relies on advanced metrics (average accuracy on base classes, and average accuracy on new classes) to better understand and assess the trade-off between base and new classes, for the benchmarked models. We also use those metrics to illustrate that the mean F1 score is better suited to evaluate FSCIL methods, as it is a metric commonly used in imbalanced classification tasks.

### 3. Problem set-up

In this work, we design a novel ensemble learning framework applicable to the problem of few-shot class incremental learning (FSCIL). Let us recall the FSCIL problem set-up, in this Section. FSCIL, recently introduced in [1], focuses on designing machine learning methods that can deal with both incremental [13, 14] and few-shot situations [7, 8]. Specifically, FSCIL aims at including an increasing number of categories in a classification problem with the extra constraint that only a small number of training samples are available for upcoming classes. This is motivated by the frequent practical situation in computer vision when a model, built to classify certain categories seen during training phase, must be adjusted to classify images belonging to new classes with only (very) few annotations. The main difficulty revolves around the ability to learn new classes while preventing catastrophic forgetting of classes previously learnt, yielding poor performance of standard incremental learning methods.

The FSCIL setting considered in this paper consists of successive sessions that incrementally provide new categories of images to be classified. First session, also called base session, is a standard classification problem with a substantial number of training samples for each base class. The number of base classes, that we will denote $n_{class\_base}$, is usually large relatively to the total number of classes, that we will denote $n_{class}$. In subsequent sessions, only a few training samples are provided for a limited number of novel classes. Reusing standard incremental and few-shot classification notations, let $K$ be the total number of sessions, $n_{way}$ the number of new classes and $n_{shots}$ the number of samples for each new class. Let $D_k^{train}$ (respectively $D_k^{test}$) be the sets of training (respectively test) images associated with session $k$ with $k \in \{1, \dots, K\}$. Similarly, let us denote $C_k^{train}$ (respectively $C_k^{test}$) the classes labels available at session $k$. During training, for sessions $k \in \{2, \dots, K\}$, the number of new classes $|C_k^{train}|$ equals $n_{way}$ and the number of sample images $|D_k^{train}|$ equals $n_{shots}$. For base session (i.e., $k = 1$), we have a relatively
large number of base classes $|C_1^{\text{train}}| = n_{\text{class\_base}}$, and a large number of samples $|D_1^{\text{train}}|$. After each session $k \in \{1, \dots, K\}$, the goal is to predict, for each image $i \in D_k^{\text{test}}$, its class label $c$ among the available $C_k^{\text{test}}$. Let us emphasise that $c$ can be any class from base to current session, i.e. that $C_k^{\text{test}} = \bigcup_{j=1}^k C_j^{\text{train}}$. At each session $k$, the number of classes to predict $|C_k^{\text{test}}|$ which we will denote $N_k$ is thus,

$$N_k = n_{\text{class\_base}} + (k-1)n_{\text{way}}. \qquad (1)$$

# 4. Methodology

We now present our main contribution, that is the proposition of a novel ensemble learning approach, and its specification for tackling the problem of FCSIL. We first recall, in Sec. 4.1, the principle of ensembling based on averaging operations. Then, in Sec. 4.2, we introduce the concept of f-average, generalizing the Kolmogorov's mean framework, that will be at the core of our ensemble learning model. In Sec. 4.3, we present our proposed ensembling method called AFA (aggregated f-average), and finally, Sec. 4.4 describes our proposed strategy to integrate ensemble learning in the context of FCSIL.

## 4.1. Ensembling through averaging

Consider $K$ machine learning models trained for a common task (e.g., classification), producing $K$ outputs $(x_k)_{1 \le k \le K}$, assumed to be vectors in $\mathbb{R}^N$. In ensemble learning, those $K$ outputs are combined during an output fusion phase in order to produce a single, expectably better, prediction for the task at hand. A naive method is to average the outputs with appropriate weights. We summarize in Table 1 common expressions for weighted averages, with $(\omega_k)_{1 \le k \le K}$ nonnegative reals such that $\sum_{k=1}^K \omega_k = 1$.

<table><caption>Table 1: Examples of weighted averages along with their validity domains.</caption><thead><tr><th>Mean</th><th>Arithmetic</th><th>Geometric</th><th>Harmonic</th><th>Power-q</th></tr></thead><tbody><tr><th>Formula</th><td>$\sum_{k=1}^{K} \omega_k x_k$</td><td>$\prod_{k=1}^{K} x_k^{\omega_k}$</td><td>$\left(\sum_{k=1}^{K} \frac{\omega_k}{x_k}\right)^{-1}$</td><td>$\left(\sum_{k=1}^{K} \omega_k x_k^q\right)^{1/q}$</td></tr><tr><th>Validity</th><td>$x_k \in \mathbb{R}^N$</td><td>$x_k \in [0, +\infty)^N$</td><td>$x_k \in (0, +\infty)^N$</td><td>$x_k \in [0, +\infty)^N, q > 0$</td></tr></tbody></table>
### 4.2. f-average

Following Kolmogorov's mean framework [26], we can rewrite the above examples under the general form:

$$ \tilde{x} = f^{-1}\left(\sum_{k=1}^{K} \omega_k f(x_k)\right). \qquad (2) $$

Hereabove, $f$ is a bijective function from $[0, +\infty)^N$ to some convex $C$ of $\mathbb{R}^N$, and $f^{-1}$ is its inverse function from $C$ to $[0, +\infty)^N$. We will subsequently assume that $f$ operates component-wise, in the sense that it consists of the application of the same scalar function to each of the components of its argument. Let us now express functions $f$ and $f^{-1}$ to retrieve the popular averaging rules from Table 1. We denote $(\xi_n)_{1\le n\le N}$ the components of $x \in \mathbb{R}^N$. We set $\epsilon \in (0, +\infty)$ and, to circumvent the indefiniteness of the harmonic mean for vectors with a zero component, we define the *leaky hyperbolic function* $h_\epsilon$ as

$$ (\forall \xi \in \mathbb{R}) \quad h_{\epsilon}(\xi) = \begin{cases} \frac{1}{\xi + \epsilon} - \epsilon & \text{if } \xi \in [0, 1/\epsilon - \epsilon] \\ -\frac{\xi}{\epsilon^2} + \frac{1}{\epsilon} - \epsilon & \text{if } \xi < 0 \\ -\epsilon^2 \left(\xi - \frac{1}{\epsilon} + \epsilon\right) & \text{if } \xi > 1/\epsilon - \epsilon. \end{cases} \qquad (3) $$

Under these notations, Table 2 summarizes the expressions for $f$, $f^{-1}$, as well as their associated definition domains, recovering the averaging rules from Table 1. Exact geometric and harmonic means formula are retrieved when $\epsilon$ goes to zero.

<table><caption>Table 2: Examples for $f$, $f^{-1}$, and definition domains, for $x = (\xi_n)_{1\le n\le N}$. Geometric and harmonic formulas are retrieved when $\epsilon \to 0$.</caption><thead><tr><th>Mean</th><th>f(x)</th><th>f domain</th><th>f<sup>-1</sup>(x)</th><th>f<sup>-1</sup> domain</th></tr></thead><tbody><tr><td>Arithmetic</td><td>Id</td><td>[0, +∞)<sup>N</sup></td><td>Id</td><td>[0, +∞)<sup>N</sup></td></tr><tr><td>Geometric</td><td>(ln(ξ<sub>n</sub> + ϵ))<sub>1≤n≤N</sub></td><td>[0, +∞)<sup>N</sup></td><td>(exp(ξ<sub>n</sub>) - ϵ)<sub>1≤n≤N</sub></td><td>[ln ϵ, +∞)<sup>N</sup></td></tr><tr><td>Harmonic</td><td>(h<sub>ϵ</sub>(ξ<sub>n</sub>))<sub>1≤n≤N</sub></td><td>[0, +∞)<sup>N</sup></td><td>(h<sub>ϵ</sub>(ξ<sub>n</sub>))<sub>1≤n≤N</sub></td><td>(-∞, ϵ<sup>-1</sup> - ϵ]<sup>N</sup></td></tr><tr><td>Power-q</td><td>(ξ<sub>n</sub><sup>q</sup>)<sub>1≤n≤N</sub></td><td>[0, +∞)<sup>N</sup></td><td>(ξ<sub>n</sub><sup>1/q</sup>)<sub>1≤n≤N</sub></td><td>[0, +∞)<sup>N</sup></td></tr></tbody></table>

We now propose to extend the generalised average framework (2) to the case when scalars $(\omega_k)_{1\le k\le K}$ are replaced by matrices $(\Omega_k)_{1\le k\le K}$ in $\mathbb{R}^{N \times N}$, so
as to allow a full mixing of the weak learners. Given some functions $(f, f^{-1})$ defined as previously, we define the $f$-average output $\tilde{x} \in \mathbb{R}^N$ as

$$ \tilde{x} = f^{-1}\left(\sum_{k=1}^{K} \Omega_k f(x_k)\right) = f^{-1}(W \mathbf{f}(\mathbf{x})). \quad (4) $$

Hereabove,

$$ W = [\Omega_1, \dots, \Omega_K] \in \mathbb{R}^{N \times KN}, \quad (5) $$

and function

$$ \mathbf{f}: [0, +\infty)^{KN} \to C^K \quad (6) $$

$$ \mathbf{x} = (x_k)_{1 \le k \le K} \mapsto (f(x_k))_{1 \le k \le K}, \quad (7) $$

applies $f$ in a parallel (i.e., block-wise) manner to the vector inputs $(x_k)_{1 \le k \le K}$.

In order to ensure the interpretability of the averaging operation in (4), we propose to set $W$ such that

$$ \begin{cases} W \in [0, +\infty)^{N \times KN}, \\ W 1_{KN} = 1_N. \end{cases} \quad (8) $$

This guarantees, in particular, that $W \mathbf{f}(\mathbf{x})$ belongs to the definition domain of $f^{-1}$ (since this domain has been assumed to be convex). For instance, if $\mathbf{x} \in [0, 1]^{KN}$ (e.g., in a classification context), the constraint (8) on $W$ ensures that the output $\tilde{x}$ also belongs to $[0, 1]^N$.

<figure><img src="image_2.png" /><figcaption>Figure 1: Structure of the $f$-average network, i.e. a neural network that performs an $f$-average for ensembling</figcaption></figure>

Remarkably, operation (4), that we call an $f$-average, can be represented as the application, on $\mathbf{x}$, of a two-layer neural network whose structure is drawn in Figure 1. This neural network is parametrized by the choice of $f$ (along with its inverse function $f^{-1}$) and by the weight matrix $W$. The former can be set by the user, while the latter can be determined through supervised learning, by minimizing a loss associated to the task at hand, over a training set. Once trained, the $f$-average network in Figure 1 is interpretable, as the contribution of each output in the final prediction can easily be retrieved using the weights in matrix $W$, and the averaging operation is simply determined by the choice of $f$.
### 4.3. Aggregated f-averages

The previous approach requires the prior choice for the average rule (i.e., $f$). To wave this restriction, we suggest aggregating $J > 1$ f-averages, associated to different functions $(f_j)_{1 \le j \le J}$. Resorting to the same structure as the one presented in the previous section, we define

$$ (\forall j \in \{1, \dots, J\}) \quad \tilde{x}_j = f_j^{-1}(W_j f_j(\boldsymbol{x})), \qquad (9) $$

where, for every $j \in \{1, \dots, J\}$,

$$ \left\{ \begin{array}{l} W_j \in [0, +\infty)^{N \times KN} \\ W_j 1_{KN} = 1_N, \end{array} \right. \qquad (10) $$

and $f_j$ is a function operating component-wise from $[0, +\infty)^{KN}$ to $C_j$, associated to a given averaging function $f_j$, $f_j^{-1}$ is its inverse function operating component-wise from some convex set $C_j \subset \mathbb{R}^N$ to $[0, +\infty)^N$, and $\boldsymbol{x}$ concatenates columnwise the inputs $(x_k)_{1 \le k \le K}$. The resulting joint aggregate estimate of the $J$ outputs $(\tilde{x}_k)_{1 \le j \le J}$ is defined as

$$ \hat{x} = \sum_{j=1}^{J} A_j \tilde{x}_j = A \begin{bmatrix} \tilde{x}_1 \\ \vdots \\ \tilde{x}_J \end{bmatrix}, \qquad (11) $$

with, for every $j \in \{1, \dots, J\}$, $A_j \in [0, +\infty)^{N \times N}$, and $A \in [0, +\infty)^{N \times N J}$ is the rowwise stacking of $(A_j)_{1 \le j \le J}$ matrices. Operations (9)-(11) are equivalent to plug $\boldsymbol{x}$ as the input of a neural network with $J$ sub-networks (i.e., branches) of the form presented in Figure 1, operating in parallel, followed by a linear layer involving the weight matrix $A$. We further propose to add a final activation function, $g: \mathbb{R}^N \to \mathbb{R}^N$, to control the domain of the output. For instance, a softmax activation can be used to get nonnegative outputs summing to one, in a classification context. The resulting network, called aggregated f-averages (AFA), is displayed in Figure 2. It has a limited number $JN^2(K + 1)$ of linear parameters, namely the entries of matrices $W_1, \dots, W_J$, and $A$. The training of these parameters can follow a classical supervised learning approach. Given a sample and its ground truth, the task model loss is computed (e.g., a cross-entropy loss for classification), before updating the weights from all layers using a backpropagation algorithm (e.g., Adam optimizer [27]). Constraints (10) on weight matrices $(W_j)_{1 \le j \le J}$ can
simply be imposed through a projection step of each row of these matrices
on the convex unit simplex set [28], performed after each backpropagation
update. Let us remark that this model maintains the interpretability prop-
erties of the *J* *f*-average sub-models it contains. Furthermore, weights from
matrix *A* can be viewed as the contribution level of each type of average
model.

<figure><img src="image_3.png" /><figcaption>Figure 2: Structure of the proposed aggregated *f*-average (AFA) neural network. It aggregates *J* *f*-averages for ensembling, with *A* ∈ [0, +∞)^(N×NJ). The activation function *g*: R^N → R^N is selected according to the task (e.g, softmax for classification, linear for regression).</figcaption></figure>

4.4. Application of Aggregated *f*-averages to FSCIL

We now present our approach, to address FSCIL, using ensemble learn-
ing by AFA. We propose to consider FSCIL as a combination of successive
few-shot learning problems. To do so, for each session, we train a few-shot
learning classifier. Each of these weak classifiers can be viewed as a model
specialised on its own session data. This enables to build a diverse set of
models which are then gathered by our AFA ensembling model. By combin-
ing predictions from every weak classifiers, the ensembling model will be able
to output a prediction for all classes from base to current sessions. Let us
now describe the detailed architecture, and our specific strategy for padding
unseen classes predictions.

4.4.1. Architecture of AFA for FSCIL

The complete architecture is displayed in Fig. 3. First, and similarly to
[3, 1], we consider a CNN classification model (here, a ResNet-18), as the
composition of a feature extractor $f_{\theta}$ parametrised by some network weights gathered in a vector $\theta$, and a classification head $H_{W_k}$ with $W_k$ its weights for session $k$. The CNN is trained during base session ($k = 1$), namely the only session with a substantial amount of data. The CNN's backbone $f_{\theta}$ (dark blue box in Fig. 3) is frozen, and will serve the purpose of a feature extractor for following sessions ($k \in \{2, \dots, K\}$). Its last layer, i.e., the classification head $H_{W_1}$ (top light blue box in Fig. 3) is a weak classifier specialised on the base session.

<figure><img src="image_4.png" /><figcaption>Figure 3: AFA ensembling for FSCIL. The CNN feature extractor $f_{\theta}$ is trained in base session and frozen for following sessions. Its classification head $H_{W_1}$ is a standard classification layer. For subsequent sessions (for $k \in \{2, \dots, K\}$) SimpleShot classifications heads $H_{W_k}$ are used, based on nearest-neighbour classifiers, and fine-tuned using $D_k^{\text{train}}$. A padding $p$ is added to compensate the size of outputs $x$ from classification heads.</figcaption></figure>
Hence, for $k \in \{2, \dots, K\}$, we replace it by new classification heads $H_{W_k}$. Specifically, instead of a CNN last layer, we rely on the few-shot learning method SimpleShot [4]. SimpleShot is a nearest-neighbour classifier which, given a session $k$, determines an estimated class $\hat{c}(i)$ of a test image $i \in D_k^{\text{test}}$ by retrieving the closest mean centroid in the feature space

$$ \hat{c}(i) = \arg \min_{c \in C_k^{\text{test}}} \|z - \bar{z}_c\|^2, \qquad (12) $$

where $z$ is the $l_2$-normalised feature vector of a sample image $i$

$$ z = \frac{f_{\theta}(i)}{\|f_{\theta}(i)\|_2} \qquad (13) $$

and $\{\bar{z}_c | c \in C_k^{\text{test}}\}$ are the mean centroid of each class

$$ (\forall k \in \{1, \dots, K\})(\forall c \in C_k^{\text{train}}) \quad \bar{z}_c = \frac{1}{n_{\text{shots}}} \sum_{n=1}^{n_{\text{shots}}} z_{n,c}. \qquad (14) $$

Hence, the SimpleShot classification head coupled with the feature extractor forms the weak classifier for each session $k \in \{2, \dots, K\}$. All these weak classifiers should perform well on images whose classes belong to their respective session since they are trained only on these classes. However, they might not generalise on (unseen) classes from other sessions. To overcome this issue, we propose to ensemble all weak classifiers outputs, using our AFA model (orange box in Fig. 3), so as to produce a final model capable of predicting any class. At session $k$, input of the ensembling model consists of $k$ outputs, each from a different weak classifier. However, except the CNN from base session, weak classifiers output distances and not probabilities. To convert them, we use the following formula:

$$ x = \text{softmax}(\tau \|z - \bar{z}_c\|^2) \qquad (15) $$

where $\tau > 0$ is a learnable temperature scaling parameter. Since SimpleShot relies on the smallest distance to predict classes, we need to transform low distances into high probabilities, hence the softmax function. Once learnt, the temperature parameter scales the output distance so that the softmax operation produces meaningful numerical values.
### 4.4.2. Padding and inlierness

As it can be seen in Fig. 3, once session K is reached, we obtain K weak classifiers outputs, with a vector of prediction of a different size $N_k$ ($\forall k \in \{1, \dots, K\}$) (see also Eq. (1)). However, our AFA ensembling only averages inputs of the same length. To cope with this situation, we must add a padding at the end of each prediction vector (except, obviously, for the last one which is already of size $N_K$) in order to obtain K prediction vectors which are all of the same size, i.e. predicting the same number of classes, namely the number of classes $N_K$ of session K (cf. Eq. (16)). The size of the padding is $N_K - N_k$, and $(p_i)_{1 \le i \le N_K - N_k}$ denotes the padding vector so that

$$x = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_{N_k} \\ p_1 \\ p_2 \\ \vdots \\ p_{N_K - N_k} \end{pmatrix} \qquad (16)$$

We initially set the components of $p$ to be all equal to $p_i = 1/(N_K - N_k)$, considering a uniform probability among the remaining classes. However, by doing so, $x$ in Eq. (16) becomes the concatenation of two probabilities vector, not summing to one, with an obvious lack of interpretability. We thus propose a corrective approach, where we multiply the predictions $(x_i)_{1 \le i \le N_k}$ and the padded values $(p_i)_{1 \le i \le N_K - N_k}$ by appropriate scaling factors, ensuring both normalization and meaningfulness of the output. To do so, we design an inlierness indicator assessing whether a given input is regarded as probably being in the set of learnt classes of that particular weak classifier or not. The inlierness indicator relies on the quantity $\max_{1 \le i \le N_k} x_i$, which indeed indicates the weak classifier level of confidence of the prediction $x$ from Eq. (15). If the latter value is higher than a certain threshold $t > 0$ (set as an hyperparameter), then the prediction is considered to be reliable and likely to belong to the set of learnt classes for this particular weak classifier. In such a case, the normalisation factor reflects the confidence by increasing the weight of corresponding predictions $x_i$ with respect to padding values $p_i$. Otherwise, if the prediction confidence is lower than the threshold, the normalisation
factor emphasizes padding values instead of the predictions. More precisely, new prediction values $x'_i$ are computed as follows, for every $i \in \{1, \dots, N_k\}$,

$$x'_{i} = \begin{cases} \frac{1}{2}(\frac{N_k}{N_K} + 1)x_i, & \text{if } \max_{1 \le i \le N_k} x_i \ge t \\ \frac{1}{2}(\frac{N_k}{N_K})x_i, & \text{otherwise.} \end{cases} \quad (17)$$

Furthermore, the corrected values $(p'_i)_{1 \le i \le N_K - N_k}$ are computed using the complimentary probability, in order to obtain a prediction vector summing to one. For every $i \in \{1, \dots, N_K - N_k\}$,

$$p'_{i} = \begin{cases} (1 - \frac{1}{2}(\frac{N_k}{N_K} + 1))p_i, & \text{if } \max_{1 \le i \le N_k} x_i \ge t \\ (1 - \frac{1}{2}(\frac{N_k}{N_K}))p_i, & \text{otherwise,} \end{cases} \quad (18)$$

where we recall that $p_i = 1/(N_K - N_k)$ for all $i \in \{1, \dots, N_K - N_k\}$.

Finally, to train the AFA ensembling model, we adopt a *prototype re-hearsal* method similarly to [29], i.e., weak classifiers perform predictions on all training prototypes used during all sessions in order to provide training inputs for our final ensembling model.

# 5. Experiments

We now present our experimental results, with the aim to show (i) the validity of the combined FSCIL and ensembling pipeline, (ii) the benefits of our AFA method when compared to state-of-the-art ensemble learning methods, in that context.

We conduct experiments on four image classification datasets typically used in few-shot or incremental learning, namely mini-ImageNet [11], CUB-200 [30], tiered-ImageNet [31], and FGVC-Aircraft [32]. Let us first describe each dataset specificity.

*mini-ImageNet*. Based on ImageNet [33], this dataset provides 500 training samples and 100 test samples for each of the 100 classes. We replicate [1] setting parameters by having 9 sessions with a number of base classes $n_{class\_base}$ of 60 and 5-way 5-shots for following sessions.

*tiered-ImageNet*. Also based on ImageNet, this dataset provides 506 classes of images. This allows us to evaluate our method on a larger scale than what mini-ImageNet allows us to. To do so, we set the total number of sessions to 11 with a number of base classes $n_{class\_base}$ of 100 and 10-way 5-shots for following sessions, similarly to the CUB-200 dataset.
*CUB-200*. As opposed to ImageNet (and its smaller versions), this dataset is devised for fine-grained image classification. For this dataset too, we re-produce [1] setting with 11 sessions, a number of base classes $n_{class\_base}$ of 100 and 10-*way* 5-*shots* for following sessions.

*FGVC-Aircraft*. To complete previous datasets, we add FGVC-Aircraft as it is another fine-grained image classification dataset, with a similar size compared to mini-ImageNet. We set it up similarly to the latter with 10 sessions, a number of base classes $n_{class\_base}$ of 50 and 5-*way* 5-*shots* for following sessions.

These four datasets allow us to provide experimental results for standard and fine-grained image classification and, for both, with a smaller and a larger setting size.

For every experiment, we use ResNet-18 [34] as base model. Trained on base session, we set the mini-batch size to 128, the learning rate to 0.1 and the momentum to 0.9. The learning rate is decreased by a factor 10 on epoch 30 and again on epoch 40. For subsequent sessions, its classification head is removed and replaced as described in Section 4.4 and feature extractor weights are frozen. Our method is implemented using Pytorch [35] and models were trained on a Nvidia Tesla V100 GPU.

Experimental results of our proposed AFA pipeline, are compared against state-of-the-art methods dedicated to FSCIL, namely SPPR [16], CEC [17], and FACT [36]. We also display results from the incremental learning methods LUCIR [14] and EEIL [13], for reference. Regarding ensemble learning benchmarks, we perform comparisons with standard output fusion strategies, namely classic averaging rules (arithmetic, geometric, and harmonic) and a majority vote scheme. In addition, stacked NNs performing the output fusion are also experimented (a shallow NN, a deep NN, and a weighted average NN).

We first discuss in Section 5.1 the choice for the comparative metrics, which is particularly challenging in the FSCIL context. We then provide our experimental results and discuss those in Section 5.2.

## 5.1. *Setting comparison metrics*

Let us first discuss the comparison metrics, used in our subsequent experiments. In most experimental settings, $n_{way}$ being much smaller than $n_{class\_base}$, the FSCIL problem is an imbalanced classification one. While the imbalance is especially strong in earlier sessions due to the very few novel
<table><caption>Table 3: Detailed metrics on last session of mini-ImageNet and CUB-200, for 3 compared methods. Here, the model ‘No fine-tuning’, only trained on base session and ignoring new sessions displays a very close mean accuracy over all classes, hiding the fact that it can hardly handle new classes. This table illustrates that F1 score better reflects performance in an imbalanced classification setting.</caption><thead><tr><th rowspan="2">Method</th><th colspan="4">mini-ImageNet</th><th colspan="4">CUB-200</th></tr><tr><th>mean acc.</th><th>acc. base</th><th>acc. new</th><th>F1</th><th>mean acc.</th><th>acc. base</th><th>acc. new</th><th>F1</th></tr></thead><tbody><tr><td>No fine-tuning</td><td>39.26</td><td>65.43</td><td>0.00</td><td>30.08</td><td>33.33</td><td>67.42</td><td>0.00</td><td>24.64</td></tr><tr><td>FACT [36]</td><td>49.51</td><td>73.80</td><td>13.07</td><td>44.77</td><td>56.43</td><td>72.93</td><td>40.30</td><td>56.03</td></tr><tr><td>AFA</td><td>48.11</td><td>66.96</td><td>27.80</td><td>48.75</td><td>52.22</td><td>70.53</td><td>34.33</td><td>51.58</td></tr></tbody></table>

classes compared to the bigger base training set, this can also be observed when reaching last session where $n_{class}$ can be inferior to $n_{class\_base}$. Preliminary experiments show that some methods might reach a fairly strong mean accuracy over all (base and new) classes by focusing only on base classes. While this can prevent efficiently catastrophic forgetting, it can also be counterproductive in the learning of new classes. Table 3 illustrates this behaviour with an extreme example.

For example, we devise a model called ‘No fine-tuning’ trained on base session only, i.e. that ignored following sessions training. For those sessions, vector predictions are completed with zeros as the model does not recognize any class beyond base session. We can observe that such model retains a decent mean accuracy thanks to its performance on base session classes. In contrast, both FACT and our AFA appear to generalize well from the base to the new session. The bias of the mean accuracy metric is well-known [37]. Instead, in such imbalanced settings, the F1 score is preferred, defined as

$$F1 = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \qquad (19)$$

where Precision = TruePositive / (TruePositive + FalsePositive) and Recall = TruePositive / (TruePositive + FalseNegative). As Table 3 shows, imbalanced performance between base and new classes are better accounted for with this metric than with average accuracy over all classes. Following results will be reported using only F1 score, based on this discussion. In all the forthcoming tables, best (i.e., highest) F1 scores among all benchmarks are highlighted in bold, while the top F1 scores among ensembling methods are underlined.
<table><caption>Table 4: Benchmark on mini-ImageNet. Methods from various strategies are included: incremental learning, FSCIL and ensemble learning.</caption><thead><tr><th rowspan="2">Strategy</th><th rowspan="2">Method</th><th colspan="9">mini-ImageNet<br>F1 score (%) for each session</th></tr><tr><th>0</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td rowspan="2">Incremental</td><td>LUCIR [14]</td><td><strong>75.93</strong></td><td>74.21</td><td>56.88</td><td>44.02</td><td>31.86</td><td>25.83</td><td>22.47</td><td>18.48</td><td>14.75</td></tr><tr><td>EEIL [13]</td><td>70.92</td><td>69.93</td><td>5.64</td><td>1.63</td><td>1.88</td><td>2.60</td><td>2.74</td><td>1.27</td><td>1.05</td></tr><tr><td rowspan="3">FSCIL</td><td>SPPR [16]</td><td>62.40</td><td>62.60</td><td>57.64</td><td>52.90</td><td>49.01</td><td>45.75</td><td>42.81</td><td>39.34</td><td>36.90</td></tr><tr><td>CEC [17]</td><td>72.12</td><td>66.95</td><td>62.94</td><td>59.63</td><td>56.91</td><td>53.87</td><td>51.37</td><td>49.36</td><td>47.72</td></tr><tr><td>FACT [36]</td><td>75.37</td><td>70.49</td><td>65.99</td><td>62.21</td><td><strong>58.89</strong></td><td>55.71</td><td>52.50</td><td><strong>50.40</strong></td><td>48.59</td></tr><tr><td rowspan="8">Ensemble<br>Learning</td><td>Arithmetic mean</td><td>75.37</td><td>69.51</td><td>64.58</td><td>60.05</td><td>55.43</td><td>51.75</td><td>48.56</td><td>45.26</td><td>42.74</td></tr><tr><td>Geometric mean</td><td>75.37</td><td>69.98</td><td>65.27</td><td>59.87</td><td>55.41</td><td>51.13</td><td>47.93</td><td>44.23</td><td>41.26</td></tr><tr><td>Harmonic mean</td><td>75.37</td><td><strong>70.82</strong></td><td><strong>66.72</strong></td><td>62.21</td><td>57.53</td><td>52.44</td><td>48.64</td><td>44.66</td><td>41.26</td></tr><tr><td>Majority vote</td><td>75.37</td><td>67.52</td><td>62.50</td><td>58.29</td><td>53.36</td><td>50.08</td><td>46.69</td><td>43.94</td><td>40.88</td></tr><tr><td>Shallow NN</td><td>75.37</td><td>65.93</td><td>63.90</td><td>59.57</td><td>56.77</td><td>54.35</td><td>50.73</td><td>43.81</td><td>46.67</td></tr><tr><td>Deep NN</td><td>75.37</td><td>60.55</td><td>61.12</td><td>59.26</td><td>55.68</td><td>53.51</td><td>46.96</td><td>47.04</td><td>43.77</td></tr><tr><td>Weighted avg.</td><td>75.37</td><td>64.24</td><td>62.82</td><td>53.77</td><td>53.17</td><td>50.58</td><td>47.50</td><td>44.18</td><td>37.46</td></tr><tr><td>AFA</td><td>75.37</td><td>68.83</td><td>66.54</td><td><strong>63.09</strong></td><td>57.86</td><td><strong>56.40</strong></td><td><strong>53.50</strong></td><td>50.36</td><td><strong>48.80</strong></td></tr></tbody></table>

<table><caption>Table 5: Benchmark on CUB-200. Methods from various strategies are included: incremental learning, FSCIL and ensemble learning.</caption><thead><tr><th rowspan="2">Strategy</th><th rowspan="2">Method</th><th colspan="10">CUB-200<br>F1 score (%) for each session</th></tr><tr><th>0</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th></tr></thead><tbody><tr><td rowspan="2">FSCIL</td><td>LUCIR [14]</td><td>72.13</td><td>72.90</td><td>65.74</td><td>60.38</td><td>57.06</td><td>49.51</td><td>44.17</td><td>41.29</td><td>40.01</td><td>37.57</td><td>37.27</td></tr><tr><td>EEIL [13]</td><td>68.40</td><td>68.44</td><td>51.27</td><td>43.59</td><td>39.45</td><td>36.22</td><td>34.12</td><td>31.27</td><td>29.19</td><td>27.10</td><td>26.47</td></tr><tr><td rowspan="3">Ensemble<br>Learning</td><td>SPPR [16]</td><td>68.66</td><td>63.18</td><td>58.70</td><td>54.44</td><td>51.67</td><td>47.92</td><td>46.22</td><td>45.59</td><td>42.34</td><td>40.35</td><td>38.74</td></tr><tr><td>CEC [17]</td><td>75.69</td><td>71.56</td><td>67.94</td><td>63.15</td><td>62.22</td><td>57.95</td><td>57.26</td><td>55.61</td><td>54.49</td><td>53.30</td><td>52.12</td></tr><tr><td>FACT [36]</td><td><strong>76.75</strong></td><td><strong>73.53</strong></td><td><strong>70.34</strong></td><td><strong>65.39</strong></td><td><strong>64.79</strong></td><td><strong>61.54</strong></td><td><strong>61.07</strong></td><td><strong>59.43</strong></td><td><strong>57.49</strong></td><td><strong>57.30</strong></td><td><strong>56.29</strong></td></tr><tr><td rowspan="8">Ensemble<br>Learning</td><td>Arithmetic mean</td><td>75.70</td><td>62.01</td><td>63.93</td><td>61.47</td><td>58.19</td><td>55.74</td><td>53.73</td><td>52.63</td><td>50.40</td><td>48.59</td><td>47.22</td></tr><tr><td>Geometric mean</td><td>75.70</td><td>52.08</td><td>51.77</td><td>50.58</td><td>49.82</td><td>48.56</td><td>48.37</td><td>47.44</td><td>46.79</td><td>45.62</td><td>44.82</td></tr><tr><td>Harmonic mean</td><td>75.70</td><td>52.17</td><td>50.64</td><td>47.17</td><td>45.26</td><td>43.57</td><td>42.24</td><td>41.10</td><td>39.33</td><td>38.59</td><td>37.27</td></tr><tr><td>Majority vote</td><td>75.70</td><td>58.64</td><td>61.42</td><td>60.39</td><td>57.92</td><td>54.05</td><td>51.42</td><td>49.03</td><td>47.56</td><td>45.38</td><td>44.29</td></tr><tr><td>Shallow NN</td><td>75.70</td><td>55.29</td><td>57.19</td><td>60.17</td><td>59.84</td><td>57.98</td><td><strong>57.28</strong></td><td><strong>55.08</strong></td><td><strong>53.85</strong></td><td><strong>53.62</strong></td><td>50.57</td></tr><tr><td>Deep NN</td><td>75.70</td><td>49.09</td><td>57.48</td><td>57.72</td><td>55.00</td><td>53.15</td><td>47.52</td><td>46.94</td><td>42.41</td><td>41.83</td><td>38.43</td></tr><tr><td>Weighted avg.</td><td>75.70</td><td>36.73</td><td>40.86</td><td>56.16</td><td>57.00</td><td>52.25</td><td>49.97</td><td>43.64</td><td>49.60</td><td>39.21</td><td>46.52</td></tr><tr><td>AFA</td><td>75.70</td><td><strong>63.08</strong></td><td>61.53</td><td>61.13</td><td><strong>60.01</strong></td><td><strong>57.99</strong></td><td>56.13</td><td>53.79</td><td>51.95</td><td>51.45</td><td><strong>51.58</strong></td></tr></tbody></table>

## 5.2. *Experimental results*

We now provide all our comparative results, namely the performance metrics for standard datasets mini-ImageNet and CUB-200 in Tables 4 and 5, and performance curves for tiered-ImageNet and FGVC-Aircraft in Figure 4. As already mentioned, this variety of datasets allows us to benchmark the methods on standard image classification datasets (mini-ImageNet and tiered-ImageNet) and on fine-grained image classification datasets (FGVC-Aircraft and CUB-200). It also allows us to compare differences between
<figure><img src="image_5.png" /><figcaption>Figure 4: F1 comparison with the state of the art on FGVC-Aircraft and tiered-ImageNet</figcaption></figure>

smaller (100 classes in total for mini-ImageNet and FGVC-Aircraft) and
larger (200 classes in total for tiered-ImageNet and CUB-200) datasets.

*Comparison of AFA with state-of-the-art FSCIL approaches.* First, a global inspection of the results show that our proposed method AFA shows comparable results with respect to FSCIL-specific methods such as CEC [17] and FACT [36]. The F1 score of our method is higher on mini-ImageNet and close to the state-of-the art on CUB-200, tiered-ImageNet and FGVC-Aircraft. This confirms the ability of our proposed ensemble learning pipeline to tackle the FSCIL challenging scenario. An advantage of our method is its simple design, with interpretable and easy to learn modules.

*Comparison of AFA with standard average.* We now focus on the comparison between AFA with classic ensembling strategies, based on simple averaging using either arithmetic, geometric, or harmonic mean. We also display results obtained with a majority vote. As Tables 4 and 5 (bottom part), and Fig. 5 show, classic averaging methods show quite inconsistent results. The best type of average seems to depend on session number and dataset. Similarly, the majority vote scheme seems to produce worse results in early sessions, when few voters are available, but a gain in performance with an increasing number of voters. The AFA ensemble method we propose models and
<figure><img src="image_6.png" /><figcaption>Figure 5: Comparison of ensemble learning output fusion methods, on FSCIL datasets, in terms of averaged F1 score over all classes, for various sessions *k*. The closer the F1 score is to 1, the better. As sessions are incrementally added, the classification task becomes increasingly more difficult because of the additional categories to be handled.</figcaption></figure>
combines, by design, those different types of average. It learns their optimal balancing, thus producing significantly better performance in all sessions.

*Comparison of AFA with standard neural networks.* Performance of our proposed method were also compared against three different types of ensembling neural network, namely (1) a shallow neural network with similar number of parameters and layers as our AFA model, (2) a deeper neural network with five fully connected layers, and (3) a neural network specifically designed for ensembling including a weighted average layer followed by a fully connected layer for the output [38]. All neural network models, including the AFA model, were trained with the same process with only slight adjustments on learning rate parameters to adapt to each model architecture, for the sake of fair comparisons. The results, in terms of accuracy and F1 scores (the larger, the better), are summarized in Table 4 and Figure 5.

<figure><img src="image_7.png" /><figcaption>Figure 6: Loss versus epoch number for various NN-based ensemble learning methods, including our proposed AFA, on the last (10th) session of the CUB-200 setting.</figcaption></figure>

AFA outperforms those other NN approaches. The second best performing is the shallow NN, indicating that the limited number of parameters from both AFA and the shallow NN produces a decisive advantage when compared to the deep NN. The latter shows inconsistent results across different datasets. For example, it performs better in last session of mini-ImageNet than the weighted average NN (F1 score of 43.77% against 37.46%) but worse
in last session of CUB-200 (38.43% against 46.52%). Indeed, these later sessions drastically increase its architecture size due to the higher number *K* of models to ensemble and the higher number of classes *N*<sub>*K*</sub> to predict, while not providing a significantly larger training set because of the few-shot constraint, making the training of larger models difficult.

Figure 6 illustrates these behaviours. It displays the training losses of the different stacked NN models, including the proposed AFA, on the last (10th) session for the CUB-200 setting. The plots demonstrate our model stability and fast convergence to a lower loss value than other methods which exhibit a plateau at a higher value. The deep NN loss behaviour shows the inability to train a large number of parameters on such limited amount of data, with a validation loss diverging. The shallow NN, having a similar number of parameters as our AFA model, also converges but with a lower rate and to a higher loss value, as demonstrated by its F1 score which is 1.01 % lower than the AFA model. Finally, having only *k* parameters, the Weighted Average NN weights converge in a single epoch as shown by its loss that does not evolve along subsequent epochs.

## 6. Conclusion

In this work, we have proposed a novel approach for ensemble learning that we have applied to the FSCIL problem. The FSCIL setting requires to produce methods able to handle an increasing number of classes with the extra constraint that only a few number of training samples are available for novel classes. FSCIL state-of-the-art extensively adopts meta-learning strategies, similarly to the few-shot classification setting. We propose here a conceptually simpler approach based on the ensembling of a set of nearest-neighbour classifiers. While basic ensembling strategies do not perform well in a consistent manner, our innovative Aggregated *f*-averages (AFA) ensembling model reaches significantly better and more stable performance than all its ensemble learning competitors, while achieving results comparable to the most sophisticated state-of-the-art FSCIL models. AFA is a supervised neural network with a specific architecture, specific activation functions, and specific weight constraints that models and combines various types of averages. AFA has a reduced number of trainable weights and present the additional advantage of being interpretable. In future work, we intend to further illustrate the flexibility and efficiency of our AFA model on regression tasks.
References

[1] X. Tao, X. Hong, X. Chang, S. Dong, X. Wei, Y. Gong, Few-shot class-incremental learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, Washington, 16th June - 18th June 2020, pp. 12183-12192. doi: 10.1109/cvpr42600.2020.01220.

[2] R. Hou, H. Chang, M. Bingpeng, S. Shan, X. Chen, Cross attention network for few-shot classification, in: Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, 8th December - 14th December 2019, pp. 4003-4014. arXiv:1910.07677.

[3] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, J.-B. Huang, A closer look at few-shot classification, in: Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, Louisiana, 6th May - 9th May 2019.

[4] Y. Wang, W. Chao, K. Q. Weinberger, L. van der Maaten, Simpleshot: Revisiting nearest-neighbor classification for few-shot learning (2019). arXiv:1412.5068, doi:10.48550/arxiv.1911.04623.

[5] J. Kim, T. Kim, S. Kim, C. D. Yoo, Edge-labeling graph neural network for few-shot learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, California, 16th June - 20th June 2019, pp. 11-20. doi:10.1109/cvpr.2019.00010.

[6] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, T. M. Hospedales, Learning to compare: Relation network for few-shot learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, Utah, 18th June - 22nd June 2018, pp. 1199-1208. doi:10.1109/cvpr.2018.00131.

[7] S. T. Martin, M. Boudiaf, E. Chouzenoux, J.-C. Pesquet, I. B. Ayed, Towards practical few-shot query sets: Transductive minimum description length inference, in: Advances in Neural Information Processing Systems (NeurIPS), New Orleans, Louisiana, 28th November - 2nd December 2022, pp. 34677-34688. doi:10.5555/3600270.3602783.
[8] M. Boudiaf, I. Ziko, J. Rony, J. Dolz, P. Piantanida, I. Ben Ayed, Information maximization for few-shot learning, in: Advances in Neural Information Processing Systems (NeurIPS), Vol. 33, (online), 6th December - 12th December 2020, pp. 2445–2457. doi:10.5555/3495724.3495930.

[9] J. Snell, K. Swersky, R. Zemel, Prototypical networks for few-shot learning, in: Advances in Neural Information Processing systems (NeurIPS), Long Beach, California, 4th December - 9th December 2017, pp. 4080–4090. arXiv:1703.05175.

[10] C. Finn, P. Abbeel, S. Levine, Model-agnostic meta-learning for fast adaptation of deep networks, in: Proceedings of the International Conference on Machine Learning (ICML), Sydney, Australia, 6th -11th August 2017, pp. 1126–1135.

[11] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al., Matching networks for one shot learning, in: Advances in Neural Information Processing Systems (NeurIPS), Barcelona, Spain, 5th December - 10th December 2016, pp. 3637–3645. arXiv:1606.04080.

[12] K. Lee, S. Maji, A. Ravichandran, S. Soatto, Meta-learning with differentiable convex optimization, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, California, 16th June - 20th June 2019, pp. 10657–10665. doi:10.1109/cvpr.2019.01091.

[13] F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, K. Alahari, End-to-end incremental learning, in: Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8th September -14th September 2018, pp. 233–248. doi:10.1007/978-3-030-01258-8_15.

[14] S. Hou, X. Pan, C. C. Loy, Z. Wang, D. Lin, Learning a unified classifier incrementally via rebalancing, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, California, 16th June - 20th June 2019, pp. 831–839. doi:10.1109/cvpr.2019.00092.
[15] K. Chen, C.-G. Lee, Incremental few-shot learning via vector quantization in deep embedded space, in: Proceedings of the International Conference on Learning Representations (ICLR), (online), 26 April - 1st May 2020.

[16] K. Zhu, Y. Cao, W. Zhai, J. Cheng, Z.-J. Zha, Self-promoted prototype refinement for few-shot class-incremental learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (online), 19th June - 25th June 2021, pp. 6801–6810. doi: 10.1109/cvpr46437.2021.00673.

[17] C. Zhang, N. Song, G. Lin, Y. Zheng, P. Pan, Y. Xu, Few-shot incremental learning with continually evolved classifiers, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (online), 19th June - 25th June 2021, pp. 12455–12464. doi:10.1109/cvpr46437.2021.01227.

[18] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio, Graph attention networks, in: Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, Canada, 30th April - 3rd May 2018. doi:10.48550/arXiv.1710.10903.

[19] M. Hersche, G. Karunaratne, G. Cherubini, L. Benini, A. Sebastian, A. Rahimi, Constrained few-shot class-incremental learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, Louisiana, 19th June - 24th June 2022, pp. 9057–9067. doi:10.1109/cvpr52688.2022.00885.

[20] A. Kukleva, H. Kuehne, B. Schiele, Generalized and incremental few-shot learning by explicit learning and calibration without forgetting, in: Proceedings of the International Conference on Computer Vision (ICCV), Montréal, Canada, 11th October 2021 – 17th October 2021, pp. 9020–9029. doi:10.1109/iccv48922.2021.00889.

[21] O. Sagi, L. Rokach, Ensemble learning: A survey, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8 (4) (2018) e1249. doi:10.1002/widm.1249.

[22] L. Breiman, Bagging predictors, Machine Learning 24 (2) (1996) 123–140. doi:10.1007/bf00058655.
[23] Y. Freund, R. E. Schapire, A decision-theoretic generalization of online learning and an application to boosting, Journal of Computer and System Sciences 55 (1) (1997) 119–139. doi:10.1006/jcss.1997.1504.

[24] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, G. E. Hinton, Adaptive mixtures of local experts, Neural Computation 3 (1) (1991) 79–87. doi:10.1162/neco.1991.3.1.79.

[25] D. H. Wolpert, Stacked generalization, Neural Networks 5 (2) (1992) 241–259. doi:10.1016/s0893-6080(05)80023-1.

[26] A. Kolmogorov, Mathematics and Mechanics, Kluwer, 1930.

[27] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization (2014). arXiv:1412.6980.

[28] L. Condat, Fast projection onto the simplex and the $\ell_1$ ball, Mathematical Programming Series A 158 (1) (2016) 575–585.

[29] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, C. H. Lampert, Icarl: Incremental classifier and representation learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, 21st July - 26th July 2017, pp. 2001–2010. doi:10.1109/cvpr.2017.587.

[30] Wah, Branson, Welinder, Perona, Belongie, The caltech-ucsd birds-200-2011 dataset, Tech. Rep. CNS-TR-2010-001, California Institute of Technology (2011).

[31] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, R. S. Zemel, Meta-learning for semi-supervised few-shot classification, in: Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, Canada, 30th April - 3rd May 2018.

[32] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, A. Vedaldi, Fine-grained visual classification of aircraft, Tech. rep., Oxford University (2013). arXiv:1306.5151, doi:10.48550/arxiv.1306.5151.
[33] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei, ImageNet Large Scale Visual Recognition Challenge, International Journal of Computer Vision (IJCV) (2015) 211–252 doi:10.1007/s11263-015-0816-y.

[34] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, 26th June - 1st July 2016, pp. 770–778. doi:10.1109/cvpr.2016.90.

[35] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-performance deep learning library, in: Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, 8th December - 14th December 2019, pp. 8024–8035.

[36] D.-W. Zhou, F.-Y. Wang, H.-J. Ye, L. Ma, S. Pu, D.-C. Zhan, Forward compatible few-shot class-incremental learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, Louisiana, 19th June - 24th June 2022, pp. 9046–9056. doi:10.1109/cvpr52688.2022.00884.

[37] P. Branco, L. Torgo, R. P. Ribeiro, A survey of predictive modeling on imbalanced domains, ACM Computing Surveys (CSUR) 49 (2) (2016) 1–50. doi:10.1145/2907070.

[38] S. Ozechi, Ensembling neural network models with tensorflow, https://blog.paperspace.com/ensembling-neural-network-models/, accessed: 2023-09-05 (2023).
<figure><img src="image_8.png" /><figcaption>Figure 7:</figcaption></figure>