EEG Classification with Limited Data: A Deep
Clustering Approach

Mohsen Tabejamaat, Hoda Mohammadzade, Farhood Negin, Francois
Bremond

► To cite this version:

Mohsen Tabejamaat, Hoda Mohammadzade, Farhood Negin, Francois Bremond. EEG Classification with Limited Data: A Deep Clustering Approach. Pattern Recognition, 2025, Volume 157 (110934), 10.1016/j.patcog.2024.110934. hal-05458806

HAL Id: hal-05458806

https://hal.science/hal-05458806v1

Submitted on 14 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
EEG Classification with Limited Data: A Deep Clustering
Approach

Mohsen Tabejamaatª, Hoda Mohammadzadeᵇ, Farhood Neginª, Francois Bremondª

ªInria, France,

ᵇDepartment of Electrical Engineering, Sharif University of Technology, Iran,

**Abstract**

In this paper, we propose a novel training strategy designed to prevent deep neural
networks from overfitting when trained on a very limited number of samples. This
approach enables the use of very deep networks in applications like EEG classification,
where data collection is challenging, leading to significant overfitting issues. To address
this issue, we introduce a classification network that leverages a deep clustering operation
in its latent space. The network utilizes a set of dictionary elements as auxiliary inputs
and learns to guide each input sample toward one of the clusters, each derived from a
dictionary element. This process facilitates the grouping of samples from the same class
within the same region of the latent space, thereby reducing the risk of complex class
boundaries and overfitting. Additionally, we introduce two probabilistic models within
our clustering approach to mitigate outlier issues and address augmentation challenges
commonly encountered with EEG signals. Experimental evaluations on two widely
recognized databases, EEG-ImageNet and BCIIV2a, demonstrate the effectiveness and
superiority of our method compared to state-of-the-art approaches, achieving 97.9%
accuracy on EEG-ImageNet and 84.1% accuracy on BCIIV2a.

**Keywords:** Deep neural network, overfitting issue, Electroencephalography, data scarcity.

PACS: 0000, 1111

2000 MSC: 0000, 1111

<figure><img src="image_2.png" /></figure>
# 1. Introduction

Deep learning has made significant progress over the last decade, leading to breakthroughs in many applications, including image and video processing [1, 2]. Despite this progress, the exceptional performance of deep learning models is highly dependent on large training datasets, which are often not feasible for many applications, such as in medicine. However, even with a small dataset, there are cases where a very deep network is necessary to handle the significant nonlinearity in data distribution. Yet, training a highly complex network with only a few samples leads to overfitting, where the network performs well on training data but poorly on unseen samples.

EEG classification is one such application that faces challenges in data collection [3, 4]. As user-specific data, EEG signals present high nonlinearity, which necessitates the use of highly expressive deep neural networks for effective processing. This challenge is even more pronounced in applications like Visually Evoked EEG Classification (VEEC), where nonlinearity exists in both the temporal dynamics and spatial correlation of different channels [5, 6, 7]. However, due to the limited number of training samples and the resulting overfitting, deep neural models for this task often struggle to achieve satisfactory performance (e.g., ~48% accuracy [5]).

As a remedy, some studies propose to classify these samples within an evoke-specific framework [8], where the same stimuli are used for both training and testing. This approach significantly reduces data complexity, allowing the problem to be solved with a shallow network that has fewer parameters, thereby mitigating overfitting even with a limited number of training samples. However, the challenge persists for cross-stimuli tasks, where different stimuli are used for training and testing. In this work, we propose a model that alleviates the overfitting problem in deep neural networks. Specifically, we introduce a feature generator that operates in an adversarial game against a domain discriminator. We also introduce a dictionary as an auxiliary input to the generator, consisting of a fixed set of EEG samples, one sample per class. This setup includes multiple samples in each forward pass of the network, such that two variants of the same class—one as the input to the network and another as a dictionary element—are pulled together in the bottleneck layer of the generator while being pushed away from
the dictionary elements of other classes. This process encourages the network to extract
shared characteristics from these variants, preventing it from overfitting by memorizing
samples, and allowing for a deeper network that supports a more expressive mapping of
input samples.

To achieve an end-to-end implementation, we propose grouping the samples by
assigning them to the same cluster. In this setup, dictionary elements serve as the centers
of the clusters, and the assignment is supervised by the class labels.

To mitigate the impact of outliers<fnref n="1" />, we propose an image-guided clustering model, where two signals (the input and one of the dictionary elements) are encouraged to be in the same cluster only if their corresponding images are strongly similar. The less similar the images, the less encouragement for their corresponding signals to cluster together. Image similarity is calculated using deep representations.

We also propose performing this assignment based on the similarity between dic-
tionary elements and an augmented view of the input image. This view is randomly
generated at each training epoch, introducing a probabilistic element into the augmenta-
tion process. Additionally, this augmentation is applied to 2D images rather than 1D
EEG signals, a task that presents challenges in direct human interpretation of the results.

We experimentally demonstrate the overfitting challenges of existing networks, even when augmented with residual connections, leading to a significant gap between training and test errors. Our results confirm the benefits of using deeper networks for EEG classification, even with limited training samples. We also verify that other strategies, such as dropout and regularization, are not as effective when training a deep neural network on small datasets.

The main contributions of our method can be summarized as follows:

* A clustering-based strategy is introduced that significantly reduces overfitting in very deep neural networks, enabling their deployment in applications that typically suffer from a limited number of training samples (e.g., EEG classification).

* An image-guided clustering approach is introduced to address potential outliers

<footnote n="1">¹An outlier is an EEG signal whose stimulus image is not a good representative of its class.</footnote>
in EEG data.

* To address the difficulty of augmentation techniques for 1D EEG samples, we propose a new strategy that benefits from affine transformations of stimulus images to introduce 2D augmentation into the clustering process.

* Extensive experiments demonstrate the superiority of our method in two different tasks: visually evoked and motor imagery classification.

The remaining parts of this paper are organized as follows: In Section 2, we provide a brief overview of related work. Our main strategy is presented in Section 3. Experimental results are detailed in Section 4, and we conclude the paper in Section 5.

## 2. Related work

EEG has been extensively introduced in various applications, including those in computer science [9, 10, 11], psychology [12, 13, 14], and medicine [15, 16, 17]. In this review, we provide an overview of two EEG-based classification tasks: Visual Stimulus Classification (VSC) and Motor Imagery Classification (MIC).

### 2.1. Visually Evoked EEG Classification

EEGs have been used to recognize a variety of cognitive, motor, and visual stimuli. While cognitive and motor imagery tasks have been extensively studied [18, 19], the application of EEG for visual stimuli requires further exploration. Current Visual Stimulus Classification (VSC) solutions often rely on off-the-shelf classifiers that do not account for the specific characteristics of EEG signals, such as their highly nonlinear distribution and the subject-specific nature of their patterns. Moreover, the limited number of training samples is another significant challenge, which has not been well-studied in the literature.

#### 2.1.1. Classification of a Visual Stimulus

Early work relied on hand-crafted spatiotemporal representations of signals and primarily used kernel classifiers to detect non-linearity in samples [20, 21]. However, due to their limited number of parameters, these methods were restricted to simple stimuli
such as directional edges or basic shapes. To address this, Ref. [22] suggests using
hierarchical encoding of dynamic features through deep neural networks. This approach
benefits from the ability to include more parameters and leverage larger training datasets.
Recent research has focused on incorporating priors into the classification process to
manage the complexity of the task [5, 22, 23]. The priors are usually an image of the
stimuli utilized to guide the position of the signal in the latent space of a deep neural
network.

2.1.2. *Signal to image*

The aim of these methods is to visualize stimuli from corresponding EEG data. [24] demonstrated that it is possible to reconstruct facial images using EEG signals captured from the frontal lobe of the cortex. This work was extended by [25] to reconstruct freestyle images sampled from the ImageNet dataset [26]. However, these methods primarily involve classifying the input signal into one of the fixed classes and then using a Conditional Generative Adversarial Network (CGAN) [27] to generate a random sample from that class. This approach does not visualize the actual stimuli but rather produces random samples based on class labels. [23] addresses this issue by introducing a consistency loss between the generated sample and the original stimulus image. This loss encourages the network to regenerate stimuli more accurately rather than producing random class samples. [28] and [29] argue that EEG signals capture some priors about the class of the stimulus but not the stimulus itself. They propose using this information, along with the original image, to create a visual representation of our mental perception.

2.2. *Motor imagery recognition*

Motor imagery EEG recognition is one of the most important and challenging research topics in human-computer interaction. There are many different approaches that deal with this task. Among them, we review the most relevant studies that tackle the problem using a deep learning framework. Wu et al. [30] propose a multiscale convolutional network that benefits from local time convolutions. Similar to [30], Ingolfsson et al. [31] use a time convolutional layer but integrate it within a residual framework. Zhang et al. [32] propose a hybrid neural network consisting of spatial
convolutional operators along with a long short-term memory (LSTM) module [33]
to encode the temporal characteristics of signals. In a related approach, [34] propose
a hybrid network with convolutional filters and recurrent neural networks, enhanced
with an attention mechanism to emphasize the most discriminative signal parts. Ko et
al. [35] employ a sub-band attention mechanism to focus on the most discriminative
frequencies and an attentive LSTM to enhance temporal feature learning. Dai et al. [36]
use a graph sequence neural network to capture local and global relationships across
different channels of motor imagery signals.

Deng et al. [37] apply a regularization technique based on Sparse Group Lasso (SGL)
constraints to address overfitting in their convolutional network. Several strategies also
aim to reduce overfitting in general-purpose frameworks. Cogwell et al. [38] propose
minimizing cross-covariance in the hidden layers, but this can push classes with fewer
samples towards those with more data and is sensitive to noisy samples. Lu et al. propose
avoiding negative empirical risk in deep neural networks to combat overfitting, but this
approach suffers from similar drawbacks due to its relation to layer correlation. Rebuffi
et al. [39] suggest using a moving average of model parameters to reduce overfitting,
though this can introduce noise from outliers, necessitating a large number of training
samples to mitigate their influence. Zhang et al. [40] utilize an affinity-based clustering
algorithm to assign EEG features to subclasses, enhancing feature discriminativeness
within a multitask learning framework.

3. Our method

Let $\mathcal{M}$ represent an input space associated with a joint distribution over $K$ different domains, $\mathcal{X} = \{X^1, \dots, X^K\}$, where each domain $X^j$ consists of $V$ samples: $X^j = \{x_1^j, \dots, x_V^j\}$. Each sample $x_i^j \in \mathbb{R}^{m \times n}$ is an EEG signal, with $m$ indicating the number of channels and $n$ denoting the temporal duration of the signal. Since brain patterns are specific to each individual, each subject is treated as a distinct domain, resulting in $K$ domains in our dataset.

The main goal is to estimate the class label $y_i \in \mathbb{R}^C$ of each input sample $x_i^j$,
regardless of its domain-oriented features. There are $C$ different classes in our database,
each referring to a class of stimulus images (e.g. flower, golf court, etc) utilized for
collecting the signals. In fact, we aim to learn a generator $\phi(., \theta_{\phi})^2$ which is headed
by a classification network $C$, where $\theta_{\phi}$ are the parameters of our generator. The
generator maps the input sample into a domain-agnostic representation where we have
the same marginal distribution for all the samples in the same class $P_{\phi}(x_i^\alpha) = P_{\phi}(x_i^\beta)$,
$P_{x_i^\alpha} \sim X \neq P_{x_i^\beta} \sim X | \forall \alpha, \beta, \alpha \neq \beta^3$, and the classifier projects this representation to one of the $C$
class labels. $\alpha$ and $\beta$ represent different subjects. The removal of domain information
is encouraged by two extra modules; a dictionary-based clustering module $\mathcal{P}$, and a
domain discriminator $\mathcal{D}$.

<figure><img src="image_3.png" /><figcaption>Figure 1: (Please Zoom in to see the details. All the images will be rescaled to fit the camera-ready format of the Journal if our paper will be accepted.)(Left): flowchart of our method: Given an EEG signal $x_i$ and a set of dictionary elements $d_1, ..., d_C$, our method learns to cluster deep features extracted from the input signal before predicting its class label. We employ $\phi(., \theta_{\phi}) = S \circ G \circ F$, where $S, G, F$ respectively capture local temporal features, long temporal characteristics, and spatiotemporal features of the signal. This composite function $S \circ G \circ F$ constitutes the generator of our method, transforming the input signal into a domain-agnostic representation. $C$ and $\mathcal{D}$ serve as the classifier and domain discriminator, while $\mathcal{P}$ is integral to the clustering module. (Right): Comparative analysis depicting the efficacy of our proposed EEG classification method against competing algorithms.</figcaption></figure>

We randomly select a set of C training samples, one sample per class, as our

<footnote n="2">²For the sake of simplicity, hereafter we drop the domain index *j*.</footnote>

<footnote n="3">³Here, *P* denotes the marginal distribution of the data</footnote>
dictionary elements {$d_1, ..., d_C$}. These elements will be the same during all the training iterations and also at inference time. Then, the input sample and each of the dictionary elements are individually applied to the generator $\phi(., \theta_{\phi})$, providing a set of intermediate representations; $\phi(x_i^j, \theta_{\phi})$ and $\phi(d_p, \theta_{\phi})|_{p=1:C}$. Notably, without a shared generator, the network tends to ignore the dictionary elements and only includes the input sample in the classification process.

### 3.1. Feature generator

As time-series data, EEG needs to be analyzed in a spatiotemporal manner. To achieve this, we propose $\phi(., \theta_{\phi})$ to consist of three distinct parts: $\mathcal{F}(., \theta_{\mathcal{F}})$, $\mathcal{G}(., \theta_{\mathcal{G}})$, and $\mathcal{S}(., \theta_{\mathcal{S}})$. The first part, $\mathcal{F}(., \theta_{\mathcal{F}})$, extracts the micro-temporal properties of the signal using a set of convolutional layers with kernel sizes of less than 3. The second part, $\mathcal{G}(., \theta_{\mathcal{G}})$, encodes the long-term dependencies of these micro-temporal features using Gated Recurrent Units (GRUs), which efficiently capture long-term dependencies in the signals. The third part, $\mathcal{S}(., \theta_{\mathcal{S}})$, enhances the expressiveness of the domain-agnostic features through a set of fully connected layers.

To form a single vector representation, we use the output from the last hidden layer of the GRU as the input to $\mathcal{S}(., \theta_{\mathcal{S}})$. $\mathcal{S}$, which is implemented with fully connected layers, extracts the spatial dependencies between different EEG channels. The output of the final layer of $\mathcal{S}$ constitutes the feature representation generated by $\phi(., \theta_{\phi})$.

Our experiments confirm that applying an atomic feature representation before extracting micro features with $\mathcal{F}(., \theta_{\mathcal{F}})$ is beneficial. This preprocessing step is applied to both the input samples and the dictionary samples. Specifically, we first split each signal into non-overlapping blocks and compute the average value of each block to serve as its representative feature. This approach not only improves the feature extraction process but also significantly reduces the computational complexity of our method, as experimentally evaluated in Section 4.

### 3.2. Dictionary based clustering

Dictionary-based clustering is our first strategy to encourage $\phi(x_i, \theta_{\phi})$ towards being a domain agnostic feature representation while keeping the class label information of
the original sample $x_i$. We consider each projected sample of the dictionary $\phi(d_p|_{p=1}^C, \theta_\phi)$ to be a cluster center and then assign the projection of the input sample $\phi(x_i, \theta_\phi)$ to one of these clusters. The assignment is directly supervised by the class label of the input and the dictionary elements. For this purpose, we give the entire set of $\{\phi(x_i, \theta_\phi), \phi(d_p|_{p=1}^C, \theta_\phi)\}$ to the clustering module $\mathcal{P}$ (Fig. 1). To be performed in an end-to-end fashion, we propose to implement $\mathcal{P}$ as a classification sub-module on top of a subtraction operator. First, we subtract $\phi(x_i, \theta_\phi)$ from each $\phi(d_p, \theta_\phi)|_{p=1:C}$, resulting in a set of $C$ difference vectors. Then, we concatenate the vectors and input them to the classification head $\mathcal{P}_c(\phi(x_i, \theta_\phi)|-\phi(d_p, \theta_\phi)|_{p=1:C})$ which outputs a $C$-element vector. Each element of the vector indicates whether or not the input sample is assigned to that cluster. Obviously, $\mathcal{P}_c$ can be defined as a set of $C$ individual subnetworks, each to transfer one of these $C$ vectors to an "assignment decision", which is considered as one if the input sample and the corresponding dictionary element is from the same class and zero otherwise. But this way, the subnetworks have no connection to each other, leading them to only learn about the intra-cluster representations without applying any constraints between different clusters. Alternatively, we consider concatenating all the difference vectors together and projecting them together to the whole vector of $y_i$ just using a simple unified network $\mathcal{P}_c$. In this way, we can minimize the intra-cluster distances while keeping as much difference between different clusters. This allows for a discriminative representation of samples which further improves the classification performance of our method.

$$ \underset{\theta_{\mathcal{P}_c}}{\operatorname{argmin}} \mathbb{E}[y_i, \mathcal{P}_c(\mathcal{R}(\phi(x_i, \theta_\phi), C) - \operatorname{cat}(\phi(d_p, \theta_\phi)|_{p=1:C}))] \quad (1) $$

where $\mathbb{E}$ stands for the cross entropy loss and $\mathcal{R}(\phi(x_i, \theta_\phi), C)$ is a $C$-time repetition of $\phi(x_i, \theta_\phi)$.

The main advantage of this clustering approach is its ability to prevent the network from overfitting when there are insufficient training samples. By incorporating two samples (which may come from different subjects) of the same class into each training iteration, the network is encouraged to generate similar representations for these samples while differentiating them from samples of other classes. This method prevents the network from fitting too closely to any single input sample, making it possible to design
deeper networks. Such networks are necessary for addressing the extreme complexity of problems with scarce data while maintaining the required high expressiveness. In Section 4, we demonstrate the effectiveness of this strategy for increasing network depth. We observe that current methods typically restrict networks to shallow architectures to mitigate overfitting, whereas our approach allows for the use of much deeper architectures without encountering this issue.

### 3.2.1. Do we need a deep stack of layers when there is insufficient training data?

An increased number of hidden layers is not merely to accommodate more parameters for handling a large number of training samples. Sometimes, domain-specific distributions introduce extreme nonlinearity that can only be effectively managed with numerous nonlinear projections. This means that separating data into different classes requires a variety of data projections. By analyzing these projections from different viewpoints, we can classify the data based on the most discriminative features. In deep learning, these diverse projections are achieved through the hidden layers of networks. Therefore, increasing the number of hidden layers is beneficial, regardless of the number of training samples. However, limited training data can lead to overfitting, which must be addressed during the training process.

### 3.3. Label classifier

Our label classifier, $C(\cdot, \theta_C)$, estimates the class labels of input samples in an end-to-end fashion. This eliminates the need for a separate classifier at inference time. Additionally, it is part of our strategy to ensure that the network retains class label information in the intermediate representations of samples, $\phi(x_i, \theta_\phi)$.

$$ \underset{\theta_C}{\operatorname{argmin}} \mathbb{E}[y_i, C(\phi(x, \theta_\phi), \theta_C)] \quad (2) $$

Following common practice [5, 6, 22], we apply a softmax function to the last layer of our classifier. This converts the class scores into a probability distribution, rather than just providing raw class scores. Using softmax helps in parameterizing the label space, which enhances stability during training. This is because, with each training step, we leverage the entire partitioned space (resulting from the softmax re-scaling) rather than
relying on a single class score estimation. This ensures that the mapping of samples to their class labels considers inter-class dependencies.

### 3.4. Domain discriminator

We also utilize a domain discriminator to encourage the network to disregard domain-specific information in the samples. Our domain discriminator is a (K + 1)-way classifier, where K represents the number of domains (with each domain consisting of all signals collected from a specific subject). Initially, we train this discriminator to identify domain-specific features of each sample. This is accomplished using the following minimization objective:

$$ \underset{\theta_D}{\operatorname{argmin}} \mathbb{E}[\operatorname{cat}(z_i, 0), \mathcal{D}(\phi(x, \theta_\phi), \theta_D)] \quad (3) $$

where $cat(z_i, 0)$ denotes concatenating a zero to the domain index vector $z_i$. Next, we freeze the parameters of this module and train the generator to assign input samples to the (K + 1)th domain. This is formulated as the following minimization problem:

$$ \underset{\theta_\phi}{\operatorname{argmin}} \mathbb{E}[\operatorname{cat}(\bar{z}, 1), \mathcal{D}(\phi(x, \theta_\phi), \theta_D)] \quad (4) $$

where $\bar{z}$ is an all-zero vector with a length of K.

### 3.5. Overall optimization

In summary, we can optimize these objective functions using a min-max game framework, defined by the following loss functions.

$$ \begin{aligned} & \underset{\theta_D}{\operatorname{argmin}} \mathbb{E}[\operatorname{cat}(z_i, 0), \mathcal{D}(\phi(x, \theta_\phi), \theta_D)] \\ & \underset{\theta_\phi, \theta_C, \theta_P}{\operatorname{argmin}} \log(\mathbb{E}[\operatorname{cat}(\bar{z}, 1), \mathcal{D}(\phi(x, \theta_\phi), \theta_D)] \\ & \mathbb{E}[y_i, C(\phi(x, \theta_\phi), \theta_C)]. \\ & \mathbb{E}[y_i, \mathcal{P}_c(\mathcal{R}(\phi(x_i, \theta_\phi), C) - \operatorname{cat}(\phi(d_p, \theta_\phi)|_{p=1:C}))] \end{aligned} \quad (5) $$

where we train $\mathcal{D}(\cdot, \theta_D)$ to distinguish between domains of the generated samples $\phi(x_i, \theta_\phi)$, $i = 1, ..., \ell$, $\ell$ is the number of EEG signals in each batch of data. Then, we freeze all the parameters of the discriminator and train $\phi(\cdot, \theta_\phi)$, $C(\cdot, \theta_C)$, and $\mathcal{P}(\cdot, \theta_P)$ so
that the discriminator classifies all the samples in the K+1 domain. This allows for estimating a domain invariant representation of $\phi(x_i, \theta_\phi)$ that also preserves the class label information of the original sample $x_i$.

**Algorithm 1 Our clustering-based domain adaptation strategy**

**Inputs:** Training data $X = \{x_1, ..., x_{KV}\}$, number of training iterations, number of steps $\eta$, domain index vector of each training sample $z_i$, an all-zero vector $\bar{z}$, batch size $\ell$

1. Sample a batch of the training samples as the dictionary elements $d_1, ..., d_C$

**for** each training iteration **do**

**for** $\eta$ steps **do**

2. Sample a batch of EEG signals $x_1, ..., x_\ell$

3. Generate $\phi(x_i, \theta_\phi)$ for each signal of the batch

4. Update the discriminator by descending its gradient:

$$\nabla_{\theta_D} = \frac{1}{\ell} \sum_{i=1}^{\ell} \mathbb{E}[\mathrm{cat}(z_i, 0), \mathcal{D}(\phi(x_i, \theta_\phi), \theta_D)]$$

5. Freeze the parameters of the discriminator

6. Update the generator by descending its gradient:

$$\nabla_{\theta_\phi} = \frac{1}{\ell} \sum_{i=1}^{\ell} \mathbb{E}[\mathrm{cat}(\bar{z}, 1), \mathcal{D}(\phi(x_i, \theta_\phi), \theta_D)]$$

Update the classifier by descending its gradient:

$$\nabla_{\theta_C} = \frac{1}{\ell} \sum_{i=1}^{\ell} \mathbb{E}[y_i, C(\phi(x_i, \theta_\phi), \theta_C)]$$

Update the clustering head by descending its gradient:

$$\nabla_{\theta_P} = \frac{1}{\ell} \sum_{i=1}^{\ell} \mathbb{E}[y_i, \mathcal{P}_c(\mathcal{R}(\phi(x_i, \theta_\phi), C) - \mathrm{cat}(\phi(d_p, \theta_\phi)|_{p=1:C}))]$$

**end for**

**end for**

**Outputs:** $\theta_\phi, \theta_D, \theta_C, \theta_P$

### 3.6. Clustering or classification?

The critical question here is: "Why do we consider $\mathcal{P}$ as a clustering module when it essentially projects from the latent space to the class labels of samples?" The answer is twofold: (1) Projection of Distances, Not Representations: $\mathcal{P}$ is applied not directly to the latent representations of samples but to the distances between these representations
and a set of dictionary elements. Thus, what is being projected is the distance between
the latent representations and the dictionary elements, not the representations themselves.
This approach is similar to classical clustering, where the objective is to adjust cluster
centers to minimize their distance to the most similar samples. (2) Decision Index
Rather Than Class Labels: Although *P* projects latent distances to yᵢ, the elements of yᵢ
do not represent class labels. Instead, yᵢ indicates whether the input and the dictionary
elements belong to the same class. Thus, yᵢ functions as a decision index rather than a
class label. However, unlike the classical strategies, we modify both the cluster centers
and input samples by passing them through a feature generator φ(., θ<sub>φ</sub>). This allows for
generating more expressive clusters but requires to be supervised by the class label of
samples. What this module *P* learns is not the outputs but the modified representations
of φ(xᵢ, θ<sub>φ</sub>) and φ(dᵣ, θ<sub>φ</sub>) for p = 1 to C, which minimizes the distance between samples
rather than projecting them into another representation.

3.7. *Contrastive learning vs dictionary-based clustering*

Contrastive learning is a method for discriminatively projecting samples into a latent
space, performed in an end-to-end fashion [41, 42]. This approach enables unsupervised
data partitioning without relying on predefined clusters. However, these methods
often overlook class label information, which can lead to suboptimal data separation.
Additionally, they typically require a large number of training samples, which can be
challenging to obtain for many applications, including those involving EEG signals.
Another challenge is accurately estimating the temperature parameter, especially when
data comes from diverse domains. In contrast, we propose guiding the network with a
set of dictionary elements, one for each class, providing an explicit definition of clusters.
Furthermore, we suggest enhancing clustering by incorporating a classification task,
which improves optimization by leveraging clear boundary definitions.

As an additional benefit, our method eliminates the need for large batch sizes, which
contrasts with contrastive learning approaches that require many negative samples.
Furthermore, while optimal parameters such as batch size and temperature values are
well-defined for datasets like ImageNet, applying these parameters to different types of
data, such as EEG samples, is particularly complex.
### 3.7.1. Probabilistic clustering

The presence of outliers is a major issue in EEG signal processing. Recognizing these samples is challenging with straightforward comparisons due to their domain-specific features. Therefore, current methods [43] for sample refinement often fail to address this issue. This challenge is more significant when stimuli scenes contain numerous objects, as there is no certainty that the participant focus was directed toward the main object. As a result, the signals may be incorrectly labeled. Including these misclassified samples in the training process can lead to significant issues, as it encourages the network to associate their representations $\phi(x_i, \theta_\phi)$ with incorrect labels.

To address this issue, we propose leveraging the visual characteristics of the stimuli to impute the EEG representations into each cluster. We achieve this by calculating the visual similarity between the stimuli of the input sample and all dictionary elements. Subsequently, we replace the classification head $C(., \theta_C)$ with a regression head (see Fig. 2), using the visual similarity scores as the basis for assigning the input sample to each cluster. This approach provides a probabilistic assignment of samples, helping to prevent the network from mapping the sample to an irrelevant representation (e.g., an EEG signal evoked by an image of a "flower" captured in a "golf field" scene would be mapped to a "golf field" representation).

### 3.7.2. Domain augmentation

The effectiveness of data augmentation (DA) has been well established across many applications in pattern recognition and machine intelligence [44]. However, while DA techniques are well-defined for images, extending these methods to one-dimensional signals, such as EEG, especially when they are heavily contaminated by noise, remains unclear. Simple algebraic transformations can severely distort noisy signals.

To tackle this issue, we propose incorporating the augmentation process directly into the clustering phase rather than relying on data collection. The main concept is that visual transformations applied to images retain the essential features of the objects while introducing subtle changes to their semantic representations. These representations, derived from a deep layer of a pre-trained model, allow us to replace the deep image features of an input sample with those of its transformed counterpart. This
<figure><img src="image_4.png" /><figcaption>Figure 2: Please Zoom in to see all the details. All the images will be rescaled to fit the camera-ready format of the Journal if our paper will be accepted. (Left): Probabilistic and augmentation clustering schemes of our method. Here, latent clustering is carried out based on the visual similarity between the stimulus of the signal and the stimulus of each dictionary element. (Right): Difference between our clustering scheme and the contrastive (discriminative) learning algorithms</figcaption></figure>

approach offers two main advantages: first, it probabilistically shapes the inter-class distribution of samples, because of the random transformations applied during training; second, it enhances intra-cluster distributions with augmented samples, facilitating the parameterization of each cluster.

## 4. Experimental results

We evaluate our method on two well-known databases: EEGImageNet [5] and BCIIV2a [45].

EEGImageNet includes 11,926 visually evoked EEG signals with corresponding images (visual stimuli) collected from six subjects (one woman and five men). The dataset features 2,000 images from 40 different ImageNet classes, including parachute, elephant, dog, missile, and others. Each class contains 50 images, resulting in 300 EEG signals per subject. Samples are collected using a 128-channel actiCAP device while subjects view a monitor for 500 ms, with a 10-second interval between classes. Data is recorded at 1 kHz. For evaluation, we use the same split as [5], with 7,970 samples for
training and 1,997 for testing.

BCIIV2a is a motor imagery EEG database recorded from nine subjects during four tasks: imagining movement of the left hand, right hand, legs, and tongue. Signals are captured using a 22-electrode device at 250 Hz. Each session includes six runs of 48 trials, totaling 288 trials. To reduce artifacts, samples are filtered within 0.5 to 100 Hz. Following [34], we use the first session for training and the second session for testing.

## 4.1. *Implementation details*

Our method is implemented using TensorFlow. For atomic feature extraction (Section 3.1), we use a window size of 40 ms for the BCIIV2a database and 10 ms for EEGImageNet. For BCIIV2a, we report subject-specific performance by removing the domain discriminator but retaining the clustering module $\mathcal{P}$ to encourage distinctive sample representations.

For $\mathcal{F}(\cdot, \theta_F)$, we use four 1D convolutional layers with kernel sizes of 1, 2, 3, and 1, respectively. The first three layers are followed by ReLU activation, while the last layer uses a linear activation function. The recurrent network utilizes a one-way GRU [46] with Tanh activation. $S(\cdot, \theta_S)$ consists of six fully connected layers with 150, 500, 300, 200, 100, and 128 hidden units, respectively. The first five layers use ReLU activation, and the last layer uses Tanh. The second and third layers include Batch Normalization [47]. $C(\cdot, \theta_C)$ and $\mathcal{D}(\cdot, \theta_D)$ have similar structures to $S(\cdot, \theta_S)$. $C(\cdot, \theta_C)$ has six fully connected layers with 150, 500, 300, 200, 100, and 40 hidden units, with ReLU activation for the first five layers and Softmax for the final layer. $\mathcal{D}(\cdot, \theta_D)$ includes seven layers with 200, 100, 100, 200, 100, 50, and 7 hidden units, with ReLU for all layers except the last, which uses Softmax. The learning rates are set to 0.001 for $\phi(\cdot, \theta_\phi)$, $C(\cdot, \theta_C)$, and $\mathcal{P}(\cdot, \theta_\mathcal{P})$, and 0.005 for $\mathcal{D}(\cdot, \theta_D)$. We use the ADAM optimizer [48] with $\beta_1 = 0.5$ and $\beta_2 = 0.999$. The architecture is consistent across both datasets, with a batch size of 64 for EEGImageNet and BCIIV2a.

## 4.2. *Analysis*

In this section, we evaluate the performance of our method through a detailed ablation study using the EEGImageNet dataset. We consider four baseline structures:
(A1) A recurrent neural network (GRU) with a 2-layer MLP head. (A2) The feature generator $\phi(., \theta_{\phi})$ coupled with the classification module $C(., \theta_{C})$. (A3) The same as A2 but with the clustering module $\mathcal{P}$ instead of the classification module. (A4) A2 with the domain discriminator $\mathcal{D}$. For simplicity, we evaluate these configurations without the probabilistic clustering or augmentation techniques. All models are trained for 120 epochs. Table 1 shows the accuracy rates for these structures. Comparing A1 and A2, we find that increased depth can lead to performance degradation, likely due to overfitting. To further investigate, we trained A2 with half the learning rate of A1 but for approximately three times more iterations, confirming that the depth of A2 is the primary factor behind its reduced performance.

Compared to A2, A3 achieves 43.2% higher accuracy, demonstrating the effectiveness of clustering in reducing overfitting. Comparing A4 with A3 reveals that the domain discriminator does not provide the same benefits as the dictionary module, likely due to its difficulty in managing multiple domains. Overall, our full model outperforms the other strategies, highlighting the complementary benefits of combining dictionary-based clustering with a domain discriminator.

<table><caption>Table 1: Ablation study of our method on EEGImageNet</caption><thead><tr><td>Method</td><td>Rec. rate</td></tr></thead><tbody><tr><td>GRU+2l-MLP</td><td>57.3%</td></tr><tr><td>$\phi(., \theta_{\phi}) + C(., \theta_{C})$</td><td>47.2%</td></tr><tr><td>$\phi(., \theta_{\phi}) + \mathcal{P}(., \theta_{\mathcal{P}}) + C(., \theta_{C})$</td><td>90.4%</td></tr><tr><td>$\phi(., \theta_{\phi}) + \mathcal{D}(., \theta_{\mathcal{D}}) + C(., \theta_{C})$</td><td>76.9%</td></tr><tr><td>Full model</td><td>97.9%</td></tr></tbody></table>

### 4.3. Comparison with the state-of-the-art

In this section, we compare our method with state-of-the-art approaches on both the EEGImageNet and BCIIV2a databases. The training and test splits are consistent across all methods for each database. Since EEGImageNet and BCIIV2a were originally designed for different tasks, we use two distinct sets of competing approaches to evaluate our performance relative to the state-of-the-art for each specific task.

Our competing methods on the EEGImageNet database include ConTraNet [49],
CSANet [50], GATransformer [51], EEGWasserstein [52], EEGCorrNet [53], Neuro-Grasp [54], EEG-ChannelNet [5], SyncNet [55], EEGNet [56], and StackedLSTM [22].
The results are shown in Table 2$^4$.

<table><caption>Table 2: Performance comparison with the state-of-the-art on the EEGImageNet database. ADA stands for Adversarial Domain Adaptation, NLL indicates the number of nonlinear layers</caption><thead><tr><th>Method</th><th>Year</th><th>Methodology</th><th>Rec. rate</th><th>NLL</th></tr></thead><tbody><tr><td>ConTraNet [49]</td><td>2024</td><td>Attention</td><td>56.0%</td><td>12</td></tr><tr><td>CSANet [50]</td><td>2023</td><td>Attention+Sliding window</td><td>62.5%</td><td>13</td></tr><tr><td>GATransformer [51]</td><td>2023</td><td>Attention+Domain adaptation</td><td>64.3%</td><td>13</td></tr><tr><td>EEGWasserstein [52]</td><td>2023</td><td>Attention+ADA</td><td>75.3%</td><td>7</td></tr><tr><td>EEGCorrNet [53]</td><td>2023</td><td>Spatiotemporal Conv</td><td>46.1%</td><td>5</td></tr><tr><td>NeuroGrasp [54]</td><td>2021</td><td>LSTM</td><td>93.6%</td><td>11</td></tr><tr><td>EEG-ChannelNet [5]</td><td>2020</td><td>LSTM</td><td>48.1%</td><td>15</td></tr><tr><td>EEGNet [56]</td><td>2018</td><td>LSTM</td><td>31.9%</td><td>4</td></tr><tr><td>SyncNet [55]</td><td>2017</td><td>LSTM</td><td>31.7%</td><td>3</td></tr><tr><td>StackedLSTM [22]</td><td>2017</td><td>LSTM</td><td>21.9%</td><td>2</td></tr><tr><td>Our method</td><td>-</td><td>Clustering+ADA</td><td>97.9%</td><td></td></tr></tbody></table>

Among the competing strategies, our method is the only one that benefits from a deep clustering method prior to its classification head. As can be seen from Table 2, this leads to superior performance compared to other strategies. The second-best performance is achieved by NeuroGrasp [5], which is 4.3% lower than ours but 44.9% better than the average recognition rate of the other competitors. Unlike the shallow strategies like SyncNet, EEGNet, and StackedLSTM, NeuroGrasp distinguishes itself with its 11 layers, making it one of the deep strategies for multi-domain EEG classification. To prevent overfitting, NeuroGrasp combines all EEG channels into a single vector before applying its operations, significantly reducing the number of learnable parameters while maintaining network depth and nonlinearity.

EEGWasserstein is also effective with a 75.3% recognition rate. This is achieved using a domain adaptation strategy with Wasserstein GAN to address domain-specific patterns in EEG signals. However, since Wasserstein GAN lacks class label information, it struggles to retain class-specific details after domain adaptation. Similar to Neuro-

<footnote n="4">⁴The result of our method is reported as the average value of the five different sets that we randomly selected as the dictionary elements</footnote>
Grasp, EEGchannelNet, ConTraNet, CSANet, and GATransformer also benefit from deep architectures. However, unlike NeuroGrasp, these methods use a large number of parameters, making them prone to overfitting, which results in an average accuracy rate 40.6% lower than ours. In contrast, the low performance of EEGNet, SyncNet, and StackedLSTM is due to their shallow structures, which limit their expressiveness. The average accuracy rates of these networks are 19.6%, 30.2%, 36.7%, 38.5%, 49.5%, 65.1%, and 69.4% lower than those of EEGchannelNet, ConTraNet, CSANet, GATransformer, EEGWasserstein, NeuroGrasp, and our method. StackLSTM achieves the lowest performance among the shallow strategies, primarily due to the lack of spatial connections in its encoding process. The number of heads in multi-head attention modules for ConTraNet, CSANet, and GATransformer are 8, 4, and 5, respectively.

<table><caption>Table 3: Performance comparison with the state-of-the-art on the BCIIV2a database, NLL indicates the number of nonlinear layers</caption><thead><tr><th>Method</th><th>Year</th><th>Methodology</th><th>Rec. rate</th><th>NLL</th></tr></thead><tbody><tr><td>MultiViewNet [57]</td><td>2024</td><td>Transfer Learning</td><td>83.3%</td><td>8</td></tr><tr><td>NFEEG [58]</td><td>2024</td><td>Conv</td><td>81.0%</td><td>12</td></tr><tr><td>EEGCorrNet [53]</td><td>2023</td><td>Spatiotemporal Conv</td><td>83.3%</td><td>5</td></tr><tr><td>S&EAttNet [59]</td><td>2022</td><td>Attention</td><td>82.8%</td><td>5</td></tr><tr><td>AttIncNet [34]</td><td>2021</td><td>LSTM</td><td>82.8%</td><td>9</td></tr><tr><td>EEGTCNet [31]</td><td>2020</td><td>Residual Conv</td><td>77.3%</td><td>4</td></tr><tr><td>MSFBCNN [30]</td><td>2019</td><td>Conv</td><td>75.8%</td><td>3</td></tr><tr><td>HybridNet [32]</td><td>2019</td><td>LSTM</td><td>83.0%</td><td>8</td></tr><tr><td>EEGNet [56]</td><td>2018</td><td>LSTM</td><td>69.0%</td><td>4</td></tr><tr><td>Our method</td><td>-</td><td>Clustering+ADA</td><td>84.1%, std=9.1</td><td>17</td></tr></tbody></table>

We also compare our method with a set of state-of-the-art algorithms on the BCIIV2a database. The competing methods include MultiViewNet [57], EEGCorrNet [53], S&EAttNet [59], MSTNet [60], NFEEG [58], MSFBCNN [30], EEGTCNet [31], EEG-Net [56], HybridNet [32], and AttIncNet [34]. The results are shown in Table 3. As observed, increasing the depth of MultiViewNet, NFEEG, S&EAttNet, HybridNet, and AttIncNet results in respective performance improvements of 9.3%, 7%, 8.8%, 8.9%, and 8.7% compared to the average accuracy rates of shallower competitors (MSF-BCNN, EEGTCNet, and EEGNet). MSFBCNN, a fully convolutional network, utilizes a multiscale representation of samples but struggles with encoding long-term temporal dependencies. EEGTCNet, another fully convolutional network, employs dilated resid-
<figure><img src="image_5.png" /><figcaption>Figure 3: t-SNE visualization of the latent space $\phi(x_i, \theta_\phi)$ respectively using 2000 and 7970 training samples.</figcaption></figure>

ual convolutions for temporal encoding but also fails to capture long-term dependencies, leading to lower performance compared to HybridNet and AttIncNet. EEGNet, which incorporates LSTM layers for long-term dependency encoding, does not outperform EEGTCNet due to its limited parameter set. MultiViewNet, based on convolutional blocks, suffers similarly from inadequate long-term feature dependency encoding. Conversely, S&EAttNet, despite its shallower depth, benefits from its attention mechanism to better handle long-term dependencies, outperforming other shallow networks. In contrast, our method, with its deeper architecture and effective long-term dependency encoding, achieves superior performance compared to all the competitors.
### 4.4. Probabilistic vs augmented clustering

In this section, we evaluate the performance of our method with probabilistic and augmented clustering approaches, as described in Sections 3.7.1 and 3.7.2, respectively. For the probabilistic model, we first pass the input image and dictionary elements through a pre-trained VGG19 model. The activation map from the second-to-last layer of this model serves as the representation for each image, which is used as the element of $q$ in Fig. 2. For the augmented clustering, we create a synthetic view of the input image through affine transformations, guided by two random matrices with five guiding points on each image. The implementation details follow those in [61].

Table 4 presents the results of these methods on EEGImageNet. Both strategies positively impact the baseline architecture of our method. However, due to the higher dimensions of the images compared to EEG signals (150,528 vs. 64,000), both strategies significantly increase computational cost. The table also reports the forward pass time for each training epoch, demonstrating that while probabilistic clustering enhances performance, it imposes a substantial overhead in training time.

<table><caption>Table 4: Probabilistic vs augmented clustering on the EEGImageNet database</caption><thead><tr><td>Method</td><td>Rec. rate</td><td>Time</td></tr></thead><tbody><tr><td>Baseline</td><td>97.9%</td><td>43ms</td></tr><tr><td>Baseline+prob. clust.</td><td>98.1%</td><td>149ms</td></tr><tr><td>Baseline+prob. clust.+aug.</td><td>98.6%</td><td>436ms</td></tr></tbody></table>

### 4.5. Outliers

We also conduct an experiment to evaluate the effectiveness of our probabilistic clustering approach in the presence of noisy labels. Specifically, we introduce 5%, 10%, and 15% symmetric noise to the labels of the training samples to compare the performance of our probabilistic strategy with the baseline version of our method. The new labels are generated by randomly replacing the original labels with labels from other classes. The experimental setup follows the procedures outlined in Section 4.4. Table 5 presents the results of experiments using the probabilistic model with and without the augmentation strategy. The results indicate that probabilistic modeling not only improves the performance of our network but also helps mitigate the adverse effects of noisy labels.
<table><caption>Table 5: Probabilistic vs augmented clustering on the EEGImageNet database with 5%, 10%, and 15% of noisy labels</caption><thead><tr><th>Method</th><th>5%</th><th>10%</th><th>15%</th></tr></thead><tbody><tr><td>Baseline</td><td>87.5%</td><td>82.7%</td><td>78.3%</td></tr><tr><td>Baseline+prob. clust.</td><td>90.4%</td><td>84.5%</td><td>78.9%</td></tr><tr><td>Baseline+prob. clust.+aug.</td><td>91.3%</td><td>84.6%</td><td>79.1%</td></tr></tbody></table>

### 4.6. Data scarcity

In this section, we evaluate the effectiveness of our method in addressing the challenge of limited training data. Current strategies to reduce overfitting in deep neural networks include data augmentation [62], regularization [63], dropout [64], and residual connections [65]. However, data augmentation is challenging to apply to time-series data [66], and dropout and regularization have been shown to be less effective for very deep structures<fnref n="5" /> [65]. To further investigate, we assess the impact of residual connections on the performance of our method. We use a baseline model consisting of our generator $\phi(., \theta_{\phi})$ followed by the classification module $C(., \theta_{C})$. We then replace all nonlinear layers with residual connections. The baseline model achieves 47.2% accuracy for EEG classification on EEGImageNet. Our results indicate that while residual connections improve performance by 12%, our proposed model achieves a 97.9% accuracy rate, which is 50.7% higher than the baseline model. We also evaluate the impact of reducing the number of training samples on the performance of our method compared to state-of-the-art approaches. For this evaluation, we use the same split as described in Section 4.1, but with the first $k$ training samples. This approach avoids random sampling and allows for reproducible evaluations. Table 6 presents the results for training on 2000, 4000, and 7970 samples. As shown, our method demonstrates increasing superiority as the amount of training data decreases. Insufficient data often causes a simple network to memorize the data rather than learning a shared representation of similar examples. In contrast, our method promotes the extraction of this shared representation even when the number of trainable weights significantly exceeds the amount of available data. This

<footnote n="5">⁵Our experiment confirms that adding dropout (with a probability of 0.5) after the activation function of every other layer in the classification head of A2 (with batch normalization included) only increases performance by 1.4%</footnote>
advantage eliminates the need for a time-consuming grid search to find the optimal network structure for a given dataset.

<table><caption>Table 6: How a reduced training data affects the performance of different methods on EEGImageNet</caption><thead><tr><th>Method</th><th>2000</th><th>4000</th><th>7970</th></tr></thead><tbody><tr><td>Our method</td><td>77.7%</td><td>82.6%</td><td>97.9%</td></tr><tr><td>NeuroGrasp</td><td>50.6%</td><td>76.2%</td><td>93.6%</td></tr></tbody></table>

### 4.7. Understanding of hidden space

To better understand the discriminative ability of our method, we visualize the output of its feature generator, $\phi(., \theta_{\phi})$, using t-SNE. This technique minimizes the KL divergence between data points in a low-dimensional space [67]. As shown in Fig. 3, at the beginning of training, the data forms six distinct clusters, reflecting the specific nature of the data related to the six subjects in our database. As training progresses, the data is redistributed into 40 clusters, corresponding to the 40 different classes in our dataset. This demonstrates that our method effectively generates a domain-agnostic representation while preserving class-specific information.

### 4.8. Effectiveness of atomic feature extraction

Atomic feature extraction is a preprocessing step applied to each sample in our database. In this section, we experimentally evaluate the impact of window size on the average pooling operation. Table 7 shows the accuracy rates of our method with various window sizes on the EEGImageNet dataset. As demonstrated, the window size typically has a positive effect on performance. The highest accuracy is achieved with a window size of 10. However, reducing the window size further makes the resulting blocks more susceptible to noise, leading to a significant decline in performance.

<table><caption>Table 7: Performance evaluation for the window size of the atomic feature extraction on EEGImageNet</caption><thead><tr><th>Window size</th><th>No wind</th><th>4</th><th>7</th><th>10</th><th>13</th><th>17</th></tr></thead><tbody><tr><td>Accuracy rate (%)</td><td>87.3</td><td>89.1</td><td>94.6</td><td>97.9</td><td>96.3</td><td>88.4</td></tr><tr><td>Computational time (ms)</td><td>101</td><td>87</td><td>49</td><td>30</td><td>22</td><td>17</td></tr></tbody></table>
4.9. Deep K-means vs our method

Deep K-means [68] is a deep neural network that integrates the K-means algorithm to create discriminative representations of samples within a hidden layer. The training process alternates between clustering the deep representations and updating the network weights based on a downstream task. This approach results in an unsupervised algorithm that clusters samples using K-means algebraic operations. In contrast, our method is a supervised domain generalization technique that leverages clustering to produce domain-agnostic representations of samples. Our end-to-end strategy simultaneously modifies both the input representations and the cluster centers. In K-means, however, only the cluster centers are updated, leaving the input samples unchanged. This limitation can constrain the expressiveness of the clusters.

The deep K-means algorithm requires alternating between two distinct optimization tasks, which can cause convergence issues and slow down the training process. This problem becomes more critical when the data originates from multiple domains, each with its own distribution. In such cases, the algorithm is prone to getting stuck in a local minimum, necessitating extensive iterative corrections through the second optimization task. These corrections are time-consuming and impose a significant burden on the overall cost of the network. Moreover, the deep K-means algorithm does not offer a mechanism for generating domain-agnostic representations of samples. As a result, while it can be useful for enhancing the performance of downstream tasks, it does not facilitate the creation of semantically meaningful clusters.

4.10. Hard Triplet loss vs our method

In this section, we evaluate the effectiveness of our deep clustering strategy compared to the hard triplet loss function. Triplet loss aims to quantify similarity by minimizing the distance between samples of the same class (positive sample) and maximizing the distance between samples of different classes (negative sample). Each data point is paired with a positive and a negative sample, creating a triplet. Triplet loss then adjusts the embeddings to reflect these relationships. In the hard triplet loss approach, triplets are chosen to emphasize extreme similarities and dissimilarities, leading to faster convergence compared to vanilla triplet loss. However, selecting these hard triplets
involves computing distances between all pairs in a batch, which is computationally
expensive. This approach operates locally within each batch and does not establish
connections between all samples of the same class, limiting its scope.

In contrast, our method employs a global strategy to optimize cluster center locations
across the entire dataset. It uses a set of dictionary elements to mitigate overfitting by
mapping different inputs to the latent space and aligning them with dictionary elements
of the same class. As iterations progress, these dictionary elements are adjusted to
new locations based on previous loss values, resulting in updated cluster centers. Hard
triplet loss, focusing on extreme examples, can even worsen overfitting issues rather
than mitigating them. Additionally, it relies on a hyperparameter to define the margin
between similarities and dissimilarities, whereas our method does not require such
parameter tuning. Table 8 shows that our clustering method significantly outperforms
the triplet loss approach, highlighting its effectiveness in addressing overfitting issues.

<table><caption>Table 8: Performance evaluation our method using a clustering module vs a triplet loss function</caption><thead><tr><th></th><th>Our method + clustering</th><th>Our method+ triplet loss</th></tr></thead><tbody><tr><th>Accuracy rate (%)</th><td>97.9</td><td>89.7</td></tr></tbody></table>

### 4.11. Why our clustering strategy leads to enhanced classification performance

Our method is a latent clustering strategy that learns to regularize the representations
of samples in the latent space of a classification model. This regularization prevents the
network from overfitting on individual training samples, encouraging it to learn invariant
representations of samples that are relevant across all samples with the same class
label. By leveraging these invariant features, our method enhances the generalization
performance of the classification task, particularly in scenarios where training data is
limited and samples with the same class label originate from different domains.

To illustrate this concept, we consider a scenario with 18 points from three different
classes distributed across three domains (as depicted in Figure 4a). We design a classifier
network composed of two parts: the first part projects a 2D point to a latent space of
size 2 × 1, consisting of four fully connected layers with 50 hidden units each. The
second part includes a block of three fully connected layers, with 30 units per layer,
which predicts the class label of the input sample. In contrast, our proposed strategy
employs a clustering network to perform deep clustering in the latent layer of the classification network. We randomly select three points, one from each class, to serve as the dictionary elements, representing the cluster centers of our method. Figure 4c illustrates the distribution of samples in the latent layer of the classification network after 200,000 iterations, demonstrating how our clustering strategy encourages the network to group samples together, thereby mitigating the risk of overfitting during inference.

<figure><img src="image_6.png" /><figcaption>Figure 4: A toy example demonstrating the effectiveness of our deep clustering strategy in mitigating the overfitting issue in a classification network. (a) The original locations of samples in a 2D plane. (b) The locations of samples after projection to the latent space of the classification network without any clustering operation. (c) The locations of samples after projection to the latent space of the classification network when a clustering head is applied to the latent layer. Some samples of the same class are closely mapped to nearby locations in the latent space, which is why their differences are hard to discern.</figcaption></figure>

### 4.12. Limitations and future directions

Despite the promising results, our method is inherently limited to supervised scenarios and cannot be directly applied to unsupervised domain adaptation tasks. In future studies, we aim to extend our strategy to semi-supervised settings, where prior knowledge of the signal domain is limited. Moreover, the performance of our method is closely related to the quality of the clustering results and is highly dependent on the depth of the clustering network $P$. An inappropriate network depth may lead to inaccurate clusters, adversely affecting the performance of the subsequent classification task. To address this limitation, we propose exploring adaptive depth learning techniques. This approach dynamically adjusts the depth of our network during training, potentially mitigating the negative impact of depth settings on the performance of our strategy, especially on databases not covered in our study.
## 5. Conclusion

We have proposed a general framework for learning a domain-agnostic representation of samples. Our approach is a clustering strategy applied to the hidden layer of a deep neural network. This clustering method is end-to-end and learns to map latent codes toward a standardized representation of their class. Our method incorporates a domain discriminator, which further ensures that the network generates a domain-agnostic representation of the samples. Additionally, we use a classification head to preserve class information while encouraging the network to remove domain-specific information. We have also introduced two probabilistic models within our clustering approach to mitigate the effects of outlier samples and data scarcity. We demonstrated the effectiveness of our method in classifying visually evoked and motor imagery signals.

## References

[1] S. T. Krishna, H. K. Kalluri, Deep learning and transfer learning approaches for image classification, International Journal of Recent Technology and Engineering (IJRTE) 7 (5S4) (2019) 427–432.

[2] A. Rehman, S. B. Belhaouari, Deep learning for video classification: A review (2021).

[3] W. Huang, G. Yan, W. Chang, Y. Zhang, Y. Yuan, Eeg-based classification combining bayesian convolutional neural networks with recurrence plot for motor movement/imagery, Pattern Recognition 144 (2023) 109838.

[4] B. H. Kim, J. W. Choi, H. Lee, S. Jo, A discriminative spd feature learning approach on riemannian manifolds for eeg classification, Pattern Recognition 143 (2023) 109751.

[5] S. Palazzo, C. Spampinato, I. Kavasidis, D. Giordano, J. Schmidt, M. Shah, Decoding brain representations by multimodal learning of neural activity and visual features, IEEE Transactions on Pattern Analysis and Machine Intelligence 43 (11) (2020) 3833–3849.
[6] N. Cudlenco, N. Popescu, M. Leordeanu, Reading into the mind's eye: Boosting automatic visual recognition with eeg signals, Neurocomputing 386 (2020) 281-292.

[7] J. Kalafatovich, M. Lee, S.-W. Lee, Decoding visual recognition of objects from eeg signals based on attention-driven convolutional neural network, in: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, 2020, pp. 2985-2990.

[8] Z. Gao, X. Sun, M. Liu, W. Dang, C. Ma, G. Chen, Attention-based parallel multi-scale convolutional neural network for visual evoked potentials eeg classification, IEEE Journal of Biomedical and Health Informatics 25 (8) (2021) 2887-2894.

[9] Y. Guo, M. Wang, T. Zheng, Y. Li, P. Wang, X. Qin, Nao robot limb control method based on motor imagery eeg, in: 2020 International Symposium on Computer, Consumer and Control (IS3C), IEEE, 2020, pp. 521-524.

[10] L. Shao, L. Zhang, A. N. Belkacem, Y. Zhang, X. Chen, J. Li, H. Liu, Eeg-controlled wall-crawling cleaning robot using ssvep-based brain-computer interface, Journal of healthcare engineering 2020 (2020).

[11] M. Attia, I. Hettiarachchi, M. Hossny, S. Nahavandi, A time domain classification of steady-state visual evoked potentials using deep recurrent-convolutional neural networks, in: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), IEEE, 2018, pp. 766-769.

[12] M. Vázquez-Marrufo, R. Caballero-Díaz, R. Martín-Clemente, A. Galvao-Carmona, J. J. González-Rosa, Individual test-retest reliability of evoked and induced alpha activity in human eeg data, Plos one 15 (9) (2020) e0239612.

[13] S. Sheykhivand, Z. Mousavi, T. Y. Rezaii, A. Farzamnia, Recognizing emotions evoked by music using cnn-lstm networks on eeg signals, IEEE Access 8 (2020) 139332-139345.
[14] S. S. Shergill, V. Santoro, L. Rocchi, M. Di Hou, I. Premoli, Tms-eeg indexes abnormal gabaergic signalling in patients with schizophrenia, BJPsych Open 7 (S1) (2021) S52-S52.

[15] I. Lazarou, S. Nikolopoulos, P. C. Petrantonakis, I. Kompatsiaris, M. Tsolaki, Eeg-based brain-computer interfaces for communication and rehabilitation of people with motor impairment: A novel approach of the 21st century, Frontiers in human neuroscience 12 (2018) 14.

[16] A. Arroyo Ferrer, D. De Noreña, J. I. Serrano, M. Ríos Lago, J. P. Romero Muñoz, et al., Cognitive rehabilitation in a case of traumatic brain injury using eeg-based neurofeedback in comparison to conventional methods. (2021).

[17] I. Sultan, J. A. Brown, D. Serna-Gallegos, P. D. Thirumala, J. R. Balzer, S. Paras, C. Fleseriu, D. J. Crammond, K. M. Anetakis, A. Kilic, et al., Intraoperative neurophysiologic monitoring during aortic arch surgery, The Journal of thoracic and cardiovascular surgery (2021).

[18] A. Al-Saegh, S. A. Dawwd, J. M. Abdul-Jabbar, Deep learning for motor imagery eeg-based classification: A review, Biomedical Signal Processing and Control 63 (2021) 102172.

[19] Y. Zhou, S. Huang, Z. Xu, P. Wang, X. Wu, D. Zhang, Cognitive workload recognition using eeg signals and machine learning: A review, IEEE Transactions on Cognitive and Developmental Systems (2021).

[20] B. C. Hansen, T. Jacques, A. P. Johnson, D. Ellemberg, From spatial frequency contrast to edge preponderance: the differential modulation of early visual evoked potentials by natural scene stimuli, Visual neuroscience 28 (3) (2011) 221-237.

[21] D. Aminaka, S. Makino, T. M. Rutkowski, Svm classification study of code-modulated visual evoked potentials, in: 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIDA), IEEE, 2015, pp. 1065-1070.
[22] C. Spampinato, S. Palazzo, I. Kavasidis, D. Giordano, N. Souly, M. Shah, Deep learning human mind for automated visual classification, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6809–6817.

[23] Z. Jiao, H. You, F. Yang, X. Li, H. Zhang, D. Shen, Decoding eeg by visual-guided deep neural networks., in: IJCAI, 2019, pp. 1387–1393.

[24] D. Nemrodov, M. Niemeier, A. Patel, A. Nestor, The dynamics of facial identity processing: an eeg-based image reconstruction study, Journal of Vision 17 (10) (2017) 1262–1262.

[25] S. Palazzo, C. Spampinato, I. Kavasidis, D. Giordano, M. Shah, Generative adversarial networks conditioned by brain signals, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 3410–3418.

[26] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.

[27] M. Mirza, S. Osindero, Conditional generative adversarial nets, arXiv preprint arXiv:1411.1784 (2014).

[28] A. Fares, S.-h. Zhong, J. Jiang, Brain-media: A dual conditioned and lateralization supported gan (dcls-gan) towards visualization of image-evoked brain activities, in: Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 1764–1772.

[29] J. Jiang, A. Fares, S.-H. Zhong, A brain-media deep framework towards seeing imaginations inside brains, IEEE Transactions on Multimedia 23 (2020) 1454–1465.

[30] H. Wu, Y. Niu, F. Li, Y. Li, B. Fu, G. Shi, M. Dong, A parallel multiscale filter bank convolutional neural networks for motor imagery eeg classification, Frontiers in neuroscience 13 (2019) 1275.
[31] T. M. Ingolfsson, M. Hersche, X. Wang, N. Kobayashi, L. Cavigelli, L. Benini, Eeg-tcnet: An accurate temporal convolutional network for embedded motor-imagery brain-machine interfaces, in: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, 2020, pp. 2958-2965.

[32] R. Zhang, Q. Zong, L. Dou, X. Zhao, A novel hybrid deep learning scheme for four-class motor imagery classification, Journal of neural engineering 16 (6) (2019) 066004.

[33] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (8) (1997) 1735-1780.

[34] S. U. Amin, H. Altaheri, G. Muhammad, M. Alsulaiman, A. Wadood, Attention-inception and long short-term memory-based electroencephalography classification for motor imagery tasks in rehabilitation, IEEE Transactions on Industrial Informatics (2021).

[35] D.-H. Ko, D.-H. Shin, T.-E. Kam, Attention-based spatio-temporal-spectral feature learning for subject-specific eeg classification, in: 2021 9th International Winter Conference on Brain-Computer Interface (BCI), IEEE, 2021, pp. 1-4.

[36] S. Cai, H. Li, Q. Wu, J. Liu, Y. Zhang, Motor imagery decoding in the presence of distraction using graph sequence neural networks, IEEE Transactions on Neural Systems and Rehabilitation Engineering 30 (2022) 1716-1726.

[37] X. Deng, B. Zhang, N. Yu, K. Liu, K. Sun, Advanced tsgl-eegnet for motor imagery eeg-based brain-computer interfaces, IEEE Access 9 (2021) 25118-25130.

[38] M. Cogswell, F. Ahmed, R. Girshick, L. Zitnick, D. Batra, Reducing overfitting in deep networks by decorrelating representations, arXiv preprint arXiv:1511.06068 (2015).

[39] S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, T. Mann, Fixing data augmentation to improve adversarial robustness, arXiv preprint arXiv:2103.01946 (2021).
[40] Y. Zhang, T. Zhou, W. Wu, H. Xie, H. Zhu, G. Zhou, A. Cichocki, Improving eeg decoding via clustering-based multitask feature learning, IEEE Transactions on Neural Networks and Learning Systems (2021).

[41] T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A simple framework for contrastive learning of visual representations, in: International conference on machine learning, PMLR, 2020, pp. 1597-1607.

[42] K. He, H. Fan, Y. Wu, S. Xie, R. Girshick, Momentum contrast for unsupervised visual representation learning, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729-9738.

[43] M. Á. Luján, M. V. Jimeno, J. Mateo Sotos, J. J. Ricarte, A. L. Borja, A survey on eeg signal processing techniques and machine learning: Applications to the neurofeedback of autobiographical memory deficits in schizophrenia, Electronics 10 (23) (2021) 3037.

[44] G. Wilson, D. J. Cook, A survey of unsupervised deep domain adaptation, ACM Transactions on Intelligent Systems and Technology (TIST) 11 (5) (2020) 1-46.

[45] B. Blankertz, K.-R. Muller, D. J. Krusienski, G. Schalk, J. R. Wolpaw, A. Schlogl, G. Pfurtscheller, J. R. Millan, M. Schroder, N. Birbaumer, The bci competition iii: Validating alternative approaches to actual bci problems, IEEE transactions on neural systems and rehabilitation engineering 14 (2) (2006) 153-159.

[46] J. Chung, C. Gulcehre, K. Cho, Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint arXiv:1412.3555 (2014).

[47] S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: International conference on machine learning, PMLR, 2015, pp. 448-456.

[48] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
[49] O. Ali, M. Saif-ur Rehman, T. Glasmachers, I. Iossifidis, C. Klaes, Contranet: A hybrid network for improving the classification of eeg and emg signals with limited training data, Computers in Biology and Medicine 168 (2024) 107649.

[50] Y. Huang, J. Zheng, B. Xu, X. Li, Y. Liu, Z. Wang, H. Feng, S. Cao, An improved model using convolutional sliding window-attention network for motor imagery eeg classification, Frontiers in Neuroscience 17 (2023) 1204385.

[51] Y. Song, Q. Zheng, Q. Wang, X. Gao, P.-A. Heng, Global adaptive transformer for cross-subject enhanced eeg classification, IEEE Transactions on Neural Systems and Rehabilitation Engineering (2023).

[52] Q. She, T. Chen, F. Fang, J. Zhang, Y. Gao, Y. Zhang, Improved domain adaptation network based on wasserstein distance for motor imagery eeg classification, IEEE Transactions on Neural Systems and Rehabilitation Engineering 31 (2023) 1137-1148.

[53] W.-Y. Hsu, Y.-W. Cheng, Eeg-channel-temporal-spectral-attention correlation for motor imagery eeg classification, IEEE Transactions on Neural Systems and Rehabilitation Engineering 31 (2023) 1659-1669.

[54] J.-H. Cho, J.-H. Jeong, S.-W. Lee, Neurograsp: Real-time eeg classification of high-level motor imagery tasks using a dual-stage deep learning framework, IEEE Transactions on Cybernetics (2021).

[55] Y. Li, K. Dzirasa, L. Carin, D. E. Carlson, et al., Targeting eeg/lfp synchrony with neural nets, Advances in Neural Information Processing Systems 30 (2017).

[56] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, B. J. Lance, Eegnet: a compact convolutional neural network for eeg-based brain-computer interfaces, Journal of neural engineering 15 (5) (2018) 056013.

[57] W. Cui, Y. Xiang, Y. Wang, T. Yu, X.-F. Liao, B. Hu, Y. Li, Deep multiview module adaption transfer network for subject-specific eeg recognition, IEEE Transactions on Neural Networks and Learning Systems (2024).
[58] E. Arı, E. Taçgın, Nf-eeg: A generalized cnn model for multi class eeg motor imagery classification without signal preprocessing for brain computer interfaces, Biomedical Signal Processing and Control 92 (2024) 106081.

[59] G. A. Altuwaijri, G. Muhammad, H. Altaheri, M. Alsulaiman, A multi-branch convolutional neural network with squeeze-and-excitation attention blocks for eeg-based motor imagery signals classification, Diagnostics 12 (4) (2022) 995.

[60] Y. Zhang, P. Li, L. Cheng, M. Li, H. Li, Attention-based multiscale spatial-temporal convolutional network for motor imagery eeg decoding, IEEE Transactions on Consumer Electronics (2023).

[61] A. Siarohin, S. Lathuilière, S. Tulyakov, E. Ricci, N. Sebe, Animating arbitrary objects via deep motion transfer, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2377–2386.

[62] C. Shorten, T. M. Khoshgoftaar, A survey on image data augmentation for deep learning, Journal of big data 6 (1) (2019) 1–48.

[63] E. Bisong, Regularization for deep learning, in: Building Machine Learning and Deep Learning Models on Google Cloud Platform, Springer, 2019, pp. 415–421.

[64] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research 15 (1) (2014) 1929–1958.

[65] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.

[66] Q. Wen, L. Sun, F. Yang, X. Song, J. Gao, X. Wang, H. Xu, Time series data augmentation for deep learning: A survey, arXiv preprint arXiv:2002.12478 (2020).

[67] L. Van der Maaten, G. Hinton, Visualizing data using t-sne., Journal of machine learning research 9 (11) (2008).
[68] M. Caron, P. Bojanowski, A. Joulin, M. Douze, Deep clustering for unsupervised learning of visual features, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 132–149.
# Highlights

## EEG Classification with Limited Data: A Deep Clustering Approach

Mohsen Tabejamaat, Hoda Mohammadzade, Farhood Negin, Francois Bremond

* A novel training strategy is proposed to mitigate the overfitting of deep neural networks trained on an extremely limited number of samples.

* An image-guided clustering approach is introduced to address outlier issues in EEG data.
Graphical Abstract

EEG Classification with Limited Data: A Deep Clustering Approach

Mohsen Tabejamaat, Hoda Mohammadzade, Farhood Negin, Francois Bremond

<figure><img src="image_7.png" /></figure>