## 1 Introduction  
Speech is commonly referred to as the most natural way of human-to-human communication [1]. As humans, we can distinguish how others feel based on their voices. Emotion detection affects our interpretation of the spoken content, our behavior, and the consequent actions. Emotions do not only affect our speech. They also affect our mood, facial expressions, medical features, and body gestures. Based on someone's facial expressions, people can often recognize how others are feeling, even without speech content in the case of non-verbal communication. Although people can recognize emotions only from speech or only from facial expressions, commonly, both of these modalities are naturally used for overall recognition [2]. Similar concepts are employed in order to design state-of-the-art systems for human-computer interaction. Automatic emotion recognition has been a research area for more than two decades, and in that time frame, numerous methods based on machine learning (ML) and deep learning (DL) have been proposed [3, 4, 5]. In the beginning, unimodal speech emotion recognition and facial emotion recognition systems were developed due to hardware limitations and the lack of available data for creating complex models. Some of the most preferred speech emotion recognition algorithms throughout the years have been based on the hidden Markov model (HMM), Gaussian mixture models (GMMs), support vector machines (SVMs), and artificial neural networks (ANNs). Other classification techniques usually involve methods based on decision trees (DTs), k-nearest neighbors (k-NN), k-means, as well as Naive Bayes classifiers. An overview of the existing methodologies can be found in [6]. Recent efforts have been directed toward the implementation of convolutional neural networks (CNNs), deep learning, and transformer-based networks, which commonly require speech samples to be preprocessed and converted into spectrograms [7; 8]. Besides spectrograms that are obtained using Fourier transform, scalograms can be also found in the literature as a visual representation of speech in the case of applied wavelet transform for different tasks involving audio signal processing and classification using convolutional neural networks [9]. [PERSON] et al. proposed a combination of a long short-term memory (LSTM) and a transformer encoder network to learn the long-term dependencies in speech signals and classify emotions [8]. Similar classifiers have been considered for facial emotion recognition, whereby CNNs are the prevailing choice [10]. Besides audio and video modalities, an analysis of brain electroencephalography (EEG) signals, collected using wearable devices, also attracted researchers' interest [11]. Transformer-based architectures are considered cutting-edge in the field of sequence modeling, and they have achieved remarkable success in various domains. However, their effectiveness is reached at the cost of quadratic computational and memory complexity [12] and in turn they require higher energy consumption compared to traditional machine learning techniques, making them challenging for implementation on edge devices [13]. On the other hand, sentiment analysis is considered a sensitive topic and there are privacy concerns related to data processing, such as potential user profiling, making edge-based applications desirable for detection. The technology proposed in this paper is developed within the MARVEL project, whose ubiquitous edge-to-fog-to-cloud architecture and solutions are primarily designed for smart city environments [14]. In recent research, there has been a shift from unimodal to multimodal approaches, with a strong focus on designing audiovisual models to ensure higher accuracy. In [15], the authors proposed an end-to-end network that incorporates LSTM besides a CNN. In [16], the authors exploit an attention mechanism to improve the efficacy of the DL network, whereas in [17], a multimodal emotion recognition metric learning is introduced.