# EEG-BASED ART INTEREST DECODING

M. Welter¹, J. Casal Martínez², E. Redmond³, J. Baum¹, T. Ward³, F. Lotte¹

¹Inria Centre at the University of Bordeaux / LaBRI, France

²Valencia Polytechnic University, Spain

³Dublin City University, Ireland

E-mail: marc.welter@inria.fr

## 1 Synopsis

Brain-Computer-Interfaces (BCIs) that decode aesthetic Interest could improve user experience in digital spaces by personalizing aesthetic stimuli selection without requiring explicit user feedback that might interrupt aesthetic experience. However, scientific understanding of aesthetic experience remains lacking, while validated BCI methods have not yet been applied to decode aesthetic Interest from EEG. We thus conducted a study in which participants gazed at visual art in a virtual museum and graded their Interest for each of them, while their EEG was being measured. Previous research suggested that brain oscillations could be informative of aesthetic preference. Therefore, we tested Filterbank Common Spatial Patterns feature extraction together with shrinkage Linear Discriminant Analysis, in a 2-class aesthetic Interest classification problem. We report promising aesthetic Interest decoding accuracies significantly and substantially above chance level.

## 2 Background

Passive Brain-Computer-Interfaces (BCIs) allow implicit and real-time monitoring of cognitive, affective and conative states in human users (Appriou & Lotte, 2021). Aesthetic experiences are complex constructs composed of such states, notably attentional, affective and reward-related states (Schaeffer, 2015). Today, we humans are exposed to an unprecedented amount of aesthetic stimuli, especially in digital spaces such as social media. In such spaces, presentation of artworks, e.g. visual art or music, relies on recommendation systems that require explicit user feedback. However, explicit feedback necessitates cognitive effort that might lessen aesthetic experience.

Passive aesthetic preference decoding BCIs, on the other hand, could allow personalization of art presentation in digital spaces without interruption, and thus, improve user experience (Welter, Bouneau, Lotte & Ward, 2023). Previous research on aesthetic preference decoding mostly focused on hedonic Liking (Welter & Lotte, 2024). However, automated art presentation based solely on Likes can easily result in filter bubbles of low aesthetic value (Yilma & Leiva, 2023). On the other hand, recommendations based on epistemic affects such as Interest and Curiosity could offer a wider range of aesthetic experiences to users (Kennett, Humphries & Chatterjee, 2023). In order to contribute towards the development of BCI-based art recommender systems that
result in interesting aesthetic experiences, we investigated aesthetic Interest decoding
from Electroencephalography (EEG) data.

**3 Methods**

We collected EEG data from 13 healthy adults while they gazed at 120 high quality
digital reproductions of diverse visual artworks in a virtual museum. The experiment
consisted of 6 blocks of 20 artwork presentation and rating trials. At the start of each
trial, a blank screen was displayed for a randomly chosen interval between 0.5 and 0.8s.
After that, a fixation cross appeared for 5s, to washout potential emotions evoked by
previous artworks and to measure a stable baseline. Then, an artwork stimulus was pre-
sented for 10s. Finally, the participants reported their Interest evoked by artworks on a
scale from 0-100. These ratings served as ground truth for subsequent aesthetic Interest
classification. After each run, the participants could rest for a minute. Balanced Inter-
est/Disinterest classes were created by selecting the 45% highest and the 45% lowest
rated artworks respectively. This partitioning resulted in 55-58 trials/class. Thus, we
defined aesthetic Interest decoding as a 2-class supervised classification problem with
an estimated chance level (alpha=0.01) of 60.9% (Combrisson & Jerbi, 2015).

EEG data were recorded at 256 Hz using an ActiChamp amplifier (Brain Products, Gilching, Germany) with 31 active electrodes on a standard 10-20 montage. Electrodes were placed on on a broad scalp covering. The signal was grounded at Fpz and during recording the reference was placed on Fz. During offline analysis, the data was re-referenced to common average reference.

In order to decode aesthetic interest for visual art from the EEG signal, the fol-
lowing pipeline was used and validated with a 10-fold shuffled and stratified cross-
validation: First, we manually inspected the data to reject bad channels. Then, we
applied a 1-100Hz bandpass filter and a notch filter at 50 Hz to remove line noise. Af-
terwards, we created fixed epochs of 1s length and cleaned these epochs using a local
Autoreject (Jas et al., 2016). These pre-processed epochs were then given to an Ex-
tended Infomax Independent Component Analysis (Lee, Girolami & Sejnowski, 1999).
The resulting ICs were classified with ICLabel (Pion-Tonachini, Kreutz-Delgado &
Makeig, 2019). Then, non-artefactual components were used to reconstruct a clean
signal from the raw EEG. Then, trial epochs were extracted from 0.1-10s (t=0s being
the start of the artwork display) during stimulus presentation and baseline corrected
from -4s to -0.01s before stimulus appearance. We extracted discriminant features of
the EEG epochs for classification with Filterbank Common Spatial Patterns (FBCSP)
(Ang et al., 2012) A bank of 8 bandpass filters was used ( 1-4Hz, 4-8Hz, 8-13Hz,
13-16Hz, 16-20Hz, 20-30Hz, 30-50Hz, 50-70Hz). Note that we included frequencies
above 30Hz - often rejected due to artefacts - as previous studies have suggested their
link with aesthetic experience (Strijbosch et al., 2022). For each of these band-pass
filter, 6 CSP spatial filters were learned from the train set and applied on the test set
during each global cross-validation fold. During the computation of the spatial filters,
covariance matrices where estimated using Oracle Approximating Shrinkage Estimator
(Chen, Wiesel, Eldar & Hero 2010). After spatial filtering, log-transformed bandpower
features were extracted. For each cross-validation fold, optimal features were selected
using Recursive Feature Elimination (Guyon, Weston, Barnhill & Vapnik, 2002) with a
local 5-fold shuffled and stratified cross-validation (inner cross-validation) on the train-
ing set data of that fold (from the outer cross-validation). The selected features were
then fed into a shrinkage Linear Discriminant Analysis (sLDA) (Blankertz et al., 2011)
classifier and aesthetic Interest decoding performance was evaluated by computing the
mean test accuracy over global cross-validation folds for each subject.

4 Results

<figure><img src="image_1.png" /><figcaption>Figure 1: Mean aesthetic Interest FBCSP-sLDA decoding accuracy for all subjects</figcaption></figure>

The overall mean decoding performance was 75.9% ± 9.4 (see Figure 1).

5 Discussion

EEG Aesthetic Interest decoding with FBCSP-sLDA worked quite well on average. Performance was below chance for one subject only. It should be noted, however, that ‘Interest’ is used with a range of different meanings (Pekrun, 2019) and thus, participants might have understood the Interest rating task differently.

Despite the relatively high FBCSP-sLDA Interest classification performance, bridg-
ing the gap towards online aesthetic preference decoding, and thus automated art pre-
sentation BCIs, remains challenging. Many BCIs require training on balanced data
samples. Unfortunately, it is often uncertain whether an individual will find an artwork
interesting or not. Statistical artwork properties, such as symmetry or complexity, have
been correlated with subjective aesthetic ratings (Geller, Bartho, Thoemmes & Redies,
2022) and could be used to estimate aesthetic preference a priori. In conclusion, aes-
thetic Interest decoding BCIs seem feasible, but future work should investigate optimal
training data selection during online calibration.
# 6 Disclosures

This research was funded by CHISTERA project BITSCOPE (grant ANR-21-CHRA-0003-01).

## References

Ang, K. K., Chin, Z. Y., Wang, C., Guan, C., and Zhang, H. (2012). Filter Bank Common Spatial Pattern Algorithm on BCI Competition IV Datasets 2a and 2b. *Frontiers in Neuroscience* 6. DOI: 10.3389/fnins.2012.00039. URL: https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2012.00039.

Appriou, A. and Lotte, F. (2021). Tools for affective, cognitive and conative states estimation from both EEG and physiological signals. *Neuroergonomics Conference (NEC)*.

Blankertz, B., Lemm, S., Treder, M., Haufe, S., and Müller, K.-R. (2011). Single-trial analysis and classification of ERP components — A tutorial. *NeuroImage* 56.2. Multivariate Decoding and Brain Reading, pp. 814–825. ISSN: 1053-8119. DOI: https://doi.org/10.1016/j.neuroimage.2010.06.048. URL: https://www.sciencedirect.com/science/article/pii/S1053811910009067.

Chen, Y., Wiesel, A., Eldar, Y. C., and Hero, A. O. (2010). Shrinkage Algorithms for MMSE Covariance Estimation. *IEEE Transactions on Signal Processing* 58.10. ISSN: 1941-0476. DOI: 10.1109/tsp.2010.2053029. URL: http://dx.doi.org/10.1109/TSP.2010.2053029.

Combrisson, E. and Jerbi, K. (2015). Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. *Journal of Neuroscience Methods* 250. Cutting-edge EEG Methods, pp. 126–136. ISSN: 0165-0270. DOI: https://doi.org/10.1016/j.jneumeth.2015.01.010. URL: https://www.sciencedirect.com/science/article/pii/S0165027015000114.

Geller, H. A., Bartho, R., Thömmes, K., and Redies, C. (2022). Statistical image properties predict aesthetic ratings in abstract paintings created by neural style transfer. *Frontiers in Neuroscience* 16. ISSN: 1662-453X. DOI: 10.3389/fnins.2022.999720. URL: https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2022.999720.

Guyon, I., Weston, J., Barnhill, S., and Vapnik, V. (2002). Gene Selection for Cancer Classification Using Support Vector Machines. *Machine Learning* 46, pp. 389–422. DOI: 10.1023/A:1012487302797.

Jas, M., Engemann, D., Bekhti, Y., Raimondo, F., and Gramfort, A. (2016). Autoreject: Automated artifact rejection for MEG and EEG data. *NeuroImage* 159. DOI: 10.1016/j.neuroimage.2017.06.030.

Kenett, Y., Humphries, S., and Chatterjee, A. (2023). A Thirst for Knowledge: Grounding Curiosity, Creativity, and Aesthetics in Memory and Reward Neural Systems. *Creativity Research Journal* 35, pp. 1–15. DOI: 10.1080/10400419.2023.2165748.

Lee, T.-W., Girolami, M., and Sejnowski, T. (1999). Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Sub-Gaussian and Super-Gaussian Sources. *Neural Computation* 11, pp. 417–441. DOI: 10.1162/089976699300016719.
Pekrun, R. (2019). The Murky Distinction Between Curiosity and Interest: State of the Art and Future Prospects. *Educational Psychology Review* 31. DOI: 10.1007/s10648-019-09512-1.

Pion-Tonachini, L., Kreutz-Delgado, K., and Makeig, S. (2019). ICLabel: An automated electroencephalographic independent component classifier, dataset, and website. *NeuroImage* 198. DOI: 10.1016/j.neuroimage.2019.05.026.

Schaeffer, J. (2015). *L'Expérience esthétique*. Paris: Gallimard.

Strijbosch, W, Vessel, E. A., Welke, D, Mita, O, Gelissen, J, and Bastiaansen, M (2022). On the neuronal dynamics of aesthetic experience: Evidence from electroencephalographic oscillatory dynamics. *Journal of Cognitive Neuroscience* 34.3.

Welter, M, Bouneau, A, Lotte, F, and Ward, T (2023). *Abstracts from the 8th Visual Science of Art Conference (VSAC) Amsterdam, The Netherlands, August 24th–27th, 2022*. Vol. 11. Art & Perception 2. Leiden, The Netherlands: Brill.

Welter, M and Lotte, F (2024). Ecological decoding of visual aesthetic preference with oscillatory electroencephalogram features—A mini-review. *Frontiers in Neuroergonomics* 5. DOI: 10.3389/fnrgo.2024.1341790. URL: https://www.frontiersin.org/articles/10.3389/fnrgo.2024.1341790.

Yilma, B. A. and Leiva, L. A. (2023). The Elements of Visual Art Recommendation: Learning Latent Semantic Representations of Paintings. *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems*. Hamburg, Germany: Association for Computing Machinery. ISBN: 9781450394215. DOI: 10.1145/3544548.3581477. URL: https://doi.org/10.1145/3544548.3581477.