<figure><img src="image_1.png" /></figure>

HAL
open science

# EEG single-trial decoding of visual art preference

Marc Welter, J Casal Martínez, E Redmond, Jonathan Baum, Tomas Ward, Fabien Lotte

► To cite this version:

Marc Welter, J Casal Martínez, E Redmond, Jonathan Baum, Tomas Ward, et al.. EEG single-trial decoding of visual art preference. Graz BCI 2024 - 9th International Graz Brain-Computer Interface Conference, Sep 2024, Graz, Austria. hal-04746676

HAL Id: hal-04746676

https://inria.hal.science/hal-04746676v1

Submitted on 21 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# EEG SINGLE-TRIAL DECODING OF VISUAL ART PREFERENCE

M. Welter¹, J. Casal Martínez², E. Redmond³, J. Baum¹, T. Ward³, F. Lotte¹

¹Inria Centre at the University of Bordeaux / LaBRI, France

²Valencia Polytechnic University, Spain

³Dublin City University, Ireland

E-mail: marc.welter@inria.fr

## ABSTRACT:

Brain-Computer-Interfaces (BCIs) able to decode aesthetic preference could improve user experience in digital spaces by personalizing aesthetic stimuli selection without requiring explicit user feedback that might interrupt aesthetic experience. However, neuroscientific understanding of aesthetic experience remains lacking, while the tried and tested BCI classification algorithms have not yet been applied to decode aesthetic preferences from EEG signals. We thus conducted an experiment in which participants where exposed to visual artworks in a virtual museum and requested to grade their preferences for each of them, all this while their EEG was being measured. Previous neuroaesthetic research suggested that oscillatory modulations in different neural frequency bands could be informative of aesthetic preference. Therefore, we tested a time-frequency feature classification method widely used in BCIs, i.e. Filterbank Common Spatial Patterns feature extraction together with shrinkage Linear Discriminant Analysis, in a 2-class aesthetic preference classification problem. We report promising aesthetic preference decoding accuracies significantly and substantially above chance level.

## INTRODUCTION

Passive Brain-Computer-Interfaces (BCIs) allow implicit and real-time monitoring of cognitive, affective and conative mental and embodied states in human users [1, 2]. Aesthetic experiences are complex experiences that are composed of such states, notably attentional, affective and reward-related states [3]. Humans in the 21st century are exposed to an unprecedented amount of aesthetic stimuli, especially in digital spaces such as social media. In such spaces, presentation of aesthetic stimuli, e.g. visual art or music, relies on recommendation systems that require explicit user feedback. However, giving explicit feedback requires cognitive effort that might interrupt aesthetic experience.

Passive aesthetic preference decoding BCIs, on the other hand, could allow personalization of art presentation in digital spaces without interruption, and thus, improve user experience [4]. Furthermore, aesthetic preference decoding BCIs could be applicable in other domains such as neuro-marketing in order to improve personalized advertising

[5], and it might even help improve positive effects of art exposure on health and well-being [6].

However, to our knowledge, only two studies have investigated single trial aesthetic preference decoding from EEG. One using Deep Learning classifiers [7] and one using Temporal Decision Trees [8]. Neither of them reported any artefact removal procedure which renders the interpretation of their results difficult [9].

Thus, there has been a lack of Electroencephalography (EEG) single trial aesthetic preference decoding studies with validated and effective EEG classification methods. In order to alleviate this lack, this article aims to contribute towards the development of aesthetic preference decoding BCIs by applying validated BCI methods on EEG data recorded in a virtual art museum environment. The different components of aesthetic experience have been shown to be related to oscillatory brain modulations in various frequency bands [9]. Therefore, we used Filter Bank Common Spatial Patterns (FBCSP) [10] and shrinkage Linear Discriminant Analysis (sLDA) [11] in order to decode aesthetic preference from oscillatory EEG features.

In the following sections, we will describe the EEG and subjective aesthetic preference data collection, as well as the offline aesthetic preference decoding pipeline. Then, we respectively report and discuss the aesthetic preference decoding results. Finally, we offer prospects for future research and summarize our findings.

## MATERIALS AND METHODS

### Participants:

14 healthy adult participants (7 women, aged 26.77 ± 8.5) completed the whole experiment. All participants grew up in Western cultures and, thus, were most familiar with Western art. None of them reported a history of neurological or psychiatric disorder. Participants gave informed consent prior to the study. The study was conducted in accordance with the ethical research guidelines in the Declaration of Helsinki and was approved by Inria's ethics committee, the COERLE (approval number: 2023-11). For one participant, subjective aesthetic appreciation ratings were not save correctly, due to a bug in the recording. Thus, the following analyses are based on 13 participants.
Experimental protocol:

<figure><img src="image_3.png" /><figcaption>Figure 1: A participant taking part in the aesthetic preference decoding BCI experiment</figcaption></figure>

Each participant participated in one session of 2 hours. The session was organized as follows: (1) consent form signature and completion of several questionnaires (around 20 min), (2) installation of the EEG cap (around 20 min), (3) 3 test trials to familiarize themselves with the procedure, (4) 6 runs during which participants were presented artworks (around 60 min in total, including breaks between the runs), (5) completion of post-session questionnaires (around 5 min), and (6) uninstallation and debriefing (around 10 min).

During each run, participants had to perform 20 trials in a virtual museum environment. This Virtual Exhibition Environment (VEE) has been developed through the Unity3D software, which contains the textured 3D models for the visualisation of the environment, artwork and lighting. The first version of the VEE (VEE1) is a desktop application that allows studies in the field of neuroscience, which can be configured through a settings screen (it allows selecting the library of images, modifying the lighting, texturing the walls, adding screens and/or fixation crosses, among others). The application captures real-time Eye-Tracking data (in a format readable by the OGAMA analysis software), and sends signals (using Lab Streaming Layer - LSL [12]) to the OpenVIBE software [13] to synchronise EEG data captured with the experiment's timeline. Figure 1 shows the experimental setup with one participant wearing an EEG cap while gazing at a painting in the VEE.

At the start of each trial, a blank screen was displayed for a randomly sampled duration between 0.5 and 0.8s. After that, a fixation cross appeared for 5s, in order to measure a stable baseline, as well as to washout potential emotions evoked by previous artworks. Then, an artwork stimulus was presented for 10s. Finally, the subject rated their aesthetic appreciation (liking and interest) of the artwork on a scale from 0-100 using a slider. After each run, the participants could rest for a minute. The timeline of one trial of data collection with our experimental protocol is shown in Figure 2.

Participants were instructed to gaze naturally at the art work. Instructions were written in advance so that all the participants started with the same standardized

<figure><img src="image_4.png" /><figcaption>Figure 2: The data collection included 6 blocks of 20 visual art stimuli presentations and subsequent subjective aesthetic evaluations</figcaption></figure>

information.

### Questionnaires:

In addition to the participant's general demographic information, we asked the participants to complete the following questionnaires:

* AREA [14] translated into French, to measure the participant's responsiveness to aesthetic experiences in general. This questionnaire was filled out before the EEG measurements.

* NeXT-Q [15] to measure the participant's mental states before and after the experiment.

### Artworks:

The artworks displayed in the museum were 120 high quality digital reproductions of diverse artworks originating from almost all continents and ranging from pre-historic to contemporary time periods in the public domain. The artworks had a minimum resolution of 450x669, with a mean of 11392920 pixels. We chose artworks that were relatively unknown to a general audience, in order to avoid familiarity effects. The Artworks' brightness levels were normalized in order to avoid different brightness levels affecting the EEG signal [16].

### EEG Recordings & Signal Processing:

EEG data were sampled at 256 Hz using an ActiChamp amplifier (Brain Products, Gilching, Germany) with 31 active electrodes on a standard 10-20 montage. Electrodes were placed on the following scalp locations: Fp1, F3, F7, FT9, FC5, FC1, C3, T7, TP9, CP5, CP1, Pz, P3, P7, O1, Oz, O2, P4, P8, TP10, CP6, CP2, Cz, C4, T8, FT10, FC6, FC2, F4, F8, Fp2, i.e., on a broad scalp covering. The signal was grounded at Fpz and the reference
was placed on Fz during recording. During offline analysis, the data was re-referenced to common average reference.

In order to decode aesthetic preference for visual art from the EEG signal, the following signal processing pipeline was used and validated with a 10-fold shuffled and stratified cross-validation:

First we manually inspected the data to reject bad channels. Then, we applied a 1-100Hz 4th order Butterworth bandpass filter and a notch filter at 50 Hz to remove line noise. Afterwards, we created fixed epochs of 1s length and cleaned these epochs using a local Autoreject [17]. These cleaned epochs were then fed into an Extended Infomax Independent Component Analysis [18]. The resulting independent components were classified with ICLabel [19]. Then, components labeled as artefacts were excluded from the components used to reconstruct a clean signal from the raw EEG.

Then, trial epochs were extracted from 0.1-10s (t=0s being the start of the artwork display) during stimulus presentation and baseline corrected from -4s to -0.01s before stimulus appearance.

Balanced Like and Dislike classes (for subsequent 2-class classification of aesthetic experience) were determined by partitioning the epochs based on quantization of subjective ratings inspired by Strijbosch et al. [20]. We chose to include the data from the 45% most liked and the 45% most disliked artworks which resulted in a margin of 10% medium liked artworks that were not included in further analyses. This partitioning procedure resulted in balanced classes with 55-58 epochs per class. Thus, we defined aesthetic preference decoding as a 2-class supervised classification problem with an estimated chance level ($\alpha$=0.01) of 60.9% [21].

We extracted discriminant features of the EEG signal for classification with FBCSP-sLDA. During the computation of the spatial filters, covariance matrices where estimated using Oracle Approximating Shrinkage Estimator [22]. A bank of 8 filters was used with 4th order Butterworth bandpass filters in the following frequency bands: 1-4Hz, 4-8Hz, 8-13Hz, 13-16Hz, 16-20Hz, 20-30Hz, 30-50Hz, 50-70Hz.

Note that EEG analyses are often done with a cutoff at 30Hz in order to remove artefacts. However, higher frequencies above 30Hz have been shown to contain discriminative information about aesthetic experience [20]. Therefore, we decided to include higher frequency bands in our analyses.

For each of these band-pass filter, 6 CSP spatial filters were learned from the train set and applied on the test set during each global cross-validation fold. After spatial filtering, log-transformed bandpower features were extracted.

For each cross-validation fold, optimal features were selected using Recursive Feature Elimination [23] with a local 5-fold shuffled and stratified cross-validation (inner cross-validation) on the training set data of that fold (from the outer cross-validation). The selected features were

then fed into a sLDA classifier.

Aesthetic preference decoding performance was evaluated by computing the mean test accuracy over global cross-validation folds for each subject. In addition, we also ran a 10-fold shuffled and stratified cross-validated CSP-sLDA classification for each band-pass filter individually, in order to investigate the discriminatory power of each frequency band.

## RESULTS

<figure><img src="image_5.png" /><figcaption>Figure 3: Mean aesthetic preference FBCSP-sLDA decoding accuracy for all subjects</figcaption></figure>

We analysed the aesthetic preference decoding performance in term of mean classification accuracy, for this 2-class BCI. Mean FBCSP-sLDA classification performance with feature selection for all subjects is shown in Figure 3. The overall mean decoding performance was $0.798 \pm 0.162\%$.

Classification on individual band-pass filters performed much worse and well below the estimated chance level on average with a mean accuracy of $0.541 \pm 0.052$ and showed a large variability across subjects. Figure 4 shows the variability in mean classification performance across subject for different band-pass filters.

<figure><img src="image_6.png" /><figcaption>Figure 4: Mean aesthetic preference CSP-LDA decoding accuracy for individual frequency bands across subjects</figcaption></figure>
# DISCUSSION

Overall, FBCSP-sLDA classification yielded surprisingly good results that were better than chance for most participants. Only for one subject, classification performance was below the estimated chance level of 60.9%. The relatively high classification accuracy suggests that EEG single trial decoding of aesthetic preference is feasible. Still, it remains possible that such relatively high classification performance relies, at least partly (and despite our artefact correction procedure), on movement artefacts in the EEG signal, as body movements can be informative of aesthetic experience [24]. In the following paragraph we will discuss possible correlates of those frequency bands that performed above chance for at least one subject: Oscillations in the alpha band were most informative which could be related to visual processing, but might also be generated by eye movements. Theta band features also seemed to contain some discriminatory information which might be due to the activation of the Default Mode Network during aesthetic experience [25]. Yet, theta band modulations can also be produced by eye blinks [26]. Low and mid beta frequencies exhibited the next best mean performance and have been implicated in emotional processing during art perception [27]. Finally, features in the gamma bands also performed relatively well for some subjects. Although, gamma bands are frequently excluded from EEG analyses, they have been implicated in aesthetically moving experiences [20]. However, both beta and gamma frequencies are also known to be commonly affected by muscular activity [28].

### future work:

Aesthetic preference decoding with EEG BCIs could potentially be improved by using more advanced BCI classification algorithms, such as Deep Learning [29] or Riemannian Geometry-based classifiers [30]. Furthermore, the inclusion of features from other physiological modalities such as electrodermal activity, heart rate or eye movement might increase classification performance [31].

Although we report good performances for offline single trial aesthetic preference decoding with passive BCI, bridging the gap towards online classification remains challenging. A major limitation remains the development of the calibration phase, as we do not necessarily know a user's aesthetic preferences beforehand which complicates the selection of optimal art stimuli in the training data during calibration. Potentially, statistical image properties of the artworks that have been correlated with subjective ratings [32] could inform stimuli selection in order to train a generalizable aesthetic preference decoder.

# CONCLUSION

We reported the first neuroaesthetic study using tested BCI algorithms for single-trial aesthetic preference decoding from EEG.

Our results revealed better than chance classification accuracies for most subjects in discriminating preferred vs non-preferred artwork, using a FBCSP-sLDA classification pipeline. Individual bands analyses suggested that the alpha, theta, high beta and gamma bands were the most informative.

Although further work is required to develop online aesthetic preference decoding BCIs, the promising classification results above chance level suggest that decoding of aesthetic experience is feasible with EEG-based BCIs. Future work should focus on improving the accuracies obtained as well as in better identifying the possible contributions of cortical EEG and possibly of muscle or eye artifacts to the obtained decoding accuracies.

# REFERENCES

[1] Appriou A, Lotte F. Tools for affective, cognitive and conative states estimation from both eeg and physiological signals. In: Neuroergonomics Conference (NEC). 2021.

[2] Zander TO, Kothe C. Towards passive brain-computer interfaces: Applying brain-computer interface technology to human-machine systems in general. Journal of neural engineering. 2011;8(2):025005.

[3] Schaeffer J. L'expérience esthétique. Gallimard: Paris (2015).

[4] Welter M, Bouneau A, Lotte F, Ward T. In: Abstracts from the 8th Visual Science of Art Conference (VSAC) Amsterdam, The Netherlands, August 24th–27th, 2022. Brill: Leiden, The Netherlands, 2023.

[5] Ferrer GG. A neuroaesthetic approach to the search of beauty from the consumer's perspective. Advances in Marketing, Customer Relationship Management, and E-Services. 2019.

[6] Fancourt D, Finn S. What is the evidence on the role of the arts in improving health and well-being? a scoping review. Nordic Journal of Arts, Culture and Health. 2019;2:77-83.

[7] Fraiwan M, Alafeef M, Almomani F. Gauging human visual interest using multiscale entropy analysis of eeg signals. Journal of Ambient Intelligence and Humanized Computing. 2023;12:2435-2447.

[8] Mazzacane S et al. Towards an objective theory of subjective liking: A first step in understanding the sense of beauty. PLOS ONE. 2023;18(6).

[9] Welter M, Lotte F. Ecological decoding of visual aesthetic preference with oscillatory electroencephalogram features—a mini-review. Frontiers in Neuroergonomics. 2024;5.

[10] Ang KK, Chin ZY, Wang C, Guan C, Zhang H. Filter bank common spatial pattern algorithm on bci competition iv datasets 2a and 2b. Frontiers in Neuroscience. 2012;6.

[11] Blankertz B, Lemm S, Treder M, Haufe S, Müller KR. Single-trial analysis and classification of erp components — a tutorial. NeuroImage. 2011;56(2):814-825.
[12] Kothe C et al. The lab streaming layer for synchronized multimodal recording. bioRxiv. 2024.

[13] Renard Y et al. OpenViBE: An Open-Source Software Platform to Design, Test, and Use Brain-Computer Interfaces in Real and Virtual Environments. Presence: Teleoperators and Virtual Environments. 2010;19(1):35-53.

[14] Schlotz W, Wallot S, Omigie D, Masucci M, Hoelzmann S, Vessel E. The aesthetic responsiveness assessment (area): A screening tool to assess individual differences in responsiveness to art in english and german. Psychology of Aesthetics, Creativity, and the Arts. 2020;15.

[15] Bismuth J, Vialatte F, Lefaucheur JP. Relieving peripheral neuropathic pain by increasing the power-ratio of low-β over high-β activities in the central cortical region with eeg-based neurofeedback: Study protocol for a controlled pilot trial (smrpain study). Neurophysiologie Clinique. 2020;50(1):5-20.

[16] Eroğlu K, Kayıkçıoğlu T, Osman O. Effect of brightness of visual stimuli on eeg signals. Behavioural brain research. 2020;382.

[17] Jas M, Engemann D, Bekhti Y, Raimondo F, Gramfort A. Autoreject: Automated artifact rejection for meg and eeg data. NeuroImage. 2016;159.

[18] Lee TW, Girolami M, Sejnowski T. Independent component analysis using an extended infomax algorithm for mixed sub-gaussian and super-gaussian sources. Neural Computation. 1999;11:417-441.

[19] Pion-Tonachini L, Kreutz-Delgado K, Makeig S. Ilabel: An automated electroencephalographic independent component classifier, dataset, and website. NeuroImage. 2019;198.

[20] Strijbosch W, Vessel EA, Welke D, Mita O, Gellissen J, Bastiaansen M. On the neuronal dynamics of aesthetic experience: Evidence from electroencephalographic oscillatory dynamics. Journal of Cognitive Neuroscience. 2022;34(3).

[21] Combrisson E, Jerbi K. Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. Journal of Neuroscience Methods. 2015;250:126-136.

[22] Chen Y, Wiesel A, Eldar YC, Hero AO. Shrinkage algorithms for mmse covariance estimation. IEEE Transactions on Signal Processing. 2010;58(10):5016-5029.

[23] Guyon I, Weston J, Barnhill S, Vapnik V. Gene selection for cancer classification using support vector machines. Machine Learning. 2002;46:389-422.

[24] Kühnapfel C et al. How do we move in front of art? how does this relate to art experience? linking movement, eye tracking, emotion, and evaluations in a gallery-like setting. Empirical Studies of the Arts,. 2023;42(1):86-146.

[25] Vessel EA, Starr GG, Rubin N. The brain on art: Intense aesthetic experience activates the default mode network. Frontiers in Neuroscience. 2012;7.

[26] Roy RN, Charbonnier S, Bonnet S. Eye blink characterization from frontal eeg electrodes using source sep-

aration and pattern recognition algorithms. Biomedical Signal Processing and Control. 2014;14:256-264.

[27] Herrera-Arcos G et al. Modulation of neural activity during guided viewing of visual art. Frontiers in Human Neuroscience. 2017;11.

[28] Pope K et al. Managing electromyogram contamination in scalp recordings: An approach identifying reliable beta and gamma eeg features of psychoses or other disorders. Brain and Behavior. 2022;12.

[29] Lawhern V, Solon A, Waytowich N, Gordon S, Hung C, Lance B. Eegnet: A compact convolutional network for eeg-based brain-computer interfaces. Journal of Neural Engineering. 2016;15.

[30] Congedo M, Barachant A, Bhatia R. Rieman-nian geometry for eeg-based brain-computer interfaces; a primer and a review. Brain-Computer Interfaces. 2017;4:1-20.

[31] Hogervorst M, Brouwer A, Erp J. Combining and comparing eeg, peripheral physiology and eye-related measures for the assessment of mental workload. Frontiers in Neuroscience. 2014;8(322).

[32] Geller HA, Bartho R, Thömmes K, Redies C. Statistical image properties predict aesthetic ratings in abstract paintings created by neural style transfer. Frontiers in Neuroscience. 2022;16.