PDAML: A Pseudo Domain Adaptation Paradigm for Subject-independent EEG-based Emotion Recognition

Domain adaptation ( DA ) and domain generalization ( DG ) methods have been successfully adopted to alleviate the domain shift problem caused by the subject variability of EEG signals in subject-independent affective brain-computer interfaces ( aBCIs ) . Usually , the DA methods give relatively promising results than the DG methods but require additional computation resources each time a new subject comes . In this paper , we first propose a new paradigm called Pseudo Domain Adaptation ( PDA ) , which is more suitable for subject-independent aBCIs . Then we propose the pseudo domain adaptation via meta-learning ( PDAML ) based on PDA . The PDAML consists of a feature extractor , a classifier , and a sumdecomposable structure called domain shift governor . We prove that a network with a sum-decomposable structure can compute the divergence between different domains effectively in theory . By taking advantage of the adversarial learning and meta-learning , the governor helps PDAML quickly generalize to a new domain using the target data through a few self-adaptation steps in the test phase . Experimental results on the public aBICs dataset demonstrate that our proposed method not only avoids the additional computation resources of the DA methods but also reaches a similar generalization performance of the state-of-the-art DA methods . 1 INTRODUCTION . Affective brain-computer interfaces ( aBCIs ) , which focus on developing machines to recognize human emotion automatically and provide more humanized interaction , have attracted widespread attention from academics and industries ( Mühl et al . ( 2014 ) ; Shanechi ( 2019 ) ) . Various studies have demonstrated that the electroencephalography ( EEG ) signal is especially reliable to recognize human emotion in the subject-dependent emotion model , in which the training and test data are from one subject ( Jenke et al . ( 2014 ) ; Alarcão & Fonseca ( 2019 ) ) . However , due to the non-stationary nature of EEG signal and structural variability between different subjects , subject-independent model based on the assumption of Independent Identity Distribution ( i.i.d . ) usually shows bad generalization performance in real aBCIs applications , which is called the problem of Domain Shift ( Sugiyama et al . ( 2007 ) ; Samek et al . ( 2013 ) ; Sussillo et al . ( 2016 ) ; Zheng & Lu ( 2016 ) ; Li et al . ( 2018d ) ) . Domain adaptation ( DA ) is one of the promising ways to solve this problem . DA uses data from both source and target domain to promote the adaption performance . One of the most sufficient studied DA is mapping the two distributions to one common feature space where they have the same marginal distribution . Though DA has demonstrated significant success in subject-independent EEG-based emotion recognition ( Zheng & Lu ( 2016 ) ; Li et al . ( 2018b ; c ) ; Luo et al . ( 2018 ) ) , the additional computation and time to apply the DA methods is an exasperating problem in real-world scenarios and causes poor user experience . As a consequence , the concept of domain generalization ( DG ) arises in situations where multiple source domains can be accessed but unlabeled target samples are not available . The DG methods have also been successfully adopted to build the subjectindependent emotion model ( Ma et al . ( 2019b ; a ) ) . However , since there is no prior information about the target domain during training , it ’ s challenging for DG to perform as promising as DA . A compromise solution is Fast Domain Adaptation ( FDA ) , which trains the main model in advance and uses a small amount of test samples to adjust efficiently . Even though FDA methods can avoid consuming time cost by adaptation , most of them need to storage both source and target domains in the test phase ( Chai et al . ( 2017 ) ; Zhao et al . ( 2021 ) ) , which needs extra storage space and causes poor portability . However , in real-world applications , whether a EEG-based affective model can adapt to different subjects quickly as well as keep its portability does matter . In this paper , we propose a new paradigm called Pseudo Domain Adaptation ( PDA ) for subjectindependent EEG-based emotion recognition . Compared to typical FDA , only target domain is required in the test phase of PDA , which makes PDA capable to output the prediction more quickly than DA and FDA . From the perspective of the real-world application , PDA is more suited to build subject-independent affective model . To implement PDA , we first prove that with the sum-decomposable structure , a network is equivalent to domain discrepancy metrics in traditional DA methods like MMD orH-divergence . Based on our theory , we propose a method named Pseudo Domain Adaptation via Meta Learning ( PDAML ) . The PDAML consists of a feature extractor , a classifier , and a sum-decomposable structure called domain shift governor . By taking advantage of the adversarial learning and meta-learning , the governor helps PDAML quickly generalize to a new domain using the target data through a few self-adaptive steps in the test phase . The main contributions of this paper can be summarized as follows : • We propose a more suitable paradigm to build subject-independent affective model . • We prove that a sum-decomposable network is equivalent to domain discriminator for rep- resenting any type of domain discrepancy in theory . • We propose PDAML that is portable and can fast adapt to different subjects in EEG-based emotion recognition . • We conduct extensive experiments on the SEED dataset1 , which is a public available EEG- based affective dataset . The experimental results demonstrate that our method has better performance than DG methods . From the perspective of time and storage cost , the PDAML performs as well as DG methods . 2 RELATED WORK . aBCIs have received considerable attention very recent . Mühl et al . presented the definition of aBCIs ( Mühl et al . ( 2014 ) ) by introducing the affective factors into traditional brain-computer interfaces ( BCIs ) ( Zander & Jatzev ( 2011 ) ) . More applied works focus on studying EEG-based emotion recognition . Zheng and Lu recruited 15 subjects to watch 15 selected Chinese movie clips to elicit three emotions : happy , neutral and sad . They developed a public emotion dataset called SEED by recording the EEG signals of the subjects ( Zheng & Lu ( 2015 ) ) . Based on the SEED dataset , researchers have made great progress in developing EEG-based emotion recognition model , especially for subject-dependent model . Due to the non-stationary nature of EEG signal and structural variability between different subjects , it is hard to develop subject-independent EEG-based emotion recognition model by directly using typical machine learning approaches . Researchers have focused on applying DA and DG methods to subject-independent EEG-based emotion recognition . Typical DA methods are discrepancy-based , and they alleviate the domain shift problem by minimizing traditional metrics , such as Maximum Mean Discrepancy ( MMD ) ( Pan et al . ( 2011 ) ; Long et al . ( 2017 ) ; Wang et al . ( 2018 ) ) , KullbackLeibler ( KL ) divergence ( Zhuang et al . ( 2015 ) ) , andH-divergence ( Bendavid et al . ( 2010 ) ) , between the different domains . Zheng and Lu adopted transfer component analysis ( TCA ) ( Pan et al . ( 2011 ) ) , which minimizes MMD ( Gretton et al . ( 2007 ) ) between two domains by constructing kernel matrix , and successfully built personalized EEG-based emotion models ( Zheng & Lu ( 2016 ) ) . Recently , adversarial DA methods have made great successes in different fields ( Ganin & Lempitsky ( 2015 ) ; Tzeng et al . ( 2017 ) ; Shen et al . ( 2018 ) ) . The basic idea of the adversarial training is similar to generative adversarial networks ( GANs ) ( Goodfellow et al . ( 2014 ) ) , which play an adversarial 1http : //bcmi.sjtu.edu.cn/∼seed/index.html game to make the generated distribution approximate to the real distribution . After the adversarial training , the data distribution of the target domain is similar to the source domain , and the domain shift is diminished . Researchers have successfully adopted adversarial DA methods to aBCIs . Li et al . ( Li et al . ( 2018b ) ) adopted domain-adversarial neural networks ( DANN ) ( Ganin & Lempitsky ( 2015 ) ) to EEG-based emotion recognition and improved the recognition accuracies of subjectindependent models . And they ( Li et al . ( 2018c ) ) also achieved significant performance in subjectindependent vigilance estimation by implementing DANN and adversarial discriminative domain adaptation ( ADDA ) ( Tzeng et al . ( 2017 ) ) . Luo et al . proposed Wasserstein GAN ( Arjovsky et al . ( 2017 ) ) adversarial domain adaptation ( WGANDA ) and successfully adopted it to build subjectindependent emotion recognition models ( Luo et al . ( 2018 ) ) . In real-world aBCIs applications , each subject can be viewed as an individual domain . DA methods , which require high additional computation resources for each new domain , hinder the development of aBCIs from lab to real scenarios . DG methods , which can be utilized by data manipulation , representation learning , or meta-learning ( Wang et al . ( 2021 ) ) , aim to generalize to unseeing target domains without additional data collection from target domains . Researchers have focused on adopting DG methods to aBCIs very recently . Ma et al . generalized the structure of DANN into DG and proposed an adversarial structure called Domain Residual Network ( DResNet ) . They adopted DResNet to subject-independent EEG-based vigilance estimation and emotion recognition ( Ma et al . ( 2019b ; a ) ) . The experimental result demonstrated that the DG method could improve the generalization ability without data collection from the target domains . However , the DA methods usually give relatively promising results than DG methods in aBCIs applications . As a compromise way , FDA was adopted in aBCIs . Zhao et al . proposed a Plug-and-Play Domain Adaptation ( PPDA ) method to fast adapt the model in EEG-based emotion recognition ( Zhao et al . ( 2021 ) ) . Meta-learning , also known as learning to learn , has received a resurgence in interest recently with applications , one of which is domain generalization . Meta-learning aims to learn episodes sampled from the related tasks ( Finn et al . ( 2017 ) ) . Meta-Learning for Domain Generalization ( MLDG ) first introduces meta-learning strategy to DG ( Li et al . ( 2018a ) ) , then MetaReg ( Balaji et al . ( 2018 ) ) and Feature-Critic ( Li et al . ( 2019 ) ) are subsequently proposed to enhance the model ’ s generalizing ability by introducing an auxiliary loss in the training . Compared to most previous DG work that designs a specific model , meta-learning-based DG methods focus on model agnostic training strategy by exposing the model to domain shift in the training phase . To the best of our knowledge , we are the first to introduce meta-learning based methods to aBCIs tasks . 3 THEORY . The motivation of PDAML is using a simple network to compute domain shift , taking only the target domain as input . Traditional DA methods usually taking two domains as input when comparing the target domain with a specific source domain . Thus they need extra storage space for source data and sophisticated methods ( e.g . GAN ) to represent domain shift in the test phase , which is time-consuming and storage-consuming . Either or both of these problems obstruct the practical application of DA method in EEG-based diagnosis . However , in multi-source setting , we will show that minimizing the discrepancy between all pairwise domains is equivalent to minimizing the discrepancy between each domain and an implicit domain . Additionally , we will prove that a network with what is termed sum-decomposition form can represent any domain shift metrics in theory .

Summary: The paper proposes an affective emotion recognition method using EEG data with an aim to perform in the subject independent paradigm. Towards this, domain adaptation and generalisation methods are evaluated and in particular a pseudo domain adaptation method is proposed for the said purpose. Experiments are performed on SEED dataset, and a method that use pseudo domain adaptation and meta learning is proposed to improve upon computational and memory requirements in realising practical aBCIs.