PDAML: A Pseudo Domain Adaptation Paradigm for Subject-independent EEG-based Emotion Recognition

Domain adaptation ( DA ) and domain generalization ( DG ) methods have been successfully adopted to alleviate the domain shift problem caused by the subject variability of EEG signals in subject-independent affective brain-computer interfaces ( aBCIs ) . Usually , the DA methods give relatively promising results than the DG methods but require additional computation resources each time a new subject comes . In this paper , we first propose a new paradigm called Pseudo Domain Adaptation ( PDA ) , which is more suitable for subject-independent aBCIs . Then we propose the pseudo domain adaptation via meta-learning ( PDAML ) based on PDA . The PDAML consists of a feature extractor , a classifier , and a sumdecomposable structure called domain shift governor . We prove that a network with a sum-decomposable structure can compute the divergence between different domains effectively in theory . By taking advantage of the adversarial learning and meta-learning , the governor helps PDAML quickly generalize to a new domain using the target data through a few self-adaptation steps in the test phase . Experimental results on the public aBICs dataset demonstrate that our proposed method not only avoids the additional computation resources of the DA methods but also reaches a similar generalization performance of the state-of-the-art DA methods . 1 INTRODUCTION . Affective brain-computer interfaces ( aBCIs ) , which focus on developing machines to recognize human emotion automatically and provide more humanized interaction , have attracted widespread attention from academics and industries ( Mühl et al . ( 2014 ) ; Shanechi ( 2019 ) ) . Various studies have demonstrated that the electroencephalography ( EEG ) signal is especially reliable to recognize human emotion in the subject-dependent emotion model , in which the training and test data are from one subject ( Jenke et al . ( 2014 ) ; Alarcão & Fonseca ( 2019 ) ) . However , due to the non-stationary nature of EEG signal and structural variability between different subjects , subject-independent model based on the assumption of Independent Identity Distribution ( i.i.d . ) usually shows bad generalization performance in real aBCIs applications , which is called the problem of Domain Shift ( Sugiyama et al . ( 2007 ) ; Samek et al . ( 2013 ) ; Sussillo et al . ( 2016 ) ; Zheng & Lu ( 2016 ) ; Li et al . ( 2018d ) ) . Domain adaptation ( DA ) is one of the promising ways to solve this problem . DA uses data from both source and target domain to promote the adaption performance . One of the most sufficient studied DA is mapping the two distributions to one common feature space where they have the same marginal distribution . Though DA has demonstrated significant success in subject-independent EEG-based emotion recognition ( Zheng & Lu ( 2016 ) ; Li et al . ( 2018b ; c ) ; Luo et al . ( 2018 ) ) , the additional computation and time to apply the DA methods is an exasperating problem in real-world scenarios and causes poor user experience . As a consequence , the concept of domain generalization ( DG ) arises in situations where multiple source domains can be accessed but unlabeled target samples are not available . The DG methods have also been successfully adopted to build the subjectindependent emotion model ( Ma et al . ( 2019b ; a ) ) . However , since there is no prior information about the target domain during training , it ’ s challenging for DG to perform as promising as DA . A compromise solution is Fast Domain Adaptation ( FDA ) , which trains the main model in advance and uses a small amount of test samples to adjust efficiently . Even though FDA methods can avoid consuming time cost by adaptation , most of them need to storage both source and target domains in the test phase ( Chai et al . ( 2017 ) ; Zhao et al . ( 2021 ) ) , which needs extra storage space and causes poor portability . However , in real-world applications , whether a EEG-based affective model can adapt to different subjects quickly as well as keep its portability does matter . In this paper , we propose a new paradigm called Pseudo Domain Adaptation ( PDA ) for subjectindependent EEG-based emotion recognition . Compared to typical FDA , only target domain is required in the test phase of PDA , which makes PDA capable to output the prediction more quickly than DA and FDA . From the perspective of the real-world application , PDA is more suited to build subject-independent affective model . To implement PDA , we first prove that with the sum-decomposable structure , a network is equivalent to domain discrepancy metrics in traditional DA methods like MMD orH-divergence . Based on our theory , we propose a method named Pseudo Domain Adaptation via Meta Learning ( PDAML ) . The PDAML consists of a feature extractor , a classifier , and a sum-decomposable structure called domain shift governor . By taking advantage of the adversarial learning and meta-learning , the governor helps PDAML quickly generalize to a new domain using the target data through a few self-adaptive steps in the test phase . The main contributions of this paper can be summarized as follows : • We propose a more suitable paradigm to build subject-independent affective model . • We prove that a sum-decomposable network is equivalent to domain discriminator for rep- resenting any type of domain discrepancy in theory . • We propose PDAML that is portable and can fast adapt to different subjects in EEG-based emotion recognition . • We conduct extensive experiments on the SEED dataset1 , which is a public available EEG- based affective dataset . The experimental results demonstrate that our method has better performance than DG methods . From the perspective of time and storage cost , the PDAML performs as well as DG methods . 2 RELATED WORK . aBCIs have received considerable attention very recent . Mühl et al . presented the definition of aBCIs ( Mühl et al . ( 2014 ) ) by introducing the affective factors into traditional brain-computer interfaces ( BCIs ) ( Zander & Jatzev ( 2011 ) ) . More applied works focus on studying EEG-based emotion recognition . Zheng and Lu recruited 15 subjects to watch 15 selected Chinese movie clips to elicit three emotions : happy , neutral and sad . They developed a public emotion dataset called SEED by recording the EEG signals of the subjects ( Zheng & Lu ( 2015 ) ) . Based on the SEED dataset , researchers have made great progress in developing EEG-based emotion recognition model , especially for subject-dependent model . Due to the non-stationary nature of EEG signal and structural variability between different subjects , it is hard to develop subject-independent EEG-based emotion recognition model by directly using typical machine learning approaches . Researchers have focused on applying DA and DG methods to subject-independent EEG-based emotion recognition . Typical DA methods are discrepancy-based , and they alleviate the domain shift problem by minimizing traditional metrics , such as Maximum Mean Discrepancy ( MMD ) ( Pan et al . ( 2011 ) ; Long et al . ( 2017 ) ; Wang et al . ( 2018 ) ) , KullbackLeibler ( KL ) divergence ( Zhuang et al . ( 2015 ) ) , andH-divergence ( Bendavid et al . ( 2010 ) ) , between the different domains . Zheng and Lu adopted transfer component analysis ( TCA ) ( Pan et al . ( 2011 ) ) , which minimizes MMD ( Gretton et al . ( 2007 ) ) between two domains by constructing kernel matrix , and successfully built personalized EEG-based emotion models ( Zheng & Lu ( 2016 ) ) . Recently , adversarial DA methods have made great successes in different fields ( Ganin & Lempitsky ( 2015 ) ; Tzeng et al . ( 2017 ) ; Shen et al . ( 2018 ) ) . The basic idea of the adversarial training is similar to generative adversarial networks ( GANs ) ( Goodfellow et al . ( 2014 ) ) , which play an adversarial 1http : //bcmi.sjtu.edu.cn/∼seed/index.html game to make the generated distribution approximate to the real distribution . After the adversarial training , the data distribution of the target domain is similar to the source domain , and the domain shift is diminished . Researchers have successfully adopted adversarial DA methods to aBCIs . Li et al . ( Li et al . ( 2018b ) ) adopted domain-adversarial neural networks ( DANN ) ( Ganin & Lempitsky ( 2015 ) ) to EEG-based emotion recognition and improved the recognition accuracies of subjectindependent models . And they ( Li et al . ( 2018c ) ) also achieved significant performance in subjectindependent vigilance estimation by implementing DANN and adversarial discriminative domain adaptation ( ADDA ) ( Tzeng et al . ( 2017 ) ) . Luo et al . proposed Wasserstein GAN ( Arjovsky et al . ( 2017 ) ) adversarial domain adaptation ( WGANDA ) and successfully adopted it to build subjectindependent emotion recognition models ( Luo et al . ( 2018 ) ) . In real-world aBCIs applications , each subject can be viewed as an individual domain . DA methods , which require high additional computation resources for each new domain , hinder the development of aBCIs from lab to real scenarios . DG methods , which can be utilized by data manipulation , representation learning , or meta-learning ( Wang et al . ( 2021 ) ) , aim to generalize to unseeing target domains without additional data collection from target domains . Researchers have focused on adopting DG methods to aBCIs very recently . Ma et al . generalized the structure of DANN into DG and proposed an adversarial structure called Domain Residual Network ( DResNet ) . They adopted DResNet to subject-independent EEG-based vigilance estimation and emotion recognition ( Ma et al . ( 2019b ; a ) ) . The experimental result demonstrated that the DG method could improve the generalization ability without data collection from the target domains . However , the DA methods usually give relatively promising results than DG methods in aBCIs applications . As a compromise way , FDA was adopted in aBCIs . Zhao et al . proposed a Plug-and-Play Domain Adaptation ( PPDA ) method to fast adapt the model in EEG-based emotion recognition ( Zhao et al . ( 2021 ) ) . Meta-learning , also known as learning to learn , has received a resurgence in interest recently with applications , one of which is domain generalization . Meta-learning aims to learn episodes sampled from the related tasks ( Finn et al . ( 2017 ) ) . Meta-Learning for Domain Generalization ( MLDG ) first introduces meta-learning strategy to DG ( Li et al . ( 2018a ) ) , then MetaReg ( Balaji et al . ( 2018 ) ) and Feature-Critic ( Li et al . ( 2019 ) ) are subsequently proposed to enhance the model ’ s generalizing ability by introducing an auxiliary loss in the training . Compared to most previous DG work that designs a specific model , meta-learning-based DG methods focus on model agnostic training strategy by exposing the model to domain shift in the training phase . To the best of our knowledge , we are the first to introduce meta-learning based methods to aBCIs tasks . 3 THEORY . The motivation of PDAML is using a simple network to compute domain shift , taking only the target domain as input . Traditional DA methods usually taking two domains as input when comparing the target domain with a specific source domain . Thus they need extra storage space for source data and sophisticated methods ( e.g . GAN ) to represent domain shift in the test phase , which is time-consuming and storage-consuming . Either or both of these problems obstruct the practical application of DA method in EEG-based diagnosis . However , in multi-source setting , we will show that minimizing the discrepancy between all pairwise domains is equivalent to minimizing the discrepancy between each domain and an implicit domain . Additionally , we will prove that a network with what is termed sum-decomposition form can represent any domain shift metrics in theory .

Summary: A new paradigm is called Pseudo Domain Adaptation (PDA) is proposed for subject-independent EEG-based affective computing. The method performs the pseudo-domain adaptation via meta-learning based on PDA and uses a feature extractor, a classifier, and a domain shift governor.  The results show that the proposed method doesn't require the additional computation resources of DA methods but achieves similar generalization performances.