MetaPhys: Few-Shot Adaptation for Non-Contact Physiological Measurement

1 INTRODUCTION . The importance of scalable health sensing has been acutely highlighted during the SARS-CoV-2 ( COVID-19 ) pandemic . The virus has been linked to increased risk of myocarditis and other serious cardiac ( heart ) conditions ( Puntmann et al. , 2020 ) . Contact sensors ( electrocardiograms , oximeters ) are the current gold-standard for measurement of heart function . However , these devices are still not ubiquitously available , especially in low-resource settings . The development of video-based contactless sensing of vital signs presents an opportunity for highly scalable physiological monitoring . Furthermore , in clinical settings non-contact sensing could reduce the risk of infection for vulnerable patients ( e.g. , infants and elderly ) and the discomfort caused to them ( Villarroel et al. , 2019 ) . While there are compelling advantages of camera-based sensing , the approach also presents unsolved challenges . The use of ambient illumination means camera-based measurement is sensitive to environmental differences in the intensity and composition of the incident light . Camera sensor differences mean that hardware can differ in sensitivity across the frequency spectrum . People ( the subjects ) exhibit large individual differences in appearance ( e.g. , skin type , facial hair ) and physiology ( e.g , pulse dynamics ) . Finally , contextual differences mean that motions in a video at test time might be different from those seen in the training data . One specific example is that there exists biases in performance across skin types Nowara et al . ( 2020 ) . This problem is not isolated to physiological measurement as studies have found systematic biases in facial gender classification , with error rates up to 7x higher on women than men and poorer performance on people with darker skin types ( Buolamwini & Gebru , 2018 ) . Moreover , there are several challenges in collecting large corpora of high-quality physiological data : 1 ) recruiting and instrumenting participants is often expensive and requires advanced technical expertise , 2 ) the data can reveal the identity of the subjects and/or sensitive health information meaning it is difficult for researchers to share such datasets . Therefore , training supervised models that generalize well across environments and subjects is challenging . For these reasons we observe that performance on cross-dataset evaluation is significantly worse than within-dataset evaluation using current state-of-the-art methods ( Chen & McDuff , 2018 ; Liu et al. , 2020 ) . Calibration of consumer health sensors is often performed in a clinic , where a clinician will collect readings from a high-end sensor to calibrate a consumer-level device the patient owns . The reason for this is partly due to the variability within readings from consumer devices across different individuals . Ideally , we would be able to train a personalized model for each individual ; however , standard supervised learning training schemes require large amounts of labeled data . Getting enough physiological training data of each individual is difficult because it requires using medical-grade devices to provide reliable labels . Being able to generate a personalized model from a small amount of training samples would enable customization based on a few seconds or minutes of video captured while visiting a clinic where people have access to a gold-standard device . Furthermore , if this process could be achieved without even the need for these devices ( i.e. , in an unsupervised manner ) , that would have even greater impact . Finally , combining remote physiological measurement with telehealth could provide patients ’ vital signs for clinicians during remote diagnosis . Given that requests for telehealth appointments have increased more than 10x during COVID-19 , and that this is expected to continue into the future ( Smith et al. , 2020 ) , robust personalized models are of growing importance . Meta-learning , or learning to learn , has been extensively studied in the past few years ( Hospedales et al. , 2020 ) . Instead of learning a specific generalized mapping , the goal of meta-learning is to design a model that can adapt to a new task or context with a small amount of data . Due to the inherent ability for fast adaption , meta-learning is a good candidate strategy for building personalized models ( e.g. , personalization in dialogue and video retargeting ( Madotto et al. , 2019 ; Lee et al. , 2019 ) . ) However , we argue that meta learning is underused in healthcare where clinicians can quickly adapt their clinical knowledge to different patients . The goal of this work is to develop a meta-learning based personalization framework in remote physiological measurement with a limited amount of data from an unseen individual ( task ) to mimic how a clinician manually calibrates sensor readings for a specific patient . When meta-learning is applied to remote physiological measurement , there are two kinds of scenarios : 1 ) supervised adaptation with few samples of labeled data from a clinical grade sensor and 2 ) unsupervised adaptation with unlabeled data . We hypothesize that supervised adaptation is more likely to yield a robust personalized model with only a few labels , while unsupervised adaptation may personalize the model less effectively but with much lower effort and complexity . In this paper , we propose a novel meta-learning approach to address the aforementioned challenges called MetaPhys . Our contributions are : 1 ) A meta-learning based deep neural framework , supporting both supervised and unsupervised few-shot adaptation , for camera-based vital sign measurement ; 2 ) A systematic cross-dataset evaluation showing that our system considerably outperforms the state-of-the-art ( 42 % to 52 % reduction in heart rate error ) ; 3 ) To perform an ablation experiment , freezing weights in the temporal and appearance branches to test sensitivity during adaptation ; 4 ) An analysis of performance for subjects with different skin types . Our code , example models , and video results can be found on our github page.1 2 BACKGROUND . Video-Based Physiological Measurement : Video-based physiological measurement is a growing interdisciplinary domain that leverages ubiquitous imaging devices ( e.g. , webcams , smartphones ’ cameras ) to measure vital signs and other physiological processes . Early work established that changes in light reflected from the body could be used to capture subtle variations blood volume and motion related to the photoplethysmogram ( PPG ) ( Takano & Ohta , 2007 ; Verkruysse et al. , 2008 ) and ballistocardiogram ( BCG ) ( Balakrishnan et al. , 2013 ) , respectively . Video analysis enables non-contact , spatial and temporal measurement of arterial and peripheral pulsations and allows for magnification of theses signals ( Wu et al. , 2012 ) , which may help with examination ( e.g. , ( Abnousi et al. , 2019 ) ) . Based on the PPG and BCG signal , heart rate can be extracted ( Poh et al. , 2010b ; Balakrishnan et al. , 2013 ) . However , the relationship between pixels and underlying physiological changes in a video is complex and neural models have shown strong performance compared to source separation techniques ( Chen & McDuff , 2018 ; Yu et al. , 2019 ; Zhan et al. , 2020 ) . Conventional supervised learning requires a large amount of training data to produce a generalized model . However , obtaining a large body of physiological and facial data is complicated and expensive . Current public datasets have limited 1https : //github.com/anonymous0paper/MetaPhys numbers of subjects and diversity in regards of appearance ( including skin type ) , camera sensors , environmental conditions and subject motions . Therefore , if the subject of interest is not in the training data or the video is otherwise different , performance can be considerably degraded , a result that is not acceptable for a physiological sensor . Lee et al . ( Lee et al. , 2020 ) recognized the potential for meta-learning applied to imaging-based cardiac pulse measurement . Their method ( Meta-rPPG ) focuses on using unsupervised meta-learning and a LSTM encoder-decoder architecture which to our knowledge was not validated in previous work . Instead , our proposed meta-learning framework is built on top of a state-of-the-art on-device network ( Liu et al. , 2020 ) and aims to explore the potential of both supervised and unsupervised ondevice personalized meta-learning . Meta-rPPG uses a synthetic gradient generator and a prototypical distance minimizer to perform transductive inference to enable unsupervised meta-learning . This learning mechanism requires a number of rather complex steps . We propose a relatively simpler mechanism that is physiologically and optically grounded ( Wang et al. , 2016 ; Liu et al. , 2020 ) and achieves greater accuracy . Meta-Learning and Person Specific Models : The ability to learn from a small number of samples or observations is often used as an example of the unique capabilities of human intelligence . However , machine learning systems are often brittle in a similar context . Meta-learning approaches tackle this problem by creating a general learner that is able to adapt to a new task with a small number of training samples , inspired by how humans can often master a new skill without many observations ( Hospedales et al. , 2020 ) . However , most of the previous work in meta-learning focuses on supervised vision problems ( Zoph et al. , 2018 ; Snell et al. , 2017 ) and in the computer vision literature has mainly been applied to image analysis ( Vinyals et al. , 2016 ; Li et al. , 2017 ) . Supervised regression in video settings has received less attention . One of few examples is object or face tracking ( Choi et al. , 2019 ; Park & Berg , 2018 ) . In these tasks , the learner needs to adapt to the individual differences in appearance of the target and then track it across frames , even if the appearance changes considerably over time in the video . Choi et al . ( 2019 ) present a matching network architecture providing the meta-learner with information in the form of loss gradients obtained using the training samples . The property of fast adaptation makes meta-learning a good candidate for personalizing models , it has been used in various applications such as dialogue agents ( Madotto et al. , 2019 ) , gaze estimation ( He et al. , 2019 ) , sleep stage classification ( Banluesombatkul et al. , 2020 ) , activity recognition ( Gong et al. , 2019 ) , and video retargeting ( Lee et al. , 2019 ) . For example , Banluesombatkul et al . proposed a MAML-based meta-learning system to perform fast adaption of a sleep stage classification model using biosignals ( Banluesombatkul et al. , 2020 ) . More recently , MetaPix ( Lee et al. , 2019 ) leveraged a meta-learning training schema with a small amount of video to adapt a universal generator to a particular background and human in the problem of video retargeting . Similarly , our proposed meta-learning framework is also capable of personalizing a universal remote physiological model to a new person or an environmental setting . 3 METHOD . 3.1 PHYSIOLOGICAL META-LEARNING . In camera-based cardiac measurement , the goal is to separate pixel changes due to volumetric variations in blood and pulsatile motions from other variations that are not related to the pulse signal . Examples of “ noise ” in this context that might impact the performance on the task include : changes in the environment ( illumination ) and changes in appearance of the subject and motions ( e.g. , facial expressions , rigid head motions ) . A model trained within a traditional supervised learning regime might perform well if illumination , non-pulsatile motions , and appearances in the test set are similar to those in the training set . However , empirical evidence shows that performance usually significantly degrades from one dataset to another , suggesting that traditional training is likely to overfit to the training set to some extent ( Chen & McDuff , 2018 ) . Therefore , to achieve state-of-the-art performance in remote physiological measurement on cross-dataset evaluation , the system should have : 1 ) a good initial representation of the mapping from the raw video data to pulse signal , and 2 ) a strategy for adapting to unseen individuals and environments . To achieve this , we propose a system called MetaPhys , an adaptable meta-learning based on-device framework aimed at efficient and personalized remote physiological sensing . MetaPhys uses a pretrained convolutional attention network as the backbone ( described below ) and leverages a novel personalized meta-learning schema to overcome the aforementioned limitations . We adopt ModelAgnostic Meta-Learning ( MAML ) ( Finn et al. , 2017 ) as our personalized parameter update schema . MAML produces a general initialization as the starting point for fast adaptation to a diverse set of unseen tasks with only a few training samples . However , applying MAML to the task of camera-based physiological measurement has differences to many previously explored meta-learning problems . Existing meta-learning approaches are often evaluated on classification or some toy regression tasks due to the lack of regression benchmark datasets ( Hospedales et al. , 2020 ) . Our problem is a nontrivial vision-based regression task due to the subtle nature of the underlying physiological signal . Algorithm 1 outlines the training process for MetaPhys , we first pretrain the backbone network to get an initial spatial-temporal representation . Then we treat each individual as a task τi . During the training , we split the data into a support set ( K video frames ) and a query set ( K ′ video frames ) for each individual ( task ) . The support set is used to update the task ’ s parameters and yield a personalized model θi . The query set is used to assess the effectiveness of the personalized model and further update the global initialization θ to make future adaptation better . A robust personalized model θi aims to provide a more accurate attention mask to the corresponding motion branch and to preform precise physiological measurement for the target individual as well as the target ’ s environment . During the testing stage , MetaPhys has the updated global initialization θ̂ , and can generate θ̂i for each test individual ( task ) by optimizing the test support set as θ̂τi ← θ̂ − α∇θ̂Lτif ( θ̂ ) . With this training and testing schema , the robust global initialization θ̂ generated from MetaPhys not only leverages the pretrained representation but also learns how to adapt to individual and environmental noise quickly .

Summary: The authors propose a system called MetaPhys for personalized remote physiological sensing from videos.  Their system combines a pre-trained CNN with an existing meta learning method (MAML). The investigated both supervised and unsupervised training of their system.  Performance evaluation of their methods on benchmark datasets show their model significantly outperforms SOTA methods using multiple metrics as well as for different skin types. They further show that the unsupervised model achieves comparable results to the supervised model.