Pain estimation with integrating global‐wise and region‐wise convolutional networks

IET Image ProcessingVolume 17, Issue 3 p. 637-648 ORIGINAL RESEARCHOpen Access Pain estimation with integrating global-wise and region-wise convolutional networks Dong Huang, Dong Huang orcid.org/0000-0002-9746-0032 School of Biomedical Engineering, Air Force Medical University, Xi'an, China Shaanxi Provincial Key Laboratory of Bioelectromagnetic Detection and Intelligent Perception, Xi'an, China Contribution: Methodology, Writing - original draft, Writing - review & editingSearch for more papers by this authorZhaoqiang Xia, Zhaoqiang Xia School of Electronics and Information, Northwestern Polytechnical University, Xi'an, China Contribution: Methodology, Project administrationSearch for more papers by this authorLei Li, Lei Li School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, China Contribution: Software, ValidationSearch for more papers by this authorYupeng Ma, Corresponding Author Yupeng Ma [email protected] School of Future Information Technology, Shijiazhuang University, 288 Zhufeng Street, Shijiazhuang, China Correspondence Yupeng Ma, School of Future Information Technology, Shijiazhuang University, 288 Zhufeng Street, Shijiazhuang, China Email: [email protected] Contribution: Formal analysis, Funding acquisitionSearch for more papers by this author Dong Huang, Dong Huang orcid.org/0000-0002-9746-0032 School of Biomedical Engineering, Air Force Medical University, Xi'an, China Shaanxi Provincial Key Laboratory of Bioelectromagnetic Detection and Intelligent Perception, Xi'an, China Contribution: Methodology, Writing - original draft, Writing - review & editingSearch for more papers by this authorZhaoqiang Xia, Zhaoqiang Xia School of Electronics and Information, Northwestern Polytechnical University, Xi'an, China Contribution: Methodology, Project administrationSearch for more papers by this authorLei Li, Lei Li School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, China Contribution: Software, ValidationSearch for more papers by this authorYupeng Ma, Corresponding Author Yupeng Ma [email protected] School of Future Information Technology, Shijiazhuang University, 288 Zhufeng Street, Shijiazhuang, China Correspondence Yupeng Ma, School of Future Information Technology, Shijiazhuang University, 288 Zhufeng Street, Shijiazhuang, China Email: [email protected] Contribution: Formal analysis, Funding acquisitionSearch for more papers by this author First published: 14 December 2022 https://doi.org/10.1049/ipr2.12639AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Abstract Pain is a common phenomenon in clinical patients, which indicates patients are suffering from uncomfortable conditions for necessary treatments. So the assessment of pain status becomes a significant task in current medical institutions. Of late, various conventional hand-crafted or deep learning methods on face images are presented to estimate pain intensity automatically. However, these approaches usually feed the whole face into the automatic estimation system and explore little information on the interdependencies of related regions during the formation of pain expression. In this paper, a hierarchical deep network (HDN) involving regional and holistic information simultaneously is proposed via two scale branches. In HDN, a region-wise branch is designed to extract features from pain related regions of face images while a global-wise branch explores the interdependencies of pain related regions. Besides, in global-wise branch, a multi-task learning method is employed to detect action units while estimating pain intensity. Finally, the pain estimation outputs of two branches are fused in a decision level. On current pain estimation benchmarks, it is empirically shown that the proposed HDN outperforms the existing methods and the essential components in HDN have key influences on final prediction. 1 INTRODUCTION In clinical medicine, pain is a common response of patients to medical conditions. Given the harm that pain brings to patients, it is defined as the fifth vital sign following the four basic vital signs. If the medical institutions have no effective way to evaluate pain states of patients but to implement some treatments blindly, it may lead to catastrophic medical blunders [1]. Therefore, an accurate pain estimation is crucial in medical treatments. In current hospitals, the self-report and observer assessment are main physical ways to estimate pain intensity. Self-report requires patients to dictate 0 -- 10 $0\text{--}10$ pain level experience within a specified time [2]. But this method severely affects patients with limited communication abilities, who cannot report their pain feelings or whose report has low validity. Such vulnerable groups, who are often under or over-treated, include infants, toddlers, children, adults with cognitive impairments (for instance, advanced dementia), people with intellectual disability, critical ill or unconscious persons, and patients who have terminal ill [3]. On the other hand, the observer assessment requires uninterrupted escorts by professional medical personnel or relatives, which is extremely inefficient. Therefore, the pain estimation system that could automatically evaluate the pain intensity of patients by analysing some images or videos becomes necessary for modern medical care [4]. In contrast to traditional assessment methods, they could monitor pain states continuously. This may improve clinical outcomes, e.g. by providing timely intervention for patients who suffer emergency but cannot call for help by themselves. Moreover, an automatic system is more objective than a human observer, whose judgement is influenced by personal factors, such as the relation to the suffers or the patient's attractiveness [5]. These techniques are not only important to provide comforts but to prevent both immediate and long-lasting consequences that are harmful to the person's overall health. Consequently, the automatic pain estimation system is the most significant part in current intelligent health care. Over the past decades, various computer vision techniques are used to address the issue of pain estimation. Like earlier traditional approaches, they used hand-crafted features, e.g. local binary patterns (LBP) [6, 7], histograms of topographical (HoT) [8], discrete cosine transform (DCT) [6], shape features [6, 9, 10], appearance features [6, 9, 11], Gabor features [12], and even biological features [7, 13]. Correspondingly, some classical classifiers were introduced, such as conditional ordinal random field (CORF) [14], support vector machines (SVM) [6, 9, 10, 15], relevance vector regression (RVR) [6], weakly supervised learning [16], auto-encoders [11], and multi-task learning [17]. Of late, due to the compelling performance of deep neural networks, deep model-based pain research has been received more attention. Such as, convolutional neural networks (CNNs) [18], recurrent convolutional neural network (RCNN) [19], the combination of CNNs and RNNs [20, 21], and 3D CNNs [22, 23] were leveraged in this task. They used deep learning techniques and achieved more impressive results than traditional methods. So, these deep works lead the trend of pain estimation research. However, pain estimation is still a challenging task. Because the generation of pain actually is the synthesis of several regional movements on face, and pain expression often involves the relationship of pain related regions [24]. In deep pain works above [18-21, 23], they simply feed the entire face image into the network, overlooking the essence of pain expression. In ref. [25], they started investigating separated related regions in face images, while they ignored the interdependencies of these regions. Consequently, one key cue to this issue is that local region movements and the relationship of pain related regions need to be involved simultaneously. Based on aforementioned analysis, in this paper, we propose a hierarchical deep network (HDN) for pain estimation. In our framework, there are two scale branches: region-wise branch and global-wise branch. The region-wise branch utilises a multi-stream CNN to extract local features that describe the regional pain movements on face. In this branch, pain is estimated based on pain related regions. In the global-wise branch, holistic features are extracted by a pre-trained CNN model from faces. Significantly, a multi-task learning method that consists of pain estimation and action units (AUs) detection is also employed when extracting holistic features. The task of AUs detection allows pain estimation consider the relationship of pain related regions. On the other hand, the results of auxiliary AU detection are used to further improve the performance of pain estimation. These two branch outputs of pain estimation are fused in a decision-level for the final prediction. Among the salient contributions of this paper, we can cite: We present a HDN that involves local information and holistic information in an end-to-end framework. This framework analyses pain expression from the related regions and regional relationships simultaneously. In the region-wise branch of HDN, a multi-stream CNN is utilised to extract different regional features with various methods. Pain estimation can be implemented on pain-related regions. The global-wise branch of HDN fine-tunes a pre-trained CNN model to represent holistic features. Two related tasks are completed by a multi-task architecture at the same time. And a promoter is devised for the use of auxiliary-task information, so as to make our holistic pain estimation more accurate. The remaining of this paper is organised as follows. In Section 2, some earlier works are reviewed for pain estimation. Section 3 presents the proposed model architectures and theories in detail, and the results of comparative experiments are illustrated in Section 4. Finally, Section 5 concludes this paper. 2 RELATED WORK Over the past few decades, there have been many works for pain estimation. From the perspective of methods, they mainly include two categories: traditional hand-crafted methods and deep learning methods. Traditional hand-crafted methods Since hand-crafted features are stable and effective, inspired by the works of facial expression, many pain researches introduced these features in the task of pain estimation. For instance, in ref. [9], active appearance model (AAM) was employed to extract shape and appearance features, which were classified by SVM. Littlewort et al. [26] trained several classifiers of AUs and used the outputs of classifiers to distinguish the real or fake pain. Their automatic recognition system achieved better results than human discrimination, and gave the most discriminative AUs about real pain. As stated by Niese et al. [27], their experiments on pain not only used the colour and gradient information but also introduced 3D geometric features at the first time. Lucey et al. [10] used the same method to give a baseline experiment on UNBC database which was released by their team. Kaltwang et al. [6] extracted three kinds of features: shape features, LBP, DCT. As a classifier, RVR was used to train the features as well as the combinations of them. In [15], Werner et al. gave an intensity model based on SVM and a function model to recognise pain and estimate intensity. It has been approached by Hammal et al. in ref. [28], where they used the combination of facial features, sensor signals, and environmental background as dynamic pain features. Similar to them, in ref. [29] and [7], Kächele et al. also applied multimodal features which consisted of LBP and bio-physiological features. According to ref. [30], a set of log-normal filters were used to extract features which were used to detect pain level. Sikka et al. in refs. [16] and [31] used weakly supervised learning to estimate pain. In their approach, only sequence-level labels were used. Latter, in order to overcome individual variances, Rudovic et al. [32] employed the heteroscedastic CORF in a pain estimation model for being adaptive to individual changes. Then, they extended the above experiment and proposed a novel CORF model for AU to estimate the intensity of AU [14]. Appearance-based features and auto-encoders were employed to detect pain in ref. [11]. Werner et al. [13] extracted features from the video sequence, and they involved physiological signals in the database, like galvanic skin response, electromyography, and electrocardiogram. In 2014, Bartlett et al. [33] focused on the identification of genuine and faked pain expressions in social interactions. In the experiment, they developed computer expression recognition toolbox (CERT) [34] to extract features. In ref. [35], a steerable and separable filter was used to measure the energy released by the facial muscles. And the measured results were employed as a representative feature of pain to distinguish dynamic pain and their levels. As stated in ref. [36], they paid more attention on the methods of machine learning. An efficient optimisation algorithm based on alternating direction method of multipliers (ADMM) was developed. Then, they tested their methods under fully supervised, weakly supervised, and unsupervised conditions. Werner et al. [37] analysed the relative positional changes of the face due to the movement of some points during the pain production process. Based on this, facial activity descriptors were proposed to describe the pain. Focusing on self-reported pain, Liu et al. [17] used a set of hand-crafted personal features and multi-task learning to personalise the pain of each subject. As can be seen, various hand-crafted features are applied in the task of pain estimation. Correspondingly, many classical classifiers are leveraged to discriminate these features. They implemented the proposed approaches to a sufficiently detailed level, for instance, in refs. [14, 26], the relationship between AUs and pain have been considered. And in refs. [32, 36], some impressive mathematical optimisation methods have been proposed. However, it is obvious that there have been fewer hand-crafted methods in recent years. Since these features are designed for a certain problem, they have limited the generalisation ability to some extent. Instead of continuing this direction, we could find some inspirations from these works, such as the exploration of the relationship between AUs and pain. Deep learning methods Compared to traditional hand-crafted works, the deep learning methods obtain targeted features and use an adaptive network to classify the pain categories. The end-to-end automatic learning approach makes many tasks in computer vision easier and could achieve better performance [38]. With the increase of pain data, the works based on deep models develop rapidly for the task of pain estimation. For instance, Zhou et al. [19] proposed a RCNN framework to consider the information of successive image frames and used a regression method to express pain intensity. In ref. [39], they devised a combination of hand-crafted and deep-learned features that makes the most of deep learning techniques in a small database. Similar to ref. [17], Martinez et al. [40] employed LSTM to estimate pain intensity. And the hidden conditional random field (HCRF) was used to output the result of visual analogue scales (VAS). Rodriguez et al. [20] approached pain task by using the combination of CNNs and long short-term memory (LSTM). Specifically, a pre-trained VGG network is used to fine tune in the pain database, and LSTM classifies the features of the VGG network output. Then, Bellantonio et al. [41] employed the super-resolution (SR) method before the combined network, which further improved the performance of strong pain recognition. In ref. [18], they leveraged a CNN to extract features from segments. Then, these features were aggregated by exploiting statistics to obtain a pain intensity. Also, in ref. [42], they employed a novel 3D CNNs architecture to capture a wide range of spatiotemporal variations on the faces. Wang et al. [23] employed the support vector regression (SVR) to estimate pain from three kinds of features: 3D CNN features, histogram of oriented gradients features and geometric features. It has been approached by ref. [21], where they used two RNNs to exploit the dynamic information from the VGGface features. In ref. [25], they tended to separate the pain areas of the face image to form the pain-awareness features. Huang et al. [43] proposed a hybrid network which consists of 1D CNN, 2D CNN, and 3D CNN to extract multidimensional features from faces for pain estimation. As stated in ref. [44], various popular deep architectures were tried for the comparison. In ref. [45], they employed spatial transformation network and attention mechanisms to find pain-related features. Also, there are some deep learning works for physiological signals. Like in ref. [46], they implemented a multi-task learning approach based on physiological signals. As stated in ref. [47], they employed skin conductance deconvolution and point process heart rate variability analysis to continuously evaluate time-varying autonomic parameters, and presented a pain intensity estimation regression algorithm based on RNNs. Jiang et al. [48] trained some artificial neural network classifiers for four physiological signals: heart rate, breath rate, galvanic skin response, and facial surface electromyogram. From these works, we can see that in recent years, the research of pain estimation becomes a concern. However, these mentioned studies still have some shortcomings. Some works involving physiological cues, for example, have received little attention. Because the increased expense of pain quantification by medical institutions will result from the continued usage of physiological signal collection equipment. Furthermore, the patient's treatment experience must be harmed by lugging a variety of equipment for an extended period of time. Besides, the deep pain methods for visual signals becomes a trend, since visual information is the easier to acquire. And these often obtain better experimental performance than traditional approaches, since the learnt features are more targeted than hand-crafted features. However, in these deep pain works for visual signals, most of them take the whole face as the input of the deep network, only [25] limits the input to the local regions in the face. Whereas, some traditional hand-crafted methods, e.g. refs. [6, 14], have shown the relationship of these pain related regions is helpful for estimating pain. In addition, different from ref. [45], we would like try a more straightforward solutions from the definition of pain intensities. For this reason, the cue to pain estimation task is to devise a deep network that could extract deep features from pain related regions and also analyse the interdependencies of these regions automatically, which may have a better performance than the existing methods. 3 PROPOSED METHOD In this paper, we propose a HDN to estimate pain intensity more accurate. In the following, we introduce our proposed HDN from four parts, i.e. the framework, the region-wise branch, the global-wise branch and the parameter learning of entire model. The framework Four facial actions, namely brow lowering (AU4), orbital tightening (AU6 and AU7), levator contraction (AU9 and AU10), and eye closure (AU43), are carried during the formation of pain [49]. Then, a criterion called Prkachin and Solomon Pain Intensity (PSPI) was presented to express the information of pain [50]. Equation (1) can be used to express the proposed PSPI. PSPI = AU 4 + max AU 6 , AU 7 + max AU 9 , AU 10 + AU 43 $$\begin{eqnarray} \text{PSPI}& =& \text{AU}4+\max\left(\text{AU}6,\text{AU}7\right)\nonumber\\ & & +\, \max\left(\text{AU}9,\text{AU}10\right)+\text{AU}43 \end{eqnarray}$$ (1) These actions regarding the pain is also visualised in Figure 1. From the above equation and figure, there are two conclusions. Firstly, pain expression is regional, i.e. pain is usually related to some specific regions. Secondly, pain intensity is decided by the relationship of these regions. Therefore, we propose a two-scale framework to describe these regions and their relationship for pain estimation, which is shown in Figure 2. The HDN consists of two different-scale branches: region-wise branch and global-wise branch. In the region-wise branch, we isolate the most relevant regions of pain and then estimate pain from these regions. In the global-wise branch, we perform two related tasks to explore the relevance of pain related regions. In the end, two branches fuse their results in a decision-level way for a more comprehensive pain estimation. FIGURE 1Open in figure viewerPowerPoint Pain expression FIGURE 2Open in figure viewerPowerPoint The framework of proposed HDN. There are two scale branches for pain analysis: region-wise branch and global-wise branch. The region-wise branch estimates pain from related regions in the face. The global-wise branch estimates pain from the relationship of related AUs The region-wise branch Owing to the locality of pain on face, this section focuses on pain related regions to estimate the intensity of pain. From the analysis, we conclude that pain expression is often related to the actions of eyes, nose, and mouth regions. They are input into region-wise branch to describe local variations. Therefore, we first segment these pain-related regions. Because the AAM landmarks provided by database describe the location of organs in the face, and they are always arranged in a certain order. Thus, according to the landmarks contained in each organ, these points could be used to segment the pain related regions. Then, we devise a multi-stream CNN for each pain-related region, as shown in Figure 2. These streams have same network structure since these regions have same input size. With the deep networks training, different parameters will be learned in each stream. Hence, each region has individual feature extraction method. In this way, these pain-related regions are fed to corresponding stream to obtain each local feature. These local features represent pain information of each region. Another question is how to integrate these regional features. For the answer, we consider the influence of these regions on pain estimation. It is worth noting that the contribution of each region to the pain estimation is different. For this reason, these local features are concatenated with adaptive weights. And these weights are able to obtain by the network training. With this method, the importance of each region enables the multi-stream CNN to extract pain-awareness features. Then, we employ these pain-awareness features to estimate pain intensity in the region-wise branch. Notably, in this branch, we employ the same architecture in our previous work [25]. The basic units are described here, while more information can be referred in ref. [25]. The global-wise branch This paper focuses more on the global-wise branch, which is the vitally important improvement for the branch in previous section. In this section, we employ the global-wise branch of HDN to estimate pain from the whole face. This branch is a supplement for region-wise branch. As for the face segmentation, the method of [51] is taken. The purpose of holistic pain estimation is to explore the relationship of related action units. We use a pretrained model (VGGface) [52] to initialise this branch, since transfer learning can accelerate the convergence and improve the performance of the network. After our network extracts deep features, we perform two tasks: pain estimation (Task I) and action unit detection (Task II). The former is consistent with our region-wise network task, and the latter is different. When our global-wise branch estimates pain intensity, the subtask of action units detection allows it to consider the relationship of pain related action regions. Specifically, we propose two techniques in global-wise branch. The first is a multi-task learning method. Two relevant tasks are implemented to increase the comprehension of pain estimation. Section 3.3.1 details the settings of these two tasks. Also, we devise a promoter module that uses the information of Task II to improve the accuracy of Task I. This promoter module will be introduced in Section 3.3.2. 3.3.1 Multi-task learning for pain This subsection implements a multi-task learning for pain. Multi-task learning is a machine learning method based on shared representation. Multiple related tasks are put together to implemented. In the learning process (training), multi-task learning utilises a low-level shared representation to complement the task-related information for each other. This way could promote the learning process of each task, and enhance the ability of generalisation. In our case, it is expected to use pain relevant tasks to understand the nature of pain by our network. As described in the overview of this section, two related tasks are considered in this paper: pain estimation (Task I) and action unit detection (Task II). Correspondingly, we need two loss functions. Since the task of pain estimation is a classification problem, we employ a softmax classification loss. As for the task of action unit detection, it can be regarded as a multi-label binary classification problem. We introduce a multi-label loss for this task. These loss functions will be shown in Section 3.4. Thus, our global-wise network could explore the relationship of the action units when it estimates the pain intensity, so as to achieve more accurate estimation. 3.3.2 Promoter module After two relevant tasks are implemented, we consider that the information of Task II may enhance the performance of pain estimation further. In this subsection, we propose a promoter module (PM) to achieve this objective. Three structures of PM are experimented in this paper, as shown in Figure 3. We devise a PM with addition operation (P-A), concatenation (P-C), and a fully connected layer (P-F), respectively. In these structures, there are corresponding feature extraction layers in different tasks. In this figure, fa and fp mean AU detection feature and pain feature. In our experiment, they have same dimension. FIGURE 3Open in figure viewerPowerPoint Three structures of promoter module (PM). "P-A", "P-C", and "P-F" represent PM with addition operation, concatenation and a fully connected layer, respectively. And "fa" and "fp" mean AU detection feature and pain feature In P-A, the features from different tasks are added as follows: f ̂ p = f a + f p $$\begin{equation} {\widehat{\mathrm{f}}}_{p}={\mathrm{f}}_{a}+{\mathrm{f}}_{p} \end{equation}$$ (2)where f ̂ p ${\widehat{\mathrm{f}}}_{p}$ is the final pain feature that are used to estimate pain intensity. In P-C, the features of two tasks are contacted by Equation (3). f ̂ p = [ f a , f p ] $$\begin{equation} {\widehat{\mathrm{f}}}_{p}=[{\mathrm{f}}_{a},{\mathrm{f}}_{p}] \end{equation}$$ (3) The P-F is different from the previous two operations. It is employed to output pain intensity probabilities from AU detection feature. In P-F, a fully connected layer is used to accomplish this goal. The operation of P-F is described as follows: p ̂ pain = p pain ⊙ ( W PM f a + b PM ) $$\begin{equation} {\widehat{\mathrm{p}}}_{\text{pain}}={\mathrm{p}}_{\text{pain}}\odot ({\mathrm{W}}_{\textit{PM}}{\mathrm{f}}_{a}+{\mathrm{b}}_{\textit{PM}}) \end{equation}$$ (4)where W PM ${\mathrm{W}}_{\text{PM}}$ and b PM ${\mathrm{b}}_{\text{PM}}$ are the weights and bias parameter of the fully connected layer in PM, ⊙ represents the element multiplication, p pain ${\mathrm{p}}_{\text{pain}}$ and p ̂ pain ${\widehat{\mathrm{p}}}_{\text{pain}}$ denote the original and promoted pain intensity probabilities, respectively. Essentially, P-F uses the feature of AU detection to influence the decision of pain estimation. All the PM structures are tested in the experiment section (Section 4.3.3). Parameter learning Suppose a training database D = { x i , y i , p i } i = 1 , ⋯ , N $D={\{{x}_{i},{y}_{i},{p}_{i}\}}_{i=1,\text{\ensuremath{\cdots}},N}$ contains N image frames. x i ${x}_{i}$ , y i ${y}_{i}$ , p i ${p}_{i}$ denotes facial images, pain intensity ground-truth and AU ground-truth, respectively. Firstly, we train the local network with x i ${x}_{i}$ and y i ${y}_{i}$ for the local pain estimation. As for global-wise branch, to make this deep network converge faster, a pre-trained VGGface [53] model is used to initialise the parameters. Then, we illustrate the loss functions in train global-wise branch with x i ${x}_{i}$ , y i ${y}_{i}$ , and p i ${p}_{i}$ . Firstly, we employ a softmax classification loss for pain estimation (Task I) that is shown as follows: L PE = − ∑ j = 1 m y j log e a j ∑ k = 1 N e a k $$\begin{equation} {L}_{\text{PE}}=-\sum _{j=1}^{m}{y}_{j}\log\frac{{e}^{{a}_{j}}}{\sum _{k=1}^{N}{e}^{{a}_{k}}} \end{equation}$$ (5)where m denotes the size of one batch, and N is the clas