# Designing Compact Convolutional Filters for Lightweight Human Pose Estimation

Authors: Shili Niu; Weihua Ou; Shihua Feng; Jianping Gou; Fei Long; Wenchuan Zhang; Wu Zeng
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

Existing methods for human pose estimation usually use a large intermediate tensor, leading to a high computational load, which is detrimental to resource-limited devices. To solve this problem, we propose a low computational cost pose estimation network, MobilePoseNet, which includes encoder, decoder, and parallel nonmaximum suppression operation. Specifically, we design a lightweight upsampling block instead of transposing the convolution as the decoder and use the lightweight network as our downsampling part. Then, we choose the high-resolution features as the input for upsampling to reduce the number of model parameters. Finally, we propose a parallel OKS-NMS, which significantly outperforms the conventional NMS in terms of accuracy and speed. Experimental results on the benchmark datasets show that MobilePoseNet obtains almost comparable results to state-of-the-art methods with a low compilation load. Compared to SimpleBaseline, the parameter of MobilePoseNet is only 4%, while the estimation accuracy reaches 98%.


---

## Body

## 1. Introduction

Human pose estimation is also called human key point detection. Its main task is to detect the key points of human body (eyes, nose, shoulders, elbows, etc.) in a given RGB picture. Human pose estimation is one of the basic tasks of computer vision and has many practical applications, such as human-computer interaction [1], human tracking [2], and motion analysis [3]. In recent years, with the quick development of neural networks, human pose estimation based on deep neural networks [4–9] has gained a high accuracy. However, these works have focused only on improving the accuracy of pose estimation through the use of complex and computationally expensive models, while largely ignoring the issue of the cost of model inference. Many methods already require computational resources beyond the capabilities of many mobile and embedded devices. At the same time, information security is a growing concern for people, and it is important to deploy applications directly on edge devices for personal information protection, which leads to high requirements for the computational volume and complexity of human pose estimation models.Many works have been proposed to solve this problem by building human pose estimation networks with small model size and low computing cost [7, 10, 11]. For example, there is a recent attempt [10] to construct pose estimation models with fewer parameters using quantitative methods, but the performance of the obtained model largely degraded. Also, some researchers [7] try to use knowledge distillation to reduce the parameters of the model, but the model training time and deployment time are increased. On the other hand, some works attempt [12] to find a lightweight pose estimation model using neural structure search methods. However, the obtained model has a complex structure and slow inference speed. The key problem is how to balance the model accuracy and the inference efficiency.To address this problem, in this paper, we propose a lightweight human pose estimation network specifically for mobile and resource-constrained environments by designing compact convolutional filters. As shown in Figure1, our model contains three main parts: an encoder, a decoder, and a heat map regressor that estimates each key point. To keep the model lightweight, we use the first 13 layers of MobileNetV3 [13] as our encoder. Intuitively, high resolution is beneficial for human pose estimation, so we design the encoder with less downsampling, and the structure of the specific model and the comparison of SimpleBaseline can be seen in Figure 2. In the decoder part, inspired by the bottleneck block, we propose a lightweight upsampling module, whose concrete structure is shown in Figure 1. The detailed structure of the overall model is shown in Table 1. Finally, we also propose a parallel OKS-based NMS to further improve the operation speed of pose estimation. Experimental results show that our method can achieve 69.0 AP with only 1.5M model parameters and 1.23 GFLOP calculation amount under the condition of less cost. The contributions of the proposed method are summarized as follows:
(i)
We design a lightweight upsampling block that integrates separable transpose convolution and channel-based attention. This is achieved by extensively examining the upsampling modules in existing state-of-the-art deep convolutional networks(ii)
We reduce the number of upsampling and use lightweight upsampling blocks to achieve a lightweight pose estimation network. In particular, we balance the accuracy of the model and the inference speed of the model, which is a key issue to be addressed in extending existing depth-pose estimation methods to practical applications(iii)
We propose a parallel OKS-NMS by combining Matrix-NMS [14] and OKS-NMS [15] to further improve the efficiency of the human pose estimation systemFigure 1
The architecture of the presented MobilePoseNet.Figure 2
The differences between our proposed network and the SimpleBaseline structure. We can see from the figure that we choose C4 as the input for upsampling, and the implement time of our network is less compared to SimpleBaseline.Table 1
Specification for MobilePoseNet. SE denotes whether there is a squeeze-and-excite in that block. NL denotes the type of nonlinearity used. Here, HS denotes h-swish and RE denotes relu. LPB is our proposed lightweight upsampling block. bneck is the bottleneck block in MobileNetV3.k is the number of key points.
Input channelInput sizeOperatorExp size#outAttentionNLs3256×192Conv2d—16—HS216128×96bneck,3×31616—RE116128×96bneck,3×36424—RE22464×48bneck,3×37224—RE12464×48bneck,5×57240SERE24032×24bneck,5×512040SERE14032×24bneck,5×512040SERE14032×24bneck,3×324080—HS28016×12bneck,3×320080—HS18016×12bneck,3×318480—HS18016×12bneck,3×318480—HS18016×12bneck,3×3480112SEHS111216×12bneck,3×3672112SEHS111216×12bneck,5×5672160SEHS116016×12bneck,5×5960160SEHS116016×12bneck,5×5960160SEHS116016×12LPB,4×4320160SERE216032×24LPB,4×4320160SERE216064×48Conv2d1×1—k—RE1The rest of this paper is organized as follows. We briefly review the related work in the second section and followed by description of the proposed method. Then, we conduct experiments on the MSCOCO and MPII datasets and conclude this work.


## 2. Related Works

### 2.1. Human Posture Estimation

In recent years, deep learning-based pose estimation methods [16] have made great progress. Despite significant performance improvements, these prior works focused only on improving the accuracy of pose estimation by using complex networks and large tensors, while largely ignoring the cost issues of model inference. This state of affairs significantly limits their deploy ability in real-world applications, especially when the available computational budget is very limited.In the literature, there are some recent works aimed at improving model efficiency. Bulat and Tzimiropoulos [10] designed a binary hourglass network using quantitative methods, but the restricted binary network has weak information representation and low accuracy of the model. Zhang et al. [7] proposed a new fast pose distillation (FPD) model learning strategy. A pretrained teacher network can be used to obtain a computationally fast and computationally inexpensive student network. However, it requires too much time to train. Yu et al. [17] proposed conditional channel weighting blocks and constructed the HR-Lite network, which achieves a great advantage in model accuracy and scale. However, the network structure is too complex, resulting in slow model inference. Zhang and Tang [11] proposed lightweight bottleneck block with depthwise convolution and attention mechanism, while the model size is still up to 2.7M parameters.Compared with previous methods, comprehensively considering the accuracy of the model, the speed of inference, and the complexity of the model, we directly designed a model with simple structure and low complexity, which makes the model more practical and reliable in practical application scenarios.


### 2.2. Efficient Upsampling Module

Recent work [13, 18–20] has shown that deep convolutional neural networks have reached state-of-the-art performance. For advanced vision problems such as semantic segmentation [21], pose estimation [16], and object detection [22], existing approaches pass inputs through a network, usually consisting of high- to low-resolution subnetworks and a main network of raised resolutions. Many approaches have been designed to improve the resolution of the main network in different ways. For example, networks such as hourglass [6] reduce the input high-resolution features to low-resolution features and then use interpolation upsampling to scale the low-resolution features to the original input features, fuse the information with the previously input high-resolution features, and finally expect to generate fused high semantic and high resolution. Although it achieved very good results, the large tensor is used in the process of feature fusion. Zhou et al. [23, 24] constructed an attention-driven feature fusion upsampling network in an attempt to reduce the complexity of the model and reduce the use of large tensor using heterogeneous convolution. However, the network structure is complex and does not solve the problem of slow model inference fundamentally. SimpleBaseline [25] uses several transposed convolutional layers to generate high-resolution representations and achieves very good results. Although the model structure is simple, however, transposed convolution introduces a large number of parameters and computational effort, which is not friendly to small devices.Therefore, we propose an efficient upsampling module that achieves a significant reduction in the number of parameters and computation during upsampling while ensuring the simplicity of the model structure and inference accuracy.


## 2.1. Human Posture Estimation

In recent years, deep learning-based pose estimation methods [16] have made great progress. Despite significant performance improvements, these prior works focused only on improving the accuracy of pose estimation by using complex networks and large tensors, while largely ignoring the cost issues of model inference. This state of affairs significantly limits their deploy ability in real-world applications, especially when the available computational budget is very limited.In the literature, there are some recent works aimed at improving model efficiency. Bulat and Tzimiropoulos [10] designed a binary hourglass network using quantitative methods, but the restricted binary network has weak information representation and low accuracy of the model. Zhang et al. [7] proposed a new fast pose distillation (FPD) model learning strategy. A pretrained teacher network can be used to obtain a computationally fast and computationally inexpensive student network. However, it requires too much time to train. Yu et al. [17] proposed conditional channel weighting blocks and constructed the HR-Lite network, which achieves a great advantage in model accuracy and scale. However, the network structure is too complex, resulting in slow model inference. Zhang and Tang [11] proposed lightweight bottleneck block with depthwise convolution and attention mechanism, while the model size is still up to 2.7M parameters.Compared with previous methods, comprehensively considering the accuracy of the model, the speed of inference, and the complexity of the model, we directly designed a model with simple structure and low complexity, which makes the model more practical and reliable in practical application scenarios.


## 2.2. Efficient Upsampling Module

Recent work [13, 18–20] has shown that deep convolutional neural networks have reached state-of-the-art performance. For advanced vision problems such as semantic segmentation [21], pose estimation [16], and object detection [22], existing approaches pass inputs through a network, usually consisting of high- to low-resolution subnetworks and a main network of raised resolutions. Many approaches have been designed to improve the resolution of the main network in different ways. For example, networks such as hourglass [6] reduce the input high-resolution features to low-resolution features and then use interpolation upsampling to scale the low-resolution features to the original input features, fuse the information with the previously input high-resolution features, and finally expect to generate fused high semantic and high resolution. Although it achieved very good results, the large tensor is used in the process of feature fusion. Zhou et al. [23, 24] constructed an attention-driven feature fusion upsampling network in an attempt to reduce the complexity of the model and reduce the use of large tensor using heterogeneous convolution. However, the network structure is complex and does not solve the problem of slow model inference fundamentally. SimpleBaseline [25] uses several transposed convolutional layers to generate high-resolution representations and achieves very good results. Although the model structure is simple, however, transposed convolution introduces a large number of parameters and computational effort, which is not friendly to small devices.Therefore, we propose an efficient upsampling module that achieves a significant reduction in the number of parameters and computation during upsampling while ensuring the simplicity of the model structure and inference accuracy.


## 3. Proposed Method

In this section, we detail a simple and low computational cost human pose estimation network (MobilePoseNet), which designs a lightweight upsampling block (LPB) and directly uses high-resolution features to achieve high-resolution representation while maintaining lightweight features.


### 3.1. Lightweight Upsampling Block

Transposed convolution was first introduced into pose estimation by SimpleBaseline and achieved excellent performance. However, this operation brings a model with nearly a third of the parameters and calculations. Specifically, given the input of feature mapsCin×Win×Hin and the output of feature maps Cout×Wout×Hout, the amount of computation for conventional transposed convolution is
(1)Cin×Cout×Wout×Hout×K×K.The number of parameters of traditional transpose convolution is(2)Cin×Cout×K×K,where K is the kernel size of traditional transposed convolution.To reduce the burden of calculation and the number of parameters, while maintaining the effect of transpose convolution, we designed a lightweight upsampling block inspired by the intuition that the bottlenecks actually contain all the necessary information, as shown in Figure3(b), which composed of three parts: depthwise transposed convolution, 1×1 point convolution, and attention module. Specifically, we first expand the low-latitude information to high-latitude information by 1×1 point convolution and use depth transpose convolution on each channel of the feature map for the spatial transformation. Finally, we use 1×1 point convolution to fuse the information between each channel and compress the high-latitude information to the original input latitude.Figure 3
The comparison of two different upsampling methods. (a) The traditional transposed convolution, which has a large computational overhead. (b) The proposed lightweight upsampling block, which includes depthwise transposed convolution operation, pointwise convolution operation, and attentionF. In (a), the features are amplified directly by transposed convolution. In (b), we first use 1×1 point convolution to expand the number of channels of the feature so that the number of channels goes from C to C∗ and then use the depthwise transposed convolution to generate high-resolution feature maps. Finally, we use a 1×1 point convolution to change the number of channels to C and the attention mechanism to make feature map stronger.
(a)(b)As shown in Figure3(b), the computation of the lightweight upsampling block is the sum of the depth transpose convolution and the two point convolution computations:
(3)C∗×Cin×Win×Hin×1×1+Wout×Hout×K×K+Cout×Wout×Hout×1×1.The number of parameters of the lightweight upsampling block is(4)C∗×K×K+Cin+Cout,where C∗ is the number of channels for high-latitude features. Compared to the traditional transposed convolution, our method reduces the calculation amount to 83.2% and the number of parameters reduces to 74%.Since LPB separates space operation and channel operation into two independent steps, the decoding effect of transpose convolution will be weakened. To solve this problem, we enhance the feature responses through channel attention mechanism. Here, we directly use SENet [26] as our channel attention mechanism to dynamically adjust the weight of each channel, as shown in Figure 3(b). To sum up, we assumed the input feature map X∈RCin×Win×Hin, the feature output through the LPB is X′∈RCout×Wout×Hout as input to the channel attention mechanism.The feature output through the channel attention mechanism isXatt∈RCout×1×1. Then, the feature output of LPB and the feature output of the channel attention mechanism are multiplied and summed to obtain the final fusion information Y, i.e.,
(5)Y=X′+X′Xatt.


### 3.2. Lightweight Human Pose Estimation

Usually, the pipeline [5, 6, 27, 28] for poses estimation consists of three parts: the upsampling, the downsampling, and estimation of the heat map. In this work, we focus on the design of a lightweight upsampling and downsampling.Different from SimpleBaseline which uses a ResNet backbone as the downsampling and three traditional deconvolutional layers as the upsampling, we use MobileNetV3 as our downsampling, which reduces the size of the parameters up to 96% and reduces the computation load up to 79%. For the upsampling, we replace each traditional deconvolution layer with a lightweight upsampling block. The details of the model are shown in Table1.As shown in Figure2, different with SimpleBaseline, we use a higher resolution feature map as the input for upsampling. The rationale behind this that it is beneficial to maintain high-resolution representations before upsampling.


### 3.3. Parallel Pose NMS

In pose estimation, human body detectors inevitably generate redundant detection, and pose estimation also generates redundant poses. Therefore, nonmaximum suppression (NMS) is required to eliminate redundant postures.Given posePi with m joints <k1i,s1i>,<k2i,s2i>,⋯,<kmi,smi> where kji and sji are the location and confidence score of the jth joint, respectively. Corresponding detection boxes bi with bsi confidence score. The general pose NMS is as follows: firstly, the pose with the highest confidences was chosen as the reference, and the poses similar to it were suppressed or discarded. This process is repeated for the rest of the pose set until only one pose is left.However, the main problems for this process are sequential and cannot be implemented in parallel, resulting in slower speeds. Inspired by Matrix-NMS, we proposed parallel nonmaximum suppression considering following two key factors:(1)
The confidence of pose: the higher the confidence of pose, the lower the probability of joints being suppressed, i.e., if the posePi and Pj with confidence (pi>pj), Pj will have a high probability of being suppressed(2)
Similarity between the pose and other poses: the lower the similarity between one pose and other poses, the lower the suppression ratio of the posesFor the pose confidence, we set the product of the average of the confidence of the key points and the confidence of the human detector as the final pose confidence below(6)pi=∑msmiδsmi,threshold∑mδsmi,threshold·bsi,where bsi is the confidence of the detection box and δ is defined as follows:
(7)δs,threshold=1,s>threshold,0,s≤threshold.We consider the key point prediction to be true ifsmi is bigger than threshold and otherwise to be false.For the similarity between two poses, we use the object key point similarity (OKS) [29] to measure the pose distance function as follows:
(8)fPi,Pj=1−Oi,j,where Oi,j is given by
(9)Oi,j=∑mexp−kmi−kmj2/2baibaj∑mδsmi,threshold∙δsmj,threshold,where bai is the area of the detection box.We define a new decay factor for pose NMS. ForfPi,Pj=1−Oi,j, we can get a new decayj, where
(10)decayj=min∀pi>pjfPi,Pjf:,Pi,where f:,Pi=min∀pl>pifpl,pi.Finally, we get a new pose confidencepj⟵decayj·pj. For usage, we just need threshold and selecting top-k scoring masks as the final predictions.Like Matrix-NMS, all the operations in pose NMS could be implemented in one shot without recurrence. We first get aNpose confidence and then compute a N×N pairwise OKS matrix for the N pose sorted descending by pose confidence score. The decay factors of each pose can be obtained by looking up the table of the OKS matrix. Finally, the pose scores are updated by the decay factors. For usage, we just need threshold and select top-k pose scoring as the final predictions. The whole procedure is summarized in Algorithm 1.Algorithm 1: Parallel pose NMS.
Input: the area of the detection boxer ba=bai, the confidence of the detection boxer bs=bsi, the location of the key point K=kji, the confidence of the key point S=sji, and parameter threshold. Here, i is the i-th person, i∈1,2,3,⋯,n, j is the j-th key point, and j∈1,2,3,⋯,m.Output: the confidence of the key point p=pii=1,2,3,⋯n.1: Initializethreshold=0.12: Calculatep by equation (6) and parameter bs, S, threshold3: Sortba, bs, and K in descending order by p=p1,p2,⋯pn4: CalculateB+ using bs×bsT5: CalculateK+ using K+KT2−4×K+KT6: Calculate OKS matrixO by equation (9) and parameter K+, B+, threshold7: UpdateO=Oi,j=1i≥j,i,j=1,2,3,⋯n8: Setom=maxjO:,jj=1,2,3,⋯n9: SetOm by repeating omn times10: Calculate decay matrixD using I−O/I−Om11: Set decayod=minD:,jj=1,2,3,⋯n12: Update the confidence of the key pointp by od⊙p


## 3.1. Lightweight Upsampling Block

Transposed convolution was first introduced into pose estimation by SimpleBaseline and achieved excellent performance. However, this operation brings a model with nearly a third of the parameters and calculations. Specifically, given the input of feature mapsCin×Win×Hin and the output of feature maps Cout×Wout×Hout, the amount of computation for conventional transposed convolution is
(1)Cin×Cout×Wout×Hout×K×K.The number of parameters of traditional transpose convolution is(2)Cin×Cout×K×K,where K is the kernel size of traditional transposed convolution.To reduce the burden of calculation and the number of parameters, while maintaining the effect of transpose convolution, we designed a lightweight upsampling block inspired by the intuition that the bottlenecks actually contain all the necessary information, as shown in Figure3(b), which composed of three parts: depthwise transposed convolution, 1×1 point convolution, and attention module. Specifically, we first expand the low-latitude information to high-latitude information by 1×1 point convolution and use depth transpose convolution on each channel of the feature map for the spatial transformation. Finally, we use 1×1 point convolution to fuse the information between each channel and compress the high-latitude information to the original input latitude.Figure 3
The comparison of two different upsampling methods. (a) The traditional transposed convolution, which has a large computational overhead. (b) The proposed lightweight upsampling block, which includes depthwise transposed convolution operation, pointwise convolution operation, and attentionF. In (a), the features are amplified directly by transposed convolution. In (b), we first use 1×1 point convolution to expand the number of channels of the feature so that the number of channels goes from C to C∗ and then use the depthwise transposed convolution to generate high-resolution feature maps. Finally, we use a 1×1 point convolution to change the number of channels to C and the attention mechanism to make feature map stronger.
(a)(b)As shown in Figure3(b), the computation of the lightweight upsampling block is the sum of the depth transpose convolution and the two point convolution computations:
(3)C∗×Cin×Win×Hin×1×1+Wout×Hout×K×K+Cout×Wout×Hout×1×1.The number of parameters of the lightweight upsampling block is(4)C∗×K×K+Cin+Cout,where C∗ is the number of channels for high-latitude features. Compared to the traditional transposed convolution, our method reduces the calculation amount to 83.2% and the number of parameters reduces to 74%.Since LPB separates space operation and channel operation into two independent steps, the decoding effect of transpose convolution will be weakened. To solve this problem, we enhance the feature responses through channel attention mechanism. Here, we directly use SENet [26] as our channel attention mechanism to dynamically adjust the weight of each channel, as shown in Figure 3(b). To sum up, we assumed the input feature map X∈RCin×Win×Hin, the feature output through the LPB is X′∈RCout×Wout×Hout as input to the channel attention mechanism.The feature output through the channel attention mechanism isXatt∈RCout×1×1. Then, the feature output of LPB and the feature output of the channel attention mechanism are multiplied and summed to obtain the final fusion information Y, i.e.,
(5)Y=X′+X′Xatt.


## 3.2. Lightweight Human Pose Estimation

Usually, the pipeline [5, 6, 27, 28] for poses estimation consists of three parts: the upsampling, the downsampling, and estimation of the heat map. In this work, we focus on the design of a lightweight upsampling and downsampling.Different from SimpleBaseline which uses a ResNet backbone as the downsampling and three traditional deconvolutional layers as the upsampling, we use MobileNetV3 as our downsampling, which reduces the size of the parameters up to 96% and reduces the computation load up to 79%. For the upsampling, we replace each traditional deconvolution layer with a lightweight upsampling block. The details of the model are shown in Table1.As shown in Figure2, different with SimpleBaseline, we use a higher resolution feature map as the input for upsampling. The rationale behind this that it is beneficial to maintain high-resolution representations before upsampling.


## 3.3. Parallel Pose NMS

In pose estimation, human body detectors inevitably generate redundant detection, and pose estimation also generates redundant poses. Therefore, nonmaximum suppression (NMS) is required to eliminate redundant postures.Given posePi with m joints <k1i,s1i>,<k2i,s2i>,⋯,<kmi,smi> where kji and sji are the location and confidence score of the jth joint, respectively. Corresponding detection boxes bi with bsi confidence score. The general pose NMS is as follows: firstly, the pose with the highest confidences was chosen as the reference, and the poses similar to it were suppressed or discarded. This process is repeated for the rest of the pose set until only one pose is left.However, the main problems for this process are sequential and cannot be implemented in parallel, resulting in slower speeds. Inspired by Matrix-NMS, we proposed parallel nonmaximum suppression considering following two key factors:(1)
The confidence of pose: the higher the confidence of pose, the lower the probability of joints being suppressed, i.e., if the posePi and Pj with confidence (pi>pj), Pj will have a high probability of being suppressed(2)
Similarity between the pose and other poses: the lower the similarity between one pose and other poses, the lower the suppression ratio of the posesFor the pose confidence, we set the product of the average of the confidence of the key points and the confidence of the human detector as the final pose confidence below(6)pi=∑msmiδsmi,threshold∑mδsmi,threshold·bsi,where bsi is the confidence of the detection box and δ is defined as follows:
(7)δs,threshold=1,s>threshold,0,s≤threshold.We consider the key point prediction to be true ifsmi is bigger than threshold and otherwise to be false.For the similarity between two poses, we use the object key point similarity (OKS) [29] to measure the pose distance function as follows:
(8)fPi,Pj=1−Oi,j,where Oi,j is given by
(9)Oi,j=∑mexp−kmi−kmj2/2baibaj∑mδsmi,threshold∙δsmj,threshold,where bai is the area of the detection box.We define a new decay factor for pose NMS. ForfPi,Pj=1−Oi,j, we can get a new decayj, where
(10)decayj=min∀pi>pjfPi,Pjf:,Pi,where f:,Pi=min∀pl>pifpl,pi.Finally, we get a new pose confidencepj⟵decayj·pj. For usage, we just need threshold and selecting top-k scoring masks as the final predictions.Like Matrix-NMS, all the operations in pose NMS could be implemented in one shot without recurrence. We first get aNpose confidence and then compute a N×N pairwise OKS matrix for the N pose sorted descending by pose confidence score. The decay factors of each pose can be obtained by looking up the table of the OKS matrix. Finally, the pose scores are updated by the decay factors. For usage, we just need threshold and select top-k pose scoring as the final predictions. The whole procedure is summarized in Algorithm 1.Algorithm 1: Parallel pose NMS.
Input: the area of the detection boxer ba=bai, the confidence of the detection boxer bs=bsi, the location of the key point K=kji, the confidence of the key point S=sji, and parameter threshold. Here, i is the i-th person, i∈1,2,3,⋯,n, j is the j-th key point, and j∈1,2,3,⋯,m.Output: the confidence of the key point p=pii=1,2,3,⋯n.1: Initializethreshold=0.12: Calculatep by equation (6) and parameter bs, S, threshold3: Sortba, bs, and K in descending order by p=p1,p2,⋯pn4: CalculateB+ using bs×bsT5: CalculateK+ using K+KT2−4×K+KT6: Calculate OKS matrixO by equation (9) and parameter K+, B+, threshold7: UpdateO=Oi,j=1i≥j,i,j=1,2,3,⋯n8: Setom=maxjO:,jj=1,2,3,⋯n9: SetOm by repeating omn times10: Calculate decay matrixD using I−O/I−Om11: Set decayod=minD:,jj=1,2,3,⋯n12: Update the confidence of the key pointp by od⊙p


## 4. Experiments

We conduct experiments on the MSCOCO and MPII datasets to evaluate the performance of our method in multiperson pose estimation.


### 4.1. Datasets

(i)
The MSCOCO dataset contains over 200K images, 250K human body instances, and 17 key points. We trained our model on the MSCOCO train2017 dataset, including 57K images and 150K person instances and evaluated our approach on val2017 and test-dev2017, which contained 5000 images and 20K images, respectively(ii)
The MPII Human Pose dataset contains about 25K images of more than 40,000 people with annotated human joints, which are taken from a wide range of real-world activities with full-body pose annotationsWe selected the object key point similarity (OKS) as an evaluation metric for the MSCOCO dataset. The standard metric [30], the PCK (probability of correct key point normalized by head) score, was used to evaluate the MPII dataset.


### 4.2. Implement Details

In MSCOCO, we extend the human detection box into a fixed aspect ratio with 4 : 3, and crop the box from the image with fixed size,256×192 or 384×288. In MPII, the input size is cropped to 256×256 for fair comparison with other methods. In addition, the same data augmentation and the training strategy are utilized for both datasets. The data augmentation includes random rotation ([-45, 45]), random scale ([0.65, 1.35]), and flipping. In MSCOCO, half body data augmentation is also involved.We all use the Adam optimizer with initial learning rate1e−3. The model was trained on a single Nvidia TITAN RTX GPU with a minibatch size 32 and stop at 210 epochs.


### 4.3. Experimental Results

#### 4.3.1. Results on MSCOCO Dataset

From the results, as shown in Table2, we can see that our method has a significant advantage in terms of model size and complexity with comparable accuracy. For input size 256×192, our method achieved comparable accuracy with less than 6% the parameters with respect to hourglass network. Compared with MobileNetV2 and ShuffleNetV2, our method obtained better accuracy with low complexity. For the small network HRNet-W16 and Lite-HRNet-18, our model is also better in terms of accuracy although the model size is slightly large. For the input 382×288, we can also derive the same conclusion.Table 2
Comparisons of results on the MSCOCO validation set.
MethodBackboneInput#ParamsGFLOPsAPAP50AP75APMAPLAR8-stage hourglass [6]Hourglass256×19225.6M26.266.9—————CPN [31]ResNet-50256×19227.0M6.268.4—————SimpleBaseline [25]ResNet-50256×19234.0M8.970.488.678.367.177.276.3HRNet-W32 [5]ResNet-50256×19228.5M12.473.489.580.770.280.179.8DARK [32]HRNetV1-W48128×9663.6M3.671.989.179.669.27877.9MobileNetV2 [19]MobileNetV2256×1929.6M1.4864.687.472.361.171.270.7MobileNetV2 1×MobileNetV2384×2889.6M3.3367.387.974.362.874.772.9ShuffleNetV2 [33]ShuffleNetV2256×1927.6M1.2859.985.466.356.666.266.4ShuffleNetV2 1×ShuffleNetV2384×2887.6M2.8763.686.570.559.570.769.7Small HRNetHRNet-W16256×1921.3M0.5455.283.762.452.36162.1Small HRNetHRNet-W16384×2881.3M1.215683.86352.462.662.6Lite-HRNetLite-HRNet-18256×1921.1M0.2064.886.77362.170.571.2Lite-HRNetLite-HRNet-18384×2881.1M0.4567.687.87564.573.773.7MobilePoseNetMobilNetV3256×1921.5M0.5566.287.374.263.172.572.4MobilePoseNetMobilNetV3384×2881.5M1.236988.275.965.575.574.9Figure4 illustrates the comparison of accuracy and complexity of small networks. Figure 5 shows the visualization results of our method in MSCOCO. It can be seen that our model achieved better balance between complexity and accuracy and can estimate the accurate joints under different complex scenes.Figure 4
The complexity and accuracy comparison of MSCOCO val for384×288 input size.Figure 5
Visualization results of human pose estimation based on MobilePoseNet for MSCOCO validation set.Table3 lists the mAP, input size, Params, and GFLOP values of compared methods and our method on the MSCOCO dataset.Table 3
Comparisons of results on MSCOCO test-dev2017 set. #Params and flops are calculated for the pose estimation network, and those for human detection are not included.
MethodBackboneInput#ParamsGFLOPsAPAP50AP75APMAPLARBottom-up: key point detection and groupingOpenPose [34]————61.884.967.557.168.266.5Associative embedding [35]————65.586.672.360.672.670.2PersonLab [4]————68.78975.464.175.575.4MultiPoseNet [36]————69.686.376.665.076.373.5HigherHRNet [37]HRNet-w32512×51228.6M47.966.487.572.861.274.2—Top-down: human detection and single-person key point detectionLarge networkMask-RCNN [22]ResNet-50-FPN———63.187.368.757.871.4—G-RMI [15]ResNet-101353×25742.6M5764.985.571.362.370.069.7IPR [27]ResNet-101256×25645.0M1167.888.274.863.974.0—RMPE [38]PyraNet [39]320×25628.1M26.772.389.279.168.078.6—CPN [28]—384×288——72.191.480.068.777.278.5SimpleBaseline [25]ResNet-152384×28868.6M35.673.791.981.170.380.079.0Small networkMobileNetV2 [19]MobileNetV2384×2889.8M3.3366.890.074.062.673.372.3ShuffleNetV2 [33]ShuffleNetV2384×2887.6M2.8762.988.569.458.969.368.9Small HRNet [17]HRNet-W16384×2881.3M1.2155.285.861.451.761.261.5Lite-HRNet [17]Lite-HRNet-18384×2881.1M0.4566.989.474.464.072.272.6MobilePoseNetMobileNetv3 [13]256×1921.5M0.5564.888.872.461.970.270.7MobilePoseNetMobileNetv3384×2881.5M1.2367.489.474.264.173.373.3


#### 4.3.2. Results on MPII Human Pose Dataset

Table4 reports the results of our network and other lightweight networks on MPII val data. Compared with MobileNetV2, MobileNetV3, ShuffleNetV2, and SmallHRNet-W16, our model achieves better accuracy with lower number of parameters and calculation weights. Compared to Lite-HRNet-30, our model achieves 87.3 PCKh@0.5 in terms of the number of parameters with 0.3M less than Lite-HRNet-30. Compared to MobileNetV2, MobileNetV3, ShuffleNetV2, and Small HRNet-W16, our model improved by 1.9%, 3.0%, 4.5%, and 7.1%, respectively. Figure 6 illustrates the comparison of accuracy and complexity.Table 4
Comparisons on the MPII val set. The GFLOPs is computed with the input size256×256.
Model#ParamsGFLOPsPCKhMobileNetV2 1×9.6M1.9785.4MobileNetV3 1×8.7M1.8284.3ShuffleNetV2 1×7.6M1.7082.8Small HRNet-W161.3M0.7280.2Lite-HRNet-181.1M0.2786.1MobilePoseNet1.5M0.7487.3Figure 6
The complexity and accuracy comparison of MPII val sets for256×256 input size.


### 4.4. Inference Speed

FLOPs and Param are only the properties that measure the size and complexity of the model. In this section, we study the actual inference speed of the human pose estimation network by inference items per second (Inference Items Per Second). The speed is tested on devices with GPU and without GPU, respectively, with a batch size of 32 and full precision (fp32). We use the Nvidia TITAN TRX as the GPU device and the Intel Core I9-10900k device without GPU as the non-GPU device. To better reflect the running speed of the model, all methods are tested on the MSCOCO validation set. We use the same person detector provided by the SimpleBaseline validation set. In the tests without GPU, a thread was used for evaluation. As can be seen in Table5, thanks to the simple structure of our model, our actual inference is 3 times faster than the less computationally intensive Lite-HRNet on the GPU speed test. In the GPU-free speed test, our method is faster than a large network like HRNet. Also, our model has a significant advantage in complexity and computational power compared to other models, which means easier deployment to embedded devices.Table 5
Inference speed comparisons on the MSCOCO validation set. Speed∗ refers to the result on non-GPU device. Speed refers to the result on GPU device. Bold values are the optimal results.
MethodBackbone#ParamsGFLOPsInput sizeAPSpeed∗SpeedHRNetHRNetV1-W3228.5M7.1256×19274.47.519.2HRNetHRNetV1-W3228.5M16384×28875.8418.8SimpleBaselineResNet-5034.0M8.9256×19270.48.1273.1NLite-HRNet-18HRNet-W160.7M0.19256×19262.81118.9WNLite-HRNet-18HRNet-W161.3M0.3256×192661218.6ShuffleNetV2 1×ShuffleNetV27.6M1.28256×19259.91771.3ShuffleNetV2 1×ShuffleNetV27.6M2.87384×28863.61064.1MobileNetV2 1×MobileNetV29.6M1.48256×19264.66.883.1MobileNetV2 1×MobileNetV29.6M3.33384×28867.34.573.1Lite-HRNetLite-HRNet-181.1M0.2256×19264.81217.4Lite-HRNetLite-HRNet-181.1M0.45384×28867.67.116.3MobilePoseNetMobileNetV31.5M0.55256×19266.27.854.8MobilePoseNetMobileNetV31.5M1.23384×28869.05.150.8


## 4.1. Datasets

(i)
The MSCOCO dataset contains over 200K images, 250K human body instances, and 17 key points. We trained our model on the MSCOCO train2017 dataset, including 57K images and 150K person instances and evaluated our approach on val2017 and test-dev2017, which contained 5000 images and 20K images, respectively(ii)
The MPII Human Pose dataset contains about 25K images of more than 40,000 people with annotated human joints, which are taken from a wide range of real-world activities with full-body pose annotationsWe selected the object key point similarity (OKS) as an evaluation metric for the MSCOCO dataset. The standard metric [30], the PCK (probability of correct key point normalized by head) score, was used to evaluate the MPII dataset.


## 4.2. Implement Details

In MSCOCO, we extend the human detection box into a fixed aspect ratio with 4 : 3, and crop the box from the image with fixed size,256×192 or 384×288. In MPII, the input size is cropped to 256×256 for fair comparison with other methods. In addition, the same data augmentation and the training strategy are utilized for both datasets. The data augmentation includes random rotation ([-45, 45]), random scale ([0.65, 1.35]), and flipping. In MSCOCO, half body data augmentation is also involved.We all use the Adam optimizer with initial learning rate1e−3. The model was trained on a single Nvidia TITAN RTX GPU with a minibatch size 32 and stop at 210 epochs.


## 4.3. Experimental Results

### 4.3.1. Results on MSCOCO Dataset

From the results, as shown in Table2, we can see that our method has a significant advantage in terms of model size and complexity with comparable accuracy. For input size 256×192, our method achieved comparable accuracy with less than 6% the parameters with respect to hourglass network. Compared with MobileNetV2 and ShuffleNetV2, our method obtained better accuracy with low complexity. For the small network HRNet-W16 and Lite-HRNet-18, our model is also better in terms of accuracy although the model size is slightly large. For the input 382×288, we can also derive the same conclusion.Table 2
Comparisons of results on the MSCOCO validation set.
MethodBackboneInput#ParamsGFLOPsAPAP50AP75APMAPLAR8-stage hourglass [6]Hourglass256×19225.6M26.266.9—————CPN [31]ResNet-50256×19227.0M6.268.4—————SimpleBaseline [25]ResNet-50256×19234.0M8.970.488.678.367.177.276.3HRNet-W32 [5]ResNet-50256×19228.5M12.473.489.580.770.280.179.8DARK [32]HRNetV1-W48128×9663.6M3.671.989.179.669.27877.9MobileNetV2 [19]MobileNetV2256×1929.6M1.4864.687.472.361.171.270.7MobileNetV2 1×MobileNetV2384×2889.6M3.3367.387.974.362.874.772.9ShuffleNetV2 [33]ShuffleNetV2256×1927.6M1.2859.985.466.356.666.266.4ShuffleNetV2 1×ShuffleNetV2384×2887.6M2.8763.686.570.559.570.769.7Small HRNetHRNet-W16256×1921.3M0.5455.283.762.452.36162.1Small HRNetHRNet-W16384×2881.3M1.215683.86352.462.662.6Lite-HRNetLite-HRNet-18256×1921.1M0.2064.886.77362.170.571.2Lite-HRNetLite-HRNet-18384×2881.1M0.4567.687.87564.573.773.7MobilePoseNetMobilNetV3256×1921.5M0.5566.287.374.263.172.572.4MobilePoseNetMobilNetV3384×2881.5M1.236988.275.965.575.574.9Figure4 illustrates the comparison of accuracy and complexity of small networks. Figure 5 shows the visualization results of our method in MSCOCO. It can be seen that our model achieved better balance between complexity and accuracy and can estimate the accurate joints under different complex scenes.Figure 4
The complexity and accuracy comparison of MSCOCO val for384×288 input size.Figure 5
Visualization results of human pose estimation based on MobilePoseNet for MSCOCO validation set.Table3 lists the mAP, input size, Params, and GFLOP values of compared methods and our method on the MSCOCO dataset.Table 3
Comparisons of results on MSCOCO test-dev2017 set. #Params and flops are calculated for the pose estimation network, and those for human detection are not included.
MethodBackboneInput#ParamsGFLOPsAPAP50AP75APMAPLARBottom-up: key point detection and groupingOpenPose [34]————61.884.967.557.168.266.5Associative embedding [35]————65.586.672.360.672.670.2PersonLab [4]————68.78975.464.175.575.4MultiPoseNet [36]————69.686.376.665.076.373.5HigherHRNet [37]HRNet-w32512×51228.6M47.966.487.572.861.274.2—Top-down: human detection and single-person key point detectionLarge networkMask-RCNN [22]ResNet-50-FPN———63.187.368.757.871.4—G-RMI [15]ResNet-101353×25742.6M5764.985.571.362.370.069.7IPR [27]ResNet-101256×25645.0M1167.888.274.863.974.0—RMPE [38]PyraNet [39]320×25628.1M26.772.389.279.168.078.6—CPN [28]—384×288——72.191.480.068.777.278.5SimpleBaseline [25]ResNet-152384×28868.6M35.673.791.981.170.380.079.0Small networkMobileNetV2 [19]MobileNetV2384×2889.8M3.3366.890.074.062.673.372.3ShuffleNetV2 [33]ShuffleNetV2384×2887.6M2.8762.988.569.458.969.368.9Small HRNet [17]HRNet-W16384×2881.3M1.2155.285.861.451.761.261.5Lite-HRNet [17]Lite-HRNet-18384×2881.1M0.4566.989.474.464.072.272.6MobilePoseNetMobileNetv3 [13]256×1921.5M0.5564.888.872.461.970.270.7MobilePoseNetMobileNetv3384×2881.5M1.2367.489.474.264.173.373.3


### 4.3.2. Results on MPII Human Pose Dataset

Table4 reports the results of our network and other lightweight networks on MPII val data. Compared with MobileNetV2, MobileNetV3, ShuffleNetV2, and SmallHRNet-W16, our model achieves better accuracy with lower number of parameters and calculation weights. Compared to Lite-HRNet-30, our model achieves 87.3 PCKh@0.5 in terms of the number of parameters with 0.3M less than Lite-HRNet-30. Compared to MobileNetV2, MobileNetV3, ShuffleNetV2, and Small HRNet-W16, our model improved by 1.9%, 3.0%, 4.5%, and 7.1%, respectively. Figure 6 illustrates the comparison of accuracy and complexity.Table 4
Comparisons on the MPII val set. The GFLOPs is computed with the input size256×256.
Model#ParamsGFLOPsPCKhMobileNetV2 1×9.6M1.9785.4MobileNetV3 1×8.7M1.8284.3ShuffleNetV2 1×7.6M1.7082.8Small HRNet-W161.3M0.7280.2Lite-HRNet-181.1M0.2786.1MobilePoseNet1.5M0.7487.3Figure 6
The complexity and accuracy comparison of MPII val sets for256×256 input size.


## 4.3.1. Results on MSCOCO Dataset

From the results, as shown in Table2, we can see that our method has a significant advantage in terms of model size and complexity with comparable accuracy. For input size 256×192, our method achieved comparable accuracy with less than 6% the parameters with respect to hourglass network. Compared with MobileNetV2 and ShuffleNetV2, our method obtained better accuracy with low complexity. For the small network HRNet-W16 and Lite-HRNet-18, our model is also better in terms of accuracy although the model size is slightly large. For the input 382×288, we can also derive the same conclusion.Table 2
Comparisons of results on the MSCOCO validation set.
MethodBackboneInput#ParamsGFLOPsAPAP50AP75APMAPLAR8-stage hourglass [6]Hourglass256×19225.6M26.266.9—————CPN [31]ResNet-50256×19227.0M6.268.4—————SimpleBaseline [25]ResNet-50256×19234.0M8.970.488.678.367.177.276.3HRNet-W32 [5]ResNet-50256×19228.5M12.473.489.580.770.280.179.8DARK [32]HRNetV1-W48128×9663.6M3.671.989.179.669.27877.9MobileNetV2 [19]MobileNetV2256×1929.6M1.4864.687.472.361.171.270.7MobileNetV2 1×MobileNetV2384×2889.6M3.3367.387.974.362.874.772.9ShuffleNetV2 [33]ShuffleNetV2256×1927.6M1.2859.985.466.356.666.266.4ShuffleNetV2 1×ShuffleNetV2384×2887.6M2.8763.686.570.559.570.769.7Small HRNetHRNet-W16256×1921.3M0.5455.283.762.452.36162.1Small HRNetHRNet-W16384×2881.3M1.215683.86352.462.662.6Lite-HRNetLite-HRNet-18256×1921.1M0.2064.886.77362.170.571.2Lite-HRNetLite-HRNet-18384×2881.1M0.4567.687.87564.573.773.7MobilePoseNetMobilNetV3256×1921.5M0.5566.287.374.263.172.572.4MobilePoseNetMobilNetV3384×2881.5M1.236988.275.965.575.574.9Figure4 illustrates the comparison of accuracy and complexity of small networks. Figure 5 shows the visualization results of our method in MSCOCO. It can be seen that our model achieved better balance between complexity and accuracy and can estimate the accurate joints under different complex scenes.Figure 4
The complexity and accuracy comparison of MSCOCO val for384×288 input size.Figure 5
Visualization results of human pose estimation based on MobilePoseNet for MSCOCO validation set.Table3 lists the mAP, input size, Params, and GFLOP values of compared methods and our method on the MSCOCO dataset.Table 3
Comparisons of results on MSCOCO test-dev2017 set. #Params and flops are calculated for the pose estimation network, and those for human detection are not included.
MethodBackboneInput#ParamsGFLOPsAPAP50AP75APMAPLARBottom-up: key point detection and groupingOpenPose [34]————61.884.967.557.168.266.5Associative embedding [35]————65.586.672.360.672.670.2PersonLab [4]————68.78975.464.175.575.4MultiPoseNet [36]————69.686.376.665.076.373.5HigherHRNet [37]HRNet-w32512×51228.6M47.966.487.572.861.274.2—Top-down: human detection and single-person key point detectionLarge networkMask-RCNN [22]ResNet-50-FPN———63.187.368.757.871.4—G-RMI [15]ResNet-101353×25742.6M5764.985.571.362.370.069.7IPR [27]ResNet-101256×25645.0M1167.888.274.863.974.0—RMPE [38]PyraNet [39]320×25628.1M26.772.389.279.168.078.6—CPN [28]—384×288——72.191.480.068.777.278.5SimpleBaseline [25]ResNet-152384×28868.6M35.673.791.981.170.380.079.0Small networkMobileNetV2 [19]MobileNetV2384×2889.8M3.3366.890.074.062.673.372.3ShuffleNetV2 [33]ShuffleNetV2384×2887.6M2.8762.988.569.458.969.368.9Small HRNet [17]HRNet-W16384×2881.3M1.2155.285.861.451.761.261.5Lite-HRNet [17]Lite-HRNet-18384×2881.1M0.4566.989.474.464.072.272.6MobilePoseNetMobileNetv3 [13]256×1921.5M0.5564.888.872.461.970.270.7MobilePoseNetMobileNetv3384×2881.5M1.2367.489.474.264.173.373.3


## 4.3.2. Results on MPII Human Pose Dataset

Table4 reports the results of our network and other lightweight networks on MPII val data. Compared with MobileNetV2, MobileNetV3, ShuffleNetV2, and SmallHRNet-W16, our model achieves better accuracy with lower number of parameters and calculation weights. Compared to Lite-HRNet-30, our model achieves 87.3 PCKh@0.5 in terms of the number of parameters with 0.3M less than Lite-HRNet-30. Compared to MobileNetV2, MobileNetV3, ShuffleNetV2, and Small HRNet-W16, our model improved by 1.9%, 3.0%, 4.5%, and 7.1%, respectively. Figure 6 illustrates the comparison of accuracy and complexity.Table 4
Comparisons on the MPII val set. The GFLOPs is computed with the input size256×256.
Model#ParamsGFLOPsPCKhMobileNetV2 1×9.6M1.9785.4MobileNetV3 1×8.7M1.8284.3ShuffleNetV2 1×7.6M1.7082.8Small HRNet-W161.3M0.7280.2Lite-HRNet-181.1M0.2786.1MobilePoseNet1.5M0.7487.3Figure 6
The complexity and accuracy comparison of MPII val sets for256×256 input size.


## 4.4. Inference Speed

FLOPs and Param are only the properties that measure the size and complexity of the model. In this section, we study the actual inference speed of the human pose estimation network by inference items per second (Inference Items Per Second). The speed is tested on devices with GPU and without GPU, respectively, with a batch size of 32 and full precision (fp32). We use the Nvidia TITAN TRX as the GPU device and the Intel Core I9-10900k device without GPU as the non-GPU device. To better reflect the running speed of the model, all methods are tested on the MSCOCO validation set. We use the same person detector provided by the SimpleBaseline validation set. In the tests without GPU, a thread was used for evaluation. As can be seen in Table5, thanks to the simple structure of our model, our actual inference is 3 times faster than the less computationally intensive Lite-HRNet on the GPU speed test. In the GPU-free speed test, our method is faster than a large network like HRNet. Also, our model has a significant advantage in complexity and computational power compared to other models, which means easier deployment to embedded devices.Table 5
Inference speed comparisons on the MSCOCO validation set. Speed∗ refers to the result on non-GPU device. Speed refers to the result on GPU device. Bold values are the optimal results.
MethodBackbone#ParamsGFLOPsInput sizeAPSpeed∗SpeedHRNetHRNetV1-W3228.5M7.1256×19274.47.519.2HRNetHRNetV1-W3228.5M16384×28875.8418.8SimpleBaselineResNet-5034.0M8.9256×19270.48.1273.1NLite-HRNet-18HRNet-W160.7M0.19256×19262.81118.9WNLite-HRNet-18HRNet-W161.3M0.3256×192661218.6ShuffleNetV2 1×ShuffleNetV27.6M1.28256×19259.91771.3ShuffleNetV2 1×ShuffleNetV27.6M2.87384×28863.61064.1MobileNetV2 1×MobileNetV29.6M1.48256×19264.66.883.1MobileNetV2 1×MobileNetV29.6M3.33384×28867.34.573.1Lite-HRNetLite-HRNet-181.1M0.2256×19264.81217.4Lite-HRNetLite-HRNet-181.1M0.45384×28867.67.116.3MobilePoseNetMobileNetV31.5M0.55256×19266.27.854.8MobilePoseNetMobileNetV31.5M1.23384×28869.05.150.8


## 5. Ablation Study

We study the effect of each component of our approach on the validation set of MSCOCO.


### 5.1. Deconvolution Blocks

In this section, we analyzed the impact of reducing the number of upsampling and using different upsampling blocks in terms of accuracy with resolution384×288. From Table 6, it can be seen that the number of parameters and the computation of our model are reduced compared to other models, while the precision has indeed been improved.Table 6
Ablation experiments on reduced downsampling with the use of lightweight upsampling blocks, on the MSCOCO val dataset. V1 denote the model that uses C5 as the input for upsampling, using the first 16 layers of MobileNetV3 as the downsampling and three layers of deconvolution as the upsampling part. V2 denote the model that uses C4 as the input for upsampling, using the first 13 layers of MobileNetV3 as the downsampling part, then uses three layers of5×5 bottleneck with a stride of 1, and finally uses two layers of the same deconvolution as V1 as the upsampling part.
ModelInput sizeFLOPs#ParamsAPAP50AP75APMAPLARV1256×192604M2.5M65.2287.0573.0562.271.4771.45V2256×192684M2.1M66.2387.2374.1963.2172.4872.38V1384×2881.33G2.5M68.4487.7175.4164.8975.0174.47V2384×2881.5G2.1M68.9787.7275.4765.3275.774.85Ours256×192557M1.5M66.2387.3874.2563.1372.5272.4Ours384×2881.23G1.5M69.0387.7275.9565.5275.5574.98


### 5.2. OKS-Based Nonmaximum Suppression

We compared the proposed OKS-based nonmaximum suppression and other OKS-based nonmaximum suppression methods on the accuracy and speed with the same pose estimator. As shown in Table7, we can find that our proposed OKS-based nonextreme suppression has significant advantages in terms of accuracy and speed.Table 7
Performance (AP) and speed (ms) of the MSCOCO validation set for different pose NMS. Matrix-OKS-NMS outperforms both Hard- and Soft-OKS-NMS methods in terms of speed and accuracy. Input size is384×288.
MethodAPAP50ARTime (ms)Hard-OKS-NMS67.787.191.46.14Soft-OKS-NMS67.786.791.56.8Parallel-OKS-NMS67.886.791.56.0


## 5.1. Deconvolution Blocks

In this section, we analyzed the impact of reducing the number of upsampling and using different upsampling blocks in terms of accuracy with resolution384×288. From Table 6, it can be seen that the number of parameters and the computation of our model are reduced compared to other models, while the precision has indeed been improved.Table 6
Ablation experiments on reduced downsampling with the use of lightweight upsampling blocks, on the MSCOCO val dataset. V1 denote the model that uses C5 as the input for upsampling, using the first 16 layers of MobileNetV3 as the downsampling and three layers of deconvolution as the upsampling part. V2 denote the model that uses C4 as the input for upsampling, using the first 13 layers of MobileNetV3 as the downsampling part, then uses three layers of5×5 bottleneck with a stride of 1, and finally uses two layers of the same deconvolution as V1 as the upsampling part.
ModelInput sizeFLOPs#ParamsAPAP50AP75APMAPLARV1256×192604M2.5M65.2287.0573.0562.271.4771.45V2256×192684M2.1M66.2387.2374.1963.2172.4872.38V1384×2881.33G2.5M68.4487.7175.4164.8975.0174.47V2384×2881.5G2.1M68.9787.7275.4765.3275.774.85Ours256×192557M1.5M66.2387.3874.2563.1372.5272.4Ours384×2881.23G1.5M69.0387.7275.9565.5275.5574.98


## 5.2. OKS-Based Nonmaximum Suppression

We compared the proposed OKS-based nonmaximum suppression and other OKS-based nonmaximum suppression methods on the accuracy and speed with the same pose estimator. As shown in Table7, we can find that our proposed OKS-based nonextreme suppression has significant advantages in terms of accuracy and speed.Table 7
Performance (AP) and speed (ms) of the MSCOCO validation set for different pose NMS. Matrix-OKS-NMS outperforms both Hard- and Soft-OKS-NMS methods in terms of speed and accuracy. Input size is384×288.
MethodAPAP50ARTime (ms)Hard-OKS-NMS67.787.191.46.14Soft-OKS-NMS67.786.791.56.8Parallel-OKS-NMS67.886.791.56.0


## 6. Conclusion

In this paper, we propose a lightweight pose estimation network, which can achieve an AP score of 69.0 on the MSCOCO val set with only 1.5M parameters and 1.23 GFLOPs. However, we found that our model has some gaps compared to high-performance algorithms, mainly because we are missing the fusion of multiscale information. Designing complex networks and introducing the fusion of multiscale information will increase the inference speed of the model. In future work, we will redesign the backbone network for human pose estimation by introducing multiscale information to balance accuracy and speed.



---

*Source: 1333250-2021-12-17.xml*