Tilting at windmills: Data augmentation for deep
pose estimation does not help with occlusions
Rafal Pytel

Osman Semih Kayhan

Jan C. van Gemert

Computer Vision Lab
Delft University of Technology

Computer Vision Lab
Delft University of Technology

Computer Vision Lab
Delft University of Technology

Abstract—Occlusion degrades the performance of human pose
estimation. In this paper, we introduce targeted keypoint andbody part occlusion attacks. The effects of the attacks are systematically analyzed on the best performing methods. In addition, we
propose occlusion specific data augmentation techniques against
keypoint and part attacks. Our extensive experiments show thathuman pose estimation methods are not robust to occlusion anddata augmentation does not solve the occlusion problems. 1

Original image

Occluded image(blackout nose)

Original image

Occluded image(blurring corpus)

I. I NTRODUCTION
Human Pose Estimation is the task of localizing anatomical
keypoints such as eyes, hips, knees and localizing bodyparts like head, limbs, corpus, etc., with many applications
in segmentation [24], [25], [52], action recognition [28], [30],[40], pose tracking [14], [54], gait recognition [39], [44],autonomous driving [12], [32], [50], elderly monitoring [10],[31] and social behaviour analysis [22], [48]. All these applications rely on correct and robust pose estimation. In thispaper we investigate the robustness of human pose estimation
methods to a natural and common effect: Occlusions.
Occlusions are common and occur frequently in the wild
as for example by a random object, another person [15], andself-occlusion [18]. Prior works address occlusion in a general
way and exploits segmentation [32] or depth information [33].
Where [36] evaluates robustness with image and domainagnostic universal perturbations. In contrast, we systematicallyanalyze targeted occlusion attacks not only for keypoints, butalso for and body parts and investigate the sensitivity of pose
estimation to occlusion attacks.
A promising solution to occlusions is data augmentation,
which is practically a default setting for deep learning applications [37] where image flipping, rotation, and scaling
offer endless data variations [6], [37], [45]. As such, regional
dropout and mixup methods improve the generalization performance of image classification [9], [16], [41], [46], [49],[55], [59], [60], object localization and detection [7], [11], [38]and segmentation [13]. In pose estimation, [19] applies region
based augmentation by exchanging a single keypoint patchwith a random background patch. More recent approaches[42], [53] use half-body augmentation wherewith the presenceof more than 8 keypoints, by choosing upper or lower body
keypoints. We implement systematic data augmentation meth1 For the code and the extended version:

https://github.com/rpytel1/occlusion-vs-data-augmentations

Fig. 1: Qualitative example how HRNet-32 [42] predictions
change after keypoint blackout on the nose (first row) andpart blurring on the corpus (second row). For both examples
keypoints change for head, nose, eyes and ears.
ods for occlusion for keypoint and body parts to investigate
how data augmentation can remedy occlusion attacks.
We have the following contributions: First, we conduct
a structured investigation on the occlusion problem of pose
estimation and introduce occlusion attacks. Second, we investigate occlusion-based data augmentation methods. Third, weshow that data augmentation does not provide robustness to
occlusion attacks.
II. R ELATED WORK
Human Pose Estimation. Deep learning methods in humanpose estimation can be divided into 2 categories: bottom-up
and top-down. Bottom-up approaches [3], [6], [21], firstly localize identity-free keypoints and then group them into person
instances. Top-down approaches [5], [29], [42], [53] firstly
detect a person in the image and then perform a single person
estimation within the bounding box. The top-down approaches
achieve the state of the art results on various multi-person
benchmarks such as COCO [26], MPII [1]. Within top-downright eye

right side

left eyeleft earright earnose
right shoulder

left shoulder

head
corpusright arm

Keypoint attack
HRNet [42]SimpleBaseline [53]right hipblackout (P)0.0

left arm

HRNet [42]SimpleBaseline [53]

−8

−0.5

−6

−11

−1.0

left elbow
left wrist

right elbow

Part attack

blackout (K)left side
upper body

−1.5

−15

hips

left hipleft kneeright wrist

right legright knee

blurring (K)left leg

left ankle

meanout (K)blurring (P)meanout (P)Deprivation in mAP

Deprivation in mAP

(a)

(b)

right ankle

lower body(a) COCO keypoints (b) Part mapping for (c) Part mapping forsmaller parts.
larger parts.
annotations.

Fig. 2: Visualization of keypoint annotations in COCO dataset
and proposed part mapping.

approaches 2 categories can be distinguished: regressing direct
location of each keypoint [4], [47] and keypoint heatmaps
estimation [8], [29], [42], [51], [53] followed by choosing thelocations with the highest heat values as the keypoints. The
best performing methods on COCO keypoint challenge use
a cascade network [5], [23] to improve keypoint prediction.
The ’SimpleBaseline’ [53] proposes simple but effective improvement by adding few deconvolutional layers to enlarge
the resolution of output features. HRNet [42] which is built
from multiple branches can produce high-resolution featuremaps with rich semantics and performs well on COCO. Someworks advance performance of HRNet via improvement overstandard encoding and decoding of heatmaps [58] and basingdata processing on the unit length instead of pixels [17] with anadditional off-set strategy for encoding and decoding. Becauseof their good accuracy and wide adaptation, we focus on topdown methods, HRNet and SimpleBaseline and bottom-up
approach Higher HRNet.
Occlusion in pose estimation. Occlusion in pose estimation
is an under-studied problem. In [36] analyses of occlusions aredone for deep pose estimators by domain-agnostic universal
perturbations. More recently, attempts to solve the occlusion
problem in pose estimation are suggested via the usage ofsegmentation of occluded parts [32] and depth of in an image[33]. OcclusionNet [34] predicts occluded keypoints via graphneural networks yet it is applied only on vehicles. Different
from these methods, in our paper we introduce keypoint
occlusion attacks and body part occlusion attacks and givea structured analysis of occlusion on human pose estimation.
Data augmentation. Data augmentation is a strong,
simple and popular approach to increase model robustness.
Removing part of the image improves generalization of image
classification [9], [55], [60] and object localization-detection[7], [11], [38]. Mixup [16], [46], [59] approaches which
create a combination of two images are often used in image
classification. [13][57] combine regional dropout and MixUp
methods for image segmentation [13] and image classification[57] task. [19] proposes a cutmix-like approach where a small
patch from the background is pasted on the single keypoint
or vice versa. For the human pose estimation methods [4],Fig. 3: Robustness comparison of HRNet [42] and SimpleBaseline [53] against (a) keypoint and (b) part occlusion
attacks. HRNet is more robust against both attacks, yet both
attacks drop performance, where part attacks deteriorate more.

[51], [56], scaling, rotation and flipping is commonly used asdata augmentation. Random cropping is also used in bottomup approaches [3], [6], [21]. More recent top-down approaches[5], [42], [53] employ the usage of half body transform bya probability of 0.3 choosing either upper or lower body
keypoints. We introduce and evaluate new data augmentation
methods for keypoint and for body parts specifically designed
against occlusion attacks for human pose estimation.
III. S ENSITIVITY TO OCCLUSION ATTACKS
We investigate the effect of occlusion attacks on MS COCO
dataset [26]. COCO contains challenging images with theunconstrained environment, different body scales, variety ofhuman poses and occlusion patterns. The dataset contains over
200k images with 250k person instances labelled with 17
keypoints. Models are trained on COCO train2017 datasets
which includes 57k images and 150k person instances. The
evaluation is done on val2017 set which contains 5k images.
The occlusion attack experiments are conducted with HRNet [42] and Simple Baseline [53] for two aspects: (i) keypoint
attacks, where the occlusion area is a centred circle on thechosen keypoint, (ii) body part attacks, where the occlusionarea is the minimum rectangle covering all keypoints of achosen part. The COCO keypoints and the proposed groupsof body parts can be seen in Figure 2. For the analyses,
COCO pretrained HRNet and Simple Baseline are evaluatedby the performance of the network against keypoint and partocclusion attacks on COCO validation set.
HRNet and SimpleBaseline produce heatmap instead ofpredicting direct single location for each keypoint. The groundtruth heatmaps are generated by using 2D Gaussian of size
13x13. Thus, as a default, we choose the size of the occlusion
circle with a radius of 6 pixels for keypoint attacks to cover
the keypoint heatmap. We have 3 different keypoint attacks: (i)Gaussian blur (blurring) attack, (ii) attack by filling with black
pixels (blackout), (iii) attack by filling with a mean intensity
value of a given image (meanout).
Body parts occlusion attacks are designed to draw a minimum rectangle which covers all the keypoints of a chosenpart. Similar to the keypoint attacks, we have 3 different part

A. How sensitive to key point occlusion attacks?
First, we analyze the effect of the occlusion size on the
average performance of the pose estimator on all keypoints.
Figure 4 indicates that pose estimator performance is inversely
proportional to the occlusion size and blurring, blackout, andmeanout attacks on average perform similarly. The size of theocclusion decreases the average performance of the estimatorby approximately 3% when the radius of the occlusion circle
is chosen as 18 pixels.
Second, we show the class-specific performance drops foreach individual keypoints for each attack. In Figure 5, attacking nose causes serious loss in mAP, almost 5% for blackout,
4.4% for meanout and 1.2% for blurring. The empirical results
indicate that the nose is the most important keypoint since
the occlusion of the nose causes notable performance drop.
After the nose, each eye influences the performance of otherkeypoints mostly by approximately 1% with each occlusion
attack. Keypoints from less densely annotated places likeankles or wrists are the least influential.
If we check the analysis of the reduced accuracy per
keypoint for the case of attacking nose (Figure 6a), the mostaffected keypoints are the ones within close distance, which
are eyes and ears due to being a part of the head. Interestingly,
occluding nose affects the performance of the left eye estimation more than occluding the left eye itself, respectively byapproximately 10% and 5% (Figure 6a, 6b). If we investigate
per keypoint performance for occluding left ankle, it can beseen that the deprivation is by several magnitudes smaller thanin case of the nose or left eye occlusions. From the observationof the analyses, it can be drawn that HRNet [42] is not robust
to keypoint occlusion attacks.
B. How sensitive to part occlusion attacks?
We analyze the effect of the part occlusion attacks on each
body parts given in Figure 2. Attacking the upper body, left andright sides influence the overall performance the most, by morethan 44%, 24% and 24% with blackout attack respectively
since these three parts include the majority of the keypoints(Figure 7). When we examine keypoint-specific accuracy drops
for the remaining keypoints of the upper body, it is clear thatblackout is the most influential attack, with a drop of almost
3% for left and right ankle (Figure 8a). If we investigate perkeypoint behaviour for the corpus (Figure 8b), we observe
significant degradation of the performance on all the keypoints,

01

Loss in mAP

attacks which are applied to the occlusion area: blurring withthe kernel size 31 and sigma 5, blackout and meanout. These
attacks can be applied on both small parts such as head, arms,
hips and larger parts like upper body, lower body, left andright side (Figure 2 b and c).
We compare HRNet and Simple Baseline according to theirrobustness to keypoint and part occlusion attacks. Figure 3
shows that both attacks are quite successful as occlusioncauses the performance to drop. HRNet is more robust against
keypoint and part occlusion attacks. For further analyses, weonly use HRNet as a baseline for our investigations.

methodblurring
meanout
blackout

2

3

4

5
4

6

8

10

12

14

16

18

Occlusion size

Fig. 4: The relation between occlusion size and average loss
in performance for keypoint level methods. Occlusion size
greatly affects the performance.
with left and right ankle affected the most. Interestingly,
attacking on one side improves performance of the the otherside (Figure 8c). Attacking on left side increases the mAPscore of right side such as shoulder, ear, elbow keypoints. The
analysis demonstrates that the pose estimator is sensitive topart occlusion attacks.
IV. O CCLUSION AUGMENTATION AGAINST ATTACKS
We evaluate two main human pose estimation datasets:
COCO [26] has 200k images with 250k person instances,labelled with 17 keypoints and MPII [1] has 40k persons,each labelled with 16 joints. The train, validation and test sets
include 22k, 3k and 15k person instances respectively. For theevaluation of MPII dataset, the validation set is used since thelabels of the test set are not available.
For training HRNet [42] models on COCO [26] and MPII
[1] we follow the original pipeline of HRNet. For COCO
dataset, human detection boxes are extended to fit 4:3 aspectratio, and cropped from the image and resized to 256x192.
The pose estimator is trained with the keypoint location of
the joints. The data augmentations that are used in HRNet
training include random rotation ∈ [−45o , 45o ], random scale∈ [0.65, 1.35], random flipping and half-body augmentations.
The Adam optimizer [20] is used to train the network withthe learning rate schedule following [53], starting with 1e − 3and reduced to 1e − 4 and 1e − 5 at 170th and 200th
epochs respectively and the training is completed at the 210th
epoch. For MPII dataset, the training procedure of HRNet
is as followed: 256x256 input size is used and half-bodyaugmentations are discarded. For the evaluation of the models,
Object Keypoint Similarity (OKS) for COCO and Percentage
of Correct Keypoints (PCK) for MPII are used.
During testing, HRNet firstly employs an object detection
algorithm to obtain boxes with a single person. Afterwards thepose estimator produces the keypoint location of the joints.
A. Occlusion augmentation
We investigate the following three methods: (i) Targeted
Blurring, (ii) Targeted Cutout, (iii) Targeted PartMix. The
augmentation techniques are called as targeted, because weOverall deprivation (mAP)

0
−1
−2
−3

blurring
meanout
cutout−4−5nose

L. eyeR. eyeL. earR. earL. shoulder R. shoulder

L. elbow

R. elbow

L. wrist

R. wrist

L. hipL. kneeR. kneeL. ankle

R. ankle

Occluded keypoint−10−15
blurring
meanout
cutout

−20
−25
nose

L. eyeR. eyeR. ear

0
−5
−10
−15
blurring
meanout
cutout−20−25

L. earnose

L. eyeKeypoint

R. eyeL. ear

Keypoint-specificdeprivation (mAP)0−5Keypoint-specificdeprivation (mAP)

Keypoint-specificdeprivation (mAP)

Fig. 5: Overall loss in mAP after performing keypoint level occlusion. L. and R. correspond to the left and right side respectively.
To note that, the occluded keypoint is included in the evaluation. Occluding nose causes the highest loss in performance.
0.2
0.1
0.0
−0.1
−0.2

−0.4
−0.5

R. ankle

blurring
meanout
cutout−0.3

L. ankle

L. knee

Keypoint

R. ankle

L. ear

L. wrist

Keypoint(a) The nose is the most influential keypoint (b) When we occlude the left eye, there is a (c) Left ankle is one of the least influential
causes a significant drop in the performance smaller loss in keypoint-specific performance keypoints with loss only visible for meanout
for the closest keypoints - left eye and right for the left eye than while occluding nose.
for occluded keypoint.
eye by around 10%.

Overall deprivation (mAP)

Fig. 6: Loss in AP for top 5 keypoints with largest deprivation, when an individual key point is occluded.

1
0−1
blurring
meanout
cutout−2−3
upper body corpus

L. legR. legL. armR. armhipheadlower bodyL. sideR. sideOccluded part0
−1
blurring
meanout
cutout−2−3
L. ankle

R. ankle

R. kneeL. kneeL. elbow1
0−1
blurring
meanout
cutout−2−3
R. ankle

L. ankle

Keypoint(a) Significant loss in performance for all of
the remaining keypoints. Blackout affects themethod most.

R. wrist

L. kneeL. wrist

Keypoint(b) Similar loss across remaining keypoints,
indicating that corpus is one of the mostinfluential parts.

Keypoint-specificdeprivation (mAP)1

Keypoint-specificdeprivation (mAP)

Keypoint-specificdeprivation (mAP)

Fig. 7: Change in mAP for various parts occluded. Upper body and sides are the parts that cause the highest loss in theperformance.
10−1
blurring
meanout
cutout−2−3
R. shoulder

R. earR. elbow

R. kneeR. wrist

Keypoint(c) Occluding the left side of the body improves the performance of right shoulder, ear
and elbow.

Fig. 8: Change in AP for top 5 keypoints with the largest difference, when chosen part is occluded.

apply them on target locations of keypoints or parts insteadof random location in the image. It is important to state thatthe proposed augmentation techniques are introduced after thebounding box person detection, and it thus does not affect theobject detection method.
Targeted Blurring. We use Gaussian blur for two types oftargeted blurring: (i) keypoint blurring with a kernel size of 9

pixels (Figure 9a) and (ii) part blurring with a kernel size of
31 pixels shown in Figure 9d.
Targeted Cutout. The size of the keypoint cutout (Figure 9b-9c) and part cutout (Figure 9e) are similar to theblurring equivalents. Instead of blurring, the area is coloredwith mean value of the image.
Targeted PartMix. The method is designed to mitigate

(a) Blurring on left
hip (almost not visible).

(b) Cutout on left hip.

(c) Multi keypoint
Cutout.

(d) Part Blurring.

(e) Part Cutout.

(f) PartMix on theright leg with apasted random arm.

Fig. 9: Targeted keypoint augmentations: a, b, c and targeted part augmentations: d, e, f.

B. Analyses of occlusion augmentation
All the following augmentation methods, except baselines,
already include flipping, rotation, scaling and half-body augmentations. Each network obtains the boxes from Cascade
RCNN [2] detector which has ResNet50 backbone. The resultsof each method can be seen in Table I.
Baselines. Table I indicates 3 baseline variants. Firstly,
HRNet without any augmentations obtains only 65.3% mAPscore. Secondly, adding flipping, rotation and scaling augmentations improve non-augmented baseline by 8.6%. Last
variant is half body augmentation which adds only 0.4%
improvements on rotation and scaling augmentations.
Single keypoint augmentations. We check the performanceof 3 different augmentations: blurring, cutout and a combination of two of them which are applied on a single keypointwith the varying probability of 0.2 and 0.5 (Figure 9a-9b). We
observe the highest improvement for blurring and cutout by
0.2% when the probability is chosen as 0.5 (Table I). Other
single keypoint variants do not improve the performance.
Multi-keypoint augmentations. We applied random multikeypoint variant blurring and cutout with a maximum of 5
keypoints with a probability of 0.2 (Figure 9c). The augmentation decreases the model performance by 0.4%.
Part augmentations. 4 different part augmentation methods
are used: part blurring, part cutout, a combination of boththem and PartMix (Figure 9d, 9e and 9f respectively). To
demonstrate the effect of each augmentation, we apply themwith a probability of 0.2 and 0.5. In addition, the effectof removing the labels of the occluded keypoint is also
investigated as removal column in Table I.
In the bottom part of Table I, cutout and PartMix show
0.2% and 0.1% improvements respectively. In all the variantsof blurring, small degradation or no improvement is observed.

The combination of part level variants of cutout and blurring
indicate some decreases of the performance for the removal
configuration with probability of 0.2 and 0.5 and do not
improve in non-removal configuration.
To conclude to findings from the Table I, flipping, rotationand scaling augmentations add a huge performance gain to
the HRNet. However, including half-body, the occlusion based
augmentation methods do not improve the performance of thepose estimator significantly.
Difference over baseline
model
Baseline
Blurring keypoint
PartMix (no remove)
Cutout part (no remove)

0.5

Difference in mAP

the occlusions caused by another person (Figure 9f). In thisapproach, a different part from a random image is pasted in theplace of a body part area. In this process, the keypoint labelsof newly pasted part are not introduced to heatmap labels.
This augmentation is only performed on body parts. Similar
to the part level blurring and cutout augmentation methods, theoccluded keypoints under the pasted area are still predicted.

0.4
0.3
0.2
0.1

76.5

75.0

74.4

74.3

74.1

Efficient Det D7

Faster RCNN
(Xception 101)

Cascade RCNN
(R50)

RetinaNet(101)

0.0
-0.1
-0.2
Ground truth

Fig. 10: Performance of chosen augmentations for HRNet-32
on various detection backbones and ground truth boxes. The
ground truth bounding box performs best. Yet, none of thedata augmentation methods help to improve performance over
0.2% for any object detector.
The effect of the object detection algorithms. HRNet [42]is a top-down approach which utilizes an object detection algorithm to obtain human instances. Therefore, the performanceof the pose estimation considerably depends on the detection
performance, namely detected human instances.
By the evidence of the Table I, we choose keypoint blurring,part cutout and PartMix methods for further analysis as they
are the most promising augmentations.
We evaluate the pose estimation performances of vanilla
HRNet and also of HRNet with the chosen augmentation
methods with two 2-stage detectors, Faster RCNN [35] with
XCeption 101 backbone and Cascade RCNN [2]; 2 singlestage detectors, RetinaNet [27] and EfficientDet D7 [43]; andby using ground truth boxes of human instances (Figure 10).

Evaluation results
50

AP 75

AP M

AP L

AR

level

removalpAP

APBaseline (no augments)
Baseline (flip, rot, scale)
Baseline (flip, rot, scale, half-body)

-

-

-

65.3
73.9
74.3

86.4
90.0
90.6

72.6
80.9
81.7

62.6
70.4
70.7

70.7
80.3
80.7

70.2
78.3
78.8

Blurring

k
k

7
7

0.2
0.5

74.3
74.5

90.4
90.4

81.6
81.8

70.8
70.8

80.6
80.8

78.7
78.7

Cutout

k
k

7
7

0.2
0.5

74.3
74.5

90.4
90.5

81.7
81.7

71.0
70.9

80.3
80.7

78.7
78.8

Cutout + Blurring

k
k

7
7

0.2
0.5

74.0
74.3

90.4
90.5

81.1
81.1

70.4
70.8

80.3
80.6

78.4
78.6

Blurring

p
p
p

X
X
7

0.2
0.5
0.5

74.3
74.0
74.1

90.5
90.5
90.3

81.7
81.1
81.1

70.6
70.5
70.6

80.8
80.4
80.2

78.6
78.4
78.5

Cutout

p
p
p

X
X
7

0.2
0.5
0.5

74.2
74.2
74.5

90.5
90.3
90.5

81.2
81.1
81.6

70.8
70.6
70.9

80.4
80.4
80.7

78.6
78.6
78.8

Cutout + Blurring

p
p
p

X
X
7

0.2
0.5
0.5

73.4
73.9
74.3

90.3
90.4
90.4

80.8
81.0
81.2

69.9
70.5
70.6

79.5
80.0
80.5

77.8
78.3
78.6

Augmentation

Multikeypoint (max. 5)

-

-

0.2

73.9

90.1

80.9

70.5

80.2

78.3

PartMix

-

X
7

0.5
0.5

74.3
74.4

90.5
90.7

81.1
81.5

70.7
71.1

80.6
80.5

78.7
78.8

TABLE I: Comparison of augmentation variants on COCO validation set for HRNet using CascadeRCNN bounding boxes.
Upper-part indicates single-keypoint augmentation and bottom-part shows multiple-keypoint augmentation. k and p in thelevel column represent keypoint and part augmentations respectively. Removal column indicates if the occluded keypoints are
removed from prediction. Column p is the probability of augmentation. Keypoint cutout and blurring, and part cutout andPartMix improve the performance. Other variants obtain results either on a par with baseline or worse than baseline.
Evaluation results
Augmentation
Baseline
Blurring
Cutout
PartMix

level

removep

HeadShoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

k
p
-

7
7
7

0.5
0.5
0.5

97.1
97.3
97.2
97.4

95.9
95.9
96.3
96.2

90.4
90.5
90.7
91.0

86.4
86.2
86.7
86.8

89.1
89.2
89.4
89.2

87.2
86.4
86.7
86.7

83.3
83.1
83.3
83.0

90.3
90.3
90.5
90.5

TABLE II: Results on MPII dataset. Keypoint blurring obtains on a par with the HRNet baseline, yet part cutout and PartMix
increase the performance.

All the augmentations indicate improvements using groundtruth bounding boxes by 0.2% for keypoint blurring and
PartMix, and 0.4% for part cutout. All the chosen augmentation methods obtain better result with Cascade RCNN andRetinaNet 0.1 − 0.2% depending on the augmentation. With
EfficientDet D7 detector, keypoint blurring and part cutout
result in similar to baseline except 0.1% improvement by
PartMix. For Faster-RCNN, keypoint blurring shows 0.2%
increase, yet part cutout degrades the performance by 0.1%.
The performances of baseline and the augmentations varydepending on the object detector. The augmentation methods
improves the results slightly, yet the gain is insignificant.
Performance on MPII. We also test the data augmentation
methods on MPII dataset (Table II). If we check the total
contribution of the proposed augmentations, keypoint blurring
result in on a par with baseline, yet part cutout and PartMix
increase the performance by 0.2% for the metric PCK@0.5.

The largest improvement per keypoint is observed for elbows
by 0.6% and wrists by 0.3%, with the degradation on knees
and ankles by 0.4% and 0.2% respectively.
Similar to analyses on the COCO dataset, the proposed
augmentations can only improve the performance slightly.
How occlusion robust is data augmentation? Figure 11
shows the robustness of the baseline and the proposed augmentations to the occlusion attacks. The analysis is done on
COCO dataset and the results are shown as mAP score of all
keypoints. We can clearly see that training with the keypoint
blurring augmentation makes the network more robust against
blurring attack, but there is no significant improvement forthe other keypoint attacks. In case of part attacks, we observe
an improvement across all augmentation methods over thebaseline. For the part augmentations, there is a significant
improvement against all part level attacks in comparison
to baseline. Specifically, PartMix has almost no advantages

Part attacks

Keypoint attacks

blackout (P)blackout (K)HRNet Baseline
PartMix
Cutout (P)Blurring (K)HRNet Baseline
PartMix
Cutout (P)Blurring (K)

1
0

−9

−1

−6

−3

−14

−2

meanout (K) blurring (P)blurring (K)0meanout (P)Deprivation in mAP

Deprivation in mAP

(a)

(b)

Fig. 11: Robustness comparison of proposed methods against(a) keypoint and (b) part occlusion attacks. Part augmentations
improve the baseline but does not solve occlusion.
Evaluation results
50

AP 75

AP M

AP L

73.0
72.1
72.9
73.0

61.5
60.6
60.7
61.3

76.1
75.7
75.6
75.8

Augmentation

AP

AP

Higher HRNet
Blurring (K)Cutout part (no remove)
PartMix (no remove)

67.1
66.5
66.6
67.0

86.2
86.3
86.4
86.4

TABLE III: Results for bottom-up method, Higher HRNet [6].
The keypoint blurring, part cutout and Partmix degrade theperformance of bottom-up methods. The augmentations do not
help Higher HRNet.

against keypoint attacks, however, it improves part level methods about more than 5% in comparison to baseline. Part cutout
obtains similar performance with PartMix against part attacks.
Proposed augmentations reduce the performance deprivations
when we apply occlusion attacks, yet data augmentation still
does not solve the occlusion problem.
C. Augmentation on bottom-up method: Higher HRNet
We also apply occlusion augmentations on Higher HRNet [6], a bottom-up method. Higher HRNet is built on
HRNet-32 and inputs 512x512 sized images. The training
procedure follows Higher HRNet implementation from thepaper. Unlike top-down methods, Higher HRNet operates on
full-image and try to obtain the keypoints of each instancefrom the full-image. When applying the augmentations on
Higher HRNet, we target all the human instances in the image.
Results in Table III show the augmentation methods to improve AP50 score slightly. For AP, all augmentations degrade
performance by 0.6% for keypoint level blurring, by 0.5% forpart level cutout and by 0.1% for PartMix. Hence, using partand keypoint augmentations do not improve the performanceof a bottom-up method.
V. D ISCUSSION AND C ONCLUSION
In this study, we investigate the sensitivity of human pose
estimators to occlusion. Firstly, we introduce targeted keypoint
and body part occlusion attacks to show how much occlusion
affects the performance. Secondly, keypoint and part based
data augmentation techniques against occlusion are investigated. The structured analyses indicate that deep pose estimators are not robust to occlusion. With all the bells and whistles,

the current and proposed data augmentation methods do notbring significant improvements on the performance of the topdown pose estimators and even reduce the performance for thebottom-up approaches. Our paper is important because it helps
data scientists looking for improvements against occlusions to
not work on data augmentation. Battling occlusions is still an
open problem for human pose estimation.
Part based attacks and augmentation are applied as a rectangle shape. This fact can introduce unusual artefacts because
natural occlusions can have arbitrary shapes. Each keypoint
augmentation is applied as a circle that covers the related
keypoint, yet in reality, keypoint occlusions can occur withnumerous shapes and ways e.g. self occlusion, occlusion byother object. Moreover, for bottom-up approaches, the input
image into the network may have more perturbations since thefull image can contain multiple instances. This fact can harm
the learning process.
R EFERENCES[1] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt
Schiele. 2d human pose estimation: New benchmark and state of theart analysis. In IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2014. 1, 3
[2] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into highquality object detection. 2018 IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 6154–6162, 2018. 5[3] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh.
Openpose: Realtime multi-person 2d pose estimation using part affinity
fields. CoRR, abs/1812.08008, 2018. 1, 2
[4] J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik. Human pose
estimation with iterative error feedback. In 2016 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pages 4733–4742,
2016. 2[5] Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, GangYu, and Jian Sun. Cascaded pyramid network for multi-person pose
estimation. In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2018. 1, 2
[6] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S.
Huang, and Lei Zhang. Higherhrnet: Scale-aware representation learningfor bottom-up human pose estimation. In The IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), June 2020. 1, 2, 7
[7] J. Choe, S. Lee, and H. Shim. Attention-based dropout layer for weakly
supervised single object localization and semantic segmentation. IEEE
Transactions on Pattern Analysis and Machine Intelligence, pages 1–1,
2020. 1, 2
[8] Xiao Chu, Wei Yang, Wanli Ouyang, Cheng Ma, Alan Yuille, andXiaogang Wang. Multi-context attention for human pose estimation.
pages 5669–5678, 07 2017. 2[9] Terrance Devries and Graham W. Taylor. Improved regularization of
convolutional neural networks with cutout. CoRR, abs/1708.04552,
2017. 1, 2
[10] Philipe Ambrozio Dias, Damiano Malafronte, Henry Medeiros, andFrancesca Odone. Gaze estimation for assisted living environments.
In The IEEE Winter Conference on Applications of Computer Vision,
pages 290–299, 2020. 1[11] Nikita Dvornik, Julien Mairal, and Cordelia Schmid. Modeling visualcontext is key to augmenting object detection datasets. In Proceedings ofthe European Conference on Computer Vision (ECCV), pages 364–380,
2018. 1, 2
[12] Zhijie Fang and Antonio M. López. Intention recognition of pedestrians
and cyclists by 2d pose estimation. ArXiv, abs/1910.03858, 2019. 1[13] Geoffrey French, Timo Aila, Samuli Laine, Michal Mackiewicz, andGraham D. Finlayson. Consistency regularization and cutmix for semisupervised semantic segmentation. CoRR, abs/1906.01916, 2019. 1,
2[14] Rohit Girdhar, Georgia Gkioxari, Lorenzo Torresani, Manohar Paluri,
and Du Tran. Detect-and-Track: Efficient Pose Estimation in Videos. In
CVPR, 2018. 1

[15] Thomas Golda, Tobias Kalb, Arne Schumann, and Jürgen Beyerer.
Human pose estimation for real-world crowded scenarios. CoRR,
abs/1907.06922, 2019. 1[16] Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear
out-of-manifold regularization. CoRR, abs/1809.02499, 2018. 1, 2
[17] Junjie Huang, Zheng Zhu, Feng Guo, and Guan Huang. The devil isin the details: Delving into unbiased data processing for human pose
estimation. In The IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), June 2020. 2[18] Ying Huang, Bin Sun, Haipeng Kan, Jiankai Zhuang, and Zengchang
Qin. Followmeup sports: New benchmark for 2d human keypoint
recognition. ArXiv, abs/1911.08344, 2019. 1[19] Lipeng Ke, Ming-Ching Chang, Honggang Qi, and Siwei Lyu. Multiscale structure-aware network for human pose estimation. CoRR,
abs/1803.09894, 2018. 1, 2
[20] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic
optimization. International Conference on Learning Representations,
12 2014. 3[21] Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. Pifpaf: Composite
fields for human pose estimation. CoRR, abs/1903.06593, 2019. 1, 2
[22] L. Ladický, P. H. S. Torr, and A. Zisserman. Human pose estimation
using a joint pixel-wise and part-wise formulation. In 2013 IEEE
Conference on Computer Vision and Pattern Recognition, pages 3578–
3585, 2013. 1[23] Wenbo Li, Zhicheng Wang, Binyi Yin, Qixiang Peng, Yuming Du, Tianzi
Xiao, Gang Yu, Hongtao Lu, Yichen Wei, and Jian Sun. Rethinking on
multi-stage networks for human pose estimation. ArXiv, abs/1901.00148,
2019. 2[24] Zhong Li, Xin Chen, Wangyiteng Zhou, Yingliang Zhang, and Jingyi
Yu. Pose2body: Pose-guided human parts segmentation. In 2019 IEEE
International Conference on Multimedia and Expo (ICME), pages 640–
645. IEEE, 2019. 1[25] Kevin Lin, Lijuan Wang, Kun Luo, Yinpeng Chen, Zicheng Liu, andMing-Ting Sun. Cross-domain complementary learning using pose formulti-person part segmentation. IEEE Transactions on Circuits andSystems for Video Technology, 2020. 1[26] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev,
Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr
Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects incontext. CoRR, abs/1405.0312, 2014. 1, 2, 3[27] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr
Dollár. Focal loss for dense object detection. 2017 IEEE International
Conference on Computer Vision (ICCV), pages 2999–3007, 2017. 5[28] Diogo Luvizon, David Picard, and Hedi Tabia. Multi-task deep learning
for real-time 3d human pose estimation and action recognition. IEEE
Transactions on Pattern Analysis and Machine Intelligence, page 1–1,
2020. 1[29] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass
networks for human pose estimation. In ECCV, 2016. 1, 2[30] Xiaohan Nie, Caiming Xiong, and Song-Chun Zhu. Joint action
recognition and pose estimation from video. 2015 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pages 1293–1301,
2015. 1[31] Štěpán Obdržálek, Gregorij Kurillo, Ferda Ofli, Ruzena Bajcsy, Edmund
Seto, Holly Jimison, and Michael Pavel. Accuracy and robustness of
kinect pose estimation in the context of coaching of elderly population.
In 2012 Annual International Conference of the IEEE Engineering in
Medicine and Biology Society, pages 1188–1193. IEEE, 2012. 1[32] Sai Perla, Sudip Das, Partha Mukherjee, and Ujjwal Bhattacharya.
Cluenet : A deep framework for occluded pedestrian pose estimation.
12 2019. 1, 2
[33] U. Rafi, J. Gall, and B. Leibe. A semantic occlusion model for humanpose estimation from a single depth image. In 2015 IEEE Conference on
Computer Vision and Pattern Recognition Workshops (CVPRW), pages
67–74, 2015. 1, 2
[34] N. D. Reddy, M. Vo, and S. G. Narasimhan. Occlusion-net: 2d/3d
occluded keypoint localization using graph networks. In 2019 IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR), pages
7318–7327, 2019. 2[35] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster RCNN: towards real-time object detection with region proposal networks.
CoRR, abs/1506.01497, 2015. 5[36] Sahil Shah, Naman Jain, Abhishek Sharma, and Arjun Jain. On therobustness of human pose estimation, 08 2019. 1, 2[37] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data
augmentation for deep learning. Journal of Big Data, 6(1):60, 2019. 1[38] K. K. Singh and Y. J. Lee. Hide-and-seek: Forcing a network to be
meticulous for weakly-supervised object and action localization. In
2017 IEEE International Conference on Computer Vision (ICCV), pages
3544–3553, 2017. 1, 2[39] Anna Sokolova and Anton Konushin. Pose-based deep gait recognition.
CoRR, abs/1710.06512, 2017. 1[40] Khurram Soomro, Haroon Idrees, and Mubarak Shah. Online localization and prediction of actions and interactions. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 41:459–472, 2019. 1[41] C. Summers and M. J. Dinneen. Improved mixed-example data augmentation. In 2019 IEEE Winter Conference on Applications of Computer
Vision (WACV), pages 1262–1270, 2019. 1[42] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep HighResolution Representation Learning for Human Pose Estimation. 2019.
1, 2, 3, 5[43] Mingxing Tan, Ruoming Pang, and Quoc V. Le. Efficientdet: Scalable
and efficient object detection. ArXiv, abs/1911.09070, 2019. 5[44] H. L. Tavares, J. B. C. Neto, J. P. Papa, D. Colombo, and A. N. Marana.
Tracking and re-identification of people using soft-biometrics. In 2019
XV Workshop de Visão Computacional (WVC), pages 78–83, 2019. 1[45] Luke Taylor and Geoff Nitschke. Improving deep learning using genericdata augmentation. arXiv preprint arXiv:1708.06020, 2017. 1[46] Y. Tokozume, Y. Ushiku, and T. Harada. Between-class learning forimage classification. In 2018 IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 5486–5494, 2018. 1, 2
[47] A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deepneural networks. In 2014 IEEE Conference on Computer Vision and
Pattern Recognition, pages 1653–1660, 2014. 2[48] Jagannadan Varadarajan, Ramanathan Subramanian, Samuel Rota Bulò,
Narendra Ahuja, Oswald Lanz, and Elisa Ricci. Joint estimation ofhuman pose and conversational groups from social scenes. International
Journal of Computer Vision, 126(2-4):410–429, 2018. 1[49] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis
Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup:
Better representations by interpolating hidden states. In Kamalika
Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th
International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6438–6447, Long Beach,
California, USA, 09–15 Jun 2019. PMLR. 1[50] Sijia Wang, Fabian Flohr, Hui Xiong, Tuopu Wen, Bao feng Wang,
Mengmeng Yang, and Diange Yang. Leverage of limb detection inpose estimation for vulnerable road users. 2019 IEEE Intelligent
Transportation Systems Conference (ITSC), pages 528–534, 2019. 1[51] S. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh. Convolutional posemachines. In 2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pages 4724–4732, 2016. 2[52] Fangting Xia, Peng Wang, Xianjie Chen, and Alan L. Yuille. Joint multiperson pose estimation and semantic part segmentation. In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), July 2017. 1[53] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for humanpose estimation and tracking. CoRR, abs/1804.06208, 2018. 1, 2, 3, 9[54] Yuliang Xiu, Jiefeng Li, Haoyu Wang, Yinghong Fang, and Cewu Lu.
Pose Flow: Efficient online pose tracking. In BMVC, 2018. 1[55] Huayong Xu, Yangyan Li, Wenzheng Chen, Dani Lischinski, Daniel
Cohen-Or, and Baoquan Chen. A holistic approach for data-driven object
cutout. CoRR, abs/1608.05180, 2016. 1, 2
[56] Wei Yang, Shuang Li, Wanli Ouyang, Hongsheng Li, and Xiaogang
Wang. Learning feature pyramids for human pose estimation. CoRR,
abs/1708.01101, 2017. 2[57] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk
Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to trainstrong classifiers with localizable features. CoRR, abs/1905.04899, 2019.
2[58] Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu.
Distribution-aware coordinate representation for human pose estimation.
ArXiv, abs/1910.06278, 2019. 2[59] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David
Lopez-Paz. mixup: Beyond empirical risk minimization. CoRR,
abs/1710.09412, 2017. 1, 2[60] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.
Random erasing data augmentation. CoRR, abs/1708.04896, 2017. 1, 2

A PPENDIX
M ORE RESULTS ON COCO VAL SET
HRNet results. For this experiment, we increase the input
resolution of images from 256x192 to 384x256. The training
process follows the aforementioned scheme for COCO dataset.
According to the analysis of the performance across avariety of detection backbones shown in Figure 12, we noticethat PartMix is consistently improving performance - with thegreatest boost of 0.4% for Cascade R-CNN and 0.3% forFaster RCNN. For both keypoint blurring and part cutout, weobserve no significant improvement or even the performancedecreases - for part cutout using EfficientDet, Faster RCNN
and RetinaNet and for Blurring using RetinaNet. All the presented augmentations show largest gain for Cascade RCNN.
Occlusion augmentations do not help to solve occlusion problems when higher resolution input is used.

Ground truth

Baseline

T. Blurring (K)Difference over Baseline

0.8

Difference in mAP

across detection backbones, with 0.4% boost in the performance for the ground truth boxes and the boxes produced by
Cascade RCNN, 0.2% for EfficientDet and Faster RCNN and
0.1 % for RetinaNet. Cutout and Blurring improve at most
0.2% across all the detection backbones, apart from 0.4%
for Cutout using ground truth bounding boxes. According to
the results, proposed augmentation techniques do not solve
occlusion problems of SimpleBaseline method.

model
HRNet Baseline
Blurring (K)PartMix (no remove)
Cutout part (no remove)

0.6
0.4
0.2

77.7

76.3

75.8

75.6

75.5

0.2
Ground truth

EfficientDet D7

Faster RCNN(XCeption 101)

RetinaNet(R101)

Cascade RCNN
(R50)

0.0

Fig. 12: Higher resolution input for HRNet 32: the resolution
is changed from 256x192 to 384x256. The best performanceacross detection backbones is observed for PartMix.
Difference over Baseline
model
SimpleBaseline Baseline
Blurring (K)PartMix (no remove)
Cutout part (no remove)

Difference in mAP

0.6
0.4
0.2

72.4

70.9

70.4

70.2

70.3

0.2
Ground truth

EfficientDet D7

Faster RCNN(XCeption 101)

RetinaNet(R101)

Cascade RCNN
(R50)

0.0

Fig. 13: Performance of chosen augmentations for SimpleBaseline on various detection backbones and ground truthboxes. Using the ground truth bounding boxes outperforms
all the SimpleBaseline methods with a detection backbones.
SimpleBaseline results. The usability of occlusion augmentations are not only limited to HRNet, yet they can beused with other top-down methods like SimpleBaseline [53].
In this experiment, we apply the occlusion augmentations
on SimpleBaseline method with different object detection
backbones. The training procedure of the network follows theoriginal implementation.
By checking the performance across the various detection
backbones we observe either small or no improvement at all(Figure 13). PartMix show the most significant improvement

Fig. 14: Qualitative comparison between ground truth (left),baseline (middle) and keypoint Blurring (K) (right). 1st and2nd rows respectively - misplacement of left wrist keypoint
and mismatch between knee keypoints in the baseline andkeypoint blurring fixes the mistakes. 3rd row - both baseline
and proposed method produce wrong keypoints. 4th row baseline produces near-optimal keypoints whilst keypoint blurring makes mistake on left ankle keypoint. Data augmentation
does not solve occlusion problem.
V ISUALIZATION OF RESULTS
Figure 14 presents a qualitative comparison between groundtruth, HRNet-32 Baseline and keypoint blurring augmentation.
In the first and second rows, keypoint blurring outperforms
the baseline by obtaining the position of the left wrist andknee keypoints respectively. In the third row, both baseline andkeypoint blurring produce wrong keypoint predictions. Fourth
row presents failure case when baseline produces near-optimal
annotations, while the method with keypoint blurring predicts
left ankle in place of the right one.