# Local and Global Orientation Correction for Oriented Human (Pose) Detection

Rottana Ly, Dominique Vaufreydaz, Eric Castelli, Sethserey Sam

► To cite this version:

Rottana Ly, Dominique Vaufreydaz, Eric Castelli, Sethserey Sam. Local and Global Orientation Correction for Oriented Human (Pose) Detection. The ASEAN Conference On Emerging Technologies 2022, Dec 2022, Phnom Penh, Cambodia. hal-03857012

HAL Id: hal-03857012

https://hal.science/hal-03857012v1

Submitted on 11 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Local and Global Orientation Correction for Oriented Human (Pose) Detection

Rottana Ly†‡ Dominique Vaufreydaz‡ Eric Castelli‡ Sethserey Sam†

†Institute of Digital Research and Innovation, CADT, Phnom Penh, Cambodia

‡Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France

`{firstname.lastname}@cadt.edu.kh`

`{firstname.lastname}@univ-grenoble-alpes.fr`

## Abstract

Detecting people and/or their pose in real life condition is of interest for several tasks notably for home care for elderly or frail people for instance. In such contexts, the perception system must be able to detect usual but also unusual poses of people in various adversarial conditions. Even if the performance of neural networks on human (pose) detection has significantly increased recently, the human detection in different poses or positions, with partial occlusions, and at multiple scales remains a challenge. In this research, a step towards rotation invariance of human detection is proposed, i.e. to identify a person in a robust way on images containing rotated or oriented human poses. After confirming that data augmentation could not solve this problem, this research explores three ways to address the rotation problem in human pose detection: steerable networks, global rotation correction, and local person orientation approaches. From reported experiments on rotated generated corpora from COCO dataset, there is no significant improvement from integrating steerable approaches into existing architectures. While the state-of-the-art approaches lose up to 67.2 mAP on rotated images, the global rotation correction keeps almost intact the performance for all angles but does not solve the problem of images crowded with several people at various orientations. The local orientation approach permits an average of 4.3 mAP[.5:.95] and 9.5 mAP[0.5:1.5] gain, and even more when combined with the global approach. The reported experiments indicate that it is possible to achieve rotation invariance. The paper ends by discussing possible improvements to strengthen the rotation invariance in perceiving humans.

**Keywords:** human detection, pose detection, orientation correction, rotation invariance

## 1 Introduction

Recent improvements in robotics let envision in a near future an increased role of companion robots in home care of elderly/frail people at home. In this real-life application, a companion robot perceives his human partners to monitor their daily-life activities or to interact with them. For security reasons, the robot must be able to detect some emergency events like fall/discomfort. Perceiving people and/or their pose within an image is of interest in this context.

During the last decade, human detection has attracted significant attention in computer vision and pattern recognition, and there has been great breakthroughs in the fields such as human pose estimation, pedestrian detection, and face recognition [1]. However, there are still some major challenges in human detection in adversarial conditions due to the flexibility of body limbs and the wide range of possible body poses, the variability of camera view-point, the environmental conditions [2] (partial occlusions, illumination conditions, scales), etc.

Although algorithms to perceive human significantly advanced recently [3; 4; 5], this article focuses on the unsolved problem of rotation invariance. Figure 1 illustrates the rotation problem in fallen body detection. On this figure, the dummy and the actual person cannot be perceived by state-of-the-art systems until a rotation correction is applied. The aim of research is to strengthen rotation invariance in perception algorithms to approach human-level performances.

The paper is organized as follows. Section 2 states the rotation invariance problem. Related works on rotation invariance for convolutional neural networks, as well as representative approaches in human detection tasks are depicted
<figure><img src="image_2.png" /><figcaption>Figure 1. Illustration of non-invariance to rotation of convolutional neural networks with Yolov3 [6] (bouding box) and OpenPose [7] (skeleton). (a) Detection result of a puppet from a companion robot point-of-view [8]. The puppet cannot be detected when it is not in usual pose in the camera view point, but when the puppet appears in a standard position, after a 140° rotation, both networks detect it (source [9]). (b) Detection result on a fallen person [10]. The networks are not able to detect (or partially) the fallen person without applying a 90° rotation.</figcaption></figure>

in section 3. The generation process to train and evaluate on rotated datasets, and the evaluation metrics are presented in section 4. Section 5 details the proposed solutions, describing experiments and results. Section 6 provides a discussion on these results and considers possible improvements before conclusion in section 7.

## 2 Problem statement

Current state-of-the-art results on classification or detection benchmarks [1] may let people think that simple image transformation like rotation is no longer a problem for neural networks. Figure 1 illustrates using OpenPose and Yolo networks that it is not the case. The first example, on the left, is extracted from the MobileRGBD corpus [8; 11]. This corpus contains sequences with fallen dummies on the ground that can be used to evaluate, for instance, person-search algorithms embedded on a companion robot. On this example, both neural architectures provide no detection. One might think that the problem lies in color/texture of the dummy, but it is not: if the dummy is oriented as people in the networks' training data, i.e. head up and feet down after a 140° rotation, the detection operates. OpenPose even manages to infer a relevant position for eyes and ears, although there are none on the dummy. The second example on figure 1 is taken from the UR Fall Detection Dataset [10], a corpus containing videos of people recorded while they are simulating falls. Like the previous one, the non-usual body position on the image prevents a correct detection. Yolo does not detect the person and OpenPose only detects a small part of the skeleton. After a 90° rotation, the person is correctly detected by both networks, even if a false positive from OpenPose appears on the

chair. The two depicted examples show that neural networks cannot detect properly a person in non-standard human pose. These results remain identical with other neural network architectures for human (pose) detection (see columns “Orig.” in table 2): for all systems, performance drops drastically.

Two issues concur to prevent networks from being rotation invariant. The first one comes from the very nature of convolutional networks which impedes them to handle rotation invariance. Convolution filters are rectangular, often square. They were designed to efficiently do massive parallel computation on images made of pixel matrices. One can write the general equation of a 2D convolutional layer for an image:

$$ Conv(\mathit{IM}) = W \otimes IT + B \\ \text{with} $$

$$ W \otimes IT = \sum_{j} \sum_{k} W[j, k] * IT[m - j][n - k] \quad (1) $$

where W, IT, B, m and n denote respectively the convolutional kernel weights, the input tensor, the bias and the shape of the convolution kernel. Let $IT^r$ be any rotation of IT, as $\exists_{r,i,j} IT[i][j] \neq IT^r[i][j]$, it is straightforward to write the inequality:

$$ \forall_{r \ in} [0,360^{\circ}] [Conv(\mathit{IT}) \neq Conv(\mathit{IT}^r)] \quad (2) $$

The inequality 2 indicates that, in the general case, the result of a convolution on a rotated tensor $IT^r$ (or image) has no reason to be equal to the same convolution applied on the original one (IT): computed features differ. The same reasoning applies to other layer types.

The second issue concerns the content of the training corpora. Like ImageNet [12] and
MSCOCO [13] datasets, they are drawn from real-life pictures where the majority of people are standing. Even if corpora containing sport images exists [14; 15], they are not sufficiently representative of the great diversity of poses that people can take in adverse conditions, resulting from a fall/discomfort for example (figure 1). Fall dedicated corpora [10; 8; 16] are not big enough and usually not annotated for people detection tasks.

Straightforward solutions to the rotation invariance problem imply data augmentation, i.e. to train applying several transformations on existing data to generate more diversity. Among the available transformations (cropping, scaling, flipping, occlusion management...), rotation is the most noteworthy [17] in this research. Using a rotation transformation, training occurrences in any orientation are expected to be covered by the augmented data. Another solution is learning transformation, which allows to transform the input image or feature map in order to get a better result for the training. These methods suits for tasks such as image classification since they work on data representations relying on random rotations and flips of the training samples [18; 19]. To the best of our knowledge, the aforementioned solutions did not solve rotation problem.

# 3 Related Work

## 3.1 Rotation invariance in CNNs

To tackle rotation problem in object detection, several convolutional neural network architectures can be found in the literature. *Steerable approach* [20] modifies the network filters to produce steerable representations of the input, changing it into a predictable linear transformation. Several significant works related to the rotation of filters, including *Steerable CNNs* [20] which uses symmetry groups compounded of a flip and four 90° rotations and performs pooling within the group to merge the features. *Harmonic Networks (H-Nets)* [21] also enriches feature maps by replacing regular CNN weights with a circular harmonic computation. *H-Nets* are limited to learn filters that are a combination of circular harmonic wavelets. *Oriented Response Network (ORN)* [22] offers feature maps with orientation channels that explicitly encode the location and orientation information of discriminative patterns. ORN approach gives sig-

nificant reduction of the network parameters and improvement in classification performance. *Rotation Equivariant Vector Field Networks (RotEqNet)* [18] is a CNN architecture aiming at dealing with the problem of rotation invariance, equivariance or covariance. In *RotEqNet*, each convolutional filter is applied at multiple orientations and returns a vector field representing magnitude and angle of the highest scoring orientation at every spatial location. It provides both compactness and flexibility suitable to deal with problem with limited training samples. Other significant approaches are based on *Spatial Transformer Networks (STN)* [23; 24]. These learnable modules aim at increasing the spatial invariance of CNN in a computationally and parameter efficient manner. STN remove spatial invariance from images by allowing a neural network to learn how to perform spatial transformations on the input images (i.e. crop a region of interest, scale and correct image orientation, etc.).

As reported by authors, these approaches improve object classification tasks for quite simple objects (MNIST, CIFAR [19]) and/or low amplitude rotations. Conducted experiments confirm that these approaches are not sufficient to tackle rotation problem for human (pose) detection.

## 3.2 State-of-the-art approaches in human detection tasks

This section presents three approaches used as backbone architecture in the reported experiments. These neural network architectures represent different paradigms employed in human (pose) detection. The first one is Yolo [25], which is dedicated to fast object detection regressing bounding boxes. The second one is OpenPose [7], which detects 18 body joints before reconstructing the person skeleton associating them. Finally, AlphaPose [26] is a hybrid top-down approach: it uses Yolo as a person detector and later performs joint detection on the person thumbnail.

### 3.2.1 YOLO

You Only Look Once [25] is one of the fastest bottom-up object detection algorithms computing in one forward pass object bounding boxes and classes. Input image is divided into S × S grid, each grid cell predicts only one object. Since it imposes strong spatial constraints on bounding box predictions, the number of nearby
objects predicted is limited and the localization error is relatively high. Several improvements has been done to increase the number of supported classes and the overall performance in version 2 and 3 [27; 6]. YOLOv4 [28] has been developed to offer a faster and more accurate detector. Several data augmentation strategies were deployed during the training, including CutMix [29], Mosaic data augmentation, Class label smoothing, and Self-Adversarial Training (SAT). As Alphapose (see section 3.2.3) uses YOLOv3 as backbone, our experiments will focus on improving this version.

### 3.2.2 OpenPose

The OpenPose architecture is a real-time multi-person approach to human pose, building bodies from detected keypoints [7]. It has been extended to hand and facial keypoint detections. Based on the bottom-up approach, it first extracts a feature maps and then propagates it into a two-branch CNN. It jointly predicts confidence maps for body parts, and another for so-called part affinity fields (PAF), i.e. pairwise relationships between body parts. After propagating though the network, a bipartite algorithm associates body-part candidates to assemble full body poses for each person in the image.

### 3.2.3 AlphaPose

RMPE (Regional Multi-person Pose Estimation) is a top-down method of pose estimation [26]. First, bounding boxes for humans are gathered using YOLOv3 and fed into the network for pose estimation. Errors in localization and duplicate predictions affect the performance of the pose extractor. To solve this issue, several subparts of the AlphaPose architecture are involved. Symmetric Spatial Transformer Network (SSTN) extracts high quality single person region from inaccurate detections. A Single Person Pose Estimator (SPPE) and a Spatial De-Transformer Network (SDTN) estimate and remap the human pose back to the original image coordinate system. Finally, a parametric pose Non-Maximum Suppression (NMS) technique removes redundant pose proposals [26].

# 4 Datasets and evaluation metrics

## 4.1 Datasets

To the best of our knowledge, among corpora dedicated to fallen person detection [10; 8; 16],

<figure><img src="image_3.png" /><figcaption>Figure 2. Human skeleton in MSCOCO dataset and an example of rotated bounding box annotation. (a) Each human object is labeled with 17 keypoints. (b) The torso orientation is used as angle for the rotated bounding box in the COCOR-BB dataset, preserving inclusion of all pixels within the annotation mask.</figcaption></figure>

none are large enough to train/test deep learning systems reliably, i.e. generalizing not overfitting. Moreover, due to the difficulty of their collect, these specialized corpora contain few variations in the recording environments and human poses, and these poses are usually not labelled as bounding boxes or skeletons. To circumvent this problem, this research employs a strategy using rotation variant of more generalist datasets used to train state-of-the-art human pose detection systems [7; 26]. Indeed, referring to figure 1, one can see that systems trained on these corpora perform well on falls, provided that the right (automatic) rotation correction is applied or that they become rotation invariant. The following sections present these corpora.

### 4.1.1 Original COCO dataset

The Microsoft COCO dataset [13; 30] is an open source object recognition databases widely used in person detection and keypoint detection tasks [7; 26; 6; 31]. It will serve as un-rotated baseline in the later reported experiments.

### 4.1.2 Optimal Angle rotated images

COCO Rotation (COCOR) corresponds to a first learning strategy employed in these experiments. It is not a corpus as such because there is no precomputation to store. It is a 360° data augmentation with online recomputing of keypoint and bounding box position for learning. COCOR-OA (COCOR Optimal Angle) is more complex. The underlying idea is to set training images at the best possible orientation for person detection tasks. AlphaPose is the most efficient architecture from the three chosen representative ones (section 3.2 and table 2). Using
it, the scores for each training sample applying
rotation angles ranged from 0° to 360° with a
10° step are gathered. The angle with the best
pose score (or the smaller angle with the best
score in case of duplicates) is set as *Optimal angle* for that image. All precomputed rotated im-
ages and their associated annotations are store as
the *COCOR-OA* dataset.

**4.1.3 Oriented Bounding-Box Corpus**

Taking in account intra-frame pose variability
needs to detect each person with his/her orien-
tation within image independently. The original
annotation bounding boxes must be modified ac-
cordingly. Due to the extreme variation among
human poses, the human body can be considered
as a rigid torso with moving around head, arms
and legs. Thus, torso orientation is taken as ref-
erence angle for the bounding box computation
when it is possible, i.e. when the person torso
is visible. The figure 2 (at right) illustrates the
new bounding box annotation. In red, the bound-
ing box comes from the original annotations. In
green, the new computed bounding box is au-
tomatically associated to the torso orientation
while including all pixels in the provided mask
annotation. The *COCOR-BB* corpus results from
the whole COCO dataset re-annotation.

## 4.2 Evaluation corpus and metrics

It contains bounding boxes, keypoints and asso-
ciated images from the MSCOCO 2017 valida-
tion dataset and person keypoints 2017 annota-
tions rotated from 0° to 360° with a step of 30°.

As usual in person detection tasks, the eval-
uation metric is the mean Average Precision
(mAP), a measure combining recall and preci-
sion for ranked retrieval results. In this paper, the
reported mAP.5:.95 or mAP values corresponds
to the average AP score for IoU from 0.5 to 0.95
with a 0.05 step, and the mAP0.5 corresponds to
the AP score for IoU ≥ 0.5.

**5 Explored rotation invariance methods**

In this paper, three methods are questioned to
tackle the rotation problem in pose detection:
integrate *steerable* layers into existing architec-
tures, perform *global rotation correction*, and
detect persons taking in account their *local ori-
entation*.

5.1 Steerable approach

On human pose detection, AlphaPose [26] al-
ready includes a steerable paradigm as it uses an
STN as first stage. As illustrated in table 2, it is
nonetheless unable natively to handle rotated im-
ages. Thus, other ways to address rotation prob-
lems are to be considered. Approaches like H-
Nets [21] and RotEqNet [18] tend to compute ro-
tation invariant features by proposing dedicated
convolutional layers. These proposals work well
on some object classification like texts, vehicles
and animals, as well as in simple dataset like
MNIST and CIFAR [19] but were not experi-
mented on human pose detection. To do so, we
instrumented OpenPose and AlphaPose by in-
tegrating H-Nets and RotEqNet as first layers.
These new networks were trained using stan-
dard schema on COCO dataset, only modify-
ing data augmentation to include rotations over
360°. They are evaluated on the COCOR-eval
dataset.

All gathered results demonstrate that, even if
*Harmonic networks* and *RotEqNet* were already
reported to contribute to rotation invariance [21;
18], it is not the case for human pose detec-
tion with state-of-the-art architectures. For all
systems and all angles, the relative performance
drops from 20%, for small rotation angles, to
up to 85% for bigger ones. Knowing that the
number of parameters in these neural networks
increased significantly, the current steerable ap-
proached where abandoned, exploring rotation
correction globally on the full image or on de-
tected bounding boxes.

5.2 Global rotation correction

The global orientation of images provides in-
formation about the orientation of the elements
composing them. As already said in section 2,
the underlying assumption is that the majority
of the images in available training datasets are
horizontal real-life pictures. It is thus of inter-
est to detect that an image is not in the expected
orientation. Image reorientation [32; 33] con-
sists in first detecting and then correcting image
orientation before passing through pose detec-
tor for evaluation. Our proposal is to go fur-
ther by learning an orientation detector sensi-
ble to people orientation within the image using
*COCOR-OA*, in order to maximize pose detec-
tion after image correction. As some previous
works on this topic [34], we named this approach
RotNet (RN). This paper experiments several flavors of this network. The first one classifies the orientation among 360 classes (RotNet-360, aka RN360), the second one, RotNet-SinCos (RNSC), regresses the angle value. Both versions are evaluated using standard COCO and COCOR-OA corpora as training set.

### 5.2.1 Angle classification

RN360 is designed to classify the image into 360 orientations from 0° to 359°. The architecture of RN360 is based on ResNet50 [35]. A fully-connected layer is appended at the end of the architecture to generate the output tensor containing the 360 class probabilities. A pre-trained ResNet model on ImageNet [12] is loaded, then the network is trained on the COCO dataset. All training images are considered suitably oriented, i.e. no prepossessing is applied trying to determine a better orientation. While training, images are rotated randomly over 360°, the rotation angle serving as class label for training with a Cross Entropy loss.

For testing, the COCO-eval corpus is used, error between predicted class and actual rotation angle is computed. The table 1 presents classification results. RN360 has an average angle error around 13° with a standard deviation σ = 5.6. Some rotation angles have better than average results (150°, 240° and 270°). Some others are worst, explaining the standard deviation value, notably 180° which is always difficult to classify as seen in table. Trained using the COCOR-OA corpus, the RN360-OA network has slightly better results with a 1.5 absolute improvement on average angle error. The performance profile and standard deviation are equivalent. Difference between both network is not significant.

### 5.2.2 Angle regression

Another way to learn the orientation angle is to regress it using sine and cosine functions. Similar to RN360, the RNSC architecture is based on ResNet50 [35] pre-trained on ImageNet [12]. Three fully-connected layers with ReLU activations are appended at the end of the network to go down to the sine and cosine values. The network architecture ends by a Tanh activation. Instead of the angle value, sine and cosine are used as training labels. The loss function is the mean error of both values. The network is trained on the COCO dataset with the training

schema described in the previous section.

For evaluation purpose, the value of predicted angle is computed back using the arctan2 function. The RNSC network is evaluated on COCOR-eval. In table 1, one can see that the results are better than previous networks even if RNSC has less parameters. The average error value is around 11° (σ = 2.57). The profile among rotation angles is more stable with less extreme values, notably at 180°. Trained on COCOR-OA, RNSC-OA gets similar scores but is slightly less efficient.

<table><caption>Table 1. Angle prediction errors (in degrees) using RotNet approaches</caption><thead><tr><th rowspan="2"></th><th colspan="2">Angle Classification</th><th colspan="2">Angle Regression</th></tr><tr><th>RN360</th><th>RN360-OA</th><th>RNSC</th><th>RNSC-OA</th></tr></thead><tbody><tr><td>0°</td><td>5.67</td><td><strong>4.18</strong></td><td>4.98</td><td>4.61</td></tr><tr><td>30°</td><td>12.35</td><td>14.56</td><td><strong>10.39</strong></td><td>16.38</td></tr><tr><td>60°</td><td>13.37</td><td>13.36</td><td><strong>11.20</strong></td><td>16.81</td></tr><tr><td>90°</td><td>14.22</td><td>9.17</td><td><strong>8.36</strong></td><td>10.72</td></tr><tr><td>120°</td><td>11.34</td><td><strong>10.77</strong></td><td>11.48</td><td>13.17</td></tr><tr><td>150°</td><td><strong>9.41</strong></td><td>11.52</td><td>10.99</td><td>12.96</td></tr><tr><td>180°</td><td>26.60</td><td>23.67</td><td>11.21</td><td><strong>10.41</strong></td></tr><tr><td>210°</td><td>10.55</td><td><strong>5.51</strong></td><td>11.86</td><td>8.95</td></tr><tr><td>240°</td><td>9.02</td><td><strong>6.93</strong></td><td>15.47</td><td>12.42</td></tr><tr><td>270°</td><td><strong>7.80</strong></td><td>8.56</td><td>10.85</td><td>9.70</td></tr><tr><td>300°</td><td>17.45</td><td>13.27</td><td>13.08</td><td><strong>12.33</strong></td></tr><tr><td>330°</td><td>17.52</td><td>16.67</td><td>12.89</td><td><strong>11.25</strong></td></tr></tbody><tfoot><tr><td>Average</td><td>12.94±5.60</td><td>11.51±5.36</td><td><strong>11.06±2.57</strong></td><td>11.64±3.27</td></tr></tfoot></table>

Comparing all flavors of RotNet, there is no huge difference in terms of performance between angle classification and regression. As depicted, regression networks are more stable among rotation angles with less parameters, and are thus of interest. The best architecture is RNSC trained on standard COCO dataset.

### 5.2.3 Human (pose) detection on corrected images

Having capacity to correct image orientation, the experiments can go forward by evaluating rotation correction for human detection and human pose estimation. The three representative architectures (OpenPose, AlphaPose and Yolov3) are evaluated without dedicated retraining. Images from COCOR-eval dataset are passed through all architectures with and without orientation correction based on RotNet angle prediction. All results are summarized in table 2.

The first arising result from this table is that all architectures, as already stated, are not rotation invariance ("Orig." column). Angles ranging in [120°,240°] are the most problematic rotations for all networks, the mAP score drops up to less
<table><caption>Table 2. Experiment results of human (pose) detection with global image correction (mAP[.5:.95])</caption><thead><tr><th rowspan="2"></th><th colspan="5">OpenPose</th><th colspan="5">Alphapose</th><th colspan="5">Yolov3</th></tr><tr><th>Orig.</th><th>RN360</th><th>RN360-OA</th><th>RNSC</th><th>RNSC-OA</th><th>Orig.</th><th>RN360</th><th>RN360-OA</th><th>RNSC</th><th>RNSC-OA</th><th>Orig.</th><th>RN360</th><th>RN360-OA</th><th>RNSC</th><th>RNSC-OA</th></tr></thead><tbody><tr><td>0°</td><td>59.6</td><td>57.8</td><td>58.2</td><td>58.0</td><td>57.9</td><td><strong>73.7</strong></td><td>71.5</td><td>71.8</td><td>71.9</td><td>71.8</td><td>42.6</td><td>38.5</td><td>39.2</td><td>37.4</td><td>37.8</td></tr><tr><td>30°</td><td>50.0</td><td>56.9</td><td><u>57.3</u></td><td>56.5</td><td>55.0</td><td>54.7</td><td>70.4</td><td><strong>70.5</strong></td><td>69.6</td><td>67.1</td><td>4.2</td><td>39.6</td><td><u>40.7</u></td><td>30.4</td><td>21.4</td></tr><tr><td>60°</td><td>26.7</td><td>55.9</td><td><u>56.8</u></td><td>56.3</td><td>55.8</td><td>35.7</td><td>69.9</td><td><strong>70.5</strong></td><td>69.3</td><td>67.4</td><td>1.9</td><td>39.6</td><td><u>40.6</u></td><td>28.4</td><td>22.7</td></tr><tr><td>90°</td><td>10.3</td><td>57.3</td><td><u>57.8</u></td><td>57.2</td><td>57.0</td><td>22.3</td><td>70.3</td><td><strong>71.2</strong></td><td>71.1</td><td>70.6</td><td>5.4</td><td>39.2</td><td><u>40.0</u></td><td>36.7</td><td>36.7</td></tr><tr><td>120°</td><td>1.3</td><td>56.8</td><td><u>56.9</u></td><td>56.2</td><td>56.0</td><td>12.6</td><td><strong>70.7</strong></td><td>70.4</td><td>69.4</td><td>69.4</td><td>0.8</td><td>39.7</td><td><u>40.5</u></td><td>28.4</td><td>28.5</td></tr><tr><td>150°</td><td>0.9</td><td><u>57.3</u></td><td>57.2</td><td>56.7</td><td>56.0</td><td>7.5</td><td><strong>70.9</strong></td><td>70.6</td><td>69.2</td><td>68.0</td><td>1.3</td><td>40.3</td><td><u>40.4</u></td><td>27.5</td><td>23.6</td></tr><tr><td>180°</td><td>1.0</td><td>53.7</td><td>54.9</td><td>56.7</td><td>56.8</td><td>6.4</td><td>67.0</td><td>68.0</td><td>70.4</td><td><strong>70.6</strong></td><td>8.0</td><td><u>35.4</u></td><td><u>35.4</u></td><td>33.7</td><td>34.7</td></tr><tr><td>210°</td><td>0.9</td><td>57.0</td><td><u>57.7</u></td><td>56.4</td><td>56.5</td><td>7.0</td><td>70.6</td><td><strong>71.5</strong></td><td>69.4</td><td>70.0</td><td>1.1</td><td>39.8</td><td><u>41.2</u></td><td>30.3</td><td>31.2</td></tr><tr><td>240°</td><td>1.4</td><td>57.2</td><td><u>57.9</u></td><td>56.2</td><td>56.2</td><td>12.6</td><td>71.1</td><td><strong>71.6</strong></td><td>68.4</td><td>68.9</td><td>0.9</td><td><u>40.8</u></td><td><u>40.8</u></td><td>25.5</td><td>28.9</td></tr><tr><td>270°</td><td>9.9</td><td><u>58.3</u></td><td>58.2</td><td>57.0</td><td>57.3</td><td>22.5</td><td><strong>71.5</strong></td><td><strong>71.7</strong></td><td>70.7</td><td>70.9</td><td>5.6</td><td>39.9</td><td><u>40.3</u></td><td>36.5</td><td>37.3</td></tr><tr><td>300°</td><td>12.4</td><td>56.9</td><td><u>57.6</u></td><td>55.8</td><td>56.2</td><td>35.8</td><td>70.3</td><td><strong>70.8</strong></td><td>68.9</td><td>69.3</td><td>1.6</td><td>39.7</td><td><u>40.8</u></td><td>28.2</td><td>28.2</td></tr><tr><td>330°</td><td>26.9</td><td><u>56.9</u></td><td>56.5</td><td>56.2</td><td>56.3</td><td>55.0</td><td><strong>70.4</strong></td><td>70.1</td><td>69.4</td><td>69.3</td><td>4.3</td><td><u>39.9</u></td><td><u>39.9</u></td><td>28.9</td><td>26.6</td></tr></tbody></table>

Average 19.9±22.3 56.8±1.1 57.3±0.9 56.6±0.6 56.4±0.8 28.8±22.4 70.4±1.2 70.7±1.0 69.8±1.0 69.4±1.4 6.5±11.6 39.4±1.4 40.0±1.5 31.0±4.0 29.8±5.8

Bold values correspond to best score for each angle. Underline values highlight best score for each network per angle.

than 13 for all networks, and even to almost 0 for *OpenPose*. *AlphaPose* remains the best architecture with a lower performance lost on small rotation, probably thanks to its spatial transformer network. *Yolo* is the most sensitive to orientation penalized by its grid processing.

Applying global orientation correction, performance of the three representative architectures is improved. All architectures present a small performance reduction at 0°. As no specific retraining of backbone architectures has been done, it is not surprising. Interestingly, angle regression approaches provide better results on image orientation estimation, but their performance as input of detection tasks are worse. Particularly, (Yolov3) loses up to a quarter using regression methods where the difference are not significant on pose estimation approaches (OpenPose and AlphaPose). Performance for all angles, even 180°, and for all RotNet flavors increases. RN360-OA approach offers the best scores, validating the Optimal Angle training strategy.

## 5.3 Local person orientation approach

The former assumption that people are oriented in a similar way is not valid for all real-life pictures. To tackle the problem of intra-frame person orientation diversity, this research explores YoloR, a redesigned version of Yolov3 implementing rotation handling using oriented bounding boxes (as presented on figure 2). Several ways to include the orientation in detection can be envisioned. Trying to regress the bounding box jointly with its orientation was unsuccessful. Taking inspiration for previous works on detecting objects using keypoints [36; 37; 38; 39], YoloR regresses the coordinates of the four corners of the oriented bounding boxes

$(x1, y1, x2, y2, x3, y3, x4, y4).$

Table 3 gives performance of YoloR trained on the COCOR-BB dataset and evaluated on COCOR-eval on person detection task. As previously seen, while the performance of Yolov3 is 42.6 mAP.5:.95 and 62.6 mAP0.5 on standard images, it drops drastically on rotated images. YoloR is able to achieve a better performance on rotated images, but there is a decrease performance on original images. Notably, detection results on 180° images and standard images are almost the same. The network learns well to detect persons although they are upside down, which differs from other approaches where the performances on 180° rotated images are the worst (see table 2). Although, YoloR provides better results compare to standard Yolov3, with an absolute gain of 4.5% mAP.5:.95 and 9.8% mAP0.5 in average for all angles, rotation invariance is not achieved. Coupling global and local rotation handling adding RN360-OA as preprocessing provides the results (third column of the table) with an average performance of 27.3 mAP.5:.95 ($\sigma$ = 0.8) and 54 mAP0.5 ($\sigma$ = 0.2). The un-rotated score decreases but, as previously reported in previous experiments, the performances are stable among angles, 180° remaining the most problematic case.

<table><caption>Table 3. Experiment results of human detection approaches (mAP)</caption><thead><tr><th rowspan="2"></th><th colspan="2">Yolov3</th><th colspan="2">YoloR</th><th colspan="2">RN360-OA+YoloR</th></tr><tr><th>mAP0.5</th><th>mAP5:.95</th><th>mAP0.5</th><th>mAP5:.95</th><th>mAP0.5</th><th>mAP5:.95</th></tr></thead><tbody><tr><td>0°</td><td><strong>62.6</strong></td><td><strong>42.6</strong></td><td>54.6</td><td>28.1</td><td>54.1</td><td>27.0</td></tr><tr><td>30°</td><td>20.5</td><td>4.2</td><td>27.4</td><td>10.6</td><td><strong>53.8</strong></td><td><strong>26.6</strong></td></tr><tr><td>60°</td><td>8.8</td><td>1.9</td><td>12.5</td><td>5.5</td><td><strong>53.9</strong></td><td><strong>28.4</strong></td></tr><tr><td>90°</td><td>14.1</td><td>7.7</td><td>13.7</td><td>6.3</td><td><strong>54.0</strong></td><td><strong>27.0</strong></td></tr><tr><td>120°</td><td>4.4</td><td>0.8</td><td>11.8</td><td>5.9</td><td><strong>54.2</strong></td><td><strong>28.6</strong></td></tr><tr><td>150°</td><td>5.6</td><td>1.3</td><td>27.6</td><td>10.7</td><td><strong>54.0</strong></td><td><strong>26.8</strong></td></tr><tr><td>180°</td><td>13.4</td><td>8.1</td><td><strong>54.4</strong></td><td><strong>28.5</strong></td><td>53.6</td><td>26.2</td></tr><tr><td>210°</td><td>5.4</td><td>1.1</td><td>27.2</td><td>10.7</td><td><strong>54.3</strong></td><td><strong>26.7</strong></td></tr><tr><td>240°</td><td>4.5</td><td>0.9</td><td>12.4</td><td>4.7</td><td><strong>54.4</strong></td><td><strong>27.6</strong></td></tr><tr><td>270°</td><td>14.9</td><td>8.3</td><td>13.7</td><td>5.6</td><td><strong>54.2</strong></td><td><strong>27.1</strong></td></tr><tr><td>300°</td><td>7.7</td><td>1.6</td><td>10.7</td><td>4.7</td><td><strong>54.0</strong></td><td><strong>26.9</strong></td></tr><tr><td>330°</td><td>20.0</td><td>4.3</td><td>27.7</td><td>10.8</td><td><strong>53.9</strong></td><td><strong>28.4</strong></td></tr></tbody><tfoot><tr><td>Average</td><td>14.7±16.1</td><td>6.5±11.6</td><td>24.5±15.7</td><td>11.0±8.5</td><td><strong>54.0±0.2</strong></td><td><strong>27.3±0.8</strong></td></tr></tfoot></table>
# 6 Discussion

Reported results from previous section validate, if mandatory, our problem statement claiming that standard neural architectures are currently not rotation invariant. The steerable approaches proposed in the literature are not rotation invariant in the case of human (pose) detection. They remain confined to small angle corrections or to simpler tasks. Their integration in the chosen representative architectures does not bring any improvement. It seems that filter transformation and feature projection are not adapted in their current version to human detection tasks.

The proposed global orientation correction approach trained on optimized data allows to approach rotation invariance. Indeed, for all the tested conditions, their use allows to improve the performance with a greater stability on all angles. The different flavors of our network have similar performances, but the classifier approach trained on optimally oriented data (RN360-OA) is the one that, coupled with the state-of-the-art networks, provides the best result. Associated with AlphaPose, the absolute gain is 41.9 mAP. Despite our expectations, this step towards invariance treatment has not yet increased the overall performance of the current systems.

A contrario, the local approach addressing the orientation of people in the image has allowed a significant improvement: 20.4 mAP absolute gain for the 180° images, 4.5 for all angles with an increased stability. This result is encouraging and shows that it is possible, for people detection, to improve the accuracy by integrating at low level the orientation constraints. Moreover, coupling this local approach with the global approach, the results increase further on all angles with a score of 27.3 (+16.5 mAP absolute gain) and $\sigma < 1$.

There are several ways to improve the research presented in this paper. The evaluated networks have been trained separately, leading to a mismatch between expected input data. It is relevant to build a global architecture integrating them and to train them jointly using a compound loss including both detection and rotation invariance metrics, similarly to generative and discriminative loss for GANs. Another way is currently emerging thanks to the advent of Transformer Networks in the computer vision area [40]. Adapting them to the specific problem of orientation could tackle rotation invariance problem at the price of an increase in the necessary computing memory and power.

# 7 Conclusion

Human care for elderly people needs, for security reasons, to detect people in all body poses, including in case of fall/discomfort. Current neural networks are not rotation invariant on human (pose) detection and, thus, not reliable enough in this context. The article explores three different approaches to tackle rotation invariance: including steerable layers in current network, global rotation correction, and local person orientation. The proposed methods are trained using generated rotated corpora based on the MS COCO dataset. Integrating steerable approaches into existing networks failed to improve results. Global rotation approach keeps intact the performance on rotated images, for an average *mAP* of 70.7 using *AlphaPose*, but multi-person images with various orientations remains a challenge. Local person orientation approach produces better results on rotated images but is not as efficient as the global approach. This research proposes methods allowing to preserve performance in person detection against rotation at the price of a *mAP* loss. Despite our expectation, the global performance did not increase. Perspectives from this research include joint training of proposed networks taking into account rotation correction and accuracy into losses, as well as designing vision Transformer Networks dedicated to rotation invariance.

# Acknowledgment

This research was funded by a French Government Scholarship (BGF), and was made possible by the collaboration between the M-PSI team of the Grenoble Informatics Laboratory (LIG) and Cambodia Academy of Digital Technology (CADT).

# References

[1] Zhong-Qiu Zhao, Peng Zheng, Shou tao Xu, and Xindong Wu. Object detection with deep learning: A review. *IEEE Transactions on Neural Networks and Learning Systems*, 30:3212–3232, 2019.

[2] Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20
years: A survey. *ArXiv*, abs/1905.05055, 2019.

[3] Yucheng Chen, Yingli Tian, and Mingyi He. Monocular human pose estimation: A survey of deep learning-based methods. *Computer Vision and Image Understanding*, 192:102897, 2020.

[4] Feng Zhang, Xiatian Zhu, and Chen Wang. Single person pose estimation: A survey. *arXiv preprint arXiv:2109.10056*, 2021.

[5] Tewodros L. Munea, Yalew Z. Jembre, Halefom T. Weldegebriel, Longbiao Chen, Chenxi Huang, and Chenhui Yang. The progress of human pose estimation: a survey and taxonomy of models applied in 2d human pose estimation. *IEEE Access*, 8:133330–133348, 2020.

[6] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. *ArXiv*, abs/1804.02767, 2018.

[7] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. *2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 1302–1310, 2017.

[8] D. Vaufreydaz and A. Nègre. MobileRGBD, An Open Benchmark Corpus for mobile RGB-D Related Algorithms. In *13th International Conference on Control, Automation, Robotics and Vision, Singapore*, Singapore, December 2014.

[9] D. Vaufreydaz. *Perception multimodale et interaction sociable*. Inria, Université Grenoble Alpes, France, 2018.

[10] Bogdan Kwolek and Michal Kepski. Human fall detection on embedded platform using depth maps and wireless accelerometer. *Computer methods and programs in biomedicine*, 117 3:489–501, 2014.

[11] Vaufreydaz, D. and Nègre, A. MobileRGBD dataset website. https://mobilergbd.imag.fr/, 2017. Online; accessed 20 January 2022.

[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. *2009 IEEE Conference on Computer Vision and Pattern Recognition*, pages 248–255, 2009.

[13] Tsung-Yi Lin, M. Maire, Serge J. Belongie, James Hays, P. Perona, D. Ramanan, Piotr Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In *ECCV*, 2014.

[14] Li-Jia Li and Li Fei-Fei. What, where and who? classifying events by scene and object recognition. In *2007 IEEE 11th international conference on computer vision*, pages 1–8. IEEE, 2007.

[15] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2014.

[16] Enea Cippitelli, Francesco Fioranelli, Ennio Gambi, and Susanna Spinsante. Radar and rgb-depth sensors for fall detection: A review. *IEEE Sensors Journal*, 17(12):3585–3604, 2017.

[17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In *NIPS*, 2012.

[18] Diego Marcos, Michele Volpi, Nikos Kostodakis, and Devis Tuia. Rotation equivariant vector field networks. *2017 IEEE International Conference on Computer Vision (ICCV)*, pages 5058–5067, 2017.

[19] Facundo Quiroga, Franco Ronchetti, Laura Lanzarini, and Aurelio Fernández Barvíiera. Revisiting data augmentation for rotational invariance in convolutional neural networks. *Modelling and Simulation in Management Sciences*, 2019.

[20] Taco Cohen and Max Welling. Steerable cnns. *ArXiv*, abs/1612.08498, 2017.

[21] Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Bros tow. Harmonic networks: Deep translation and rotation equivariance. *2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 7168–7177, 2017.

[22] Yanzhao Zhou, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Oriented response networks. *2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 4961–4970, 2017.
[23] Max Jaderberg, K. Simonyan, Andrew Zisserman, and K. Kavukcuoglu. Spatial transformer networks. In NIPS, 2015.

[24] Akhil Meethal, Marco Pedersoli, Soufiane Belharbi, and Eric Granger. Convolutional stn for weakly supervised object localization. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 10157–10164. IEEE, 2021.

[25] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 779–788, 2016.

[26] Hao shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. Rmpe: Regional multi-person pose estimation. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2353–2362, 2017.

[27] Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6517–6525, 2017.

[28] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. ArXiv, abs/2004.10934, 2020.

[29] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6022–6031, 2019.

[30] Microsoft. COCO dataset website. http://cocodataset.org/, 2017. Online; accessed 20 January 2022.

[31] William McNally, Kanav Vats, Alexander Wong, and John McPhee. Evopose2d: Pushing the boundaries of 2d human pose estimation using accelerated neuroevolution with weight transfer. IEEE Access, 9:139403–139414, 2021.

[32] P. Fischer, A. Dosovitskiy, and T. Brox. Image orientation estimation with convolutional networks. In GCPR, 2015.

[33] S. Maji and S. Bose. Deep image orientation angle detection. ArXiv, abs/2007.06709, 2020.

[34] Daniel Saez. Correcting Image Orientation Using Convolutional Neural Networks. https://d4nst.github.io/2017/01/12/image-orientation/, 2017. Online; accessed 20 January 2022.

[35] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.

[36] Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. In Proceedings of the European conference on computer vision (ECCV), pages 734–750, 2018.

[37] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6569–6578, 2019.

[38] Zhiwei Dong, Guoxuan Li, Yue Liao, Fei Wang, Pengju Ren, and Chen Qian. Centripetalnet: Pursuing high-quality keypoint pairs for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10519–10528, 2020.

[39] Jingru Yi, Pengxiang Wu, Bo Liu, Qiaoying Huang, Hui Qu, and Dimitris Metaxas. Oriented object detection in aerial images with box boundary-aware vectors. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2150–2159, 2021.

[40] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.