Journal of Information Processing Vol.29 406–423 (May 2021)[DOI: 10.2197/ipsjjip.29.406]

Regular Paper

MirrorNet: A Deep Reflective Approach to
2D Pose Estimation for Single-Person Images
Takayuki Nakatsuka1,a) Kazuyoshi Yoshii2 Yuki Koyama3
Masataka Goto3 Shigeo Morishima4

Satoru Fukayama3

Received: August 26, 2020, Accepted: February 2, 2021

Abstract: This paper proposes a statistical approach to 2D pose estimation from human images. The main problems
with the standard supervised approach, which is based on a deep recognition (image-to-pose) model, are that it often
yields anatomically implausible poses, and its performance is limited by the amount of paired data. To solve these problems, we propose a semi-supervised method that can make eﬀective use of images with and without pose annotations.
Specifically, we formulate a hierarchical generative model of poses and images by integrating a deep generative modelof poses from pose features with that of images from poses and image features. We then introduce a deep recognition
model that infers poses from images. Given images as observed data, these models can be trained jointly in a hierarchical variational autoencoding (image-to-pose-to-feature-to-pose-to-image) manner. The results of experiments showthat the proposed reflective architecture makes estimated poses anatomically plausible, and the pose estimation performance is improved by integrating the recognition and generative models and also by feeding non-annotated images.
Keywords: 2D pose estimation, amortized variational inference, variational autoencoder, mirror system

1. Introduction
Human beings understand the essence of things by abstractionand embodiment. As Richard P. Feynman, the famous physicist,stated, “What I cannot create, I do not understand” [15], abstraction and embodiment are two sides of the same coin. Our hypothesis is that such a bidirectional framework plays a key role inthe brain process of recognizing human poses from 2D images,
inspired by the mirror neuron system or motor theory known in
the field of cognitive neuroscience [17]. In this paper, we focus
on the estimation of the 2D pose (joint coordinates) of a person
in an image, inspired by the human mirror system.
The standard approach to 2D pose estimation is to train a
deep neural network (DNN) that maps an image to a pose in
a supervised manner by using a collection of images with pose
annotations [3], [30], [42], [44], [45], [48], [51]. Toshev andSzegedy [45] pioneered a method called DeepPose that uses a
DNN consisting of convolutional and fully connected layers forthe nonlinear regression of 2D joint coordinates from images.
Instead of directly using 2D joint coordinates as target data,
Thompson et al. [44] proposed a heatmap representation that indicates the posterior distribution of each joint over pixels. This
representation has commonly been used in many state-of-the-art
methods for 2D pose estimation [3], [30], [42], [48], [51]. Notethat all of these methods focus only on the recognition part of the
1
2
3
4
a)

Waseda University, Shinjuku, Tokyo 169–8050, Japan
Kyoto University, Kyoto 606–8501, Japan
National Institute of Advanced Industrial Science and Technology(AIST), Tsukuba, Ibaraki 305–8568, Japan
Waseda Research Institute for Science and Engineering, Shinjuku, Tokyo
169–8050, Japant59nakatsuka@fuji.waseda.jp

c 2021 Information Processing Society of Japanhuman mirror system.
Such a supervised approach based on image-to-pose mapping
has two major drawbacks. First, the anatomical plausibility of
estimated poses is not taken into account. To mitigate this problem, the positional relationships between adjacent joints have often been considered [4], [9], [26], [32], [43], and error correction
networks [5], [6] and adversarial networks [7], [8] have been used
in a heuristic manner. Second, the performance of the supervised
approach is limited by the amount of paired pose-image data. To
overcome this limitation, data augmentation techniques [34] andthe use of metadata [46] and non-annotated data [12], [46] havebeen proposed. A unified solution to these complementary problems, however, remains an open question.
In this paper, we propose a hierarchical variational autoencoder(VAE) called MirrorNet that consists of higher- and lower-level
mirror systems (Fig. 1). Specifically, we formulate a probabilistic latent variable model that integrates a deep generative modelof poses from pose features (called primitives) with that of images from poses and foreground and background features (called
appearances and scenes). To estimate poses, pose features, andimage features from given images in the framework of amortizedvariational inference (AVI) [24], we introduce deep recognition
models of pose features from poses, foreground and backgroundimage features from poses and images, and poses from images.
These generative and recognition models can be trained jointly
even from non-annotated images.
A key feature of our semi-supervised method is to consider
the anatomical fidelity and plausibility of poses in the estimation
process. To make use of both annotated and non-annotated images, our method constructs an image-to-pose-to-image reflectivemodel (i.e., a higher-level mirror system for image understand-406

Journal of Information Processing Vol.29 406–423 (May 2021)

ducted for evaluating the proposed method. Section 6 summarizes this paper.

2. Related Work

Fig. 1

An overview of MirrorNet, which consists of generative models of
poses and images from latent features and recognition models of
poses and latent features from images. The latent features consist ofprimitives (pose features), appearances (foreground image features),and scenes (background image features). A higher-level image-topose-to-image mirror system is integrated with a lower-level pose-tofeature-to-pose mirror system in a hierarchical manner for eﬀectively
using images with and without pose annotations.

ing) by connecting the image-to-pose recognition model with thepose-to-image generative model. Even when only images withoutpose annotations are given, the generative model can be used forevaluating the anatomical fidelity of poses estimated by the recognition model (i.e., how consistent the estimated poses are withthe given images). In the same way, our method builds a pose-tofeature-to-pose reflective model (i.e., a lower-level mirror systemfor pose understanding) by connecting the pose-to-feature recognition model with the feature-to-pose generative model. Thispose VAE can be trained in advance by using a large number ofpose data (e.g., data obtained by rendering human 3D models)
and then used for evaluating the anatomical plausibility of the estimated poses. Note that the pose VAE cannot be used alone asan evaluator of an estimated pose for a non-annotated image because any plausible pose is allowed even if it does not reflect theimage. This is why conventional plausibility-aware methods stillneed paired data [6], [7], [8]. The higher- and lower- mirror systems are integrated into MirrorNet and can be trained jointly ina statistically principled manner. In practice, each component of
MirrorNet is trained separately by using paired data and then theentire MirrorNet is jointly trained in a semi-supervised mannerusing both paired and unpaired data for further optimization.
The main contribution of this paper is to realize plausibilityand fidelity-aware 2D pose estimation based on a hierarchical
VAE consisting of pose and image VAEs corresponding to thehigher- and lower-level mirror systems. To achieve this, we effectively integrate state-of-the-art DNN-based methods for supervised pose estimation, foreground/background segmentation, andimage generation into the unified VAE architecture. The VAEbased probabilistic formulation can make eﬀective use of bothannotated and non-annotated images for joint semi-supervised
learning of pose fidelity and plausibility, leading to improved
pose estimation. We experimentally show that the image-to-pose
recognition model can be improved by integrating the pose-toimage generative model and the pose VAE.
The rest of this paper is organized as follows. Section 2 reviews
related work on plausibility-aware pose estimation and fidelityaware image processing. Section 3 explains the proposed method
for unsupervised, supervised, and semi-supervised pose estimation. Section 4 describes the detailed implementation of the proposed method. Section 5 reports comparative experiments con-c 2021 Information Processing Society of Japan

Here, 2D human pose estimation refers to estimating the coordinates of joints of a person in an image. This task is challenging
because a wide variety of human appearances and backgroundscenes can exist and some joints are often occluded.
For robust pose estimation, Ramanan [37] proposed an edgebased model, and Andriluka et al. [2] introduced a pictorial structural model of human joints. Modeling the human body using tree
or graph structures has been intensively studied [11], [14], [19],[35], [40], [41], [52]. To improve the accuracy of estimation, one
needs to carefully design sophisticated models and features that
can appropriately represent the relations between joints.
Toshev and Szegedy [45] proposed a neural pose recognizer
called DeepPose that estimates the positions of joints by usinga DNN consisting of convolutional layers and fully connected
layers. DeepPose is the first method that applies deep learning
to pose estimation, resulting in significant performance improvement. Instead of directly regressing the coordinates of joints froman image as in DeepPose, Thompson et al. [44] used a heatmap(pixel-wise likelihood) for representing the distributions of eachjoint, which has recently become standard. These state-of-the-art
methods for 2D pose estimation have been examined from several
points of view. For example, intermediate supervision and multistage learning were proposed for using deep convolutional neuralnetworks (CNNs) [3], [30], [42], [48], [51]. An optimal objective function was proposed for evaluating the relations between
pairs of joints [4], [9], [26], [43]. Recently, some studies haveassessed the correctness of inferred poses using additional networks [6], [13], [29] or compensated for the lack of data samples
with data augmentation [34], [46]. Here, we review plausibilityaware methods of pose estimation and fidelity-aware methods ofimage processing.
2.1 Plausibility-Aware Pose Estimation
A standard way of improving the anatomical plausibility of estimated poses is to focus on the local relations of adjacent joints
in pose estimation [4], [9], [26], [32], [43] or to refine the estimated poses as post-processing. Carreira et al. [5] proposed a
self-correcting model based on iterative error feedback. Chenet al. [6], Fieraru et al. [13], and Moon et al. [29] proposed cascaded networks that recursively refine the estimated poses whilereferring to the original images. Adversarial networks have often been used to judge whether the estimated poses are anatomically plausible [7], [8]. In addition, Ke et al. [20] proposed ascale-robust method based on a multi-scale network with a bodystructure-aware loss function. Nie et al. [31] proposed a structured pose representation using the displacement in the positionof every joint from a root joint position. While these methods
can use only paired data for supervised learning, our VAE-based
method enables unsupervised learning. In contract to the existing autoencoding approach that aims to extract latent features ofposes [27], [47], our VAE is used for measuring the plausibilityof poses.

407

Journal of Information Processing Vol.29 406–423 (May 2021)

To compensate for a lack of training data, Ukita andUematsu [46] took a semi- and weakly-supervised approach thatuses non-annotated images and action labels of images (e.g.,baseball and volleyball) to estimate the poses of humans from apart of paired data. Peng et al. [34] proposed an eﬃcient data
augmentation method that generates hard-to-recognize images
with adversarial training. Yeh et al. [54] used the chirality transform, a geometric transform that generates an antipode of a target,
for pose regression. In this paper, we take a diﬀerent approachbased on mirror systems for unsupervised learning so that nonannotated images can be used to improve the performance.
2.2 Fidelity-Aware Image Processing
The mirror structure has been used successfully for various image processing tasks including domain conversion. A representative example is the VAE that jointly learns a generative model(decoder) of observed variables from latent variables following aprior distribution, and a recognition model (encoder) of the latentvariables from the observed variables [24]. The VAE can generate new samples by randomly drawing latent variables from theprior distribution. CycleGAN [57], DiscoGAN [21], and DualGAN [55] are popular variants of GANs using mirror structures
for image-to-image conversion. The key feature of these methods is to consider bidirectional inter-domain mappings from unpaired data. Qiao et al. [36] recently proposed MirrorGAN for
bidirectional text-image conversion. Yildiri et al. [56] proposed
an analysis-by-synthesis approach to joint 3D face generation andrecognition from a cognitive point of view. The success of thesemethods indicates the potential of the mirror structure for stablytraining a DNN with non-annotated data.
In the context of pose estimation, we propose the first mirrorstructured DNN for human pose estimation that integrates thetwo-level mirror systems in a hierarchically autoencoding manner. Recently, de Bem et al. [12] proposed a fidelity-aware pose
estimation method based on an image-to-pose-to-image model
called VAEGAN. This model can be trained in a semi-supervisedmanner, but its generalization capability has not been fully validated on a dataset with annotated and non-annotated images having various background images. The advantages of our method
are that it can simultaneously consider both the pose plausibility
and fidelity and that it can work robustly against changes in background images thanks to the foreground/background segmentation.

3. Proposed Method
This section describes the proposed method based on a fully
probabilistic model of poses and images for 2D pose estimation in images of people (Fig. 2). MirrorNet is a hierarchical
VAE that is one technique for amortized variational inference(AVI) [10], [24], [28], [38], and consists of a VAE of images (i.e.,a pose-to-image generative model and an image-to-pose recognition model), and a VAE of poses (i.e., a primitive-to-pose generative model and a pose-to-primitive recognition model). In theory, this model can be trained in an unsupervised manner byusing non-annotated images only, or by using unpaired images
and poses. In practice, the model is trained in a semi-supervisedc 2021 Information Processing Society of Japanmanner by using partially annotated images. Each model is firsttrained separately to stabilize the training, and then all models
are jointly trained for further optimization. Once the training iscompleted, only the image-to-pose recognition model is used forpose estimation. The hierarchical autoencoding architecture iseﬀective for estimating poses that are anatomically plausible andreproduce the original images with high fidelity.
3.1 Problem Specificationxs
N
N
Let X = {xn ∈ RD }n=1
and S = {sn ∈ RD }n=1
be a set of images and a set of poses corresponding to X, respectively, where
Dx is the number of dimensions of each image, Ds is the numberof dimensions of each pose, and N is the number of images. We
assume that each xn is an RGB image featuring single or multiple persons, showing all or parts of their bodies, and each sn isa set of grayscale images, each of which represents the positionof a joint using a heatmap [44]. Note that a coordinate taking the
maximum pixel value in each of the grayscale images is retrieved
as the 2D joint coordinate during a test phase.
agNN
Let A = {an ∈ RD }n=1
and G = {gn ∈ RD }n=1
be a set of
appearances representing the foreground features of X (e.g., skinand hair colors and textures) and a set of scenes representing thebackground features of X (e.g., places, color, and brightness), respectively, where Da and Dg are the number of dimensions of thelatent spaces. These latent features are used in combination withz
N
S for representing X. Let Z = {zn ∈ RD }n=1
be a set of primitives
representing the features of S (e.g., scales, positions, and orientations of joints), where Dz is the number of dimensions of thelatent space.
Our goal is to train a pose recognizer that maps X to S. Let
M be the number of annotated images. In a supervised condition,
X and S are given as observed data (M = N). In an unsupervised condition, only X is given (M = 0). In a semi-supervisedM
condition, X and a part of S, i.e., {sn }n=1
, are given.
3.2 Generative Modeling
We formulate a unified hierarchical generative model of images X, poses S, appearances A, scenes G, and primitives Z that
integrates a deep generative model of X from S, A, and G with adeep generative model of S from Z as follows:p(X, S, A, G, Z) = pθ (X|S, A, G)pφ (S|Z)p(A)p(G)p(Z)=Npθ (xn |sn , an , gn )pφ (sn |zn )p(an )p(gn )p(zn ),

(1)

n=1

where θ and φ are the sets of trainable parameters of the deep
generative models of xn and sn , respectively. The pose likelihoodpθ (xn |sn , an , gn ) evaluates the pose fidelity to the given images andthe pose prior pφ (sn |zn ) prevents anatomically implausible pose
estimates. The remaining terms are prior probability distributionsof an , gn , and zn .
The pose-to-image generation model pθ (xn |sn , an , gn ) and theprimitive-to-pose generation model pφ (sn |zn ) are both formulated
as follows:pθ (xn |sn , an , gn ) = N(xn ; μθ (sn , an , gn ), σ2θ (sn , an , gn )IDx ), (2)pφ (sn |zn ) = N(sn ; μφ (zn ), σ2φ (zn )IDs ),

(3)

408

Journal of Information Processing Vol.29 406–423 (May 2021)

Fig. 2 The proposed architecture of MirrorNet integrating a VAE for poses (lower-level mirror system)with a pose-conditioned VAE for images (higher-level mirror system) in a hierarchical Bayesianmanner. In terms of generative modeling, the decoder of the pose VAE serves as a prior distribution of poses p(S) to evaluate the pose plausibility and the decoder of the image VAE as alikelihood function of poses p(X|S) to evaluate the pose fidelity. In terms of posterior inference,the encoder of the pose VAE is used as a variational posterior distribution of poses q(S|X). Sucha statistical approach based on a complete probabilistic generative model enables semi-supervisedpose estimation using any images with/without pose annotations.

where μθ (sn , an , gn ) and σθ (sn , an , gn ) are the outputs of a DNN
with parameter θ that takes sn , an , and gn as input, and μφ (zn ) andσφ (zn ) are the outputs of a DNN with parameters φ that takes zn
as input. The priors p(an ), p(gn ), and p(zn ) are set to the standard
Gaussian distributions as follows:p(an ) = N(an ; 0Da , IDa ),(4)p(gn ) = N(gn ; 0 , I ),(5)p(zn ) = N(zn ; 0 , I ),

(6)

Dg

Dz

Dg

Dz

where 0D† (D† = {Da , Dg , Dz }) and ID† are the zero vector of sizeD† and the identity matrix of size D† , respectively.
3.3 Unsupervised Learning
We explain the unsupervised learning of the proposed modelusing only images X, which is the basis for practical semisupervised learning using partially annotated images (Section
3.4). Given a set of images X as observed data, our goal is toinfer the distribution of the latent variables Ω ≡ (S, A, G, Z). We
estimate optimal parameters θ∗ and φ∗ in the framework of maximum likelihood estimation as follows:θ∗ , φ∗ = argmax p(X),θ,φand p(X) is the marginal likelihood given byp(X) = p(X, Ω)dΩ.

(7)

(8)

where p(X, Ω) is the joint probability distribution given by
Eq. (1).
Because Eq. (8) is analytically intractable, we use an amortizedvariational inference (AVI) technique [10], [24], [28], [38] thatc 2021 Information Processing Society of Japan

introduces an arbitrary variational posterior distribution q(Ω|X)
and makes it approach as close as possible to the true posterior distribution p(Ω|X) (Section 3.3.1). The minimization of
the Kullback–Leibler (KL) divergence between these posteriors
is equivalent to the maximization of a variational lower bound L
of log p(X) with respect to q(Ω|X). Thus, the optimal parametersθ∗ and φ∗ can be obtained by maximizing the variational lower
bound L instead of log p(X) (Section 3.3.2).
3.3.1 Variational Lower Bound
Using Jensen’s inequality, a variational lower bound LX oflog p(X) can be derived as follows:p(X, Ω)defdΩ = LX ,(9)log p(X) ≥ q(Ω|X) log
q(Ω|X)
where the equality holds, i.e., LX is maximized, if and only ifq(Ω|X) = p(Ω|X). Because this equality condition cannot becomputed analytically, q(Ω|X) is approximated by a factorized
form as follows:q(Ω|X) = qα (S|X)qβ (A|S, X)qγ (G|S, X)qδ (Z|S)=

Nqα (sn |xn )qβ (an |sn , xn )qγ (gn |sn , xn )qδ (zn |sn ),

(10)

n=1

where α, β, γ, and δ are the sets of parameters of these four variational distributions, respectively.
In the statistical framework of AVI, we introduce a DNN-based
posterior distribution q(Ω|X) such that the complex true posterior
distribution p(Ω|X) can be well approximated by q(Ω|X). Specifically, we introduce a deep image-to-pose model qα (sn |xn ), a deep
image-to-appearance model qβ (an |sn , xn ), a deep image-to-scenemodel qγ (gn |sn , xn ), and a deep pose-to-primitive model qδ (zn |sn )as follows:409

Journal of Information Processing Vol.29 406–423 (May 2021)

qα (sn |xn ) = N(sn ; μα (xn ), σ2α (xn )IDs ),(11)

qβ (an |sn , xn ) = N(an ; μβ (sn , xn ), σ2β (sn , xn )IDa ),qγ (gn |sn , xn ) = N(gn ; μγ (sn , xn ), σ2γ (sn , xn )IDg ),(12)qδ (zn |sn ) = N(zn ; μδ (sn ), σ2δ (sn )IDz ),

(14)

(13)

where μα (xn ) and σα (xn ) are the outputs of a DNN with parameters α that takes xn as input, μ‡ (sn , xn ) and σ‡ (sn , xn ) (‡ = β or γ)
are the outputs of a DNN with parameters ‡ that takes sn and xn
as input, and μδ (sn ) and σδ (sn ) are the outputs of a DNN withparameters δ that takes sn as input.
Substituting both of the generative model given by Eq. (1) with
Eqs. (2)–(6) and the recognition model given by Eq. (10) with
Eqs. (11)–(14) into Eq. (9), the variational lower bound LX can XNXbe rewritten as the sum of {LX
n Ln ) as follows (Apn }n=1 (L =pendix A.1):
LX
n = Eq [log pθ (xn |sn , an , gn )+log pφ (sn |zn )+log p(an )+log p(gn )+log p(zn )−log qα (sn |xn )−log qβ (an |sn , xn )−log qγ (gn |sn , xn )−log qδ (zn |sn )]calculate the first two reconstruction terms of Eq. (15), respectively. The recognition models α, β, γ, and δ, and the generativemodels φ and θ can thus be concatenated in this order with thereparametrization trick given by Eqs. (20)–(23), and are jointly
optimized in an autoencoding manner with an objective function
given by Eq. (15).
3.4 Supervised Learning
We explain the supervised learning of the proposed model using paired data of X and S. This approach follows the mannerof the semi-supervised learning of a VAE [23]. While the variational lower bound LX of log p(X) is maximized in the unsupervised condition (Section 3.3), we aim to maximize the variational
lower bound LX,S of log p(X, S), which is given byp(X, Ω)defdΘ = LX,S , (24)log p(X, S) ≥ q(Θ|S, X) log
q(Θ|S, X)
where Θ = Ω\S = {A, G, Z}. As in Eq. (10), q(Θ|S, X) is factorized asq(Θ|S, X) = qβ (A|S, X)qγ (G|S, X)qδ (Z|S)= Eq [log pθ (xn |sn , an , gn )]+Eq [log pφ (sn |zn )]−Eq [log qα (sn |xn )]−Eq [KL(qβ (an |sn , xn )||p(an ))]−Eq [KL(qγ (gn |sn , xn )||p(gn ))]−Eq [KL(qδ (zn |sn )||p(zn ))],
(15)
where the first term represents the fidelity of a pose sn with an
original image xn having features an and gn , the second term represents the plausibility of sn , the third term prevents the overfittingof the recognition model α, and the fourth to sixth terms evaluate
the similarities between the recognition models β, γ, and δ andthe priors on an , gn , and zn , respectively.
3.3.2 Parameter Optimization
Because Eq. (15) still includes intractable expectations, we perform Monte Carlo integration using samples sn , an , gn , and zn
obtained by reparametrization trick [24] as follows:=Nqβ (an |sn , xn )qγ (gn |sn , xn )qδ (zn |sn ),

(25)

n=1

where qβ (an |sn , xn ), qγ (gn |sn , xn ), and qδ (zn |sn ) are given by
Eqs. (12)–(14), respectively. Substituting both of the probabilistic
model given by Eq. (1) with Eqs. (2)–(6) and the inference model
given by Eq. (25) with Eqs. (12)–(14) into Eq. (24), LX,S can be
N(LX,S = n LnX,S ) as follows
rewritten as the sum of {LnX,S }n=1
(Appendix A.1):
LnX,S = Eq [log pθ (xn |sn , an , gn ) + log pφ (sn |zn ) + log p(an )+ log p(gn ) + log p(zn ) − log qβ (an |sn , xn )− log qγ (gn |sn , xn ) − log qδ (zn |sn )]= Eq log pθ (xn |sn , an , gn ) + Eq [log pφ (sn |zn )]sn ∼ N(0, IDs ),an ∼ N(0, IDa ),(16)− KL(qβ (an |sn , xn )||p(an )) − KL(qγ (gn |sn , xn )||p(gn ))(17)− KL(qδ (zn |sn )||p(zn )).

g
n ∼ N(0, IDg ),(18)zn ∼ N(0, IDz ),(19)

sn = μα (xn ) + ns  σα (xn ),an = μβ (sn , xn ) +gn = μγ (sn , xn ) +zn = μδ (sn ) +an  σβ (sn , xn ),gn  σγ (sn , xn ),zn  σδ (sn ),

(20)
(21)
(22)
(23)

where  indicates the element-wise product. Although in theory
a suﬃcient number of samples should be generated to performaccurate Monte Carlo integration, we generate only one sample
for each variable as in the standard VAE [24].
Using these techniques, the lower bound LX given by Eq. (15)can be approximately computed, and can thus be maximized withrespect to θ, φ, α, β, γ, and δ (Fig. 2). First, the recognition models α, β, γ, and δ are used to deterministically generate samples
sn , an , gn , and zn in Eqs. (20)–(23), and to calculate the last four
regularization terms of Eq. (15), respectively. Given the samples sn , an , gn , and zn , the generative models θ and φ are used toc 2021 Information Processing Society of Japan(26)

A major problem in such supervised learning is that the recognition model qα (sn |xn ), which plays a central role in human pose
estimation from images, cannot be trained because it does not appear in Eq. (26). To solve this problem, we add a term to assess
the predictive performance of qα (sn |xn ) to Ln , following [23] as
X,S= LnX,S + λ log qα (sn |xn ),
Ln,λ

(27)

log qα (sn |xn )⎞⎛Ds(sn − μα,ds (xn ))2 ⎟⎟⎟1  ⎜⎜⎜⎜
2⎟⎟⎠ ,=−⎜log 2πσα,ds (xn ) +2 d =1 ⎝σ2α,d (xn )
s

(28)

s

where λ is a hyperparameter that controls the balance between
purely generative learning and purely discriminative learning. In
our method, we empirically used λ = 0.01 in all experiments.
 X,SThe new objective function LλX,S = n Ln,λcan be maximized
with respect to θ, φ, α, β, γ, and δ jointly in the same way as theunsupervised learning described in Section 3.3.2, where an , gn ,and zn are obtained by using Eqs. (21)–(23), and sn is given.

410

Journal of Information Processing Vol.29 406–423 (May 2021)

3.5 Semi-supervised Learning
In the semi-supervised condition, where X is only partially annotated, we define a new objective function L by accumulating
X,S
LX
n used for unsupervised learning or Ln,λ used for supervised
learning as follows:


def
X,S
LX
Ln,λ
,
(29)
L=
n +η
n: xn is given

Fig. 3 Layers used for implementing DNNs.

n: xn & sn are given

where η is a weighting factor, which was set to 1 in our experiments reported in Section 5, because this choice is theoreticallyreasonable in terms of probabilistic modeling and we found no
significant diﬀerence in pose estimation performance between
η = 0.25, 0.50, 1.0, 2.0, 4.0. All generation and recognition models can be trained for all samples regardless of the availability of
their annotations. We will discuss the eﬀectiveness of the curriculum learning based on the pre-training of each component of
MirrorNet and the joint training of the whole MirrorNet in Section 5.4.

Fig. 4

4. Implementation
This section describes the implementation of MirrorNet, which
is based on curriculum learning. First, we separately pre-train
the components of MirrorNet, i.e., the pose recognizer α (Section 4.1), the pose-conditioned image VAE with the generator θand the recognizers β and γ (Section 4.2), and the pose VAE withthe generator φ and the recognizer δ (Section 4.3). We then train
the whole MirrorNet under a supervised condition (Section 3.4)
and further optimize it under a semi-supervised condition (Section 3.5).
Note that, as shown in Fig. 2, the pose-conditioned image VAE
has a human mask estimator as a subcomponent for separating
an image into foreground and background images which helps tostabilize the training of MirrorNet.
4.1 Pose Recognizer
The image-to-pose recognizer α is the most fundamental partof pose estimation, and is pre-trained in a supervised manner byusing paired data of X and S. We maximize an objective function
given bydef

Ln (α) = log qα (sn |xn ),(30)

where the variance σ2α (xn ) is fixed to 0.01 for stability.
The network α can be implemented with any DNN that outputs the heatmaps of joint positions, e.g., a stack of eight residual hourglass networks (HGNet) [30], ResNet-50 [50], and highresolution sub-networks (HRNet) [42].
4.2 Pose-Conditioned Image VAE
The pose-conditioned image VAE consisting of the image-toappearance recognizer β, the image-to-scene recognizer γ (encoders), and the appearance/scene-to-image generator θ (decoder)is pre-trained in an unsupervised manner by using paired data of
X and S. We maximize a variational lower bound L(θ, β, γ) of themarginal log likelihood log p(X|S). More specifically, we havelog p(xn |sn ) ≥ Eq log pθ (xn |sn , an , gn )+log p(an )+log p(gn )c 2021 Information Processing Society of JapanThe network configuration common in the recognizers β and γ of thepose-conditioned image VAE taking as input foreground and backfg
bg
ground images xn and xn (c = 3), respectively, and the recognizer
δ of the pose VAE taking as input the pose sn (c is the number ofjoints).

− log qβ (an |sn , xn )−log qγ (gn |sn , xn )= Eq log pθ (xn |sn , an , gn ) −KL(qβ (an |sn , xn )||p(an ))−KL(qγ (gn |sn , xn )||p(gn ))def= Ln (θ, β, γ),(31)
NLn (θ, β, γ). The three networks θ, β,
where L(θ, β, γ) = n=1
and γ can be optimized jointly by using the reparametrization
tricks [24] given by Eq. (21) and Eq. (22), where the varianceσ2θ (sn , an , gn ) of the generator θ is fixed to 1 for stability.
To encourage the disentanglement between the foreground features (appearance) an and the background features (scene) gn , weseparately input foreground and background parts of the original
image xn into the two encoders β and γ, respectively, instead ofx∗
directly feeding xn into β and γ. Specifically, an image x∗n ∈ RD ,a reduced-size version of xn , is first split into foreground andfg
bg
background images xn and xn as follows:
xn = x∗n  wn ,

(32)

bg
xn = x∗n  (1 − wn ),

(33)

fg

x∗

where  indicates the element-wise product and wn ∈ RD represents a mask image estimated from x∗n with the additional information of the pose sn . In this paper, we use a neural mask
estimator ψ trained in a supervised manner such that the meansquared error between the estimated and ground-truth masks is
minimized.
The recognizers β and γ are implemented as stacks of four
residual blocks [16] (Fig. 3 and Fig. 4). Unlike the original
ResNet, a branching architecture is introduced in the last layer
to output the mean and variance of the posterior Gaussian distribution. The generator θ is implemented with a U-Net [39] thattakes as input a stack of the heatmaps of the joints given by sn
and the latent variables an and gn , where a branching architecture is introduced in the last layer to evaluate the pose fidelity
with xn (Fig. 5). The mask estimator ψ is implemented as a UNet [39] that takes as input a shrunk image x∗n and a stack of411

Journal of Information Processing Vol.29 406–423 (May 2021)

Fig. 5 The network configuration of the generator θ of the pose-conditioned image VAE taking as input the pose sn , the appearance an , and the scene gn and yielding the mean μθ (an , gn , sn ) and thevariance σθ (an , gn , sn ).

Fig. 6

The network configuration of the generator θ of the pose VAE taking
as input the primitives zn and yielding the mean μφ (zn ) and variance
σφ (zn ).

the heatmaps of the joints given by sn and outputs a mask image
wn . To obtain sharper mask images, we apply a sigmoid function,
ς(x) = (1 + exp (−10x))−1 , to every element of the output wn of
the pre-trained estimator ψ.
4.3 Pose VAE
The pose VAE consisting of the pose-to-primitive recognizer δand the primitive-to-pose generator φ (decoder) is pre-trained in
an unsupervised manner by using only S. We maximize a variational lower bound L(φ, δ) of the marginal log likelihood log p(S)evaluating the pose plausibility. More specifically, we havelog p(sn ) ≥ Eq[log pφ (sn |zn ) + log p(zn ) − log qδ (zn |sn )]Fig. 7

Sixteen joints dealt with in our experiments.

5. Evaluation
This section reports comparative experiments conducted forevaluating the eﬀectiveness of our semi-supervised plausibilityand fidelity-aware pose estimation method. Our goal is to train a
DNN-based pose recognizer α that detects the coordinates of 16
joints (left and right ankles, knees, and hips, left and right wrists,
elbows, and shoulders, pelvis, thorax, upper neck, and head top as
shown in Fig. 7) from an image. We here validate two hypotheses: (A) under a supervised condition, the joint training of therecognition and generative models α, β, γ, θ, and φ outperforms
the standalone training of the pose recognizer α, and (B) undera semi-supervised condition, the mirror architecture makes eﬀective use of non-annotated images for improving performance.

= Eq [log pφ (sn |zn )] − KL(qδ (zn |sn )||p(zn ))def= Ln (φ, δ),

(34)

N
Ln (φ, δ). The two networks φ and δ arewhere L(φ, δ) = n=1
optimized jointly with the reparametrization trick [24] (Eq. (23)),
where the variance σ2φ of the generator φ is fixed to 1 for stability.
The recognizer δ is implemented in the same way as the recognizers β and γ except for the input dimension (Fig. 4). The
generator φ is implemented as a three-layered transposed convolutional network, where a branching architecture was introducedin the last layer to evaluate the pose plausibility (Fig. 6).

c 2021 Information Processing Society of Japan

5.1 Datasets and Criteria
We used two standard datasets that have been widely used inconventional studies on pose estimation.
5.1.1 Leeds Sports Pose (LSP) Dataset
The LSP dataset with its extension [18], [19] contains 12 K images of sports activities (11 K for training and 1 K for testing) intotal. Each image originally has an annotation about the coordinates of the 14 joints except for the pelvis and thorax. In order
to use the same set of joints as the MPII Human Pose Dataset [1](see Section 5.1.2) and use the same configuration of MirrorNet,
we estimated the ground-truth coordinate of the pelvis by aver-412

Journal of Information Processing Vol.29 406–423 (May 2021)

aging the coordinates of the left and right hips. Similarly, we
estimated the ground-truth coordinate of the thorax by averaging
the coordinates of the left and right shoulders. Each image wascropped to a square region centering on a person and then scaled
to Dx = 256 × 256.
The performance of pose estimation was measured with thepercentage of correct keypoints (PCK) [53]. The estimated coordinate of a joint was judged as correct if within τlt pixels fromthe ground-truth coordinate, where lt is the torso size defined as
the diagonal length of the ground-truth bounding box of the torsoand τ is a relative error tolerance (τ = 0.2 in our experiment).
5.1.2 MPII Human Pose (MPII) Dataset
The MPII dataset [1] contains around 25 K images of daily activities (22 K for training and 3 K for testing). Each image annotated with the coordinates of the 16 joints was cropped to a squareregion centering on a person, and then scaled to Dx = 256 × 192.
The performance of pose estimation was measured with thepercentage of correct keypoints in relation to head segment length(PCKh) [1]. The estimated coordinate of each joint was judged
as correct if within τlh pixels around the ground-truth coordinate,
where τ is a constant threshold and lh is the head size corresponding to 60% of the diagonal length of the ground-truth head bounding box. We used τ = 0.5 in our experiment.
5.2 Training Procedures
We randomly selected 20%, 40%, 60%, 80%, and 100% of
the training data as annotated images and regarded the remaining part as non-annotated images. Only the annotated images
were used for supervised training and the entire training data
were used for semi-supervised training. As in the oﬃcial implementation of Ref. [42], the training data were augmented withrandom scaling, rotation, and horizontal flipping [49]. The target data of sn were made by stacking 16 reduced-size grayscale
images (heatmaps) indicating the coordinates of the 16 joints(Ds = 16 × 64 × 64 or 16 × 64 × 48). In the test phase, a coordinate taking the maximum value in each of the 16 greyscale
images was detected.
We conducted curriculum learning as described in Section 4
and shown in Fig. 8, where the dimensions of the latent foreground, background and pose features were set to Da = Dg =Dz = 256 × 8 × 8 or 256 × 8 × 6 (Fig. 8).
( 1 ) Supervised pre-training: The six sub-networks weretrained independently in a supervised manner using the annotated images. The pose recognizer α based on the residual hourglass network (HGNet) [30], ResNet-50 [50], or thehigh-resolution network (HRNet) [42] was trained for 100
epochs (Section 4.1). The pose-conditioned image VAE consisting of the generator θ and the recognizers β and γ wastrained for 200 epochs (Section 4.2). The pose VAE consisting of the generator φ and the recognizer δ was also trainedfor 200 epochs (Section 4.3). The mask estimator ψ wastrained by using the UPi-S1h dataset [25] containing humanimages with silhouette annotations (i.e., masks), where images included in the LSP or MPII datasets were excluded.
( a ) MirrorNet was built by combining the six sub-networks
and the mask estimator ψ and passed to the step ( 2 ).

c 2021 Information Processing Society of Japan

Fig. 8

Learning curves obtained with the pose recognizer α based on thehourglass network [30] on the LSP dataset, in which 40% of the training data were regarded as annotated images.

( b ) The pose recognizer α was further trained for 100
epochs (i.e., 200 epochs in total) and the parameters at
the last 10 epochs were used for evaluation (baseline).
( 2 ) Supervised fine-tuning: MirrorNet was trained in a supervised manner with the same annotated images for 50 epochs,
where the mask estimator ψ was not updated.
( a ) MirrorNet at the last epoch was passed to the step ( 3 ).
( b ) MirrorNet was further trained for 50 epochs and the parameters of the pose recognizer α at the last 10 epochs
were used for evaluation (supervised MirrorNet).
( 3 ) Semi-supervised fine-tuning: MirrorNet was further
trained in a semi-supervised manner with the annotatedand non-annotated images for 50 epochs, where the mask
estimator ψ was not updated. The parameters at the last
10 epochs were used for evaluation (semi-supervised
MirrorNet).
To make a fair comparison, the pose recognizer α was trainedfor 200 epochs in total under any conditions. The performanceof pose estimation was measured by averaging the values of
PCK@0.2 or PCKh@0.5 over the last 10 epochs.
All networks were implemented using PyTorch [33] and optimized using Adam [22] with a learning rate of 0.001. The minibatch size was set to 128 images, which are fully annotated in the
supervised training phase or consisted of 96 annotated images and
32 non-annotated images in the semi-supervised training phase.
5.3 Experimental Results
Tables 1, 2, 3, 4, 5 and 6 show the respective performanceof pose estimation obtained by the pose recognizer α (Refs. [30],[50], or Ref. [42]) trained in the three ways (baseline, supervised
MirrorNet, and semi-supervised MirrorNet) on the LSP and MPII
datasets, respectively, and Fig. 9 comparatively shows the respective performance listed in the “Total” columns of Tables 1–6.
Under all conditions, the supervised MirrorNet outperformed thebaseline method by 4.61 ± 1.70 points on the LSP dataset and4.91 ± 1.75 points on the MPII dataset, where the means and standard deviations were computed over the fifteen conditions, i.e.,
all possible combinations of the pose recognizers [30], [42], [50]and the five ratios of annotated images (20%, 40%, 60%, 80%,and 100%). The left five columns of Fig. 9 clearly show thatthe supervised MirrorNet significantly outperformed the baseline

413

Journal of Information Processing Vol.29 406–423 (May 2021)

Table 1

Pose estimation performance on the LSP dataset [18], [19] with the pose recognizer α based on
HGNet [30].
Training data

PCK@0.2#annotated#non-annotated

Head

Shoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

Baseline [30]Supervised MirrorNet
Semi-supervised MirrorNet

2200
2200
2200

8800

92.87
93.81
90.68

81.50
83.83
79.49

73.05
75.78
69.60

70.08
73.26
67.97

71.81
75.18
69.15

67.73
72.41
66.69

65.85
68.91
63.33

74.94
77.80
72.66

Baseline [30]Supervised MirrorNet
Semi-supervised MirrorNet

4400
4400
4400

6600

92.51
95.07
94.88

85.46
86.55
88.94

79.75
80.02
83.32

77.67
76.08
80.27

76.60
80.36
82.34

77.91
79.71
82.41

74.49
75.64
79.15

80.81
82.10
84.68

Baseline [30]Supervised MirrorNet
Semi-supervised MirrorNet

6600
6600
6600

4400

94.79
95.27
95.60

87.07
89.94
89.39

80.59
85.90
85.91

77.82
83.66
83.60

80.46
83.53
84.06

79.36
84.82
85.13

75.73
82.37
83.13

82.46
86.69
86.87

Baseline [30]Supervised MirrorNet
Semi-supervised MirrorNet

8800
8800
8800

2200

94.73
95.62
95.63

87.83
89.71
89.98

81.88
85.10
85.65

79.24
83.84
83.98

82.06
84.51
84.87

82.35
86.29
86.29

79.05
83.68
83.44

84.10
87.19
87.34

Baseline [30]Supervised MirrorNet

11000
11000

-

95.85
96.79

89.68
92.31

84.89
89.15

83.25
87.64

85.39
89.94

86.24
90.29

82.41
88.30

86.97
90.76

Table 2

Pose estimation performance on the LSP dataset [18], [19] with the pose recognizer α based on
ResNet [50].
Training data

PCK@0.2#annotated#non-annotated

Head

Shoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

Baseline [50]Supervised MirrorNet
Semi-supervised MirrorNet

2200
2200
2200

8800

86.69
89.25
85.15

69.26
76.55
75.49

56.62
65.97
63.86

52.32
61.14
58.69

57.22
65.67
63.58

53.37
62.69
57.17

48.04
54.02
47.66

60.83
68.19
64.89

Baseline [50]Supervised MirrorNet
Semi-supervised MirrorNet

4400
4400
4400

6600

89.59
90.82
88.39

78.16
81.23
79.58

68.97
72.19
70.78

63.28
67.94
66.42

67.74
71.68
68.87

63.31
67.53
64.67

55.02
60.05
59.78

69.68
73.30
71.54

Baseline [50]Supervised MirrorNet
Semi-supervised MirrorNet

6600
6600
6600

4400

89.59
92.50
92.36

78.40
83.15
83.38

69.26
75.29
75.31

64.51
72.23
72.24

70.32
74.39
75.55

64.47
71.70
73.27

57.23
63.62
64.85

70.82
76.35
77.00

Baseline [50]Supervised MirrorNet
Semi-supervised MirrorNet

8800
8800
8800

2200

90.29
93.06
93.98

78.70
84.04
85.24

68.40
76.79
76.51

64.70
73.48
72.78

69.70
76.81
77.08

64.79
75.11
74.96

57.62
65.80
66.84

70.90
78.14
78.40

Baseline [50]Supervised MirrorNet

11000
11000

-

92.35
94.33

82.93
86.94

75.11
80.91

69.71
76.98

78.66
84.64

72.57
80.00

62.79
72.42

76.60
82.56

Table 3

Pose estimation performance on the LSP dataset [18], [19] with the pose recognizer α based on
HRNet [42].
Training data

PCK@0.2#annotated#non-annotated

Head

Shoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

Baseline [42]Supervised MirrorNet
Semi-supervised MirrorNet

2200
2200
2200

8800

92.18
94.04
92.55

81.46
83.92
84.16

73.15
78.19
77.90

70.59
76.61
76.58

72.84
75.90
73.57

71.28
74.82
76.14

68.06
72.30
74.06

75.86
79.61
79.49

Baseline [42]Supervised MirrorNet
Semi-supervised MirrorNet

4400
4400
4400

6600

93.05
95.02
94.47

83.40
88.23
88.48

76.19
82.45
83.31

74.31
81.23
82.57

75.77
81.30
81.63

73.53
81.16
82.95

70.78
78.76
80.23

78.36
84.22
85.02

Baseline [42]Supervised MirrorNet
Semi-supervised MirrorNet

6600
6600
6600

4400

93.86
95.65
95.69

84.90
88.82
89.40

77.74
84.34
85.11

75.29
82.74
83.37

78.69
83.42
83.28

77.49
84.74
84.58

74.78
82.36
83.23

80.65
86.22
86.58

Baseline [42]Supervised MirrorNet
Semi-supervised MirrorNet

8800
8800
8800

2200

93.99
95.61
95.70

85.49
89.69
89.91

79.21
85.21
85.52

77.20
83.90
83.60

77.96
83.98
82.96

78.57
85.31
85.78

75.70
83.67
83.81

81.43
86.98
86.95

Baseline [42]Supervised MirrorNet

11000
11000

-

95.74
96.65

90.45
92.78

86.20
89.71

83.08
87.44

88.74
91.72

86.92
91.57

83.80
89.90

88.06
91.58

c 2021 Information Processing Society of Japan


414

Journal of Information Processing Vol.29 406–423 (May 2021)

Table 4

Pose estimation performance on the MPII dataset [1] with the pose recognizer α based on
HGNet [30].
Training data

PCKh@0.5#annotated#non-annotated

Head

Shoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

Baseline [30]Supervised MirrorNet
Semi-supervised MirrorNet

4449
4449
4449

17797

93.26
94.10
93.34

86.86
89.27
87.92

76.10
79.05
77.00

69.17
72.89
70.03

75.07
77.64
73.60

67.82
71.16
67.68

63.91
67.60
63.02

76.91
79.63
77.02

Baseline [30]Supervised MirrorNet
Semi-supervised MirrorNet

8899
8899
8899

13347

94.32
95.12
95.19

89.07
91.40
92.15

78.58
81.97
83.27

71.36
75.57
77.03

78.48
81.98
81.57

71.29
75.14
76.30

67.29
71.20
72.82

79.43
82.51
83.32

Baseline [30]Supervised MirrorNet
Semi-supervised MirrorNet

13347
13347
13347

8899

94.93
95.92
95.85

91.24
93.37
93.56

81.63
84.96
85.06

74.97
78.54
79.15

81.92
85.07
85.35

74.41
78.80
79.26

69.83
74.90
75.47

82.07
85.18
85.47

Baseline [30]Supervised MirrorNet
Semi-supervised MirrorNet

17797
17797
17797

4449

94.90
95.85
95.76

91.32
93.45
93.36

81.62
85.40
85.14

74.70
79.32
79.21

81.06
85.24
84.79

74.39
79.32
79.29

70.32
75.69
75.64

81.94
85.54
85.38

Baseline [30]Supervised MirrorNet

22246
22246

-

95.08
96.13

91.94
94.01

82.47
86.47

75.98
80.05

82.65
86.41

75.69
80.56

71.58
76.85

82.94
86.40

Table 5

Pose estimation performance on the MPII dataset [1] with the pose recognizer α based on
ResNet [50].
Training data

PCKh@0.5#annotated#non-annotated

Head

Shoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

Baseline [50]Supervised MirrorNet
Semi-supervised MirrorNet

4449
4449
4449

17797

88.71
90.71
89.74

80.11
82.99
82.29

64.74
69.26
67.98

54.85
60.25
58.50

66.30
70.08
66.54

54.39
59.96
57.14

52.14
56.31
53.70

67.10
71.03
69.16

Baseline [50]Supervised MirrorNet
Semi-supervised MirrorNet

8899
8899
8899

13347

90.56
92.68
92.94

82.91
87.16
87.31

68.39
74.86
74.76

58.58
65.74
66.50

70.61
76.39
75.14

59.16
66.54
66.18

55.31
61.74
61.26

70.50
76.01
75.87

Baseline [50]Supervised MirrorNet
Semi-supervised MirrorNet

13347
13347
13347

8899

90.64
92.96
93.02

83.58
87.84
88.14

68.52
74.80
75.28

58.19
65.40
66.15

71.63
77.32
77.89

59.38
66.72
67.53

55.62
62.22
62.67

70.79
76.32
76.79

Baseline [50]Supervised MirrorNet
Semi-supervised MirrorNet

17797
17797
17797

4449

89.49
92.36
92.72

81.37
86.68
87.58

65.70
73.50
74.04

54.90
64.19
64.20

68.78
76.36
76.88

56.91
65.63
65.92

53.50
60.34
60.65

68.40
75.19
75.61

Baseline [50]Supervised MirrorNet

22246
22246

-

91.07
93.76

84.86
89.36

70.31
76.86

60.31
67.90

72.61
79.12

60.84
68.87

56.72
63.65

72.11
78.06

Table 6

Pose estimation performance on the MPII dataset [1] with the pose recognizer α based on HRNet [42].
Training data

PCKh@0.5#annotated#non-annotated

Head

Shoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

Baseline [42]Supervised MirrorNet
Semi-supervised MirrorNet

4449
4449
4449

17797

92.89
93.89
93.92

86.95
89.40
88.89

75.44
79.90
79.05

68.99
73.70
72.40

74.50
77.76
75.14

66.80
72.07
70.59

62.73
67.74
66.22

76.42
80.04
78.91

Baseline [42]Supervised MirrorNet
Semi-supervised MirrorNet

8899
8899
8899

13347

93.86
95.32
95.32

88.27
91.82
91.97

77.52
83.09
83.09

70.69
76.67
76.77

77.62
82.71
81.31

69.71
76.44
76.08

66.05
72.43
72.26

78.52
83.35
83.12

Baseline [42]Supervised MirrorNet
Semi-supervised MirrorNet

13347
13347
13347

8899

93.83
95.84
95.67

88.23
92.67
92.59

78.02
84.60
84.67

70.78
78.04
78.21

77.87
84.03
84.15

69.92
77.62
77.95

66.26
73.76
73.90

78.70
84.48
84.57

Baseline [42]Supervised MirrorNet
Semi-supervised MirrorNet

17797
17797
17797

4449

93.51
95.71
95.60

87.55
92.93
92.81

76.84
84.31
84.31

69.50
77.73
77.55

75.97
83.89
84.09

67.84
77.41
77.50

63.48
73.16
73.26

77.31
84.30
84.30

Baseline [42]Supervised MirrorNet

22246
22246

-

92.55
95.91

87.22
93.77

75.95
85.85

69.69
79.33

75.68
85.48

67.11
79.41

63.67
75.35

76.92
85.69

c 2021 Information Processing Society of Japan


415

Journal of Information Processing Vol.29 406–423 (May 2021)

Fig. 9 Pose estimation performance (“Total” columns of Tables 1–6). The top and bottom rows show therespective performance on the LSP dataset [18], [19] and the MPII dataset [1], respectively. Fromleft to right, the first five columns show the performance under the conditions that 20%, 40%, 60%,
80%, and 100% of the training data were regarded as annotated images, respectively, and the lasttwo columns show the average respective performance under the 20% and 40% conditions andthose under the 60% and 80% conditions, respectively.
Table 7

Pose estimation performance on the LSP dataset [18], [19] with the pose recognizer α based on
HRNet [42] with respect to the number of training epochs.
Training epochs

Training data

PCK@0.2#annotated#non-annotated

Head

Shoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

Supervised MirrorNet
Semi-supervised MirrorNet

200
200

8800
8800

2200

95.61
95.70

89.69
89.91

85.21
85.52

83.90
83.60

83.98
82.96

85.31
85.78

83.67
83.81

86.98
86.95

Supervised MirrorNet
Semi-supervised MirrorNet

300
300

8800
8800

2200

96.26
95.92

89.83
89.85

86.29
86.40

84.85
84.81

84.20
84.50

85.79
86.52

83.81
83.81

87.47
87.61

Table 8Pose estimation performance on the MPII dataset [1] with the pose recognizer α based on
HGNet [30] with respect to the number of training epochs.
Training epochs

Training data

PCK@0.2#annotated#non-annotated

Head

Shoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

Supervised MirrorNet
Semi-supervised MirrorNet

200
200

17797
17797

4449

95.85
95.76

93.45
93.36

85.40
85.14

79.32
79.21

85.24
84.79

79.32
79.29

75.69
75.64

85.54
85.38

Supervised MirrorNet
Semi-supervised MirrorNet

300
300

17797
17797

4449

95.96
95.92

93.46
93.67

85.91
86.08

80.13
80.17

85.39
85.73

80.34
80.42

76.70
76.73

86.02
86.15

method. This strongly supports hypothesis (A) or namely that thejoint training of the generative and recognition models leads toperformance improvement. The fidelity and plausibility of estimated poses, which were evaluated by the pose-to-image generator θ and the pose VAE, respectively, were key factors for accurate
pose estimation.
We found that the semi-supervised MirrorNet outperformed
the supervised MirrorNet when the ratios of annotated images
were higher in the training data. As shown in the rightmost column of Fig. 9, the semi-supervised MirrorNet outperformed thec 2021 Information Processing Society of Japansupervised MirrorNet when the annotation ratio was 60% or 80%,
except for the conditions that the HRNet [42] and the HGNet [30]were used as the pose recognizer α on the LSP dataset and the
MPII dataset, respectively. When the number of epochs was increased from 50 to 100 in each of the steps ( 2 ) and ( 3 ) on
trial, the semi-supervised MirrorNet performed better under thoseconditions, as shown in Tables 7 and 8. The performance had
often been degraded temporarily before the pose recognizer α
learned to extract consistent representations from annotated andnon-annotated images. As shown in the second right column

416

Journal of Information Processing Vol.29 406–423 (May 2021)

Fig. 10 Examples of pose estimation obtained by the baseline method [30], the supervised and semisupervised versions of MirrorNet. Anatomically implausible poses were corrected by the MirrorNet architecture.
Table 9 Network size and computation speed.
Network
Pose recognizer α– Hourglass [30]– ResNet-50 [50]– HRNet [42]Pose-conditioned image VAE– Appear. & scene recognizers β & γ– Image generator θ– Mask estimator ψPose VAE– Primitive recognizer δ– Pose generator φ
MirrorNet (training)– α (Hourglass [30]), β, γ, δ, θ, φ, & ψ– α (ResNet-50 [50]), β, γ, δ, θ, φ, & ψ– α (HRNet [42]), β, γ, δ, θ, φ, & ψ
MirrorNet (runtime)– only α (Hourglass [30])– only α (ResNet-50 [50])– only α (HRNet [42])#params

GFLOPs(LSP)

GFLOPs(MPII)

25.59M
34.00M
28.54M

26.17
11.99
9.49

19.62
8.99
7.12

5.28M
12.84M
10.28M

1.11
3.25
1.42

0.83
2.44
1.06

5.29M
1.33M

1.13
1.13

0.85
0.85

65.89M
74.30M
68.84M

35.32
21.14
18.64

26.48
15.85
13.98

25.59M
34.00M
28.54M

26.17
11.99
9.49

19.62
8.99
7.12

of Fig. 9, in contrast, the semi-supervised MirrorNet underperformed the supervised MirrorNet when the annotation ratio was
20% or 40%, Under these conditions, the pose-to-image generator θ and the pose VAE could not appropriately evaluate the fidelity and plausibility of estimated poses, i.e., gave wrong feedback to the pose recognizer α in steps ( 2 ) and ( 3 ), leading to
the performance degradation. These results conditionally support
hypothesis (B) or namely that the semi-supervised training is effective under the condition that a suﬃcient number of annotatedimages are available and MirrorNet is suﬃciently trained in thesemi-supervised fine-tuning steps.
As shown in Fig. 10, the pose recognizer α trained by using the MirrorNet architecture yielded anatomically plausible
poses. For a better understanding of how each part of MirrorNet works, we show examples of human images generated by
the pose-conditioned VAE in Fig. A·1, pose images by the pose
VAE in Fig. A·2, and silhouette images by the mask estimator φin Fig. A·3 in the appendix. As shown in Table 9, the training of
the whole MirrorNet is computationally demanding because thegenerative and recognition models of pose and images should betrained jointly. Note that only the pose recognizer α is used in

c 2021 Information Processing Society of Japanthe runtime; the pose-conditioned VAE and the pose VAE serve
as regularizers that stabilize the training of the MirrorNet.
5.4 Discussions
Eﬀectiveness of Individual Components. To validate the effectiveness of each component of MirrorNet, we conducted anablation study. We used the semi-supervised MirrorNet with the
HRNet-based pose recognizer α [42] trained on the LSP dataset,
where the ratio of annotated images was 20%, 40%, 60%, or80%. Tables 10 and 11 show the pose estimation performanceobtained with and without the lower-level mirror system and themask estimator. This clearly shows the eﬀectiveness of the lowerlevel mirror system and the mask estimator, respectively. Figure A·4 shows examples of human images generated by the poseconditioned VAE with and without the mask estimator. These
results strongly support the design of MirrorNet.
Eﬀectiveness of Curriculum Learning. To validate the eﬀectiveness of the curriculum learning consisting of the pre-trainingof the individual sub-networks and the fine-tuning of the wholenetwork, we conducted another additional experiment. We used
the semi-supervised MirrorNet with the HRNet-based pose recognizer α [42] trained on the LSP dataset, where the ratio of annotated images was 20%, 40%, 60%, or 80%. Table 12 shows
the pose estimation performance obtained with and without the
pre-training. When the ratio of annotated images was larger, MirrorNet without the pre-training outperformed that with the pretraining, i.e., training the whole network from scratch is more
eﬀective than the stability-aware gradual optimization for finding a better solution. It is thus important to select an appropriate
training procedure for extracting the full potential of the semisupervised MirrorNet.
Eﬀectiveness of Semi-supervised Learning. To investigate theimpact of the mini-batch composition in the semi-supervised
training of MirrorNet, we changed the number of annotatedand non-annotated images in each mini-batch to 32+96, 48+80,
64+64, 80+48, or 96+32. We used MirrorNet with the HRNetbased pose recognizer α [42] trained on the LSP dataset, where
the ratio of annotated images was 20%, 40%, 60%, or 80%. As
shown in Section 5.3, the semi-supervised MirrorNet underper-

417

Journal of Information Processing Vol.29 406–423 (May 2021)

Table 10

Pose estimation performance obtained with and without the lower-level mirror system.
Training data

PCK@0.2#annotated#non-annotated

Head

Shoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

MirrorNet w/o pose VAE
MirrorNet w/ pose VAE

2200
2200

8800
8800

94.43
92.55

84.02
84.16

76.25
77.90

75.09
76.58

75.64
73.57

72.62
76.14

69.10
74.06

78.31
79.49

MirrorNet w/o pose VAE
MirrorNet w/ pose VAE

4400
4400

6600
6600

94.85
94.47

86.63
88.48

80.74
83.31

78.57
82.57

79.95
81.63

78.80
82.95

76.49
80.23

82.49
85.02

MirrorNet w/o pose VAE
MirrorNet w/ pose VAE

6600
6600

4400
4400

95.58
95.69

87.75
89.40

82.95
85.11

81.07
83.37

81.32
83.28

83.08
84.58

80.12
83.23

84.77
86.58

MirrorNet w/o pose VAE
MirrorNet w/ pose VAE

8800
8800

2200
2200

95.46
95.70

88.50
89.91

83.53
85.52

81.33
83.60

83.09
82.96

83.30
85.78

80.96
83.81

85.35
86.95

MirrorNet w/o pose VAE
MirrorNet w/ pose VAE

11000
11000

-

96.98
96.65

92.51
92.78

89.30
89.71

87.89
87.44

89.28
91.72

90.35
91.57

88.38
89.90

90.81
91.58

Table 11

Pose estimation performance obtained with and without the mask estimator.
Training data

PCK@0.2#annotated#non-annotated

Head

Shoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

MirrorNet w/o mask estimator
MirrorNet w/ mask estimator

2200
2200

8800
8800

91.62
92.55

83.85
84.16

77.15
77.90

76.38
76.58

72.35
73.57

74.77
76.14

72.49
74.06

78.62
79.49

MirrorNet w/o mask estimator
MirrorNet w/ mask estimator

4400
4400

6600
6600

94.58
94.47

88.58
88.48

83.52
83.31

81.83
82.57

81.10
81.63

81.77
82.95

79.77
80.23

84.64
85.02

MirrorNet w/o mask estimator
MirrorNet w/ mask estimator

6600
6600

4400
4400

95.43
95.69

88.60
89.40

83.53
85.11

81.53
83.37

82.77
83.28

83.68
84.58

81.62
83.23

85.49
86.58

MirrorNet w/o mask estimator
MirrorNet w/ mask estimator

8800
8800

2200
2200

95.55
95.70

88.70
89.91

84.86
85.52

83.63
83.60

82.81
82.96

85.00
85.78

83.45
83.81

86.47
86.95

MirrorNet w/o mask estimator
MirrorNet w/ mask estimator

11000
11000

-

96.91
96.65

92.55
92.78

89.06
89.71

87.79
87.44

90.00
91.72

90.22
91.57

87.53
89.90

90.68
91.58

Table 12 Pose estimation performance obtained with and without the supervised pre-training.
Training data

PCK@0.2#annotated#non-annotated

Head

Shoulder

Elbow

Wrist

Hip

Knee

Ankle

Total

MirrorNet w/o supervised pre-training
MirrorNet w/ supervised pre-training

2200
2200

8800
8800

91.52
92.55

82.02
84.16

76.37
77.90

74.79
76.58

73.76
73.57

75.15
76.14

72.32
74.06

78.22
79.49

MirrorNet w/o supervised pre-training
MirrorNet w/ supervised pre-training

4400
4400

6600
6600

93.54
94.47

87.31
88.48

82.40
83.31

81.38
82.57

77.93
81.63

81.80
82.95

79.11
80.23

83.56
85.02

MirrorNet w/o supervised pre-training
MirrorNet w/ supervised pre-training

6600
6600

4400
4400

95.16
95.69

88.84
89.40

85.53
85.11

84.47
83.37

81.82
83.28

85.63
84.58

83.60
83.23

86.61
86.58

MirrorNet w/o supervised pre-training
MirrorNet w/ supervised pre-training

8800
8800

2200
2200

95.85
95.70

91.27
89.91

87.68
85.52

85.63
83.60

85.63
82.96

88.09
85.78

86.08
83.81

88.82
86.95

MirrorNet w/o supervised pre-training
MirrorNet w/ supervised pre-training

11000
11000

-

97.05
96.65

93.59
92.78

91.51
89.71

90.63
87.44

91.99
91.72

92.73
91.57

91.35
89.90

92.79
91.58

formed the supervised MirrorNet when the mini-batch size was
96+32 and the ratio of annotated images was 20%. Interestingly,
as shown in Table 13, the semi-supervised MirrorNet outperformed the supervised MirrorNet under the conditions of 32+96
and 64+64. In the objective function given by Eq. (29), the contributions of annotated and non-annotated images are directly affected by the ratio of annotated images in each mini-batch. Thus,
it is necessary to optimize it for drawing out the full potential ofsemi-supervised learning. This should be included as topics for
future work.
Future Directions. One of the most interesting research direc-

c 2021 Information Processing Society of Japantions is to investigate fully unsupervised training of MirrorNet
because it is based on the VAE architecture and can be trained
from only non-annotated images in theory. In this study, the hierarchical mirror system in single-person 2D pose estimation has
successfully been used only under the semi-supervised condition.
Another important research direction is to deal with human images in which some joints are occluded or out of view. The noticeable advantage of the fully probabilistic modeling underlying
MirrorNet is that unobserved joints could be statistically inferredduring the training. In addition, it is worth extending the current
MirrorNet for multi-person 3D pose estimation by using a hier-418

Journal of Information Processing Vol.29 406–423 (May 2021)

Table 13

Pose estimation performance with respect to the ratio of annotated images in each mini-batch.
Training data

Mini-batch composition

PCK@0.2#annotated #non-annotated #annotated #non-annotated

Head

Shoulder Elbow Wrist

Hip

Knee

Ankle

Total

Supervised MirrorNet
Semi-supervised MirrorNet

2200
2200
2200
2200
2200
2200

8800
8800
8800
8800
8800

128
32
48
64
80
96

96
80
64
48
32

94.04
94.00
94.08
93.78
92.67
92.55

83.92
84.86
82.55
84.89
84.45
84.16

78.19
78.71
78.11
79.02
78.83
77.90

76.61
77.33
76.86
77.10
76.82
76.58

75.90
78.07
73.89
77.28
75.38
73.57

74.82
77.21
74.89
76.81
75.29
76.14

72.30
74.88
72.97
73.91
72.27
74.06

79.61
80.95
79.28
80.61
79.59
79.49

Supervised MirrorNet
Semi-supervised MirrorNet

4400
4400
4400
4400
4400
4400

6600
6600
6600
6600
6600

128
32
48
64
80
96

96
80
64
48
32

95.02
94.67
95.20
95.07
94.94
94.47

88.23
87.71
87.86
88.84
88.30
88.48

82.45
80.63
81.76
83.20
81.98
83.31

81.23
79.07
80.27
81.09
80.85
82.57

81.30
80.92
81.92
82.33
81.75
81.63

81.16
80.19
81.26
82.11
83.58
82.95

78.76
77.40
78.05
79.63
80.72
80.23

84.22
83.14
83.94
84.82
84.82
85.02

Supervised MirrorNet
Semi-supervised MirrorNet

6600
6600
6600
6600
6600
6600

4400
4400
4400
4400
4400

128
32
48
64
80
96

96
80
64
48
32

95.65
95.08
95.93
95.37
95.71
95.69

88.82
86.98
89.18
88.21
88.62
89.40

84.34
80.41
84.10
82.98
84.18
85.11

82.74
78.71
81.73
82.00
82.69
83.37

83.42
80.76
82.45
81.76
83.02
83.28

84.74
80.81
83.38
84.08
84.68
84.58

82.36
77.59
80.61
81.93
82.67
83.23

86.22
83.11
85.50
85.43
86.15
86.58

Supervised MirrorNet
Semi-supervised MirrorNet

8800
8800
8800
8800
8800
8800

2200
2200
2200
2200
2200

128
32
48
64
80
96

96
80
64
48
32

95.61
94.39
94.60
94.96
95.44
95.70

89.69
86.12
88.22
88.47
89.62
89.91

85.21
80.50
83.22
83.47
85.17
85.52

83.90
78.95
81.42
82.08
83.25
83.60

83.98
78.67
82.54
82.87
83.55
82.96

85.31
78.05
81.83
83.64
84.66
85.78

83.67
76.34
80.14
82.06
82.76
83.81

86.98
82.05
84.78
85.58
86.56
86.95

archical mirror system involving the 3D pose VAE at the lower
level.

6. Conclusion
Inspired by the cognitive knowledge about the mirror neuron
system of humans, this paper proposes a deep Bayesian framework called MirrorNet for 2D pose estimation from human images. The key idea is to jointly train the generative models ofimages and poses as well as the recognition models of appearances, scenes, and primitives in a fully statistical manner. From atechnical point of view, the two-level mirror systems (VAEs) are
jointly trained with the hierarchical autoencoding manner (image→ pose → primitive → pose → image), such that the plausibility
and fidelity of poses are both considered. Thanks to the natureof the fully generative modeling, MirrorNet is the first pose estimation architecture that could, in theory, be trained from nonannotated images in an unsupervised manner when appropriate
inductive biases are introduced. We experimentally proved thatthe whole MirrorNet could be jointly trained and outperformed
a conventional recognition-model-only method in terms of pose
estimation performance. We also showed that the additional useof non-annotated images could improve the performance.
The main contribution of this paper is that we shed light on the
mirror neuron system (or motor theory) and built a robust computational model of the human vision system by leveraging theexpressive power of modern deep Bayesian models. The same
framework can be applied to 3D motion estimation from videos
by formulating recurrent versions of the pose and image VAEs
that represent the anatomical plausibility and fidelity of humanc 2021 Information Processing Society of Japan

motions, respectively. This paper also ushers the new research
field of semi-supervised pose estimation. We believe that MirrorNet inspires a new approach to multimedia understanding.
Acknowledgments We are thankful for AI Bridging Cloud
Infrastructure (ABCI) of National Institute of Advanced Industrial Science and Technology (AIST), which we used extensively
for our experiments. This work was partly supported by the
Program for Leading Graduate Schools, “Graduate Program for
Embodiment Informatics” of the Ministry of Education, Culture, Sports, Science and Technology (MEXT) of Japan, JST
ACCEL No.JPMJAC1602, JSPS KAKENHI No.19H04137, and
JST-Mirai Program No.JPMJMI19B2.
References
[1]

[2]
[3]
[4]
[5]
[6]

[7]

Andriluka, M., Pishchulin, L., Gehler, P. and Schiele, B.: 2D Human Pose Estimation: New Benchmark and State of the Art Analysis, Conference on Computer Vision and Pattern Recognition (CVPR),
pp.3686–3693 (2014).
Andriluka, M., Roth, S. and Schiele, B.: Pictorial structures revisited:
People detection and articulated pose estimation, Conference on Computer Vision and Pattern Recognition (CVPR), pp.1014–1021 (2009).
Belagiannis, V. and Zisserman, A.: Recurrent human pose estimation,
International Conference on Automatic Face & Gesture Recognition(FG), pp.468–475 (2017).
Bulat, A. and Tzimiropoulos, G.: Human pose estimation via convolutional part heatmap regression, European Conference on Computer
Vision (ECCV), pp.717–732 (2016).
Carreira, J., Agrawal, P., Fragkiadaki, K. and Malik, J.: Human pose
estimation with iterative error feedback, Conference on Computer Vision and Pattern Recognition (CVPR), pp.4733–4742 (2016).
Chen, Y., Wang, Z., Peng, Y., Zhang, Z., Yu, G. and Sun, J.: Cascaded pyramid network for multi-person pose estimation, Conference
on Computer Vision and Pattern Recognition (CVPR), pp.7103–7112(2018).
Chen, Y., Shen, C., Wei, X.-S., Liu, L. and Yang, J.: Adversarial

419

Journal of Information Processing Vol.29 406–423 (May 2021)

[8]

[9]

[10]
[11]

[12]

[13]
[14]

[15]
[16]
[17]
[18]
[19]
[20]
[21]

[22]
[23]
[24]
[25]

[26]
[27]
[28]
[29]
[30]
[31]

PoseNet: A structure-aware convolutional network for human pose
estimation, Conference on Computer Vision and Pattern Recognition(CVPR), pp.1212–1221 (2017).
Chou, C.-J., Chien, J.-T. and Chen, H.-T.: Self adversarial training
for human pose estimation, Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC),
pp.17–30 (2018).
Chu, X., Yang, W., Ouyang, W., Ma, C., Yuille, A.L. and Wang,
X.: Multi-context attention for human pose estimation, Conference
on Computer Vision and Pattern Recognition (CVPR), pp.1831–1840(2017).
Dai, Z., Damianou, A., Gonzalez, J. and Lawrence, N.: Variational
Auto-encoded Deep Gaussian Processes, International Conference on
Learning Representations (ICLR), pp.1–11 (2016).
Dantone, M., Gall, J., Leistner, C. and Van Gool, L.: Human pose
estimation using body parts dependent joint regressors, Conference
on Computer Vision and Pattern Recognition (CVPR), pp.3041–3048(2013).
de Bem, R., Ghosh, A., Ajanthan, T., Miksik, O., Boukhayma, A.,
Siddharth, N. and Torr, P.: DGPose: Deep Generative Models for Human Body Analysis, International Journal of Computer Vision (IJCV),
Vol.128, pp.1537–1563 (2020).
Fieraru, M., Khoreva, A., Pishchulin, L. and Schiele, B.: Learning
to refine human pose estimation, Conference on Computer Vision and
Pattern Recognition Workshops (CVPRW), pp.205–214 (2018).
Gkioxari, G., Arbelaez, P., Bourdev, L. and Malik, J.: Articulated
pose estimation using discriminative armlet classifiers, Conference
on Computer Vision and Pattern Recognition (CVPR), pp.3342–3349(2013).
Hawking, S.: The Universe in a Nutshell. The Inspiring Sequel to A
Brief History of Time, London: Transworld Publishers (2001).
He, K., Zhang, X., Ren, S. and Sun, J.: Deep residual learning for image recognition, Conference on Computer Vision and Pattern Recognition (CVPR), pp.770–778 (2016).
Iacoboni, M. and Mazziotta, J.C.: Mirror Neuron System: Basic Findings and Clinical Applications, American Neurological Association,
Vol.62, No.3, pp.213–218 (2007).
Johnson, S. and Everingham, M.: Clustered Pose and Nonlinear Appearance Models for Human Pose Estimation, British Machine Vision
Conference (BMVC), pp.1–11 (2010).
Johnson, S. and Everingham, M.: Learning eﬀective human pose estimation from inaccurate annotation, Conference on Computer Vision
and Pattern Recognition (CVPR), pp.1465–1472 (2011).
Ke, L., Chang, M.-C., Qi, H. and Lyu, S.: Multi-scale structure-aware
network for human pose estimation, European Conference on Computer Vision (ECCV), pp.713–728 (2018).
Kim, T., Cha, M., Kim, H., Lee, J.K. and Kim, J.: Learning to discover cross-domain relations with generative adversarial networks, International Conference on Machine Learning (ICML), pp.1857–1865(2017).
Kingma, D.P. and Ba, J.: Adam: A Method for Stochastic Optimization, International Conference for Learning Representations (ICLR),
pp.1–13 (2015).
Kingma, D.P., Rezende, D.J., Mohamed, S. and Welling, M.: Semisupervised Learning with Deep Generative Models, Advances in Neural Information Processing Systems (NIPS), pp.3581–3589 (2014).
Kingma, D.P. and Welling, M.: Auto-Encoding Variational Bayes, International Conference on Learning Representations (ICLR), pp.1–14(2014).
Lassner, C., Romero, J., Kiefel, M., Bogo, F., Black, M.J. and Gehler,
P.V.: Unite the People: Closing the Loop Between 3D and 2D Human
Representations, Conference on Computer Vision and Pattern Recognition (CVPR), pp.6050–6059 (2017).
Lifshitz, I., Fetaya, E. and Ullman, S.: Human pose estimation using deep consensus voting, European Conference on Computer Vision(ECCV), pp.246–260 (2016).
Ma, L., Sun, Q., Georgoulis, S., Van Gool, L., Schiele, B. and Fritz,
M.: Disentangled person image generation, Conference on Computer
Vision and Pattern Recognition (CVPR), pp.99–108 (2018).
Mnih, A. and Gregor, K.: Neural Variational Inference and Learningin Belief Networks, International Conference on Machine Learning(ICML), pp.1791–1799 (2014).
Moon, G., Chang, J.Y. and Lee, K.M.: Posefix: Model-agnostic general human pose refinement network, Conference on Computer Vision
and Pattern Recognition (CVPR), pp.7773–7781 (2019).
Newell, A., Yang, K. and Deng, J.: Stacked hourglass networks forhuman pose estimation, European Conference on Computer Vision(ECCV), pp.483–499 (2016).
Nie, X., Feng, J., Zhang, J. and Yan, S.: Single-stage multi-person
pose machines, International Conference on Computer Vision (ICCV),
pp.6951–6960 (2019).

c 2021 Information Processing Society of Japan


[32]
[33]

[34]

[35]
[36]
[37]
[38]

[39]

[40]
[41]
[42]
[43]
[44]

[45]
[46]
[47]
[48]
[49]

[50]
[51]
[52]
[53]
[54]

[55]
[56]

Nie, X., Feng, J., Zuo, Y. and Yan, S.: Human pose estimation withparsing induced learner, Conference on Computer Vision and Pattern
Recognition (CVPR), pp.2100–2108 (2018).
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G.,
Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: PyTorch: An
imperative style, high-performance deep learning library, Advancesin Neural Information Processing Systems (NeurIPS), pp.8024–8035(2019).
Peng, X., Tang, Z., Yang, F., Feris, R.S. and Metaxas, D.: Jointly optimize data augmentation and network training: Adversarial data augmentation in human pose estimation, Conference on Computer Vision
and Pattern Recognition (CVPR), pp.2226–2234 (2018).
Pishchulin, L., Andriluka, M., Gehler, P. and Schiele, B.: Poselet conditioned pictorial structures, Conference on Computer Vision and Pattern Recognition (CVPR), pp.588–595 (2013).
Qiao, T., Zhang, J., Xu, D. and Tao, D.: MirrorGAN: Learning Textto-image Generation by Redescription, Conference on Computer Vision and Pattern Recognition (CVPR), pp.1505–1514 (2019).
Ramanan, D.: Learning to parse images of articulated bodies, Advances in Neural Information Processing Systems (NIPS), pp.1129–
1136 (2007).
Rezende, D.J., Mohamed, S. and Wierstra, D.: Stochastic Backpropagation and Approximate Inference in Deep Generative Models, International Conference on Machine Learning (ICML), pp.1278–1286(2014).
Ronneberger, O., Fischer, P. and Brox, T.: U-Net: Convolutional networks for biomedical image segmentation, International Conference
on Medical Image Computing and Computer-Assisted Intervention(MICCAI), Springer, pp.234–241 (2015).
Sapp, B. and Taskar, B.: Modec: Multimodal decomposable models
for human pose estimation, Conference on Computer Vision and Pattern Recognition (CVPR), pp.3674–3681 (2013).
Sapp, B., Jordan, C. and Taskar, B.: Adaptive pose priors for pictorial
structures, Conference on Computer Vision and Pattern Recognition(CVPR), pp.422–429 (2010).
Sun, K., Xiao, B., Liu, D. and Wang, J.: Deep High-Resolution Representation Learning for Human Pose Estimation, Computer Vision and
Pattern Recognition (CVPR), pp.5693–5703 (2019).
Tang, W., Yu, P. and Wu, Y.: Deeply learned compositional models
for human pose estimation, European Conference on Computer Vision(ECCV), pp.190–206 (2018).
Tompson, J.J., Jain, A., LeCun, Y. and Bregler, C.: Joint training ofa convolutional network and a graphical model for human pose estimation, Advances in Neural Information Processing Systems (NIPS),
pp.1799–1807 (2014).
Toshev, A. and Szegedy, C.: DeepPose: Human pose estimation via
deep neural networks, Conference on Computer Vision and Pattern
Recognition (CVPR), pp.1653–1660 (2014).
Ukita, N. and Uematsu, Y.: Semi- and weakly-supervised humanpose estimation, Computer Vision and Image Understanding, Vol.170,
pp.67–78 (2018).
Walker, J., Marino, K., Gupta, A. and Hebert, M.: The pose knows:
Video forecasting by generating pose futures, International Conference on Computer Vision (ICCV), pp.3332–3341 (2017).
Wei, S.-E., Ramakrishna, V., Kanade, T. and Sheikh, Y.: Convolutional pose machines, Conference on Computer Vision and Pattern
Recognition (CVPR), pp.4724–4732 (2016).
Xiao, B.: An oﬃcial implementation of “Deep High-Resolution Representation Learning for Human Pose Estimation”, available fromhttps://github.com/leoxiaobin/deep-high-resolution-net.pytorch (accessed 2020-03-17).
Xiao, B., Wu, H. and Wei, Y.: Simple baselines for human pose
estimation and tracking, European Conference on Computer Vision(ECCV), pp.466–481 (2018).
Yang, W., Li, S., Ouyang, W., Li, H. and Wang, X.: Learning feature pyramids for human pose estimation, International Conference
on Computer Vision (ICCV), pp.1281–1290 (2017).
Yang, Y. and Ramanan, D.: Articulated pose estimation with flexible
mixtures-of-parts, Conference on Computer Vision and Pattern Recognition (CVPR), pp.1385–1392 (2011).
Yang, Y. and Ramanan, D.: Articulated human detection with flexible
mixtures of parts, IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI), Vol.35, No.12, pp.2878–2890 (2013).
Yeh, R.A., Hu, Y.-T. and Schwing, A.G.: Chirality Nets: Exploiting
Structure in Human Pose Regression, Conference on Advances in Neural Information Processing Systems Workshop (NeurIPSW), pp.1–11(2019).
Yi, Z., Zhang, H., Tan, P. and Gong, M.: DualGAN: Unsupervised
dual learning for image-to-image translation, International Conference on Computer Vision (ICCV), pp.2849–2857 (2017).
Yildirim, I., Belledonne, M., Freiwald, W. and Tenenbaum, J.: Eﬃ-

420

Journal of Information Processing Vol.29 406–423 (May 2021)

 ⎤xn −μθ,dx (sn , an , gn ) 2 ⎥⎥⎥⎥⎥+σ2θ,dx (sn , an , gn ) ⎦
⎤
⎡
Ds
⎢⎢⎢
(sn −μφ,ds (zn ))2 ⎥⎥⎥
1
2
⎥⎥
⎢
−
Eq (z |s ) ⎢log(2πσφ,ds (zn ))+2 d =1 δ n n ⎣σ2φ,ds (zn ) ⎦scient inverse graphics in biological face processing, Science Advances,
Vol.6, No.10, pp.1–18 (2020).
Zhu, J.-Y., Park, T., Isola, P. and Efros, A.A.: Unpaired imageto-image translation using cycle-consistent adversarial networks, International Conference on Computer Vision (ICCV), pp.2223–2232(2017).

[57]

1
+(1+log(σ2β,da (sn , xn ))−μ2β,da (sn , xn )−σ2β,da (sn , xn ))2 d =1
Da

Appendix
Lower Bound L

A.1

a

+

The variational lower bound LX of log p(X) in the unsuper
vised condition (LX = n LXn ) is given by Eq. (A.1). The variational lower bound LX,S of log p(X, S) in the supervised condition(LX,S = n LnX,S ) is given by Eq. (A.2).
LX
n = Eq(sn ,an ,gn ,zn |xn ) [log{pθ (xn |sn , an , gn )pφ (sn |zn )p(an )p(gn )p(zn )}−log{qα (sn |xn )qβ (an |sn , xn )qγ (gn |sn , xn )qδ (zn |sn )}]= Eqα (sn |xn )qβ (an |sn ,xn )qγ (gn |sn ,xn ) [log pθ (xn |sn , an , gn )]+Eqα (sn |xn )qδ (zn |sn ) [log pφ (sn |zn )]−Eqα (sn |xn ) [log qα (sn |xn )]−Eqα (sn |xn ) [KL(qβ (an |sn , xn )||p(an ))]−Eqα (sn |xn ) [KL(qγ (gn |sn , xn )||p(gn ))]−Eqα (sn |xn ) [KL(qδ (zn |sn )||p(zn ))]Dg1

(1+log(σ2γ,dg (sn , xn ))−μ2γ,dg (sn , xn )−σ2γ,dg (sn , xn ))2 d =1g+Dz
1(1+log(σ2δ,dz (sn ))−μ2δ,dz (sn )−σ2δ,dz (sn )).
2 d =1

(A.2)

z

A.2

Image Reconstruction and Prediction

Figures A·1, A·2, A·3 show the reconstruction of human images based on the pose-conditioned image VAE, the reconstruction of 16-joint heatmaps based on the pose VAE, and the prediction of silhouette images from 16 joint heatmaps based on themask estimator φ, respectively. Figure A·4 shows the reconstruction of human images based on the pose-conditioned image VAE
obtained with and without the mask estimator φ.

⎡
Dx
1
⎢⎢
Eq (s |x )q (a |s ,x )q (g |s ,x ) ⎢⎢⎢log(2πσ2θ,dx (sn , an , gn ))=−2 d =1 α n n β n n n γ n n n ⎣x ⎤xn −μθ,dx (sn , an , gn ) 2 ⎥⎥⎥⎥⎥+σ2θ,dx (sn , an , gn ) ⎦
⎤
⎡
Ds
⎢⎢
(sn −μφ,ds (zn ))2 ⎥⎥⎥1⎥⎥−Eqα (sn |xn )qδ (zn |sn ) ⎢⎢⎢⎣log(2πσ2φ,ds (zn ))+2 d =1
σ2φ,ds (zn ) ⎦s1
+(1+log(2πσ2α,ds (xn )))2 d =1
Ds

s

+

Da
1

2 d =1Eqα (sn |xn ) [1+log(σ2β,da (sn , xn ))a−μ2β,da (sn , xn )−σ2β,da (sn , xn )]1
Eq (s |x ) [1+log(σ2γ,dg (sn , xn ))+2 d =1 α n nDgg−μ2γ,dg (sn , xn )−σ2γ,dg (sn , xn )]1
Eqα (sn |xn ) [1+log(σ2δ,dz (sn ))
2 d
Dz

+

z

−μ2δ,dz (sn )−σ2δ,dz (sn )].

(A.1)

LnX,S = Eq(an ,gn ,zn |sn ,xn ) [log{pθ (xn |sn , an , gn )pφ (sn |zn )× p(an )p(gn )p(zn )}−log{qβ (an |sn , xn )qγ (gn |sn , xn )qδ (zn |sn )}]= Eqβ (an |sn ,xn )qγ (gn |sn ,xn ) [log pθ (xn |sn , an , gn )]+Eqδ (zn |sn ) [log pφ (sn |zn )]−KL(qβ (an |sn , xn )||p(an ))−KL(qγ (gn |sn , xn )||p(gn ))−KL(qδ (zn |sn )||p(zn ))

⎡
Dx
1
⎢⎢
Eq (a |s ,x )q (g |s ,x ) ⎢⎢⎢log(2πσ2θ,dx (sn , an , gn ))=−2 d =1 β n n n γ n n n ⎣x

c 2021 Information Processing Society of Japan


421

Journal of Information Processing Vol.29 406–423 (May 2021)

Fig. A·1 Reconstruction of human images based on the pose-conditioned image VAE. The LSP dataset [18],[19] was used for training. A larger amount of annotation images resulted in a better quality of generated images.

Fig. A·2

Reconstruction of 16-joint heatmaps based on thepose VAE. For the purpose of visualization, 16
heatmaps are superimposed to a single heatmap.

c 2021 Information Processing Society of Japan

Fig. A·3 Prediction of silhouette images (foreground mask images) from 16 joint
heatmaps based on the mask estimator φ. The ground truth images aretaken from Ref. [25].

422

Journal of Information Processing Vol.29 406–423 (May 2021)

Yuki Koyama is a Researcher at the National Institute of Advanced Industrial
Science and Technology (AIST), Japan.
He received his Ph.D. from the University of Tokyo in 2017. His main research
field is the intersection of computer graphics and human-computer interaction. In
particular, he is interested in developing
computational techniques for enabling new interactions, producing creative artifacts, and enhancing design processes.

Fig. A·4

Reconstruction of human images based on the pose-conditioned
image VAE. The LSP dataset [18], [19] was used for training. The
ratio of annotated images was 20%.

Takayuki Nakatsuka received a B.E. degree in the Department of Applied Physics
and an M.E. degree in the Department ofPure and Applied Physics from Waseda
University, Japan. He is currently a student in the Graduate Schools of Advanced
Science and Engineering and Graduate
Program for Embodiment Informatics at
Waseda University. His primary research interests are inphysically-based animation, human motion analysis, human computer interaction, and music information retrieval.

Kazuyoshi Yoshii received his M.S. and
Ph.D. degrees in informatics from Kyoto
University, Kyoto, Japan, in 2005 and2008, respectively. He is an Associate
Professor with the Graduate School of Informatics, Kyoto University, and concurrently the Leader of the Sound Scene Understanding Team, Center for AdvancedIntelligence Project (AIP), RIKEN, Tokyo, Japan. His researchinterests include music informatics, audio signal processing, andstatistical machine learning.

c 2021 Information Processing Society of Japan

Satoru Fukayama received his Ph.D.
degree in information science and technology in 2013 from the University ofTokyo. He is currently a senior researcher
at the National Institute of Advanced Industrial Science and Technology (AIST),
Japan. His interests are in music information retrieval, especially music
generation with probabilistic models. He has received awards,
including IPSJ Yamashita SIG Research Award, several Best
Presentation Awards, and Specially Selected Paper Award fromthe Information Processing Society of Japan.

Masataka Goto received his Doctor ofEngineering degree from Waseda University in 1998. He is currently a Prime Senior Researcher at the National Institute ofAdvanced Industrial Science and Technology (AIST), Japan. Over the past 28 years
he has published more than 270 papers
in refereed journals and international conferences and has received 51 awards, including several best paper awards, best presentation awards, the Tenth Japan Academy
Medal, and the Tenth JSPS PRIZE. In 2016, as the Research
Director he began OngaACCEL Project, a 5-year JST-funded research project (ACCEL) on music technologies.

Shigeo Morishima was born on August
1959. He received his B.S., M.S. and
Ph.D. degrees, all in Electrical Engineering from the University of Tokyo, in 1982,
1984, and 1987, respectively. He was avisiting professor of University of Toronto
from 1994 to 1995 and an invited researcher of Advanced Telecommunication
Research institute from 1999 to 2011. Currently, he is a professorof School of Advanced Science and Engineering, Waseda University. He was a General Chair of ACM VRST2018 and VR/ARadviser of SIGGRAPH ASIA 2018. He received many awards
and takes an administration board member of several societies.

423