# Towards Single 2D Image-Level Self-Supervision for 3D Human Pose and Shape Estimation
## 3 Method
### Network Architectures  
Our feature extractor \(f^{\text{Feat}}\) and mesh estimator \(f^{\text{Mesh}}\) were taken from the previous mesh reconstruction method [5], and they were responsible for extracting features and estimating the 3D human meshes. Three networks (jigsaw puzzle solver \(f^{\text{jigsaw}}\), rotation predictor \(f^{\text{Rot}}\), part in-painter \(f^{\text{Inp}}\)) were additionally involved to develop the self-supervision loss solving three different image manipulation tasks, and two additional networks (texture network \(f^{\text{Tex}}\), neural mesh renderer \(f^{\text{NMR}}\)) were involved to develop the self-supervision loss solving the image projection task. Finally, the discriminator \(f^{\text{Disc}}\) was involved to capture the distribution of real 2D images containing human images. More details on individual networks are described in the remainder of this subsection. **Feature extractor \(f^{\text{Feat}}:X\to F\subset\mathbb{R}^{2048\times 1}\). ** Similar to recent human mesh recovery works [5,6] that estimate 3D human mesh model (i.e., SMPL [13]) parameters from the RGB images \(\mathbf{x}\in X\), we involved the ResNet-50 [57] architecture as our feature extractor. It  
Figure 2: A schematic diagram of the proposed image-level self-supervised 3D human pose and shape estimation framework. First, we prepared three types of images \(\mathbf{x}_{\text{jigsaw}}\), \(\mathbf{x}_{\text{hole}}\), and \(\mathbf{x}_{\text{rot}}\) from the original image \(\mathbf{x}\). Second, we pre-trained four networks denoted as green boxes (i.e., \(f^{\text{jigsaw}}\), \(f^{\text{Inp}}\), \(f^{\text{Rot}}\), and \(f^{\text{Tex}}\)) and a network in a purple box (i.e., \(f^{\text{Disc}}\)) using five corresponding losses (i.e., \(L_{\text{jigsaw}}\), \(L_{\text{inp}}\), \(L_{\text{rot}}\), \(L_{\text{RGB}}\), and \(L_{\text{adv}}\), respectively). Two networks \(f^{\text{NMR}}\) and \(f^{\text{rcnn}}\) with pink boxes were fixed after employing the implementation of [22] and initializing weights from [58], respectively. Finally, in the mesh training stage, we trained \(f^{\text{Feat}}\) and \(f^{\text{Mesh}}\) in blue boxes, which were used to infer the 3D human meshes from 2D images. The discriminator \(f^{\text{Disc}}\) in the purple box was also further trained during the mesh training stage. Blue, black, and green arrows denote routes used for testing, training, and supervision signals, respectively. Optional loss \(L_{\text{joints}}\) was not used for the self-supervised setting (i.e., _Ours (self-sup)_); it was used via 2D skeletons for the weakly-supervised setting (i.e., _Ours (weakly-sup)_) and was used via both 2D and 3D skeletons for the semi-supervised setting (i.e., _Ours (semi-sup)_). generates 2048-dimensional feature vectors \(\mathbf{f}\in F\) from the input image \(\mathbf{x}\), whose size is resized to a \(224\times 224\times 3\) dimensional array. **Mesh estimator \(f^{\mathbf{Mesh}}:F\to M\subset\mathbb{R}^{6890\times 13,776}\). ** After extracting the feature vector \(\mathbf{f}\in F\), we mapped it to the corresponding 85-dimensional SMPL [13] parameters \(\mathbf{h}\in H\subset\mathbb{R}^{85\times 1}\) (we used 3, 10, and 72-dimensional vectors as camera, shape, and pose parameters, respectively, in the same manner as in [5]). These parameters were used to differentially generate the corresponding human body meshes \(\mathbf{m}\in M\) with 6890 vertices and 13,776 faces using the SMPL layer (refer to [5,13] for more details). **Jigsaw puzzle solver \(f^{\mathbf{Jigsaw}}:F\times F\times F\times F\to O\subset\mathbb{R}^{C \times 1}\). ** The jigsaw puzzle solver was used to solve the first self-supervision task, called the jigsaw puzzling task. In this task, we divided the input 2D image \(\mathbf{x}\) into \(D\times D\) tiles and permuted them to generate a jigsaw puzzling image \(\mathbf{x}_{\mathrm{jigsaw}}\).