# Towards Single 2D Image-Level Self-Supervision for 3D Human Pose and Shape Estimation
## 3 Method
### Network Architectures
Then, we input \(D^{2}\) tiles combined with the original image into the feature extractor \(f^{\mathrm{Feat}}\) by resizing all images into \(224\times 224\times 3\) dimensional images. The resultant features from five images were concatenated, retaining the permuted orders, and mapped into the vector \(\mathbf{o}\in O\), whose entry denotes the probability for the permutation order of the tiles in the whole images. The class number \(C=(D\times D)!\) is equivalent to the number of permutations for \(D\times D\) tiles. We found that \(D=2\) best encodes the human bodies, balancing it from the background regions (see Figure 1); thus, \(C=(2\times 2)!=24\) was used throughout the experiment. **Rotation predictor \(f^{\mathbf{Rot}}:F\to R\subset\mathbb{R}^{R\times 1}\). ** The rotation predictor was used to solve the second self-supervision task, called the rotation prediction task. In this task, we rotated the input 2D image \(\mathbf{x}\) with \(R\) fixed angles \(\theta\) to generate a new image \(\mathbf{x}_{\mathrm{rot}}\). Humans are in the upright position in images with 0 degrees. We first applied \(f^{\mathrm{Feat}}\) on the input image \(\mathbf{x}_{\mathrm{rot}}\) to obtain the feature vector \(\mathbf{f}\). Then, the rotation predictor \(f^{\mathrm{Rot}}\) was applied on \(\mathbf{f}\) to predict the vector \(\mathbf{r}\in R\), which contained the probability for \(R\) possible angles. In this work, we set \(R=4\) by setting the rotation angles to \(0,90,180,\) and \(270\) for simplicity; however, we were able to achieve a meaningful accuracy improvement using this simple setting. **Part in-painter \(f^{\mathbf{Inp}}:F\to P\subset\mathbb{R}^{P\times P\times 3}\). ** The part in-painter was used to solve the third self-supervision task, called the part in-painting task. In this task, we erased a \(P\times P\times 3\)-sized patch \(\mathbf{p}\) from the center of an input image \(\mathbf{x}\), resulting in \(\mathbf{x}_{\mathrm{hole}}\). Then, the part in-painter \(f^{\mathrm{Inp}}\) was responsible for predicting \(\hat{\mathbf{p}}\) resembling \(\mathbf{p}\). We set \(P=64\) considering the patch size compared to the size of the human bodies in images (Figure 1d describes the example in-painted patches). **Neural mesh renderer \(f^{\mathbf{NMR}}:T\times M\to X\times S\subset\mathbb{R}^{224\times 224 \times 3}\times\mathbb{R}^{224\times 224}\). ** The neural mesh renderer [22] was employed to solve the last self-supervision task, called the image projection task. In this task, 3D meshes were projected to the 2D images by an operation similar to graphics rendering [22]. Either the RGB human images \(\mathbf{x}\in X\) or a binary segmentation mask \(\mathbf{s}\in S\) could be generated from the 3D meshes \(\mathbf{m}\). To render RGB images, an additional texture array \(\mathbf{t}\in T\) was required that described RGB values for faces of the 3D meshes. To infer the texture array, we additionally involved the texture network \(f^{\mathrm{Tex}}\), as explained below. **Texture network \(f^{\mathbf{Tex}}:F\to T\subset\mathbb{R}^{13,776\times 3}\). ** As there are \(13,776\) faces in the SMPL [13] model, the dimension of the texture array \(\mathbf{t}\) is \(13,776\times 3\), as it represents RGB values corresponding to each mesh face. Furthermore, we first inferred the \(13,776\times 3\) dimensional array \(\mathbf{t}\) from \(f^{\mathrm{Tex}}\) and differentially reshaped this towards a \(13,776\times Z\times Z\times 2\times 3\) dimensional array. \(Z=2\) was used as the visual quality based on this minimum dimension, which was sufficient for our purpose. **Discriminator \(f^{\mathbf{Disc}}:\mathbf{x}\rightarrow[0,1]\). ** The discriminator was involved to capture the distribution of the real human images and give gradients to the networks to make the realistic images, in a similar manner to [23].