SlowGuess/NeurIPS_2023_OCR huggingface dataset:
D More Ablations on Backbones and Pre-training Datasets
We additionally conduct experiments on ViTPose [61], the recent state of the art on 2D HPE. The weights of ViTs [13] in ViTPose are initialized with MAE [19] pre-training, and then the model is further trained on 2D HPE datasets. The results are in Table 7, and observations are as follows.
Backbone design outweighs the results on 2D HPE. The first three rows in Table 7 demonstrate that better results on 2D HPE do not necessarily translate to better performance on 3D HPE: Although ViTPose performs best on COCO 2D HPE, it achieves worst results on Human3.6M. We attribute this result to the lack of multi-scale network design. ViTPose gains from powerful MAE pre-training with modern transformer architecture, while its network design is arguably simplified for 2D HPE. Specifically, ViTPose processes on tokenized image patches with transformers and finally increases the resolution of feature maps using 2D deconvolution layers. They use no multi-scale designs, e.g., high-resolution feature branches or multi-scale feature fusion, as in CPN and HRNet. However, such techniques may help our approach learn more task-relevant information (i.e., joint-context features) to localize joints in 3D.
More pre-training datasets on 2D HPE improve the performance on the 3D Task. A comparison between row 3 and row 4 in Table 7 shows that multi-dataset pre-training improves performance on both 2D HPE and 3D HPE. Plus, the gains on 3D HPE (5.8% error reduction) are even more significant than those on 2D HPE (1.3 points improvement on AP). We hypothesize that multi-dataset pre-training improves the generalization ability of learned backbone features. Therefore, the best performance on 2D HPE of ViTPose better transfers to the 3D task.
Figure 9: Visualization of consecutive frames on the Human3.6M test set where severe self-occlusion occurs. Deformable Context Extraction learns sampling points that attempt to discover ground truth joints given false 2D joint detection as reference.
E More Details and Visualization for Deformable Context Extraction
More details. The Deformable Context Extraction module extracts informative joint-centric context features from feature maps using deformable attention [75]. The sampling points of each attention head are initialized in different directions (w.r.t. the reference joint) to promote learning diverse
contextual clues from images. To prevent overly aggressive updates on sampling offsets (e.g., they may go outside the image), the learning rate of linear layers that generate sampling offsets is set to  $6.4\mathrm{e - 4}$  ( $1 / 10$  of that for other layers in the lifting model). We use 4 heads for deformable attention.
Visualization of learned sampling points on consecutive video frames. In Fig. 9, a subject in the Human3.6M test set raises his right arm where severe self-occlusion occurs, and the 2D pose detector fails to localize the right wrist (blue dots are detected results, and green dots are ground truth). We find that most sampling points (gray dots) are concentrated on the upper body of the subject. More interestingly, despite the unreliable 2D joint detection (reference points), some learned sampling points attempt to approach the ground truth. We indicate the sampling points that gain larger attention weights with higher brightness (we decrease the brightness of images for better visual effects). Note that we do not train sampling points using ground truth. This indicates that our adaptive context extraction strategy can learn informative contextual features based on the visual cues of images despite bad sampling references (i.e., false joint detection), which helps reduce uncertainty brought by imperfect 2D pose detectors.
F More Visualization
We provide more qualitative results on Human3.6M (Fig. 10) and MPI-INF-3DHP (Fig. 11). Our single-frame method obtains reliable estimation in hard cases, e.g., self-occlusion, and rare poses, compared to state-of-the-art multi-frame methods such as 351-frame MHFormer [29] and 81-frame P-STMO [50].
Figure 10: Qualitative comparison with MHFormer (351 frames) [29] and our context-agnostic counterpart (please refer to Sec. 4.3 for more details) on Human3.6M. Our method obtains reliable results despite severe self-occlusion, which may cause false 2D joint detection. Notable parts are indicated by arrows.
Figure 11: Qualitative comparison with P-STMO (81 frames) [50] on MPI-INF-3DHP. Our method infers correct results given rare poses (e.g., the subject is lying on the ground and relaxing on the couch). Notable parts are indicated by arrows or circles.