AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints
Authors: Xingzhe He
Bastian Wandt
Helge Rhodin
Venue: NeurIPS 2022 Accept (2022)
Keywords: self-supervised Learning
keypoints
articulated bodies
human skeleton

Abstract: Structured representations such as keypoints are widely used in pose transfer, conditional image generation, animation, and 3D reconstruction. However, their supervised learning requires expensive annotation for each target domain. We propose a self-supervised method that learns to disentangle object structure from the appearance with a graph of 2D keypoints linked by straight edges. Both the keypoint location and their pairwise edge weights are learned, given only a collection of images depicting the same object class. The resulting graph is interpretable, for example, AutoLink recovers the human skeleton topology when applied to images showing people. Our key ingredients are i) an encoder that predicts keypoint locations in an input image, ii) a shared graph as a latent variable that links the same pairs of keypoints in every image, iii) an intermediate edge map that combines the latent graph edge weights and keypoint locations in a soft, differentiable manner, and iv) an inpainting objective on randomly masked images. Although simpler, AutoLink outperforms existing self-supervised methods on the established keypoint and pose estimation benchmarks and paves the way for structure-conditioned generative models on more diverse datasets.  Project website: https://xingzhehe.github.io/autolink/.

AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints
Xingzhe He
Bastian Wandt University of British Columbia {xingzhe, wandt, rhodin)0cs.. cac
Helge Rhodin
Abstract
Structured representations such keypoints are widely used in pose transfer, conditional image generation, animation, and 3D) reconstruction. However, their supervised learning requires expensive annotation for each target domain. We propose a self-supervised method that learns to disentangle object structure from the appearance with a graph of 2D keypoints linked by straight edges. Both the keypoint location and their pairwise edge weights are learned, given only collection of images depicting the same object class. The resulting graph interpretable, for example, AutoLink recovers the human skeleton topology when applied to images showing people. Our key ingredients are i) an encoder that predicts keypoint locations in input image, ii) shared graph that links the same pairs of keypoints in every image, iii) an intermediate edge map that combines the latent graph edge weights and keypoint locations in a soft, differentiable manner, and iv) an inpainting objective on randomly masked images. Although simpler, AutoLink outperforms existing self-supervised methods on the established keypoint and pose estimation benchmarks and paves the way for structure-conditionel generative models more diverse datasets. Project website: ptttps:/xxinghetht ggthhub............. io/ aattlink/
latent variable
Introduction
Object structure representations are widely used modern computer graphics and computer vision techniques, including keypoints for image generation [67
and skeletons for 3D reconstruction However, the structure is usually supervised on large annotated datasets [1 [ B]
68
26
123
33 99.
i1
via hand-crafted parametric models 65. Rzl
X0
5 59 Neither approach generalizes well
to new domains and both require additional manual annotation whenever more detail is needed [8
Our goal is to reconstruct the keypoint locations of an object by learning from an unlabelled image collection, thereby sidestepping the generalization problem. Our key idea is to leverage that the same object shares the same topology by introducing explicit graph that links the same pairs of keypoints all instances. By contrast, existing self-supervised keypoint learning methods model objects as set of independent parts. Their consistency over different instances of the same object is encouraged
by either enforcing parts to follow cand-crafted image transformations [102] 1266 42 66
37
52 or
by adding implicit bias in the network architecture that encodes such spatial equivariances 35 +
Only 4 93] use an explicit skeleton representation. but both require predefined topology, rely on video input, and is trained in CycleGAN setting that still requires manually labeled examples.
We propose simple yet effective method learn both the keypoints and their links without supervision in terms of a sparse graph serving two purposes. First, the graph acts a bottleneck that can only store structural information disentangled from appearance. Second, forms a constraint that associates observations across training images. We enforce the same topology across instances of the same class by learning a shared graph with single set of edge weights. In the absence labels,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

8400 COORRIOA C Fore dee pood
conoitional GAN
(a) detected keypoints and visualized graph representation
(b) applications
Figure 1: Teaser. (a) AutoLink applies to diverse collections of images and automatically yields keypoints linked to graph without ground truth. It recovers animal and human poses and object shapes in settings where previous methods struggle, including cluttered backgrounds, structured stripe textures, articulated fingers, and detailed faces. (b) Example applications are conditional image generation with autoencoders (top) and GANs (bottom) that are driven by the learned keypoints.
we train AutoLink with only an autoencoder reconstruction objective. Since the graph bottleneck should not model appearance, additionally feed the decoder with the input image after masking the majority of its pixels. In turn, is important for the disentanglement that the heavily masked image contains appearance without leaking structural information. This is the case inpainting methods are unable to infer the original image precisely from only sparse pixel colors [12 [24]
unless conditioning structure representations, such as edge maps IT0 75 56 31
Therefore,
by forcing the autoencoder to reconstruct the original image, the detector converges to generate representative structures of images.
We demonstrate on benchmarks that the trained detector has a significantly improved keypoint localization accuracy and 6 additional datasets that it applies to a broader set of images spanning portraits, persons, animals, hands, and flowers, which we attribute to the explicit modeling of links in the graph. Figure shows the diverse set of image domains it applies to, including challenging textures and uncontrolled background, how both skeleton representations as well as object outlines learned by varying the number of keypoints, and exemplifies applications to controlled image generation.
Ethics Risks. The estimated keypoints and edges could be abused for deep fakes as the driving signal for generative models or for unwanted surveillance applications. However, our method works towards improved generality, including objects and animals, and does not improve upon supervised models that already exist in high detail for humans. Benefits. Since our method is entirely self- supervised, it can be applied to a diverse set of persons, objects, animals, situations that have not yet been labeled.
Related Work
Most representation learning methods focus on generic feature vectors for entire images to initialize deep networks for improved object classification [108] introduces explicit object structure. We review the most related approaches in the following.
32 33]
By contrast, our method
Self-supervised Keypoints Detection. The most common idea to discover keypoints in an unsuper- vised manner is rely on the notion that keypoints move as the image changes. Various constraints have been used to enforce that the keypoints follow known transformation. including view changes
in multi-view recording [[O 86 87 or the natural motion in videos 23
95
55
20
53 4

---

When only single images are available, artificial image deformation is applied, either from ran- domized [002 126
42 66 or learned [107 [T6 transformations within pre-defined deformation space. However, learned keypoints may model the background [126, 9 and struggle with large pose variation [77] as image deformations do not separate foreground from background, which are usually tuned for each dataset and bound to be small. Most models leverage multi-branch network architectures to encode the structure and appearance separately and utilize multiple losses that need to be balanced. By contrast, we do not apply artificial transformations, use single branch, and single loss which eases and stabilizes training. To overcome the need for artificial image deformation, He et al. G4 B51 exploit GANs to generate images along with corresponding keypoints and later use them to train a detector. However, this leads to even more complex network architectures and comes with instabilities in GAN training, limiting their applicability to complex objects like human bodies.
Skeleton Representations. Bone maps representing the keypoints connectivity as affinity fields [9] or via explicit offsets [82] are used in supervised human, animal, and object pose estimation. We use similar edge map representation but learn both the location and linking from scratch without annotations. In the weakly-supervised setting, Jakab et al. between image and edge maps. The graph connectivity is predefined to the human skeleton and edges are supervised by a large dataset of unpaired ground truth object edges, which can come from different dataset but are manually annotated. Schmidtke et al. [31 overcome the manual labeling by deforming template skeleton. However, they both require the known connectivity of the keypoints and videos for training while ours learns both the keypoints and connectivity from a collection of single images. Noguchi et al. 78 generate skeleton heuristically by linking the centers of part-wise learned Signed Distance Fields 70). However, they require videos without the background of the same object, and the learned skeleton does not generalize to other objects of the same class.
4 exploit CycleGAN |30 translate
Object Sketch Learning. Sketches are made of strokes drawn by a pen. It is a concise and abstract representation, which can be used in object recognition [221 1T5 and image retrieval
104 122
114.
There are two common sketch representations used in neural models 113
black- and
white raster images [105]]
120
often used for image-to-image translation 20
130
81
48
sequences of points (pen coordinates)
30 90), which is usually used by recurrent generation
models 300
19 28 This graph representation is similar ours. However, instead of
learning to mimic human drawings, ours directly predicts both the keypoints and their connectivity on real natural images.
Structure-enhanced Image Inpainting. When key parts are missing in an image, e.g., eyes on faces arms of humans, it is hard for inpainting networks to imagine the content accurately from scratch. Therefore, additional structural cues are detected to guide the subsequent image generation.
The cues can be supervised segmentation masks [128] 60 31 98
foreground contours [100, and
landmarks [56] 125
117], or automatically extracted edges 75 58 45 112 7 and low-frequency
image components [166 [35. Our reconstruction objective can be seen as such two-stage inpainting, but self-supervised and with the image edges replaced with the learned graph edge representation.
Self-supervised Foreground Segmentation. Traditional methods use color [ contrast [K] and hand-crafted features 44] to cluster foreground pixels. A recent trend is exploiting inpainting techniques to segment the foreground. Chen et al. 100 and Arandjelović et al. 44] use GAN to inpaint the background at the predicted segmentation mask, assuming that the object texture can be changed without changing the data distribution due to the independence of foreground and background. Yang et al. [T8 propose Contextual Information Separation (CIS), a general objective to segment the foreground by maximizing the error of inpainting the mask and its complement. It was first applied to optical flow maps and subsequently to RGB images by 192 I19 5] When the object is small compared to the background, [52] information is required. Different from these previous methods, we utilize a form of inpainting to learn sparse keypoints instead of segmentation. Our learned edges form sparse foreground shape, but further extensions would be necessary to transfer from the edge maps to the cunndar--aligned foreground segmentation.
additional object detection module [51 [
or multi-
3 Method
We leverage that the objects in the dataset share the same topology and can be represented as graph that connects keypoints by shared set of edges. To Iearn the keypoints and edges, we design an
3

---

draw odges
Detector
Decoder
image
keypoints
edge map
reconstrucied image
masking
learned graph edge weights shared by all Instances same dataset
single-Image reconstruction
masked image
Figure 2: Overview. Given an image, we detect keypoints and draw differentiable edges between keypoints according to the learned graph edge weights that is visualized as color matrix. The method is self-supervised in that the latent edge map and keypoints are learned by reconstructing the masked input images. Note that keypoints are image specific and edge maps are shared.
autoencoder that aims to accurately reconstruct the input image, with the graph as the intermediate representation. encode the input image into graph, we detect the keypoints and create an edge heatmap based on learnable edge weights. To mostly obtain appearance information, the majority of the image, which randomizes the structure information and reduces the remainder to a very low level. The edge heatmap is combined with the masked image to reconstruct the original image. Since the missing structure is important to reconstructing the original image, the network is forced learn the structure of the object in self-supervised manner. Figure 2 shows overview of our method.
mask out
Formally, given an image I E H with height H and width W we aim to learn a set of keypoints {k, 1K where k; E [-1,1] [] I2 is normalized keypoint coordinate, and K is the number of keypoints. We use ResNet with upsampling [109] to detect keypoints. Afterward, we draw a differentiable edge 73 between each pair keypoints (details below). The
edge map S E " is concatenated along the channel dimension with the randomly masked image Im E xx x3 and fed into a UNet 39 to obtain the reconstructed image I'. The detailed
network architectures can be found in Appendix D
3.1 Image Structure Representation
In this section, we introduce the generation of keypoints and the edge map from image. Let H E H xk be the K heatmaps generated by a ResNet with upsampling [109] from the image I. The keypoint k; is calculated by the differentiable soft-argmax function,
(p)) (x H(p))
k; Σ
where p E [-1,1] [-1, the normalized pixel coordinates.
Given two keypoints k, kj. we draw a differentiable edge map Sij. where values are 1 on the edge linked by the two keypoints and decrease exponentially based on the distance to the line. Formally, the edge map Sij is a Gaussian extended along the line [73], defined as
(p) = exp (p)/oo)
(2)
where a is a hyperparameter controlling the thickness of the edge, and d:y (p) is the L2 distance between the pixel p and the edge drawn by keypoints k; and k,.
pp kill2
if < 0,
where t= (p - k;) (k; - ki)
(p)(p) = ||p (k, + 1l,)
if < 1,
(3)
k; k2 11
|pp k,112
1
We assign weight wy > each edge, which is enforced to positive by SoftPlus 22 This weight is learned during training and shared across all object instances in dataset. Finally, take

---

the maximum at each pixel of the heatmaps to obtain the final edge map S E R
S()) max w/y (p)).
(4)
Taking the maximum at each pixel avoids the entanglement of the edge weights and the convolution kernel weights, which is further explained Scction 4.4
3.2 Image Reconstruction
The masked image Im is generated by first uniformly dividing the image I into a 16 x 16 grid, and randomly masking out 80% of the grid cells, similar to [33. We concatenate the masked image with the edge map and feed it into UNet decoder [8 to reconstruct the original image,
I' Decoder(aII + +SS)
(5)
where means concatenation along the channel dimension and a learnable parameter that compensates for the change of the edge weight magnitude during training. a is initialized to 1. We found this parameter to be helpful in training stability. Different to [3 we condition on an edge map. Different to [ we have no ground truth for the edge map. Our edge map is an unobserved latent variable. Thus we only minimize the difference of the original image and the reconstructed image by the perceptual loss [46].
Σ [
-
where N is the number of examples and I is the feature extractor. The perceptual loss is believed to measure the structure similarity 46 27 and leads more robust training 42 3
3.3 Implementation Details
We use the Adam optimizer [54] with a learning rate of 10-4 with 31 = 0.9, 3, = 0.99. The batch size is 64. We train for 20k iterations. It takes 3 hours to train on a single V100 GPU, All images are resized to 128 x 128. The learning rate for the edge weights is multiplied by 512 due to the small gradient of SoftPlus [22] when the value is close O. To show the robustness of our model, report all experiments on the sampling strategy of masking 80% of the 16 x 16 patches. We perform experiments with the same edge thickness of x² 5e for all benchmark datasets. We train 10 times and report the mean and the standard deviation of the evaluation metrics. Although it already outperforms other work in most experiments,
also tune thicknesses each individual dataset, as others did for their hyperparameters, which further improves the results. The tuned thicknesses can be found in Appendix B The only other hy hyperaaraeeter is the number of keypoints, which we to that of the established benchmarks for quantitative comparisons, ranging from 4 to 32 points.
Experiments
In this section, compare results the related methods, showing that model is simple yet ef- fective. Besides, we perform a number of ablation studies on hyperparameters and algorithm variants, exhibiting the robustness of our model and justifying the necessity of every model component.
4.1 Datasets and Evaluation Metrics
CelebA-aligned
[ contains 200k celebrity faces aligned in center. We follow [122] splitting it into
three subsets: CclebA training set without MAFL (160k images), MAFL training set (19k), MAFL test set (Ik). We train our network on the CelebA training set without MAFL. To quantitatively evaluate the consistency of our predicted keypoints, without bias from our detected keypoints to the ground truth keypoints on the MAFL training set and reporting the mean Lg error normalized by inter-ocular distance on the MAFL test set.
follow [022 training a linear regression
Celeb -in-the-wild [x contains celebrity faces unconstrained conditions. We follow 3i7 and first split it into three subsets as for CelebA-aligned, and then remove the images where a face covers less than 30% of the area, which results 45,609 images for model training, 5,379 with keypoint labels for regression, and 283 for testing. The evaluation metric is the same CelebA-aligneed
5

---

Oun
U2m Lodgen)
Figure 3: Qualitative comparison on detected keypoints. Our model is more robust on wild face
poses, and depicts more details human bodies compared to 34 and 35 poses in the middle four images are clearly detected.
For example, the feet
Table 1: Landmark detection on CelebA. The metric is the landmark regression (without bias)
error terms of T-2 distance normalized by inter-ocular distance (lower is better). While all methods
perform well on aligned CelebA, ours is more robust on Wild CelebA. The sign means being reported by and means being reported by [62
Method
Type
Aligned (K=10)
(K=4)
Wild (K=8)
DDF 17
Part Segmentation
31.30%
SCOPS 37
(w/o saliency)
Part Segmentation
46.62%
22.11%
SCOPS
77
(w/ salicney)
Part Segmentation
21.76%
15.01 %
Liu et al.
32
Part Segmentation
15.39%
12.26%
Huang al.
(w/ detailed label)
Segmentation
8.40%
GANSeg 35
Segmentation
3.98%
12.26%
6.18%
Thewlis
1002]
Landmark
7.95%
31.30%
7hang al.
26
I Landmark
3,46%
40.82%
LatentKeyponnGAAN
A4
Landmark
5.85%
25.81%
21.90%
Lorenz ct al.
66
Landmark
3.24%
15.49%
11.41%
IMM
42
Landmark
3.19%
19.42%
8.74% 1
LatentKeypoinGGAN-uu
Landmark
3.31%
12.10%
5,63%
Ours (general)
Landmark
3.92#t0.69%
7.72+0.47%
5.66=0.29%
Ours (thickness-tuned)
Landmark
3.54%
6.11%
5.24%
Human3.6m 39 contains human activity videos in static backgrounds. We follow [16 considering six activities (direction, discussion, posing, waiting, greeting, walking), and using subjects 1, 6, 7, 8, for training and 11 for testing. This results in 796,648 images for training and 87,975 images for testing. The evaluation metric is the regressed (without bias) mean L2 error normalized by the image size. We remove the background as in 126 66 to make fair comparison to others. underline the robustness against structured backgrounds we also report the numbers including background.
DeepFashion 4 contains 53k in-shop clothes images. We follow only keeping the full-body images. We use 10604 images for training and 1179 images for testing as in [6. We use the keypoints generated by AlphaPose 255 as the ground truth. The evaluation metric Percentage of Correct Keypoints of d=6 pixels in resolution 256 x 256.
Taichi 96 contains 3049 training videos and 285 test videos of people performing Tai-Chi, with the various appearance of foreground and background. We follow [97 using 5000 and 300 images (not contained in training data) for training linear regression and for testing, respectively. The evaluation metric, mean average error (MAE), is calculated as the sum of the L2 error in resolution 256 x 256.
CUB-200-2011 [03 consists of 11,788 images of birds. We follow two established protocols [66| [5] to evaluate our method: 1) Images are cropped based on the bird landmarks, aligned to face the left [66], and seabirds are removed; Birds cropped based the given bounding box and the train/val/test split of [15] is used. In both cases, the evaluation metric is the regressed (without bias) mean L2 error normalized by the cropped image size.
Flower 77., 11k Hands [T. Horses [30,, and Zebras [1:0] are used for qualitative experiments.
VoxCeleb2 [16] and AFHQ [4] are used for pose transfer and conditional image generation,

---

Table 2: Landmark detection on Human Body. Our model outperforms all the other unsupervised baselines. The metric for each dataset follows the corresponding description in the text. The sign means being reported by [97] and the sign means being reported by [6] The number of keypoints K 16 for Human3. 6m and DeepFashion and K 10 for Taichi.
Method
Supervision
Human3.6m
DeepFashion
Taichi
Jakah et
153
Video & unpaired ground truth
2.73
Newell et al.
paired ground truth
2.16
DFF 112
lesting dataset
494.48
SCOPS 37
saliency maps
411.38
Siarohin et al
97
videus
389.78
Zhang et
127
videns
343.67
Zhang et
12
videns
4.14
Schmidtke et
93
video & T-pose template videos
3.31
Sun et al. 1010
2.53++0.06
Thewlis al.
1022]
unsupervised
7.51
Zhang [R0
unsupervised
4.91
LatentKeypointGAAN
3
unsupervised
49%
437.69
Lorenz ct al.
36
unsupervised
2.79
57%
GANSeg
unsupervised
417.17
Ours (general)
unsupervised
2..07
65.11.2%
337.50:::25,00
Ours (thickness-tuned)
unsupervised
2.76
66%
316.10
Table 3: Landmark detection on CUB Birds. Our model outperforms most other baselines and achieves comparable results with the ones using ground truth segmentation masks. The metric is the landmark regression (without bias) error of L2 distance normalized by the image size (lower better). A star means being reported by [5]. means being reported by [77, and & means tested by us with their official code; all other numbers are taken from the respective papers. The number of keypoints is K 10 for CUB-aligned and K 4 for CUB-001, CUB CUB-002, CUB-003, and CUB-all.
Method
Supervision
CUB-alignod
CUB-001
CUB-002
CUB-003
CUB-alI
SCOPS thi
GT silhomette
18.3
17.7*
1
12.6
Choulhury at 15
GT' silhouette
11.3
15.0
10.6
9.2
DFF
testing
22.41
21.67
22.01
SCOPS 37
saliency maps
18.5
18.8
21.1
et 16
unsupervised
ULD [P5 [02]]
unsupervised
30.1f
29.47
28.2f
Zhang al. 126
unsupervised
5.36
26.91
27.6E
27.14
22.41
LatentKeypointGAN 2
unsupervised
5.211
22.61
29.12
21.2]
14.7]
GANSeg [5]
unsupervised
3.23
22.11
22.35
21.51
12.11
Ours (general)
unsupervised
4.15 1: 0.24
20.6 1. 0.54
20.3 0,96
19.7 0.91
11.6 1 0.33
Ours (thickness tuned)
unsupervised 3.51
20.2
19.2
18.5
11.3
respectively. Horses and Zebras are extracted from the CycleGAN dataset 130 by removing the images with multiple horses and aligning them to face left. Note that the horses and zebra are trained separately, yet the model learns similar structures. The train/test split of Flower follows [C] All the other datasets follow the train/test split specified by the dataset.
4.2 Qualitative Analysis
We qualitatively compare our detected keypoints with other methods and show the examples of the learned edges F For visualization purposes, we scale the edge weights obtain visible edges. We use the same number of keypoints as the previous method 35 for fair comparison, which are for CelebA-in-the-Wiin on the choice of the number of keypoints in Ablation Study 4 As shown in Figure our model not only detects consistent keypoints but also learns reasonable edges, such as human skeletons in DeepFashion and Taichi. For example, the feet are clearly connected with the corresponding knees, and there is no edge between the left and right hands. We show 105 images with detected keypoints and visualized graph structure for each dataset in Figure 2 in the Appendix, demonstrating that our model works on various classes of objects of diverse appearance and complex backgrounds.
for DeepFashion, and for Taichi. We will discuss more
4.3 Quantitative Analysis
We compare the keypoint detection results with other methods in Table I (CelebA), Table (Hu- man3.6, DeepFashion, Taichi), and Table (CUB). Our simple model outperforms all other unsuper-
7

---

CelebA
Human2
DeepRashion
Ticci
50
Number Kerpoints
5.0
70
K-d
83
Gam of Sime
30
4.0
-00
32
55
1.0
50
2720
2732 Thickness
2
2
Thidkness
Thicknese
Ihiconess
Figure 4: Ablation tests on the number of keypoints and edge thickness. While the model shows better performance with more keypoints, is robust to the edge thickness.
Celebv in The Wil
Humann3.0m 6m
DeepPaston
Thicht
Normakoa La Sirer
Fabch Resolution
axd
13 Sir
60
Cimal Erer
600
NoX @i fc
30
16x26
500
25
50
40
Masking
Masking
Figure 5: Ablation tests masking strategy. Overall, the performance increases the mask ratio increases. Too small or too large patch sizes can decrease the performance. Empirically, masking 80% of the 16 16 patches is golden rule.
vised methods all benchmarks, except [37 on three CUB subsets, which however requires saliency maps, and for the most constrained setting CelebA-aligned and CUB-aligned where all methods perform well and the results are comparable. The results on CelebA in Table [ confirms that our model is more robust to poses in the wild. Since self-supervised part segmentation methods usually more robust on wild faces [62 37 35]], we also include them for comparison, demonstrating the robustness of our model over existing baselines. The results Human3. 6m and DeepFashion in Table 2 show the capability of our model to detect keypoints on human bodies of either similar or diverse appearances. The result Taichi in Table demonstaetss the general applicability of our model to human bodies of diverse poses in complex backgrounds.
4.4 Ablation Tests
In this section, analyze the hyperparameters and demonstrate the robustness of our model. We also discuss the possible variants of our model and show the superiority of our design.
Number of Keypoints & Thickness. We show ablation test results the different numbers of
keypoints and edge thicknesses in Figure 4 The exact numbers can be found in Table Appendix B Our model shows very strong robustness to edge thickness. The accuracy remains state-of-the- while the thickness of the edges varies by multiple orders of magnitude. On the other hand, with the increasing number of keypoints the accuracy increases. This expected since more keypoints able to capture structure in more detail, as shown in Figure g Yet, some other methods fail for a large number of keypoints 35
Masking Strategy. our standard setting, the image is divided into 16 16 patches and 80% of the patches are randomly masked. We investigate how the patch size and masking ratio affect the model performance. Figure shows that a too low masking ratio enables the network to directly infer the structure from the masked image which is undesired in case. In these cases, the network would not choose to infer a set of compact keypoints from the original image. Figure Jillustaaess that the patch size cannot be too small (4 x 4) or too large (64 x 64). Although in some cases, such as masking strategy gives better results (4. vs 5.24), we choose to report unified strategy that mask 80% of 16 x 16 patches for simplicity and conciseness.
CelebA-in- in-tthhe---ee--- -hel- a different
Variants of Edge Heatmap Generation. Besides the heatmap generation described in Method 3 we test four more ways to generate the edge maps: 1) we define the thickness as a globally learnable parameter; 2) we learn each edge thickness independently as a parameter; 3) we treat each edge heatmap as independent channel of the feature map, instead of making them single channel
8

---

K=4
K=8
K=16
K=32
K=4
K=8
K=16
K=32
n
nn
Un
Un
n
Figure 6: Examples of different numbers of detected keypoints. With very few keypoints, the model only models very basic shape, such as the box on the face, and sometimes cannot fully capture the structure. For example, with K=4, only legs are modeled on humans. With abundant keypoints, is able model details.
Table Ablation tests on variants of edge heatmap generation. The original design is proved to be the most robust one. Although in some cases it is not optimal, the difference is almost trivial.
Model
CclebA The Wild
Human3.6m
DecpFashion f
Taichi
original model
5.24%
2.76%
65.8%
316
a
6.39%
2.87%
66.0%
374
shared learnable thickness
6.12%
3.25%
49.1%
425
independent learnable thickness
5.94%
3.73%
50.2%
311
edge-specific heatmap
5.65%
3.83%
65.1%
407
only using keypoints without edges
6.55%
3.58%
52.9%
722
heatmap with Equation 4 4) instead of using the edge heatmap. generate Gaussian heatmaps for the keypoints and use Equation 4 to combine them single channel heatmap. The results are listed in Table 4 Overall, these variants have worse performance. Although in some cases, the model has slightly better results on specific datasets, the performance boost does not hold general. We observe that with only keypoints without edges, the model may degenerate, shown in Figure 7a. Interestingly, assigning each edge a different channel performs worse than simply combining all edges into single channel. We believe is caused by entangling the edge weights with the convolution kernel weights. As visualized in Figure Zb, there exist dummy edges that do not model the object structure. In addition, tried remove the learnable in Equation 5 fixing 1, but the overall performance decreases as shown Table
Does Texture Matter? We trained two networks on horses and zebras separately. As shown in Figure h. the horse and the zebra share similar shape structures but only one is textured. The striped texture not having a significant impact the learned structure shows that our model primarily learns the structure instead texture features.
What if the model is trained on images with structured background? We tested on Human3.66 6m with a background, where all images are taken in a single room. The error 5.02. As shown in Figure E our model captures the entrance in the background. It is expected since we assume the foreground object is structured. If we apply spectral clustering 776) on the learned graph, the keypoints clearly divided into two clusters, one for the room and one for the person.

---

(8) w/o edges
(b) w/o shared edge channel (c) effect structured BG
(d) failure cases
Figure 7: (a) If we do not model the edges, the model may degenerate. (b) we give different edges different channel in the feature map. the model would give dummy edges. (c) If the model trained on the dataset with structured background, the background would be modeled. However, the keypoints can be separated into sets by spectral clustering. (d) failure case: left) the model cannot model the occlusion well; right) the model has left and right ambiguity.
Limitations and Future Work
If the background is highly structured, the keypoints will appear the background. Yet, we showed an avenue for future work, as already a simple graph clustering could separate the object from the background (66 97 (35), our model cannot model occlusion well. We show in Figure Z left that the occluded right arm becomes the back when the person turns to the left. In addition, for all other methods, the model cannot distinguish the left and right sides the objects as shown in Figure middle and right. We believe it is necessary model the structure in 3D solve these problems.
the Human3.61 3.6M dataset. Similar to the previous 2D self-supervised methods
6 Conclusion
presented simple approach for learning a spatial graph representation from unlabelled image collections by reconstructing masked images. The crucial part is our learnable graph design that models the relationship between different keypoints. simpler than existing alternatives and opens up path for image understanding, image editing, and learning 3D models from 2D images.
Acknowledgement
This work was supported by the Compute Canada GPU servers, and Huawei-UBC Joint Lab project.
References
[1] M. Afifi. 11k hands: gender recognition and biometric identification using large dataset of hand images. Multimedia Tools and Applications. 2019. doi: 10.1007/s1 1042-019-7422 URL httth::////d //doi. org/ 10. 1007/ s0000// 010022--99---4--400000 -
[2] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2d human pose estimation: New
benchmark and state the art analysis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
[3] M. Andriluka, U. Iqbal, E. Insafutdinov. L. Pishchulin, A. Milan, J. Gall, and S. B. PoscTrack: A benchmark for human pose estimation and tracking. CVPR, 2018.
R. Arandjelović and A. Zisserman. Object discovery with copy-pasting gan. arXiv preprint arXiv:1905.. 1369, 2019.
[5] V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187-194, 1999.
[6] Braun. umsuppervieed--iiennangggggg ttsnerrieetlleennnniiigigigtg/ge http://///ggthiioooooaaaaaaaaaaa 2020.
10

---

[7] C. Cao and Y. Fu. Learning sketch tensor space for image inpainting of man man-made scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14509-
14518, 2021.
[8] N. Cao, Yan, Y. Shi, and Chen. Ai-sketcher: A deep generative model for producing high-quality sketches. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 2564-2571, Jul, 2019, doi: 10. 1609/aiaaiii index. phppppppppppp / pbtpAAAt/aattccee/vvee wwwaiceeeiiee//11e...
URL, httpttt/////ooft http:///ss.aaaiaa.....a.....aaaa aaai org/
[9] Z. Cao, T. Simon, S.-F. Wei, and Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7291-7299 2017.
[10] M. Chen, T. Artières, and I.. Denoyer. Unsupervised object segmentation by redrawing. In Advances Neural Information Processing Systems 32 (NIPS 20/9), pages 12705-12716. 2019.
[11] T. Chen, Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[12] Y. Chen, S. Tu, Y. Yi, and L. Xu. Sketch-pix2seq: a model to generate sketches of multiple categories. arXiv preprint 17XXv:1709.0422..1 2017.
[13] M.-M. Cheng, G.-X. Zhang, J. Mitra, X. Huang, and S.-M. Hu. Global contrast based salient region detection. In CVPR 2011, pages 409 416, 2011. doi: 10.1 10.1109/CVPR2201 1.1999945444.
[14] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 81 3188-8197, 2020.
[15] Choudhury, I. Laina, C. Rupprecht, and A. Vedaldi. Unsupervised part discovery from contrastive reconstruction. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL httpps://openreviex het/fforummid=iHQQPPP
[16] J. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. In INTERSPEECH, 2018.
[17] E. Collins, R. Achanta, and S. Susstrunk. Deep feature factorization for concept discovery. In Proceedings of the European Conference on Computer Vision (ECCV), pages 336-352, 2018.
[18] E. Crawford and Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3412-3420, 2019.
[19] A. Das, Y. Yang, T. Hospedales, T. Xiang, and Y.-Z. Song. Béziersketch: A generative model for scalable vector sketches. In European Conference on Computer Vision, pages 632-644 647. Springer, 2020.
[20] X. Dong, S.-I. Yu, X. Weng. S.-E. Wei, Y. Yang, and Y. Sheikh. Supervision-by-reee byrrgisttaatiin An unsupervised approach to improve the precision of facial landmark detectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 360-368, 2018.
[21] A. Dosovitskiy and T. Brox. Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems, 29, 2016.
[22] C. Dugas, Y. Bengio, F. Bélisle, C. Nadeau, and R. Garcia. Incorporating second- order functional knowledge for better option pricing. T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neurul Information Processing Systems, volume 13. MIT Press, 2000. URL
https/eeriisseeepeee/////0..... cc/ppaeer/200/ffiee/ 1 Paper pdf

---

[23] A. Dundar, K. J. Shih, A. Garg, R. Pottorf, A. Tao, and B. Catanzaro, Unsupervised disen- tanglement of pose, appearance and background from images and videos. arXiv preprint arXiv:2001.0951 2020.
[24] M. Eitz, Hays, and M. Alexa. How do humans sketch objects? ACM Trans. Graph, (Proc. 2012.
314))44
[25] H.-S. Fang, S. Xie, Y.WW. Tai, and C. Lu, RMPE: Regional multi-person pose estimation, In ICCV, 2017.
[26] Feng, Wu, X. Shao, Wang, and X. Zhou. Joint 3d face reconstruction and dense alignment with position map regression network. In Proceedings of the European conference on computer vision (ECCV), pages 534-551, 2018.
[27] I. A. Gatys, A. S. Feker, and M. Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414-2423, 2016.
[28] S. Ge, V. Goswami,
L. Zitncck and D. Parikh. Creative sketch generation. In International Conference Learning Representations, 2021. URL htpp:///ppeeerreeeee wttpe/eviiieeeweefrrreeeeeee net/forum?
id=gwnoVHIES05
[29] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2680, 2014.
[30] D. Ha and D. Eck. A neural representation of sketch drawings. In International Conference on Learning Representations, 2018. htw.ll/pss///ppaooooe.eeeee/r/ep
[31] X. Han, Z. Wu, W. Huang, M. R. Scott, and L.. S. Davis. Finet: Compatible and diverse fashion image inpainting. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
[32] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[33] K. He, X. Chen, S. Xie, Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
[34] X. He, B. Wandt, and H. Rhodin. Latentkeypoinggan: Controlling gans via latent keypoints. arXiv preprint S..iv:2103.188121 2021.
[35] X. He, B. Wandt, and H. Rhodin. Ganseg: Learning to segment by unsupervised hierarchical image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1225-1235, 2022.
[36] Z. Huang and Y. Li. Interpretable and accurate fine-grained recognition via region grouping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8662 8672, 2020.
[37] W.-C. Hung, V. Jampani, S. Liu, P. Molchanov, M.-H. Yang, and J. Kautz. Scops: Self- supervised co-part segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 869-8788 2019.
[38] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 2015.
[39] C. Ionescu, D. Papava, Olaru, and C. Sminchisescu. Human3.66 6m:: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325- 2014. doi: 10. 109/TPAMM..
2013.248.
12

---

[40] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to to to-inage translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.
[41] A. S. Jackson, A. Bulat, V. Argyriou, and G. Tzimiropoulos. Large pose 3d face reconstruc- tion from a single image via direct volumetric enn regression. In Proceedings of the IEEE international conference on computer vision, pages 1031-1039, 2017.
[42] T. Jakab, A. Gupta, H. Bilen, and A. Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Advances in neural information processing systems, pages 4016-4027, 2018.
[43] T. Jakab, A. Gupta, H. Bilen, and A. Vedaldi. Self-supervised learning of interpretable keypoints from unlabelled videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8787-8797, 2020.
[44] H. Jiang. J. Wang, Z. Yuan, Y. Wu, N. Zheng, and S. Li. Salient object detection: A discrimi- native regional feature integration approach. In 2013 IEEE Conference on Computer Vision and Patlern Recognition, pages 2083-2090, 2013. doi: 10.1 1091///R..001..22
[45] Y. S. Jie Yang, Zhiquan Qi. Learning to incorporate structure knowledge for image inpainting. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12605-12612, 2020.
[46] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In European conference on computer vision, pages 694-711. Springer, 2016.
[47] K. Kaiyrbekov and M. Sezgin. Deep stroke-based sketched symbol reconstruction and seg- mentation. IEEE computer graphics and applications, 40(1):112-122)) 2019.
[48] M. Kampelmuhler and A. Pinz. Synthesizing human-like sketches from natural images using a conditional convolutional decoder. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages
2020.
[49] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[50] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 81 10-8119, 2020.
[51] I. Katircioglu, H. Rhodin, V. Constantin, J. Sporri, M. Salzmann, and P. Fua. Self-supervised human detection and segmentation via background inpainting. IEEE Transactions on Pattern Analysis & Machine Intelligence, 01:1-1, 2021.
[52] I. Katircioglu, H. Rhodin, J. Sporri, M. Salzmann, and P. Fua. Human detection and segmenta- tion via multi-view consensus. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 2855-2864, 2021.
[53] Y. Kim, S. Nam, I. Cho, and S. J. Kim. Unsupervised keypoint learning for guiding class- conditional video prediction. In Advances in Neural Information Processing Systems, pages 38 1814-3824, 2019.
[54] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[55] T. D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning of object keypoints for perception and control. In Advances in neural information processing systems, pages (0724-10734 2019.
[56] A. Lahiri, A. K. Jain, S. Agrawal, P. Mitra, and P. K. Biswas. Prior guided gan based semantic inpainting. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
13

---

[57] Y. LeCun, B. Boser. J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied handwritten zip code recognition. Neural Computation, 1(4): 541-551, 1989. doi: 10.1 162/neco.. 198..19891.4444 1.4.541.
[58] J. Li, F. He, L. Zhang, B. Du, and D. Tao. Progressive reconstruction of visual structure for image inpainting. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[59] T. Li, T. Bolkart, M. J. Black, H. Li, and J. Romero. Learning model of facial shape and expression from 4D scans. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6): 194:1-194:17, 2017.
[60] L. Liao, J. Xiao, Z. Wang, C.-W. Lin, and S. Satoh. Guidance and evaluation: Semantic-aware image inpainting for mixed scenes. In European Conference on Computer Vision, pages 683-700. Springer, 2020.
[61] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[62] S. Liu, L. Zhang. X. Yang, H. Su, and Zhu. Unsupervised part segmentation through disentangling appearance and shape. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8355-8364, 2021.
[63] Z. Liu, Luo, X. Wang, and X. Tang. Deep learning face attributes the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[64] Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[65] M. Loper, N. Mahmood, J. Romero, G. Mols,Moll, and M. J. Black. SMPL: A skinned multi- person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), (-6)):24::122482 Oct. 2015.
[66] D. Lorenz, L. Bereska, T. Milbich, and B. Ommer. Unsupervised part-based disentangling of object shape and appearance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10955-10964, 2019.
[67] I. Ma, X. Jia, Q. Sun, B. Schiele, T. Tuytelaars, and L.. Van Gool, Pose guided person image generation. Advances in neural information processing systems, 30, 2017.
[68] L. Ma, Q. Sun, S. Georgoulis, L. Van Gool, B. Schiele, and M. Fritz. Disentangled person image generation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 99-108, 2018.
[69] A. L. Maas, A. Y. Hannun, and A. Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In in ICML Workshop on Deep Learning for Audio, Speech and Language Processing. Citeseer, 2013.
[70] R. Malladi, J. Sethian, and B. Vemuri. Shape modeling with front propagation: a level set approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1782)1158-175. 1995. doi: 10.1 199/34..6..
[71] D. Mehta, H. Rhodin, D. Casas, Fua, O. Sotnychenko, W. Xu, and C. Theobalt. Monocular 3d human pose estimation in the wild using improved enn supervision. In 3D Vision (3DV), 2017 Fifth International Conference on. IEEE, 2017. doi: 10. 11110//d..221..006655 URL
Sttp://gvv.mpil mpi-inf mpg. de/ hahp/dhp_daaee dataset
[72] L. Mescheder, A. Geiger, and S. Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 3481-3490. PMLR, 2018.
[73] Mihai and Hare, Differentiable drawing and sketching. arXiv preprint arXiv:2103. 169914 2021.
14

---

[74] M. Minderer, C. Sun, R. Villegas. F. Cole, K. P. Murphy, and H. Lee. Unsupervised learning of object structure and dynamics from videos. In Advances in Neural Information Processing Systems, pages 92-102, 2019.
[75] K. Nazeri, E. Ng. T. Joseph, F. Qureshi, and M. Ebrahimi. Edgeconnect: image inpainting using edge prediction. In The IEEE International Conference on Computer Vision (ICCV) Workshops, Oct 2019.
Structure guided
[76] A. Y. Ng. M. Jordan, and Y. Weiss. On spectral clustering: Analysis and an algorithm. In Proceedings of the 14th International Conference Neural Information Processing Systems: Natural and Synthetic, NIPS 01, page $49-856, Cambridge, MA, USA, 2001. MIT Press.
[77] M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing. Dec 2008.
[78] A. Noguchi, U. Iqbal, J. Tremblay, T. Harada, and O. Gallo. Watch it move: Unsupervised discovery of 3D joints for re- re-posing of articulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022.
[79] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: wrrXiv:880.... 0074433788 2018
[80] A. A. Osman, T. Bolkart, and M. J. Black. Star: Sparse trained articulated human body regressor. In European Conference on Computer Vision, pages 598-613, Springer, 2020.
[81] K. Pang. Li, J. Song, Y.-Z. Song. T. Xiang, and T. M. Hospedales. Deep factorised inverse- sketching. In Proceedings of the European Conference Computer Vision (ECCV), pages 36-52, 2018.
[82] G. Papandreou, T. Zhu, L.-C. Chen, S. Gidaris, J. Tompson, and K. Murphy. Personlab: Person pose estimation and instance segmentation with a bottom-up. part-based, geometric embedding model. In Proceedings of the European conference on computer vision (ECCV), pages 269-286, 2018
[83] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10975-10985 2019.
[84] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975-10985, 2019.
[85] Y. Ren, X. Yu, R. Zhang, T. Li, S. Liu, and G. Li. Structureflow: Image inpainting via structure-awarr appearance flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 181-190, 2019.
[86] H. Rhodin, M. Salzmann, and P. Fua. Unsupervised geometry-aware representation for 3d human pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 750-767, 2018
[87] H. Rhodin, V. Constantin. I. Katircioglu, M. Salzmann, and P. Fua. Neural scene decomposition for mulli-person motion capture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[88] I. K. Riza Alp Güler, Natalia Neverova. Densepose: Dense human pose estimation in the wild. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[89] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
[90] P. Sangkloy, N. Burnell, C. Ham, and Hays. The sketchy database: Learning to retrieve badly drawn bunnies. ACM Transactions Graphics (proceedings of SIGGRAPH), 2016.
15

---

[91] K. Sasaki, S. lizuka, E. Serra, and H. Ishikawa. Learning to Restore Deteriorated Line Drawing. The Visual Computer (Proc. of Computer Graphics International 2018), 34(6-8): 1077-1085, 2018.
[92] P. Savarese, S. S. Kim, M. Maire, G. Shakhnarovich, and D. McAllester. Information-theorer segmentation by inpainting error maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[93] L. Schmidtke, Vlontzos, Ellershaw, A. Lukens, Arichi, and Kainz. Unsupervised human pose estimation through transforming shape templates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2484 2494, 2021.
[94] A. Siarohin, E. Sangineto, S. Lathuiliere, and N. Sebe. Deformable gans for pose-based human image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3408 3416, 2018.
[95] A. Siarohin, S. Lathuilière, S. Tulyakov, E. Ricci, and N. Sebe. Animating arbitrary objects via deep motion transfer. In Proceedings of the IEEE Conference on Compuler Vision and Pattern Recognition, pages 2377-2388 2019.
[96] A. Siarohin, S. Lathuilière, S. Tulyakov, E. Ricci, and Sebe. First order motion model for image animation. In Conference on Neural Information Processing Systems (NeurIPS), December 2019.
[97] A. Siarohin, S. Roy, S. Lathuilière, S. Tulyakov, E. Ricci, and N. Sebe. Motion-supervised co-part segmentation. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 9650-9653 IEEE, 2021.
[98] Y. Song, C. Yang, Y. Shen, P. Wang, Q. Huang, and C.-C. J. Kuo. Spg-net: Segmentation prediction and guidance network for image inpainting. arXiv preprint arXiv: 2018.
[99] S.-Y. Su, F. Yu, M. Zollhoefer, and H. Rhodin. A-neRF: Articulated neural radiance fields for learning human shape, appearance, and pose. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https:/ https:///ppereviee net/forum? see/:forrmmm1ee1w2hh0000611 1d=1wEE000661
[100] J. Sun, S. Ryou, R. H. Goldshmid, B. Weissbourd, J. O. Dabiri, D. J. Anderson, A. Kennedy, Y. Yue, and P. Perona. Self-supervised keypoint discovery in behavioral videos. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pages 2171-2180, 2022.
[101] S. Suwajanakorn, N. Snavely. J. Tompson, and M. Norouzi. Discovery of latent 3d keypoints via end-to-end geometric reasoning. In Advances neural information processing systems, pages 2059-20700 2070, 2018.
[102] J. Thewlis, H. Bilen, and A. Vedaldi. Unsupervised learning of object landmarks by factorized spatial embeddings. In Proceedings of the IEEE international conference on computer vision, pages 5916-5999 5925, 2017.
[103] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds- 200-2011 Dataset. Technical Report CNS-TR-2011-00 1-001, California Institute of Technology. 2011.
[104] F. Wang, L. Kang, and Y. Li. Sketch-based 3d shape retrieval using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 1875-1883, 2015.
[105] H. Wang, S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, pages 10506- 10518, 2019.
16

---

[106] J. Wang. C. Wang, Q. Huang, Y. Shi, J.-F. Cai, Q. Zhu, and B. Yin. Image Impainting Based on Multi- Frequency Probabilistic Inference Model, page 1-9. Association for Computing Machinery, New York, NY, USA, 2020. ISBN 978 9781450379885. URI. https:/tpps.iiii.. org/ 10. 10.1114/220 01114/333947111 3413891
[107] W. Wu, K. Cao, C. Li, C. Qian, and C. C. Loy. Transgaga: Geometry-aware unsupervised image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8012-8022 8021, 2019.
[108] Z. Wu, Y. Xiong, instance discrimination. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, 2018.
Stella, and Lin. Unsupervised feature learning via non-parametric
[109] B. Xiao, H. Wu, and Y. Wei. Simple baselines for human pose estimation and tracking. In European Conference on Computer Vision (ECCV), 2018.
[110] W. Xiong, J. Yu, Z. Lin, J. Yang, X. Lu, C. Barnes, and J. Luo. Foreground-aware image inpainting. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 3840-5848, 2019.
[111] H. Xu, E. Bazavan, A. Zanfir, B. Freeman, R. Sukthankar, and C. Sminchiseseu. Ghum ghuml: Generative 3d human shape and articulated pose models. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (Oral), pages 6184-6193 2020. URL thecvf.eh.. com wwwt/nttttttttceeeeeeee CVPR http:/ateeccee///oeeeeeeee///ee GHUM -HUML .tml
Generative_ 3D
[112] H. Xu, X. Su, M. Wang. X. Hao, and G. Gao. An edge information and mask shrinking based image inpainting approach. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6, Los Alamitos, CA, USA, jul 2020. IEEE Computer Society. doi: 10. 10.100//MM44422444.00 2tt.:/4662422200..90 2020.9102892. 1109/ICME46284 2020. .9002992
htlps:tps: wtw.oeeeeeecoteeecciiteeteeeeeee org//0
[113] P. Xu. Deep learning for free-hand sketch: survey. IEEE transactions on paltern analysis and machine intelligence, PP, 2022.
[114] P. Xu, Y. Huang, T. Yuan, K. Pang, Y.Z. Song, T. Xiang, T. M. Hospedales, Z. Ma, and J. Guo. Sketchmate: Deep hashing for million-scal human sketch retrieval. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8090-8098, 2018.
[115] P. Xu, C. K. Joshi, and X. Bresson. Multigraph transformer for free-hand sketch recognition. IEEE Transactions on Neural Networks and Learning Systems, 2021.
(116) Y. Xu, C. Yang. Z. Liu, B. Dai, and B. Zhou. Unsupervised landmark learning from unpaired data. arXiv preprint urXiv:2007.0105: 2020.
[117] Y. Yang and X. Guo. Generative landmark guided face inpainting. In Chinese Conference Pattern Recognition and Computer Vision (PRCV), pages 14-26. Springer, 2020.
[118] Y. Yang, A. Loquercio, D. Scaramuzza, and S. Soatto. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 879-888, 2019.
[119] Yang, B. Lai, and S. Soatto. Dystab: Unsupervised object segmentation via dynamic-static bootstrapping. In Proceedings of the IEEF/CVF Conference Computer Vision and Patlern Recognition, pages
2021.
[120] R. Yi, Y.J. Liu, Y.KK. Lai, and P. L. Rosin. APDrawingGAN: Generating artistic portrait drawings from face photos with hierarchical gans. IEEE Conference on Computer Vision and Pattern Recognition (CVPR '19), pages 10743-10752, 2019.
[121] Q. Yu, Yang, Y.-Z. Song, Xiang, and T. M. Hospedales. Sketch-a-net that beats humans. In BMVC, 2015.
17

---

[122] Q. Yu, F. Liu, Y.-Z. Song, T. Xiang, T. M. Hospedales, and C. C. Loy. Sketch me that shoe. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[123] T. Yu, K. Guo, F. Xu, Y. Dong, Z. Su, J. Zhao, Li, Q. Dai, and Y. Liu. Bodyfusion: Real-time capture of human motion and surface geometry using single depth camera. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 910-919, 2017. doi: 10. 10991CCV.2017.100
[124] Y. Yu, F. Zhan, R. J., J. Pan, K. Cui, S. Lu, Ma, X. Xie, and C. Miao. Diverse Image Inpainting with Bidirectional and Autoregressive Transformers, page 69-78. Association for Computing Machinery, New York, NY, USA, 2021. ISBN 9781450386517. URL https //doi. org/ 10. /145/3444085.. 347443653477436
3475436
[125] X. Zhang, C. Shi, X. Wang. X. Wu, X. Li, J. Lv, and I. Mumtaz. Face inpainting based on gan by facial prediction and fusion as guidance information. Applied Soft Computing. 111:107626, 2021. ISSN 1568-4946. doi: http:///o..rr//0..00oo.........
URL
fttps://www..ciincces htt.a///w......oe.oe....cccocaaa
[126] Y. Zhang, Y. Guo, Y. Jin, Y. Luo, Z. He, and H. Lee. Unsupervised discovery of object landmarks structural representations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2694-2229111 2994,,703, 2018.
[127] Y. Zhang, Q. Liang, K. Zou, Z. Li, W. Sun, and Wang. Self-supervised part segmentation via motion imitation. Image and Vision Computing, 120: 120:104393, 2022. ISSN 0262-8856. doi: https://doi..rrg/ 10. 101 imwais 2022 104393. URL https: btttps:/wwww.ccieee www. www.seeieieedrrrc....mmmmm.. com/
[128] Z. Zhao, W. Liu, Y. Xu, X. Chen, W. Luo, L. Jin, B. Zhu, T. Liu, B. Zhao, and Gao. Prior based human completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7947-7957, 2021.
[129] Zheng, T.-J. Cham, and Cai. Pluralistic image completion. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1438-1447, 2019. 2019,
[130] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pages 2223-2232 2017.
[131] W. Zhu, S. Liang, Y. Wei, and Sun. Saliency optimization from robust background detection. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 28 2814 1821, 2014. doi: 10.1109/CVPR.2014.555
18

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [No] This paper does work towards theoretical analysis.
(b) Did you include complete proofs of all theoretical results? [No] This paper does not work towards theoretical analysis.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] The code, instructions, and the pre-trained models are released our GitHub The data we used is already publicly available.
(b) Did you specify all the training details (c.g.. data splits. hyperparameters, how they were chosen)? [Yes] See Section 3.3 and Appendix B D
(c) Did report error bars (e.g.. with respect to the random seed after running experi-
ments multiple times)? [Yes] See Section 3.3 and Table I
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 3.3
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] All the datasets are commonly used in
research papers and their licenses allow the usage for research purposes.
(c) Did you include any new assets either the supplemental material or as URL? [Ye]] The code, instructions, and the pre-trained models are released in our GitHub
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] All the datasets are commonly used in research papers and the licenses allow the usage.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] All the datasets commonly used in research papers and all datasets that contain humans are established datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
19


Reviews:
[{"review_id": "rTpSZohx5mN", "reviewer": "Reviewer_aWFu", "summary": "This paper proposes to learn 2D keypoints from images, where the keypoints are linked by straight edges, from images of the same object class. The method is trained by reconstructing the original input image using masked input images and the learned edge map. The edge weights are learned and are shared across all instances in the dataset. Quantitative evaluations focus on datasets with human poses such as Human3.6M and CelebA, and qualitative results are on datasets such as Zebra, Horses, and Flowers. Results show that the proposed edge modeling is important for learning keypoints across datasets.", "questions": "I listed some questions above in weaknesses.\n\nSome more questions:\n- Could the authors further clarify \"structured background\"? Is it when the dataset have all the same background, or is it when the background have straight / patterned edges?\n\n- What is the effect of using the masked image for appearance vs. extracting an appearance feature from the image as in [38]?", "limitations": "Yes the authors included limitations and potential negative societal impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- To the best of my knowledge, the proposed method is new for learning keypoints in a self-supervised way with learned edge weights across keypoints.\n\n- The evaluation is on a range of datasets, with fairly extensive ablation studies of the method with a range of hyperparameters. \n\n- The presentation of the paper is clear, with lots of visualizations of the method, including discussions of failure cases and limitations.\n\nWeaknesses\n- The paper mentions that the training required a collection of images of the same object class (+ the graph edge weights are shared). Could the authors further discuss this requirement (perhaps in limitations in the main paper, if the class requirement is strict)? How similar do the objects need to be - would this method work on image datasets like the CUB dataset? \n\n- The keypoints visually seem to cover object edges a lot of the time - I'm not sure if the keypoints are covering meaningful locations of the image (ex: in the provided supplementary videos & especially for flowers in Figure 6 of the appendix). Maybe this is intentional by the authors - could this be clarified? Could the authors also discuss relation of this work to works on unsupervised foreground segmentation, since it looks like the edges are covering a lot of the foreground (ex: [A])?\n\n- The authors mention that their approach is simpler than alternatives (lines 13, 265) - however, there are simpler approaches for keypoint discovery that perform comparably on Human3.6M (ex: [39] [B] - both have fewer components than this model and doesn't need edges - they both need video though). Does the author mean that their method is simpler for image-based keypoint discovery? Also I think that Table 2 could be updated with [B] which is a self-supervised method that achieves 2.53 $\\pm$ 0.056 on Human3.6M. \n\n[A] Savarese et al., Information-Theoretic Segmentation by Inpainting Error Maximization, CVPR 2021\n\n[B] Sun et al., Self-Supervised Keypoint Discovery in Behavioral Videos, CVPR 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ahBXuUUp-i2", "reviewer": "Reviewer_zetA", "summary": "This work proposes a new self-supervised landmark estimation technique from image collections without any GT annotations. In contrast the existing works the discovers spatially independent landmarks, this work proposes to reason about connections/edges between the landmarks (skeletons). The core of the technique is auto-encoder with the estimated edge maps and masked images as bottleneck layers forcing the network to disentangle structure information into the learned keypoints and edges. Experiments on humans and face datasets demonstrate better results compared to existing landmark and part discovery works. Visual results demonstrate generalization of the technique to other object types (although the quality of results is not clear from the visuals on these datasets).", "questions": "Experiments to further strengthen the paper:\n- Did authors try using unsupervised saliency to make the network focus more on foreground regions? Does the improve the results?\n- If we assume a generic object skeleton is given (i.e., edge weights are given), can the proposed approach learn to attach the given skeleton to the right locations in the images?\n\nSuggestions:\n- It is better to give an intuitive explanation of edge computation (eq.3) so that authors need to look into [66] to understand this part o the paper.\n- Font sizes is too small in figures-4 and 5. Increase it for better readability.\n\nMinor corrections:\n- Eq.1: Missing opening bracket in the denominator.\n- Eq.3: Missing 'equal' sign.\n- line 264: 'a masked images' -> 'masked images'.", "limitations": "Several limitations are discussed in the paper. It is better to also add a discussion on how much camera viewpoint variations does the proposed approach can handle across the image collection.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- A simple yet effective technique for landmark discovery.\n- The use of simple masking strategy to bottleneck the structure information in an image instead of learning a separate appearance encoder.\n- The use differentiable edge estimation to represent structure with keypoint edges instead of just keypoints.\n- State-of-the-art results on face and human datasets.\n\nWeaknesses:\n- Much of the results are shown on 2D-like object images with relatively constrained viewpoint variations. There are no quantitative results on more 3D like object datasets such as Pascal-VOC where there are 3D objects such as cars, animals from different viewpoints. For instance, existing works (e.g. [33]) report results on pascal-voc parts dataset. Does the proposed approach fail on such datasets with more 3D viewpoint variations? Paper claims the technique works well on all types of objects, but metrics are limited to only certain type of 2D objects.\n- Quantitative metrics are only shown on humans and face datasets. It is difficult to assess the quality of the discovered skeletons on other object datasets (such as zebras, hands) just from visuals. Also, the visuals in Figure-1 only shows horses and zebras from very similar viewpoints.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6EGmokjWiS8", "reviewer": "Reviewer_Wivm", "summary": "This paper proposed a simple yet effective framework for self-supervised learning of human skeletons and object outlines. An encoder is firstly used to regress the key points in a heatmap manner. Then a shared edge weight graph is generated by the keypoint predictions. An edge map is drawn based on the keypoint locations and edge weights. To get rid of the structure information in the image, the authors also randomly masked the original image as an appearance material to recover the input from edge maps. The model is then trained by the perceptual reconstruction loss. Experiments on human skeletons and other object outline datasets show promising results (robust key points and edge predictions).", "questions": "As in the Weaknesses.", "limitations": "Yes. The method has limitations when the background has structures or the object is occluded.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) This paper is well-written. The idea of this paper is novel and interesting.\n2) This paper successfully built a simple yet effective self-supervised learning framework.  Different from previous self-supervised landmark prediction networks which normally enforce the key points to follow the same transformation as images, this paper constructed a shared graph and individual edge weights to reconstruct the image. The differentiable weights were inspired by the previous work on drawing and sketching, and it's very interesting to introduce them into this area for skeleton predictions.\n3) To avoid the graph bottleneck from modelling appearance, this paper additionally feeds the encoder with a marked input to enrich the required appearance for reconstructing images.\n4) The experimental results of this paper are impressive. By using linear regression to regress predicted facial key points to ground truths, the model achieved a very promising NRMSE result. On other additional datasets, the model is also able to predict very satisfying skeletons and key points, which cater to human perceptions. In the provided videos, the predictions are also stable in continuous frames. The predicted edge weight, which is shared in a specific task, can also well model the dependency of different key points.\n5) Interesting ablation study on the number of key points and edge thickness was given.\n\nWeaknesses:\n\n1) The training process of this paper should be described in more detail in the main paper. For example, do you use a concentration constraint as in [115] to help learn heatmaps? Otherwise, the keypoint detector seems to be slow to converge.\n2) Some figures in the paper are blurry when zoomed in (especially for the skeletons), which should be improved.\n3) The authors argued that texture doesn't matter as predictions for zebra are not affected by the stripes. However, this can be because the zebra is jointly trained with horses. What if the model is only trained with objects with significant textures?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept