Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos

Unsupervised landmark learning is the task of learning semantic keypoint-like representations without the use of expensive input keypoint-level annotations . A popular approach is to factorize an image into a pose and appearance data stream , then to reconstruct the image from the factorized components . The pose representation should capture a set of consistent and tightly localized landmarks in order to facilitate reconstruction of the input image . Ultimately , we wish for our learned landmarks to focus on the foreground object of interest . However , the reconstruction task of the entire image forces the model to allocate landmarks to model the background . This work explores the effects of factorizing the reconstruction task into separate foreground and background reconstructions , conditioning only the foreground reconstruction on the unsupervised landmarks . Our experiments demonstrate that the proposed factorization results in landmarks that are focused on the foreground object of interest . Furthermore , the rendered background quality is also improved , as the background rendering pipeline no longer requires the ill-suited landmarks to model its pose and appearance . We demonstrate this improvement in the context of the video-prediction task . 1 INTRODUCTION . Pose prediction is a classical computer vision task that involves inferring the location and configuration of deformable objects within an image . It has applications in human activity classification , finding semantic correspondences across multiple object instances , and robot planning to name a few . One of the caveats of this task is that annotation is very expensive . Individual object “ parts ” need to be carefully and consistently annotated with pixel-level precision . Our work focuses on the task of unsupervised landmark learning , which aims to find unsupervised pose representations from image data without the need for direct pose-level annotation . A good visual landmark should be tightly localized , consistent across multiple object instances , and grounded on the foreground object of interest . Tight localization is important because many objects ( such as persons ) are highly deformable . A landmark localized to a smaller , rigid area of the object will offer more precise pose information in the event of object motion . Consistency across multiple object instances is also important , as we wish for our landmarks to apply to all instances within a visual category . Finally , and most relevant to our proposed method , we want our landmarks to focus on the foreground objects . A landmark that fires on the background is a wasted landmark , as the background is constantly changing , and yields little information regarding the pose of our foreground object of interest . Many unsupervised landmark learning methods perturb an input training image with various transformations , then require the model to learn semantic correspondences across the transformed variants to piece together the unaltered input image . The primary issue with this approach is it penalizes the entire image reconstruction when we care only about the foreground , resulting in landmarks being allocated to the background . This poses a number of issues , including increased memory requirements ( more landmarks required to capture the foreground ) and lower landmark reliability ( landmarks assigned to background are unstable ) . Our proposed method aims to reduce the likeli- hood of landmarks being allocated to the background , thereby improving overall landmark quality and reducing the number of landmarks required to achieve state-of-the-art performance . Our work builds upon existing methods in image-reconstruction-guided landmark learning techniques ( Jakab et al. , 2018 ; Lorenz et al. , 2019 ) . We explicitly encourage our model to factorize the reconstruction task into separate foreground and background reconstructions , where only the foreground reconstruction is conditioned on learned landmarks . Our contributions are as follows : 1 . We propose an improvement to reconstruction-guided unsupervised landmark learning that allows the landmarks to better focus on the foreground . 2 . We demonstrate through empirical analysis that our proposed factorization allows for state-ofthe-art landmark results with fewer learned landmarks , and that fewer landmarks are allocated to modeling background content . 3 . We demonstrate that the overall quality of the reconstructed frame is improved via the factorized rendering , and include an application to the video-prediction task . 2 RELATED WORKS . Our work builds upon prior methods in unsupervised discovery of image correspondences ( Thewlis et al. , 2017b ; Zhang et al. , 2018b ; Suwajanakorn et al. , 2018 ; Thewlis et al. , 2017a ; Kanazawa et al. , 2016 ; Jakab et al. , 2018 ; Lorenz et al. , 2019 ) . Most relevant here are Jakab et al . ( 2018 ) and Lorenz et al . ( 2019 ) , which learn the latent landmark activation maps via an image factorization and reconstruction pipeline . Each image is factored into pose and appearance representations and a decoder is trained to reconstruct the image from these latent factors . The loss is designed such that accurate image reconstruction can only be achieved when the landmarks activate at consistent locations between an image its TPS-warped variant . Lorenz et al . ( 2019 ) specifically improves upon the method proposed by Jakab et al . ( 2018 ) such that instead of representing the appearance information as a single vector for the entire image , there is a separate appearance encoding for each landmark in the pose representation . One limitation of these works is that the appearance and pose vectors also need to encode background information in order to reconstruct the entire image . Our work attempts to resolve this limitation by introducing unsupervised foreground-background separation into the pipeline , using the pose and appearance vectors for only the foreground rendering . There are few other works that propose to separate foreground and background in image rendering tasks . Balakrishnan et al . ( 2018 ) separates foreground and background for image synthesis in an unseen pose , but their method relies on supervised 2D keypoints . Rhodin et al . ( 2018 ) and Rhodin et al . ( 2019 ) separate background from foreground for single and multi-person pose-estimation . In both works , the background images are computed by taking the median pixel value across all frames , and therefore require video sequence data with perfectly static backgrounds . Instead , our approach trains a network to synthesize a clean background from any input frame . It is therefore more forgiving with respect to background variation , and can even handle thin-plate-spline warped backgrounds after overfitting to the training data . This allows us to use our method on non-video datasets such as CelebA faces ( Liu et al. , 2015 ) . 3 METHOD . Our method extends the pipeline proposed by Lorenz et al . ( 2019 ) . At a high-level , it reconstructs an image from two perturbed variants : one where the appearance ( color , lighting , texture ) information is perturbed , and one where the pose ( position , orientation ) of the object is perturbed . The model must learn to extract the pose information from the appearance-perturbed image , and appearance information from the pose-perturbed image . The model will learn a set of landmarks in the process as a means to spatially-align the information extracted from the two sources in order to reconstruct the original image . Our extension to Lorenz et al . ( 2019 ) factorizes the final reconstruction into separate foreground and background renders , where only the foreground is rendered conditioned on the landmark positions . The background will be inferred directly from the pose-perturbed input image with a simple UNet ( Ronneberger et al. , 2015 ) . We want our UNet to have a limited capacity for handling complex changes in pose . The remaining complex pose changes ( e.g . limb motion , object rotations ) will then be captured by the more flexible landmark representations . During training , this factorization is guided by a translating foreground/static background assumption , though we demonstrate that landmark quality is improved even when this assumption is held weakly . 3.1 MODEL COMPONENTS . Our full pipeline comprises five components : the pose and appearance encoders , foreground decoder , background reconstruction subnet , and foreground mask subnet . The goal of the pose encoder Φpose = Encpose ( x ) is to take an input image x and output a set of unsupervised part activation maps . Critically , we want these part activation maps to be invariant to changes in local appearance , as well as to be consistent across deformations . A heatmap that activates on a person ’ s right hand should be invariant across varying skin tones and lighting conditions , as well track the right hand ’ s location across varying deformations and translations . The appearance encoder Φapp = Encapp ( x ; Encpose ( x ) ) extracts local appearance information , conditioned on the pose-encoder ’ s activation maps . Given an input image x , the pose encoder will first provide K × H × W part activation maps Φpose . To extract local appearance vectors , the appearance encoder projects the image to a C×H×W appearance feature mapMapp . We compute the appearance vector for the kth pose activation map as : Φappk , c = H∑ i W∑ j Φposek , i , jM app c , i , j for c = 1 ... C , ( 1 ) giving us K C-dimensional appearance vectors . Here , each activation map in Φpose is softmaxnormalized . The method pipeline attempts to reconstruct the original input image by combining the pose information from the K activation maps with the pooled appearance vectors for each of the K parts . As in Lorenz et al . ( 2019 ) , we fit a 2D Gaussian to each activation of the K activation maps by estimating their respective means and either estimating or using a pre-determined covariance . Each part is represented by Φ̃posek = ( µk , Σk ) , where µk ∈ R2 and Σk ∈ R2×2 . The 2D Gaussian approximation forces each part activation map into a unimodal representation with a simple parameterization , thereby enforcing that each landmark appears in at most one location per image . The foreground decoder ( FGDec ) and background reconstruction subnet ( BGNet ) are networks that attempt to reconstruct the foreground and background respectively . Our foreground decoder is based on the architecture proposed in SPADE ( Park et al. , 2019 ) . In SPADE , semantic maps are used to predict spatially-aware affine transformation parameters for normalization schemes such as InstanceNorm . Herein , we project the 2D Gaussian parameters from Φ̃pose to a heatmap of the target output width and height to use as semantic maps in the SPADE architecture . Following Lorenz et al . ( 2019 ) , we use the formula : s ( k , l ) = 1 1 + ( l − µk ) TΣ−1k ( l − µk ) ( 2 ) where s ( k , l ) is the heatmap value for part map k at coordinate location l. In addition to feeding s ( k , l ) as a semantic map to SPADE , individual appearence vectors are also projected onto their respective heatmap to create a localized appearance encoding to be fed into the decoder . Please see section 3.4 of Lorenz et al . ( 2019 ) for details on this projection . Unlike the foreground decoder , which is conditioned on bottlenecked pose-appearance representation , the BGNet is given direct access to image data , albeit the pose-perturbed variant of the input . Given a static background video sequence , we assume it is easier for the BGNet to learn to directly copy background pixels ( and remove the foreground when necessary ) than it is for the pose-appearance factorization to learn to model the background . In the absence of a BGNet-like module , several landmarks will be allocated to capture the “ pose ” of the background , despite being ill-suited for such a task . The final module is the foreground mask subnet ( MaskNet ) , which infers the blending mask to composite the foreground and background renders . It can be interpreted as a foreground segmentation mask and is conditioned on Φ̃pose .

Summary: The paper presents an unsupervised approach for learning landmarks in images or videos with single objects by separating the representation of the image into foreground and background and factorizing the representation of the foreground into pose and appearance. It builds upon previous work [Jakab 2018, Lorenz 2019] who proposed to train by reconstructing the original image from one version of the image with perturbed appearance and another with perturbed pose. It extends this approach by introducing an additional separation of foreground and background in the image.