Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos

Unsupervised landmark learning is the task of learning semantic keypoint-like representations without the use of expensive input keypoint-level annotations . A popular approach is to factorize an image into a pose and appearance data stream , then to reconstruct the image from the factorized components . The pose representation should capture a set of consistent and tightly localized landmarks in order to facilitate reconstruction of the input image . Ultimately , we wish for our learned landmarks to focus on the foreground object of interest . However , the reconstruction task of the entire image forces the model to allocate landmarks to model the background . This work explores the effects of factorizing the reconstruction task into separate foreground and background reconstructions , conditioning only the foreground reconstruction on the unsupervised landmarks . Our experiments demonstrate that the proposed factorization results in landmarks that are focused on the foreground object of interest . Furthermore , the rendered background quality is also improved , as the background rendering pipeline no longer requires the ill-suited landmarks to model its pose and appearance . We demonstrate this improvement in the context of the video-prediction task . 1 INTRODUCTION . Pose prediction is a classical computer vision task that involves inferring the location and configuration of deformable objects within an image . It has applications in human activity classification , finding semantic correspondences across multiple object instances , and robot planning to name a few . One of the caveats of this task is that annotation is very expensive . Individual object “ parts ” need to be carefully and consistently annotated with pixel-level precision . Our work focuses on the task of unsupervised landmark learning , which aims to find unsupervised pose representations from image data without the need for direct pose-level annotation . A good visual landmark should be tightly localized , consistent across multiple object instances , and grounded on the foreground object of interest . Tight localization is important because many objects ( such as persons ) are highly deformable . A landmark localized to a smaller , rigid area of the object will offer more precise pose information in the event of object motion . Consistency across multiple object instances is also important , as we wish for our landmarks to apply to all instances within a visual category . Finally , and most relevant to our proposed method , we want our landmarks to focus on the foreground objects . A landmark that fires on the background is a wasted landmark , as the background is constantly changing , and yields little information regarding the pose of our foreground object of interest . Many unsupervised landmark learning methods perturb an input training image with various transformations , then require the model to learn semantic correspondences across the transformed variants to piece together the unaltered input image . The primary issue with this approach is it penalizes the entire image reconstruction when we care only about the foreground , resulting in landmarks being allocated to the background . This poses a number of issues , including increased memory requirements ( more landmarks required to capture the foreground ) and lower landmark reliability ( landmarks assigned to background are unstable ) . Our proposed method aims to reduce the likeli- hood of landmarks being allocated to the background , thereby improving overall landmark quality and reducing the number of landmarks required to achieve state-of-the-art performance . Our work builds upon existing methods in image-reconstruction-guided landmark learning techniques ( Jakab et al. , 2018 ; Lorenz et al. , 2019 ) . We explicitly encourage our model to factorize the reconstruction task into separate foreground and background reconstructions , where only the foreground reconstruction is conditioned on learned landmarks . Our contributions are as follows : 1 . We propose an improvement to reconstruction-guided unsupervised landmark learning that allows the landmarks to better focus on the foreground . 2 . We demonstrate through empirical analysis that our proposed factorization allows for state-ofthe-art landmark results with fewer learned landmarks , and that fewer landmarks are allocated to modeling background content . 3 . We demonstrate that the overall quality of the reconstructed frame is improved via the factorized rendering , and include an application to the video-prediction task . 2 RELATED WORKS . Our work builds upon prior methods in unsupervised discovery of image correspondences ( Thewlis et al. , 2017b ; Zhang et al. , 2018b ; Suwajanakorn et al. , 2018 ; Thewlis et al. , 2017a ; Kanazawa et al. , 2016 ; Jakab et al. , 2018 ; Lorenz et al. , 2019 ) . Most relevant here are Jakab et al . ( 2018 ) and Lorenz et al . ( 2019 ) , which learn the latent landmark activation maps via an image factorization and reconstruction pipeline . Each image is factored into pose and appearance representations and a decoder is trained to reconstruct the image from these latent factors . The loss is designed such that accurate image reconstruction can only be achieved when the landmarks activate at consistent locations between an image its TPS-warped variant . Lorenz et al . ( 2019 ) specifically improves upon the method proposed by Jakab et al . ( 2018 ) such that instead of representing the appearance information as a single vector for the entire image , there is a separate appearance encoding for each landmark in the pose representation . One limitation of these works is that the appearance and pose vectors also need to encode background information in order to reconstruct the entire image . Our work attempts to resolve this limitation by introducing unsupervised foreground-background separation into the pipeline , using the pose and appearance vectors for only the foreground rendering . There are few other works that propose to separate foreground and background in image rendering tasks . Balakrishnan et al . ( 2018 ) separates foreground and background for image synthesis in an unseen pose , but their method relies on supervised 2D keypoints . Rhodin et al . ( 2018 ) and Rhodin et al . ( 2019 ) separate background from foreground for single and multi-person pose-estimation . In both works , the background images are computed by taking the median pixel value across all frames , and therefore require video sequence data with perfectly static backgrounds . Instead , our approach trains a network to synthesize a clean background from any input frame . It is therefore more forgiving with respect to background variation , and can even handle thin-plate-spline warped backgrounds after overfitting to the training data . This allows us to use our method on non-video datasets such as CelebA faces ( Liu et al. , 2015 ) . 3 METHOD . Our method extends the pipeline proposed by Lorenz et al . ( 2019 ) . At a high-level , it reconstructs an image from two perturbed variants : one where the appearance ( color , lighting , texture ) information is perturbed , and one where the pose ( position , orientation ) of the object is perturbed . The model must learn to extract the pose information from the appearance-perturbed image , and appearance information from the pose-perturbed image . The model will learn a set of landmarks in the process as a means to spatially-align the information extracted from the two sources in order to reconstruct the original image . Our extension to Lorenz et al . ( 2019 ) factorizes the final reconstruction into separate foreground and background renders , where only the foreground is rendered conditioned on the landmark positions . The background will be inferred directly from the pose-perturbed input image with a simple UNet ( Ronneberger et al. , 2015 ) . We want our UNet to have a limited capacity for handling complex changes in pose . The remaining complex pose changes ( e.g . limb motion , object rotations ) will then be captured by the more flexible landmark representations . During training , this factorization is guided by a translating foreground/static background assumption , though we demonstrate that landmark quality is improved even when this assumption is held weakly . 3.1 MODEL COMPONENTS . Our full pipeline comprises five components : the pose and appearance encoders , foreground decoder , background reconstruction subnet , and foreground mask subnet . The goal of the pose encoder Φpose = Encpose ( x ) is to take an input image x and output a set of unsupervised part activation maps . Critically , we want these part activation maps to be invariant to changes in local appearance , as well as to be consistent across deformations . A heatmap that activates on a person ’ s right hand should be invariant across varying skin tones and lighting conditions , as well track the right hand ’ s location across varying deformations and translations . The appearance encoder Φapp = Encapp ( x ; Encpose ( x ) ) extracts local appearance information , conditioned on the pose-encoder ’ s activation maps . Given an input image x , the pose encoder will first provide K × H × W part activation maps Φpose . To extract local appearance vectors , the appearance encoder projects the image to a C×H×W appearance feature mapMapp . We compute the appearance vector for the kth pose activation map as : Φappk , c = H∑ i W∑ j Φposek , i , jM app c , i , j for c = 1 ... C , ( 1 ) giving us K C-dimensional appearance vectors . Here , each activation map in Φpose is softmaxnormalized . The method pipeline attempts to reconstruct the original input image by combining the pose information from the K activation maps with the pooled appearance vectors for each of the K parts . As in Lorenz et al . ( 2019 ) , we fit a 2D Gaussian to each activation of the K activation maps by estimating their respective means and either estimating or using a pre-determined covariance . Each part is represented by Φ̃posek = ( µk , Σk ) , where µk ∈ R2 and Σk ∈ R2×2 . The 2D Gaussian approximation forces each part activation map into a unimodal representation with a simple parameterization , thereby enforcing that each landmark appears in at most one location per image . The foreground decoder ( FGDec ) and background reconstruction subnet ( BGNet ) are networks that attempt to reconstruct the foreground and background respectively . Our foreground decoder is based on the architecture proposed in SPADE ( Park et al. , 2019 ) . In SPADE , semantic maps are used to predict spatially-aware affine transformation parameters for normalization schemes such as InstanceNorm . Herein , we project the 2D Gaussian parameters from Φ̃pose to a heatmap of the target output width and height to use as semantic maps in the SPADE architecture . Following Lorenz et al . ( 2019 ) , we use the formula : s ( k , l ) = 1 1 + ( l − µk ) TΣ−1k ( l − µk ) ( 2 ) where s ( k , l ) is the heatmap value for part map k at coordinate location l. In addition to feeding s ( k , l ) as a semantic map to SPADE , individual appearence vectors are also projected onto their respective heatmap to create a localized appearance encoding to be fed into the decoder . Please see section 3.4 of Lorenz et al . ( 2019 ) for details on this projection . Unlike the foreground decoder , which is conditioned on bottlenecked pose-appearance representation , the BGNet is given direct access to image data , albeit the pose-perturbed variant of the input . Given a static background video sequence , we assume it is easier for the BGNet to learn to directly copy background pixels ( and remove the foreground when necessary ) than it is for the pose-appearance factorization to learn to model the background . In the absence of a BGNet-like module , several landmarks will be allocated to capture the “ pose ” of the background , despite being ill-suited for such a task . The final module is the foreground mask subnet ( MaskNet ) , which infers the blending mask to composite the foreground and background renders . It can be interpreted as a foreground segmentation mask and is conditioned on Φ̃pose .

Summary: The paper presents an unsupervised method to get disentanglement of pose, appearance, background from both images domain and video domain. 5 sub-network are used to model pose, appearance, foreground, background, and decoders.  Their methods let the network focus more on the foreground to regress the landmark and improve state-of-the-art performance on landmark regression (unsupervised.), video prediction and image reconstruction.