Title: UNSUPERVISED DISCOVERY OF PARTS, STRUCTURE, AND DYNAMICS

Abstract: Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future. In this paper, we propose a novel formulation that simultaneously learns a hierarchical, disentangled object representation and a dynamics model for object parts from unlabeled videos. Our Parts, Structure, and Dynamics (PSD) model learns to, first, recognize the object parts via a layered image representation; second, predict hierarchy via a structural descriptor that composes low-level concepts into a hierarchical structure; and third, model the system dynamics by predicting the future. Experiments on multiple real and synthetic datasets demonstrate that our PSD model works well on all three tasks: segmenting object parts, building their hierarchical structure, and capturing their motion distributions.

1. INTRODUCTION:

What makes an object an object? Researchers in cognitive science have made profound investigations into this fundamental problem; results suggest that humans, even young infants, recognize objects as continuous, integrated regions that move together (Carey, 2009; Spelke & Kinzler, 2007) . Watching objects move, infants gradually build the internal notion of objects in their mind. The whole process requires little external supervision from experts. Motion gives us not only the concept of objects and parts, but also their hierarchical structure. The classic study from Johansson (1973) reveals that humans recognize the structure of a human body from a few moving dots representing the keypoints on a human skeleton. This connects to the classic Gestalt theory in psychology (Koffka, 2013) , which argues that human perception is holistic and generative, explaining scenes as a whole instead of in isolation. In addition to being unsupervised and hierarchical, our perception gives us concepts that are fully interpretable and disentangled. With an object-based representation, we are able to reason about object motion, predict what is going to happen in the near future, and imagine counterfactuals like "what happens if?" (Spelke & Kinzler, 2007) How can we build machines of such competency? Would that be possible to have an artificial system that learns an interpretable, hierarchical representation with system dynamics, purely from raw visual data with no human annotations? Recent research in unsupervised and generative deep representation learning has been making progress along this direction: there have been models that efficiently explain multiple objects in a scene (Huang & Murphy, 2015; Eslami et al., 2016) , some simultaneously learning an interpretable representation (Chen et al., 2016) . Most existing models however either do not produce a structured, hierarchical object representation, or do not characterize system dynamics.

In this paper, we propose a novel formulation that learns an interpretable, hierarchical object representation and scene dynamics by predicting the future. Our model requires no human annotations, learning purely from unlabeled videos of paired frames. During training, the model sees videos of objects moving; during testing, it learns to recognize and segment each object and its parts, build their hierarchical structure, and model their motion distribution for future frame synthesis, all from a single image. Our model, named Parts, Structure, and Dynamics (PSD), learns to recognize the object parts via a layered image representation. PSD learns their hierarchy via a structural descriptor that composes low-level concepts into a hierarchical structure. Formulated as a fully differentiable module, the structural descriptor can be end-to-end trained within a neural network. PSD learns to model the system dynamics by predicting the future.

We evaluate our model in many possible ways. On real and synthetic datasets, we first examine its ability in learning the concept of objects and segmenting them. We then compute the likelihood that it correctly captures the hierarchical structure in the data. We finally validate how well it characterizes object motion distribution and predicts the future. Our system works well on all these tasks, with minimal input requirement (two frames during training, and one during testing). While previous state-of-the-art methods that jointly discover objects, relations, and predict future frames only work on binary images of shapes and digits, our PSD model works well on complex real-world RGB images and requires fewer input frames.

2. RELATED WORK:

Our work is closely related to the research on learning an interpretable representation with a neural network (Hinton & Van Camp, 1993 

3. FORMULATION:

By observing objects move, we aim to learn the concept of object parts and their relationships. Take human body as an example (Figure 1 ). We want our model to parse human parts (e.g., torso, hands, and legs) and to learn their structure (e.g., hands and legs are both parts of the human body).

Formally, given a pair of images {I 1 , I 2 }, let M be the Lagrangian motion map (i.e. optical flow). Consider a system that learns to segment object parts and to capture their motions, without modeling their structure. Its goal is to find a segment decomposition of I 1 = {O 1 , O 2 , . . . , O n }, where each segment O k corresponds to an object part with distinct motion. Let {M g 1 , M g 2 , . . . , M g n } be their corresponding motions.

Beyond that, we assume that these object parts form a hierarchical tree structure: each part k has a parent p k , unless itself is the root of a motion tree. Its motion M g k can therefore be decomposed into its parent's motion M g p k and a local motion component M l k within its parent's reference frame. Specifically,

M g k = M g p k + M l k , if k is not a root.

Here we make use of the fact that Lagrangian motion components M l k and M g p k are additive. Figure 2 gives an intuitive example: knowing that the legs are part of human body, the legs' motion can be written as the sum of the body's motion (e.g., moving to the left) and the legs' local motion (e.g., moving to lower or upper left). Therefore, the objective of our model is, in addition to identifying the object components {O k }, learning the hierarchical tree structure {p k } to effectively and efficiently explain the object's motion. Such an assumption makes it possible to decompose the complex object motions into simple and disentangled local motion components. Reusing local components along the hierarchical structure helps to reduce the description length of the motion map M. Therefore, such a decomposition should naturally emerge within a design with information bottleneck that encourages compact, disentangled representations. In the next section, we introduce the general philosophy behind our model design and the individual components within.

4. METHOD:

In this section, we discuss our approach to learn the disentangled, hierarchical representation. Our model learns by predicting future motions and synthesizing future frames without manual annotations. I 1 into multiple feature maps using an image encoder (Figure 3c ). Intuitively, these feature maps correspond to separate object components. Our model then performs convolutions (Figure 3d ) on these feature maps using separate kernels obtained from a kernel decoder (Figure 3b ), and synthesizes the local motions M l k of separate object components with a motion decoder (Figure 3e ). After that, our model employs a structural descriptor (Figure 3f ) to recover the global motions M g k from local motions M l k , and then compute the overall motion M. Finally, our model uses an image decoder (Figure 3g ) to synthesize the next frame I 2 from the input frame I 1 and the overall motion M.

Our PSD model can be seen as a conditional variational autoencoder. During training, it employs an additional motion encoder (Figure 3a ) to encode the motion into the latent representation z; during testing, it instead samples the representation z from its prior distribution p z (z), which is assumed to be a multivariate Gaussian distribution, where each dimension is i.i.d., zero-mean, and unit-variance. We emphasize the different behaviors of training and testing in Algorithm 1 and 2. (z mean , z var ) = motion_encoder( M). Randomly sample z from N (z mean , z var ).

for k = 1 to d do K k = kernel_decoder k (z k ). Fk = cross_convolution(F k , K k ). M l k = motion_decoder k ( Fk ). end for {M g k } = structural_descriptor({M l k }, S). I 2 = image_decoder(I 1 , d k=1 M g k ).

Algorithm 2 Evaluating PSD Inputs: a single frame I 1 .{I 1 , Î2 } Outputs: predicted future frame I 2 .

F = image_encoder(I 1 ). M = optical_flow(I 1 , Î2 ).

(z mean , z var ) = motion_encoder( M) Randomly sample z from N (0, 1).

for k = 1 to d do K k = kernel_decoder k (z k ). Fk = cross_convolution(F k , K k ). M l k = motion_decoder k ( Fk ). end for {M g k } = structural_descriptor({M l k }, S). I 2 = image_decoder(I 1 , d k=1 M g k ).

4.2. NETWORK STRUCTURE:

We now introduce each component. Dimensionality. The hyperparameter d is set to 32, which determines the maximum number of objects we are able to deal with. During training, the variational loss encourages our model to use as few dimensions in the latent representation z as possible, and consequently, there will be only a few dimensions learning useful representations, each of which correspond to one particular object, while all the other dimensions will be very close to the Gaussian noise.

Motion Encoder. Our motion encoder takes the flow field M between two consecutive frames as input, with resolution of 128×128. It applies seven convolutional layers with number of channels 2017)): it applies each of the d convolutional kernels to its corresponding channel in the feature map. The output will be a d-channel transformed feature map. Motion Decoder. Our motion decoder takes the transformed feature map as input and estimates the x-axis and y-axis motions separately. For each axis, the network applies two 9×9, two 5×5 and two 1×1 depthwise separable convolutional layers, all with 32 channels. We stack the outputs from two branches together. The output motion will have a size of d×128×128×2. Note that the local motion M l k is determined by z k only. Structural Descriptor. Our structural descriptor recovers the global motions {M g k } from the local motions {M l k } and the hierarchical tree structure {p k } using

EQUATION

Then, we define the structural matrix S as S ik = [i ∈ P k ], where each binary indicator S ik represents whether O i is an ancestor of O k . This is what we aim to learn, and it is shared across different data points. In practice, we relax the binary constraints on S to [0, 1] to make this module differentiable: S ik = sigmoid(W ik ), where W ik are trainable parameters. Finally, the overall motion can be simply computed as M = k M g k . Image Decoder. Given the input frame I 1 and the predicted overall motion M, we employ the U-Net (Ronneberger et al., 2015) as our image decoder to synthesize the future image frame I 2 .

4.3. TRAINING DETAILS:

Our objective function L is a weighted sum over three separate components: (

L = L recon + β • L reg + γ • L struct ,

EQUATION

The first component is the pixel-wise reconstruction loss, which enforces our model to accurately estimate the motion M and synthesize the future frame I 2 . We have The last component is the structural loss, which encourages our model to learn the hierarchical tree structure so that it helps the motions M l be represented in an efficient way:

L recon = M -M 2 + α • I 2 -Î2 2 ,

L struct = d k=1 M l k 2

. Note that we apply the structural loss on local motion fields, not on the structural matrix. In this way, the structural loss serves as a regularization, encouraging the motion field to have small values. In the first stage, we encourage the model to learn a disentangled representation (without structure). We set the γ in Equation 3 to 0 and fix the structural matrix S to the identity I. The β in Equation 3is the same as the one in the β-VAE (Higgins et al., 2017), and therefore, larger β's encourage the model to learn a more disentangled representation. We first initialize the β to 0.1 and then adaptively double the value of β when the reconstruction loss reaches a preset threshold.

In the second stage, we train the model to learn the hierarchical representation. We fix the weights of motion encoder and kernel decoder, and set the β to 0. We initialize the structural matrix S, and optimize it with the image encoder and motion decoder jointly. We adaptively tune the value of γ in the same way as the β in the first stage.

5. EXPERIMENTS:

We evaluate our model on three diverse settings: i) simple yet nontrivial shapes and digits, ii) Atari games of basketball playing, and iii) real-world human motions.

5.1. MOVEMENT OF SHAPES AND DIGITS:

We first evaluate our method on shapes and digits. For each dataset, we rendered totally 100,000 pairs for training and 10,000 for testing, with random visual appearance (i.e., sizes, positions, and colors).

For the shapes dataset, we use three types of shapes: circles, triangles and squares. Circles always move diagonally, while the other two shapes' movements consist of two sub-movements: moving together with circles and moving in their own directions (triangles horizontally, and squares vertically). Figure A3 demonstrates the motion distributions of each shape. The complex global motions (after structure descriptor) are decomposed into several simple local motions (before structure descriptor). These local motions are much easier to represent. We also construct an additional dataset with up to nine different shapes. We assign these shapes into four different groups: i) square and two types of parallelograms, ii) circle and two types of triangles, iii) two types of trapezoids, and iv) pentagon. The movements of shapes in the same group have intrinsic relations, while shapes in different groups are independent of each other. These nine shapes have their own different motion direction. In the first group, the tree structure is the same as that of our original shapes dataset: replacing circles with squares, triangles with left parallelograms, and squares with right parallelograms. In the second group, circle and two types of triangles form a chain-like structure, which is similar to the one in our digits dataset. In the third group, the structure is a chain contains two types of trapezoids. In the last group, there is only a pentagon.

As for the digits dataset, we use six types of hand-written digits from MNIST (LeCun et al., 1998). These digits are divided into two groups: 0's, 1's and 2's are in the first group, and 3's, 4's and 5's in the second group. The movements of digits in the same group have some intrinsic relations, while digits in different groups are independent of each other. In the first group, the tree structure is the same as that of our shapes dataset: replacing circles with 0's, triangles with 1's, and squares with 2's. The second group has a chain-like structure: 3's move diagonally, 4's move together with 3' and move horizontally at the same time, and 5's move with 4's and move vertically at the same time.

After training, our model should be able to synthesize future frames, segment different objects (i.e., shapes and digits), and discover the relationship between these objects.

Future Prediction. In Figure 4d and Figure 6d , we present some qualitative results of synthesizing future frames. Our PSD model captures the different motion patterns for each object and synthesizes multiple possible future frames. Figure A3 summarizes the distribution of sampled motion of these shapes; our model learns to approximate each shape's dynamics in the training set. Latent Representation. After analyzing the representation z, we observe that its intrinsic dimensionality is extremely sparse. On the shapes dataset, there are three dimensions learning meaningful representations, each of which correspond to one particular shape, while all the other dimensions are very close to the Gaussian noise. Similarly, on digits dataset, there are six dimensions, corresponding to different digits. In further discussions, we will only focus on these meaningful dimensions. Object Segmentation. For each meaningful dimension, the feature map can be considered as the segmentation mask of one particular object (by thresholding). We evaluate our model's ability on learning the concept of objects and segmenting them by computing the intersection over union (IoU) between model's prediction and the ground-truth instance mask. We compare our model with Neural Expectation Maximization As these two methods both take a sequence of frames as inputs, we feed two input frames repetitively (I 1 , I 2 , I 1 , I 2 , I 1 , I 2 , ...) into these models for fair comparison. Besides, as these methods do not learn the correspondence of objects across data points, we manually iterate all possible mappings and report the one with the best performance.

We present qualitative results in Figure 7 and Figure 5b , and quantitative results in Table 1 . Our PSD model significantly outperforms two baselines. In particular, R-NEM and our PSD model focus on complementary topics: R-NEM learns to identify instances through temporal reasoning, using signals across the entire video to group pixels into objects; our PSD model learns the appearance prior of objects: by watching how they move, it learns to recognize how object parts can be grouped based on their appearance and can be applied on static images. As the videos in our dataset has only two frames, temporal signals alone are often not enough to tell objects apart. This explains the less compelling results from R-NEM. We included a more systematic study in Section A.3 to verify that.

To evaluate the generalization ability, we train our PSD model on a dataset with two squares, among other shapes, and test it on a dataset with three squares. In each piece of data, all squares move together and have the same motion. Other settings are the same as the original shapes dataset. Figure 8 shows segmentation results on these two datasets. Our model generalizes to recognize the three squares simultaneously, despite having seen up to two in training. Hierarchical Structure. To discover the tree structure between these dimensions, we binarize the structural matrix S ik by a threshold of 0.5 and recover the hierarchical structure from it. We compare our PSD model with R-NEM and Neural Relational Inference (NRI) proposed by Kipf et al. (2018) .

As the NRI model requires objects' feature vectors (i.e., location and velocity) as input, we directly feed the coordinates of different objects in and ask it to infer the underlying interaction graph. In Figure 4f and Figure 6f , we visualize the hierarchical tree structure obtained from these models. Our model is capable of discovering the underlying structure; while two baselines fail to learn any meaningful relationships. This might be because NRI and R-NEM both assume that the system dynamics is fully characterized by their current states and interactions, and therefore, they are not able to model the uncertainties in the system dynamics. On the challenging dataset with more shapes, our PSD model is still able to discover the underlying structure among them (see Figure 5c ).

5.2. ATARI GAMES OF PLAYING BASKETBALL:

We then evaluate our model on a dataset of Atari games. In particular, we select the Basketball game from the Atari 2600. In this game, there are two players competing with each other. Each player can move in eight different directions. The offensive player constantly dribbles the ball and throws the ball at some moment; while the defensive player tries to steal the ball from his opponent player. We download a video of playing this game from YouTube and construct a dataset with 5,000 pairs for training and 500 for testing.

Our PSD model discovers three meaningful dimensions in the latent representation z. We visualize the feature maps in these three dimensions in Figure 9 . We observe that one dimension (in Figure 9d ) is learning the offensive player with ball, another (in Figure 9e ) is learning the ball, and the other (in Figure 9f ) is learning the defensive player. We construct the hierarchical tree structure among these three dimensions from the structural matrix S. As illustrated in Figure 9g , our PSD model is able to discover the relationship between the ball and the players: the offensive player controls the ball. This is because our model observes that the ball always moves along with the offensive player.

5.3. MOVEMENT OF HUMANS:

We finally evaluate our method on two datasets of real-world human motions: the human exercise dataset used in Xue et al. ( 2016) and the yoga dataset used in Balakrishnan et al. (2018a) . We estimate the optical flows between frames by an off-the-shelf package (Liu, 2009) . Compared with previous datasets, these two require much more complicated visual perception, and they have challenging hierarchical structures. In the human exercise dataset, there are 50,000 pairs of frames used for training and 500 for testing. As for the yoga dataset, there are 4,720 pairs of frames for training and 526 for testing.

Future Prediction. In Figure 10 and Figure 11 , we present qualitative results of synthesizing future frames. Our model is capable of predicting multiple future frames, each with a different motion. We compare with 3DcVAE (Li et al., 2018) , which takes one frame as input and predicts the next 16 frames. As our training dataset only has paired frames, for fair comparison, we use the repetition of two frames as input: (I 1 , I 2 , I 1 , I 2 , ..., I 1 , I 2 ). We also use the same optical flow (Liu, 2009) for both methods. In Figure 12 , the future frames predicted by 3DcVAE have much more artifacts, compared with our PSD model.

Object Segmentation. In Figure 13 and Figure 14 , we visualize the feature maps corresponding to the active latent dimensions. It turns out that each of these dimensions corresponds to one particular human part: full torsos (13c, 14c), upper torsos (13d), arms (13e), left arms (14d), right arms (14e), right legs (13f, 14g), and left legs (13g, 14f). Note that it is extremely challenging to distinguish different parts from motions, because different parts (e.g., arms and legs) might have similar motions (see Figure 13b ). R-NEM is not able to segment any meaningful parts, let alone structure, while our PSD model gives imperfect yet reasonable part segmentation results. For quantitative evaluation, we collect the ground truth part segmentation for 30 images and compute the intersection over union (IoU) between the ground-truth and the prediction of our model and the other two baselines (NEM, R-NEM). The quantitative results are presented in Table 2 . Our PSD model significantly outperforms the two baselines.

Hierarchical Structure. We recover the hierarchical tree structure among these dimensions from the structural matrix S. From Figure 13h , our PSD model is able to discover that the upper torso and the legs are part of the full torso, and the arm is part of the upper torso, and from Figure 14h , our PSD model discovers that the arms and legs are parts the full torso.

6. CONCLUSION:

We have presented a novel formulation that simultaneously discovers object parts, their hierarchical structure, and the system dynamics from unlabeled videos. Our model uses a layered image representation to discover basic concepts and a structural descriptor to compose them. Experiments suggest that it works well on both real and synthetic datasets for part segmentation, hierarchical structure recovery, and motion prediction. We hope our work will inspire future research along the direction of learning structural object representations from raw sensory inputs.

:



Figures and Tables:

Figure 1: Observing human moving, humans are able to perceive disentangled object parts, understand their hierarchical structure, and capture their corresponding motion fields (without any annotations).

Figure 2: Knowing that the legs are part of human body, the legs' motion can be decomposed as the sum of the body's motion and the legs' local motion. Adelson, 1993). These papers often fail to model the object hierarchy. There has also been abundant research making use of physical models for human or scene tracking (Salzmann & Urtasun, 2011; Kyriazis & Argyros, 2013; Vondrak et al., 2013; Brubaker et al., 2009). Compared with these papers, our model learns to discover the hierarchical structure of object parts purely from visual observations, without resorting to prior knowledge.

shows an overview of our Parts, Structure, and Dynamics (PSD) model. 4.1 OVERVIEW Motion can be decomposed in a layer-wise manner, separately modeling different object component's movement (Wang & Adelson, 1993). Motivated by this, our model first decomposes the input frame

Figure 3: Our PSD model has seven components: (a) motion encoder; (b) kernel decoder; (c) image encoder; (d) cross convolution; (e) motion decoder; (f) structural descriptor; and (g) image decoder.

Training PSD Inputs: a pair of frames {I 1 , Î2 }. Outputs: reconstructed second frame I 2 . F = image_encoder(I 1 ). M = optical_flow(I 1 , Î2 ).

where β and γ are two weighting factors.

where α is a weighting factor (which is set to 10 3 in our experiments).The second component is the variational loss, which encourages our model to use as few dimensions in the latent representation z as possible(Xue et al., 2016;Higgins et al., 2017). We have L reg = D KL N (z mean , z var ) || p z (z) , where D KL (• || •) is the KL-divergence, and p z (z) is the prior distribution of the latent representation (which is set to normal distribution in our experiments).

Figure 4: Results of synthesizing future frames (a-e) and learning hierarchical structure (f) on shapes

Figure 5: Results of segmenting different shapes (b) and learning hierarchical tree structure (c) on a dataset with more shapes.

Figure 6: Results of synthesizing future frames (a-e) and learning hierarchical structure (f) on digits.

Figure 7: Qualitative results of object segmentation on shapes and digits: (a) Ground-truth; (b) PSD; (c) NEM; and (d) R-NEM. In this visualization, we superimpose the segmentation masks on images.

Test on original dataset with 2 squares (b) Generalize to new dataset with 3 squares

Figure 8: Qualitative results of object segmentation when generalizing to new dataset.

Figure 9: Results of segmenting objects (d-f) and learning hierarchical structure (g) on Atari games.

(NEM) proposed by Greff et al. (2017) and Relational Neural Expectation Maximization (R-NEM) proposed by van Steenkiste et al. (2018).

Figure 10: Qualitative results of synthesizing future frames on real-world human motions (exercise).

Figure 11: Qualitative results of synthesizing future frames on real-world human motions (yoga).

Figure 12: Comparison of synthesizing future frames between our PSD model and 3DcVAE

Figure 14: Results of segmenting parts (c-g) and learning hierarchical structure (h) on human motions.

{16, 16, 32, 32, 64, 64, 64}, kernel sizes 5×5, and stride sizes 2×2. Between convolutional layers, there are batch normalizations(Ioffe & Szegedy, 2015), Leaky ReLUs (Maas et al., 2013) with slope 0.2. The output will have a size of 64×1×1. Then it is reshaped into a d-dimensional mean vector z mean and a d-dimensional variance vector z var . Finally, the latent motion representation z is sampled from N (z mean , z var ). Kernel Decoder. Our kernel decoder consists of d separate fully connected networks, decoding the latent motion representation z to the convolutional kernels of size d×5×5. Therefore, each kernel corresponds to one dimension in the latent motion representation z. Within each network, we make uses four fully connected layers with number of hidden units {64, 128, 64, 25}. In between, there are batch normalizations and ReLU layers. Image Encoder. Our image encoder applies six convolutional layers to the image, with number of channels {32, 32, 64, 64, 32, 32}, kernel sizes 5×5, two of which have strides sizes 2×2. The output will be a 64-channel feature map. We then upsample the feature maps by 4× with nearest neighbor sampling, and finally, the resolution of feature maps will be 128×128. Cross Convolution. The cross convolution layer(Xue et al., 2016) applies the convolutional kernels learned by the kernel decoder to the feature maps learned by the image encoder. Here, the convolution operations are carried out in a channel-wise manner (also known as depth-wise separable convolutions inChollet (

Table content: <table/>

Quantitative results (IoUs) of object segmentation on shapes and digits.

Table content: <table><tr><td/><td/><td>Shapes</td><td/><td/><td/><td>Digits</td></tr><tr><td/><td colspan="4">Circles Squares Triangles 0's</td><td>1's</td><td>2's</td><td>3's</td><td>4's</td><td>5's</td></tr><tr><td>NEM</td><td>0.368</td><td>0.457</td><td>0.348</td><td colspan="3">0.470 0.229 0.322 0.512 0.295 0.251</td></tr><tr><td>R-NEM</td><td>0.540</td><td>0.559</td><td>0.583</td><td colspan="3">0.323 0.416 0.339 0.448 0.352 0.326</td></tr><tr><td>PSD (ours)</td><td>0.935</td><td>0.816</td><td>0.905</td><td colspan="3">0.750 0.742 0.739 0.739 0.472 0.641</td></tr></table>

Quantitative results (IoUs) of object segmentation on human exercise dataset.

Table content: <table><tr><td/><td colspan="2">Full torso Upper torso</td><td>Arm</td><td>Left leg</td><td>Right leg</td><td>Overall</td></tr><tr><td>NEM</td><td>0.298</td><td>0.347</td><td>0.125</td><td>0.264</td><td>0.222</td><td>0.251</td></tr><tr><td>R-NEM</td><td>0.321</td><td>0.319</td><td>0.220</td><td>0.294</td><td>0.228</td><td>0.276</td></tr><tr><td>PSD (ours)</td><td>0.697</td><td>0.574</td><td>0.391</td><td>0.374</td><td>0.336</td><td>0.474</td></tr></table>

Peer review (iclr2019):
 The paper describes a method, which learns the hierarchical decomposition of moving objects into parts without supervision, based on prediction of the future. A deep neural network is structured into a sequence of encoders and decoders: the input image is decomposed into objects by a trained head, then motion is estimated from predicted convolutional kernels whose model is trained on optical flow; the latent motion output is encoded into separated motion fields for each object and then composed into a global model with a trainable structured matrix which encodes the part hierarchy. The latent space is stochastic similar to VAEs and trained with similar losses. Strengths: The idea is interesting and nicely executed. I particularly appreciated the predicted kernels, and the trainable structure matrix. Although the field of hierarchical motion segmentation is well studied, up to my knowledge this method seems to be the first of its kind based on a fully end-to-end trainable method where the motion estimators, the decomposition and the motion decoders are learned jointly. The method is evaluated on different datasets including fully synthetic ones with synthetic shapes or based on MNIST; very simple moving humans taken from ATARI games, and realistic humans from two different pose datasets. The motion decomposition is certainly not as good as the definition and the output of a state of the art human pose detector; however, given that the decomposition is discovered, the structure looks pretty good. Weaknesses I have two issues with the paper. First of all, although the related work section is rich, the methods based on hierarchical motion decompositions are rarer, although the field is quite large. Below are a couple of references: Mihir Jain, Jan Van Gemert, Hervé Jégou, Patrick Bouthemy, and Cees GM Snoek. Action localization with tubelets from motion. CVPR, 2014. Chenliang Xu and Jason J Corso. Evaluation of super-voxel methods for early video processing. CVPR, 2012. Jue Wang, Bo Thiesson, Yingqing Xu, and Michael Cohen. Image and video segmentation by anisotropic kernel mean shift. ECCV, 2004 Chenliang Xu, Caiming Xiong, and Jason J Corso. Streaming hierarchical video segmentation. ECCV 2012. Matthias Grundmann, Vivek Kwatra, Mei Han, and Irfan Essa. Efficient hierarchical graph-based video segmentation. CVPR, 2010. Peter Ochs, Jitendra Malik, and Thomas Brox. Segmentation of moving objects by long term video analysis. IEEE PAMI, 2014. Discovering motion hierarchies via tree-structured coding of trajectories Juan-Manuel Pérez-Rúa, Tomas Crivelli, Patrick Pérez, Patrick Bouthemy, BMVC 2016. Samuel J Gershman, Joshua B Tenenbaum, and Frank Jäkel. Discovering hierarchical motion structure. Vision Research, 2015. Secondly, the presentation is not perfect. The paper is densely written with lots of information thrown rapidly at the reader. Readers familiar with similar work can understand the paper (I needed a second pass). But many parts could be better formulated and presented. I understood the equations, but I needed to ignore certain thinks in order to understand them. One of them is the superscript in the motion matrices M, which does not make sense to me. “g” seems to indicate “global” and “l” local, but then again a local parent matrix gets index “g”, and this index seems to switch whether the same node is seen as the current node or the parent of its child. Figure 3 is useful, but it is hard to make the connection with the notation. Symbols like z, M etc. should be included in the figure. The three lines after equations 2 and 3 should be rewritten. They are understandable but clumsy. Also, we can guess where the binary constraints come from, but they should be introduced nevertheless. In essence, the paper is understandable with more efforts than there should be necessary. Other remarks: The loss L_struct is L_2, I don’t see how it can favor sparsity. This should be checked and discussed. A symbolic representation is mentioned in the introduction section. I am not sure that this notion is completely undisputed in science, it should at least not be presented as a fact. The ATARI dataset seems to be smallish (a single video and 5000 frames only).