Learning from Unlabelled Videos Using Contrastive Predictive Neural 3D Mapping
Authors: Adam W. Harley
Shrinidhi K. Lakshmikanth
Fangyu Li
Xian Zhou
Hsiao-Yu Fish Tung
Katerina Fragkiadaki
Venue:  (2020)
Keywords: 3D feature learning
unsupervised learning
inverse graphics
object discovery

Abstract: Predictive coding theories suggest that the brain learns by predicting observations at various levels of abstraction. One of the most basic prediction tasks is view prediction: how would a given scene look from an alternative viewpoint? Humans excel at this task. Our ability to imagine and fill in missing information is tightly coupled with perception: we feel as if we see the world in 3 dimensions, while in fact, information from only the front surface of the world hits our retinas. This paper explores the role of view prediction in the development of 3D visual recognition. We propose neural 3D mapping networks, which take as input 2.5D (color and depth) video streams captured by a moving camera, and lift them to stable 3D feature maps of the scene, by disentangling the scene content from the motion of the camera. The model also projects its 3D feature maps to novel viewpoints, to predict and match against target views. We propose contrastive prediction losses to replace the standard color regression loss, and show that this leads to better performance on complex photorealistic data. We show that the proposed model learns visual representations useful for (1) semi-supervised learning of 3D object detectors, and (2) unsupervised learning of 3D moving object detectors, by estimating the motion of the inferred 3D feature maps in videos of dynamic scenes. To the best of our knowledge, this is the first work that empirically shows view prediction to be a scalable self-supervised task beneficial to 3D object detection. 

Published as a conference paper at ICLR 2020
LEARNING FROM UNLABELLED VIDEOS USING CON- TRASTIVE PREDICTIVE NEURAL 3D MAPPING
Adam W. Harley
Shrinidhi K. Lakshmikanth
Fangyu Li
Carnegie Mellon University aharley eem. edu
Carnegie Mellon University kowshi ka cmu. edu
Carnegie Mellon University fangyul 1esyyll@cmm @cmu.
Xian Zhou
Hsiao-Yu Yu Fish Tung
Katerina Fragkiadaki Camegie Mellon University kateflcs cmu edu
Carnegie Mellon University zhouxi and cmu. edu
Carnegie Mellon University htung@cs. cmu edu
ABSTRACT
Predictive coding theories suggest that the brain Iearns by predicting observations at various levels of abstraction. One of the most basic prediction tasks is view prediction: how would given scene look from alternative viewpoint? Humans excel at this task. Our ability to imagine and fill in missing information tightly coupled with perception: we feel as if fact, information from only the front surface of the world hits our retinas. This paper explores the role of view prediction in the development of 3D visual recog- nition. We propose neural 3D mapping networks, which take as input 2.5D (color and depth) video streams captured by moving camera, and lift them stable 3D feature maps of the scene, by disentangling the scene content from the motion of the camera. The model also projects its 3D feature maps to novel viewpoints, predict and match against target views. We propose contrastive prediction losses to replace the standard color regression loss, and show that this leads to better performance on complex photorealistic data. We show that the proposed model learns visual representations useful for (1) semi-supervised learning of 3D object detectors, and (2) unsupervised learning of 3D moving object detectors, by esti- mating the motion of the inferred 3D feature maps in videos of dynamic scenes. To the best of knowledge, this is the first work that empirically shows view prediction to be a scalable self-supervised task beneficial to 3D object detection.
the world in 3 dimensions, while in
INTRODUCTION
Predictive coding theories |Rao & Ballard 1999
Friston 2003) suggest that the brain learns by pre-
Mean Average Precision
dicting observations at various levels abstrac-
tion. These thcories currently have extensive em- pirical support: stimuli are processed more quickly if they predictable (McClelland & Rumelhart 1981 Pinto et al.] 2015).. prediction error is reflected increased neural activity (Rao & Ballard 1999
0.5
0.4
0.3
View-contrastive pre-training
0.2
Brodski et al., 2015), and disproven expectations lead to learning (Schultz al. 1997). A basic pre- diction task is view prediction: from one viewpoint, predict what the scene would look like from another viewpoint. Learning this task does not require su- pervision from any annotations; supervision is freely available to mobile agent in 3D world who can estimate its egomotion (Patla 1991). Humans excel at this task: can effortlessly imagine plausible hypotheses for the occluded side of objects a pho- tograph, guess what would if we walked around our office desks. Our ability to imagine
View regression pre-training
0.1
No pre-training
0.0
10"
10"
10
10
Number of annotated 3D object boxes
Figure I: Semi-supervised 3D object de- tection. Pre-training with view-contrastive prediction improves results, especially when there are few 3D bounding box annotations.

---

Published as a conference paper at ICLR 2020
information missing from the current image view hnd necessary for predicting alternative views- is tightly coupled with visual perception. We infer a mental representation of the world that is 3 3-dimensional, in which the objects distinct, have 3D extent, occlude one another, and so Despite our 2-dimensional visual input, and despite never having been supplied a bounding box or segmentation mask supervision, our ability for 3D perception emerges early in infancy (Spelke) et al. 1982 Soska & Johnson 2008).
In this paper, we explore the link between view predictive learning and the emergence of 3D per- ception in computational models of perception, on mobile agents in static and dynamic scenes. Our models are trained to predict views of static scenes given 2.5D (color and depth; RGB-D) video streams as input, and are evaluated on their ability to detect objects in 3D. Our models map the 2.5D input streams into 3D feature volumes of the depicted scene. At every frame, the architecture estimates and accounts for the motion of the camera, so that the internal 3D representation remains stable under egomotion. The model projects its inferred 3D feature maps to novel viewpoints, and matches them against visual representations extracted from the target view. We propose contrastive losses to measure the match error, and backpropagate gradients end-to-end in our differentiable modular architecture. At test time, our model forms plausible 3D completions of the scene given RGB-D video streams or even a single RGB-D image as input: it learns to fill in information behind occlusions, and infer the 3D extents of objects.
We evaluate the trained 3D representations in two tasks. (1) Semi-supervised learning of 3D object detectors (Figure D: We show that view-contrastive pre-training helps detect objects in 3D, espe- cially in the low-annotationn regime. (2) Unsupervised 3D moving object detection (Figure r-ight:: Our model can detect moving objects in 3D without any human annotations, by forming a 3D fea- ture volume for each timestep, then estimating the motion field between volumes, and clustering the motion into objects.
We have the following contributions over prior works:
1. Novel view-contrastive prediction objectives: We show that contrastive losses outperform RGB regression (Dosovitskiy et al. 2017 Tung et al. 2019) and variational auto-encoder (VAE) alternatives (Eslami et 20118) in sumi-supervised 3D object detection.
2. A novel unsupervised 3D moving object detection method: By estimating 3D motion inside egomotion-stabilized 3D feature maps, we can detect moving objects unsupervised, outperform- ing 2.5D baselines and iterative generative what-where VAES of previous works (Kosiorek al.,
2018 Hsieh et al. 2018).
3. Simulation-to- contrastive pre-training simulation boosts the performance of 3D object detection real data.
transfer of the acquired 3D feature representations: We show that view-
Our code and data are publicly available
2 RELATED WORK
View prediction View prediction has been the center of much recent research effort. Most meth- ods test their models in single-object scenes, and aim to generate beautiful images for graphics applications (Kar et al. 2017] Sitzmann et al.) 2019: Tatarchenko et al. 2016 [Sato et al.] (2019), as opposed to learning general- -areerrlpuurose visual representations. In this work, we use view prediction to help object detection, not the inverse. The work Eslami al. (2018) attempted view prediction in full scenes, yet only experimented with toy data containing a few colored 3D blocks. Their model cannot effectively generalize beyond the training distribution, e.g., cannot generalize across scenes of variable numbers of objects. The work of Tung et al.) (2019) is the closest to our work. Their model is also an inverse graphics network equipped with a 3 d-meensionaa feature bottleneck, trained for view prediction; it showed strong generalization across scenes, number of objects, and arrange- ments. However, the authors demonstrated its abilities only in toy simulated scenes, similar to those used in [Eslami et (2018). Furthermore, they did not evaluate the usefulness of the learned features for a downstream semantic task (beyond view prediction). This raises questions on the scalability and usefulness of view prediction as objective for self-supervised visual representation learning. which our work aims to address. We compare against the state-of the-rrr model of Tung et al. (2019)
httpps://gittubb.cmmmaaaae
2

---

Published as a conference paper at ICLR 2020
and show that the features learned under our proposed view-contrastive losses are more semantically meaningful (Figure D. To the best of our knowledge, this is the first work that can discover objects in 3D from single camera viewpoint, without any human annotations of object boxes or masks.
Predictive visual feature learning Predictive coding theories suggest that much of the learning in the brain is of a predictive nature (Rao & Ballard 1999 Friston 2003). Recent work in unsupervised learning of word representations has successfully used ideas of predictive coding to learn word representations by predicting neighboring words (Mikolov et al. (013). Many challenges emerge in going from a finite-word vocabulary to the continuous high-dimenniooall image data manifold. Unimodal losses such as mean squared error are not very useful when predicting high dimensional data, due to the stochasticity of the output space. Researchers have tried to handle such stochasticity using latent variable models (Loehlin 1987) or autoregressive prediction of the output pixel space, which involves sampling each pixel value from a categorical distribution conditioned on the output thus far (Van den Oord et al. 2016). Another option is make predictions feature space which is less stochastic than the input. Recently, Oord et al. (0118) followed this direction and used an objective that preserves the mutual information between "top-down" contextual features predicted from input observations, and "bottom-up" features produced from future observations; it applied this objective in speech, text, and image crops. The view-contrastive loss proposed in this work a non- probabilistic version of their contrastive objective. However, our work focuses the video domain as opposed to image patches, and uses drastically different architectures for both the top-down and bottom-up representations, involving 3D egomotion-stabilee bottleneck.
3 CONTRASTIVE PREDICTIVE NEURAL 3D MAPPING
We consider a mobile agent that can move about the scene at will. The agent has a color camera with known intrinsics, and a depth sensor registered the camera's coordinate frame. At training time, the agent has access to its camera pose, and it learns in this stage to imagine full 3D scenes (via view prediction), and to estimate egomotion (from ground-truth poses). It is reasonable to assume that a mobile agent who moves at will has access to its approximate egomotion, since it chooses where to move and what to look at (Patla 1991). Active vision is outside the scope of this work, so our agent simply chooses viewpoints randomly. At test time, the model estimates egomotion on-the-fly from its RGB-D imputs. We use groundtruth depth provided by the simulation environment, and we will show in Sec. 1that the learned models generalize to the real world, where (sparser) depth is provided by a LiDAR unit. We describe our model architecture in Sec. 3.I] our view-contrastive prediction objectives in Sec. 3.2, and our unsupervised 3D object segmentation method in Sec. 3.3
3.1 NEURAL 3D MAPPING
Our model's 's architecture is illustrated in Figure SegureeIlet. It is recurrent neural network (RNN) with memory state tensor (+) "xxhxxxx which has three spatial dimensions (width w,, height h, and depth d) and feature dimension (c channels per grid location). The latent state aims to capture an informative and seometrically-coniisiti
3D deep feature map of the world space. Therefore, the spatial extents correspond to large cuboid of world space, defined with respect to the camera's position at the first timestep. We refer to the latent state as the model's 's imagination to emphasize that most of the grid locations in M will not be observed by any sensor, and so the feature content must be "imagined" by the model.
Our model is made up of differentiable modules that go back and forth between 3D feature imagina- tion space and 2D image space. It builds on the recently proposed geometry-aware recurrent neural networks (GRNNs) of Tung et al.] (2019), which also have a 3D coomotion-stabliaae
latent space, and are trained for RGB prediction. Our model can be considered type of GRNN. In comparison to Tung et al. 2019): (1) our egomotion module can handle general camera motion, as opposed to a dffegree-of-freedon sphere-locked camera. This is a critical requirement for handling data that comes from freely-moving cameras, such as those mounted on mobile vehicles, as opposed to only orbiting cameras. (2) Our 3D-to-2D projection module decodes the 3D map into 2D feature maps, as opposed to RGB images, and uses view-contrastive prediction as the objective, as opposed to regression. In Table 3] and Figure g we show that nearest neighbors in our learned feature space more semantically related than neighbors delivered by RGB regression. We briefly describe each neural module of our architecture next. Implementation details are in the appendix.
3

---

Published as a conference paper at ICLR 2020
3D object detector
3D moving object segmentation
Center.
scoring
Latent update
Connected components
Egomotion stabilization
M
M (n)
Egomotion stabilization
3D-to-2D projection
Magnitucte thresholding
F(C+1)
Egomotion stabilization
Egomotion stabilization
(t) view
M 12 View
3D flow estim.
Egomotion
(ni+)))
WOO
|R. g
C
AD contrast
contrast
2D-to-3D unprojection
2D-to-3D unprojection
2D-to-3D unprojection
2D CNN
2D-to-3D unprojection
2D-to-3D unprojection
+++1)
Flet
F(1I
U o o
g o
U o 14
e O (
1 (
f! m)
1++1 (++1)
((4) pit)
[++1 (++1)
Figure 2: Contrastive predictive neural 3D mapping. Left: Learning visual feature represen- tations by moving in static scenes. The neural 3D mapper learns to lift 2.5D video streams to egomotion-stabiliized 3D feature maps of the scene by optimizing for view-contrastive prediction. Right: Learning to segment 3D moving objects by watching them move. Non-zero 3D motion in the cromotion-ttabllizee 3D feature space reveals independently moving objects and their 3D extent, without any human annotations.
2D-to-3D unprojection This module converts the input RGB image (() Rwxhxx3 and point- cloud (*)) E R into 3D tensors. The RGB is "unprojected" into a 3D tensor U(1)) E B=xhxdx3 by filling each 3D grid location with the RGB value of its corresponding subpixel. The pointcloud converted to a occupancy grid O E wxxxdxx1 by assigning each voxel a value of or O, depending whether not point lands in the voxel. We then convert the concatenation of these tensors into 3D feature tensor F(!) Ruxhxdxe via 3D convolutional encoder-decoder network with skip connections. We Lg-normalize the feature in each grid cell.
Egomotion estimation This module computes the relative 3D rotation and translation between the current camera pose (at time t) and the reference pose (from time 1), allowing us to warp the reg In principle any egomotion estimator could be used here, but we find that our 3D feature tensors are well-suited to 3D coarse-to- fine alignment search, similar to the 2D process in the state-of-the-art optical flow model PWC-Net (Sun et al. 2018). Given the 3D tensors of the two timesteps, (()) and F(!) incrementally warp F into
feature tensor F(!) into registered version F(t)
alignment with (1)) by estimating the approximate transformation at coarse scale, then estimating residual transformations finer scales. This is done efficiently with 6D cross correlations and cost volumes. Following PWC-Net, we use fully-connected layers to convert the cost volumes into motion estimates. While neural architecture is trained end-to-end to optimize view prediction objective, our egomotion module by exception is trained supervised using pairs of frames with annotated egomotion. In this way, it learns to be invariant moving objects in the scene.

---

Published as a conference paper at ICLR 2020
Latent map update This module aggregates inoomotion-stabilieee (registered) feature tensors into the memory tensor ( On the first timestep, we ( F()). On later timesteps, we update the memory with a simple rumning average.
3D-to-2D projection This module "renders" the 3D feature state ( into a 2D feature map of desired viewpoint (A), We first warp the 3D feature map (()) into view-aligned version M
viewe?
then map it to a 2D feature map M Viewa with a 2-block 2D ResNet (He et al. 2016).
3D object detection Given images with annotated 3D object boxes, we train a 3D object detector that takes as input the 3D feature map ( and outputs 3D bounding boxes for the objects present. Our object detector is a 3D adaptation the state-of-the-arr 2D object detector, Faster-RCNN (Ren et al. 2015). The model outputs 3D axis-aligned boxes with objectness confidences.
3.2
CONTRASTIVE PREDICTIVE TRAINING
Given set of input RGB images (19) ..... (()).. pointclouds (DI))...,,(.)) and camera poses (V(1) v(m), we train our model to predict feature abstractions of an unseen input (1(++1) 1)+ V ((++1),, shown in Figure left. We consider two types of representations for the target view: top-down one, T f((1() D v()), ...., (1(m) D("), v(n)), v(+t)), where f is the top-down feature prediction process, and a bottom-up one, B = gf (x+11) D(n+t),, where I is the bottom-up feature extraction process. Note that the top-down representation has access to the viewpoint (a+1) but not to observations from that viewpoint (1(n Din-1)), while the bottom-up representation is only a function of those observations.
We construct 2D and 3D versions of these representation types, using our architecture modules:
We obtain 7 M (m) by encoding the set of inputs from indices I,..., n.
We obtain g = F (++1 by encoding the single input from index n + 1.
We obtain 2 M wit by rendering (*) from viewpoint (n+11)
We obtain BDD_ = ((++1) by convolving ((++1) with a 3-block 2D ResNet (He et al. 2016).
Finally, the contrastive losses pull corresponding (top-down and bottom-up) features close together in embedding space, and push non-corresponding ones beyond a margin distance:
Cocattant
Σ
max
(yip.
(IIT.))) -
B 1em a)).))
(1)
k.jom,n
Σ
max (JO00 mno (1T Bun -a),lla))),)) a).
contrast
k.J.k.m.n.o
where a the margin size, and ) is indices where T corresponds to B, and everywhere else. The losses ask tensors depicting the same scene, but acquired from different viewpoints, to con- tain the same features. The performance of a metric learning loss depends heavily on the sampling strategy used (Schroff al., 2013; Song al. 2016 Sohn 2016). We use the distance- weighted sampling strategy proposed by Wu et al. (2017) which uniformly samples "easy" and "hard" nega- tives; we find this outperforms both random sampling and semi-hard sampling (Schroff al. 2015).
3.3
UNSUPERVISED 3D MOVING OBJECT DETECTION
Upon training, our model learns to map even single RGB-D input to a complete 3D imagination, as we show in Figure Given two temporally consecutive and egomotion-stabilizee 3D maps 1(++1) predicted independently using inputs (1()) D(*)) and (1(++1, D(++)),, we train motion F())
reg
estimation module to predict the 3D motion field ((() between them, which call 3D imagination flow. Since we have accounted for camera motion, this 3D motion field should only be non-zero for independently moving objects. We obtain 3D object proposals by clustering the 3D flow vec- tors, extending classic motion clustering methods (Brox & Malik] 2010) Ochs & Brox 2011 to an egomotion-stabiliize 3D feature space, as opposed to 21D pixel space.
5

---

Published as a conference paper at ICLR 2020
3.3.1 ESTIMATING 3D IMAGINATION FLOW
Our 3D "imagination flow" module is 3D adaptation of the PWC-Net 2D optical flow model (Sun et al. 2018). Note that our model only needs to estimate motion of the independently-movin of the scene, since egomotion has been accounted for. It works by iterating across scales in a coarse- to-fine manner, at each scale, we compute 3D cost volume, convert these costs to 3D displacement vectors, and incrementally warp the two tensors to align them. We use two self-supervised tasks:
part
Synthetic rigid transformations: We apply random rotations and translations F(t) and ask the model to recover the dense 3D flow field that corresponds to the transformation.
Unsupervised 3D temporal feature matching:
Lwap
II i
W(F )..+)s
(3)
where W(F(t+)) w()) back-warps F 1t++1) to align it with F()) using the estimated flow w We apply the warp with a differentiable 3D spatial transformer layer, which does trilinear inter- polation to resample each voxel. This extends self-supervised 2D optical flow (Yu et al. 2016)
to 3D feature constancy (instead of 2D brightness constancy).
We found that both types of supervision are essential for obtaining accurate 3D flow field estimates. Since we are not interested in the 3D motion of empty air voxels, we additionally estimate 3D voxel occupancy, and supervise this using the input pointclouds. We describe our 3D occupancy estimation in more detail in the appendix. At test time, set the 3D motion of all unoccupied voxels to zero.
The proposed 3D imagination flow enjoys significant benefits over 2D optical flow or 3D scene flow. It does not suffer from occlusions and dis-occlusions of image content or projection artifacts (Sun et al. 2010), which typically transform rigid 3D transformations into non-rigid 2D flow fields. In comparison to 3D scene flow (Hornacek al. 2014), which concerns visible 3D points, 3D imagination flow is computed between visual features that may never have appeared in the field of view, but are rather filled in by the model (i.e., "imagined").
3.3.2 3D MOVING OBJECT SEGMENTATION
We obtain 3D object segmentation proposals by thresholding the 3D imagination flow magnitude, and clustering voxels using connected components. We score each component using 3D version of a center-surround motion saliency score employed by numerous works for 2D motion saliency detection (Gao et al., 2008 Mahadevan & Vasconcelos 2010). This score is high when the 3D box interior has lots motion but the surrounding shell does not. This results set of scored 3D segmentation proposals for each video scene.
4 EXPERIMENTS
We train models in CARLA (Dosovitskiy al., 2017), an open-source photorealistic simulator of urban driving scenes, which permits moving the camera to any desired viewpoint in the scene. We obtain data from the simulator as follows. We generate 1170 autopilot episodes of 50 frames each (at 30 FPS), spanning all weather conditions and all locations in both "towns" in the simulator. We define 36 viewpoints placed regularly along a 20m-radius hemisphere in front of the ego-car. This hemisphere is anchored to the ego-car (i.e., it moves with the car). In each episode, we sample random viewpoints from the 36 and randomly perturb their pose, and then capture each timestep of the episode from these 6 viewpoints. We generate train/test examples from this, by assembling all combinations of viewpoints (e-g., N S 5 viewpoints as input, and unseen viewpoint as the target). We filter out frames that have zero objects within the metric "in bounds" region of the GRNN (32m x 32m x 4m). This yields 172524 frames (each with multiple views): 124256 in Town1, and 48268 in Town2. We treat the Town data the "training" set, and the Town2 data as the "test" set, so there no overlap between the train and test images.
For additional testing with real-world data, from the KITTI dataset (Geiger et al., 2013), with the official train/val split: 3712 training frames, and 3769 validation frames.
the (single-view) object detection benchmark

---

Published as a conference paper at ICLR 2020
We evaluate our view-contrastive 3D feature representations in three tasks: (1) semi-supervised 3D object detection, (2) unsupervised 3D moving object detection, and (3) 3D motion estimation. We use Tung et al. (2019) as a baseline representing the state-of-the-an -f-th--rrt but evaluate additional related works in the appendix (see Sec. C.2 Sec.
4.1 SEMI-SUPERVISED LEARNING OF 3D OBJECT DETECTION
We use the proposed cew-contrasttvee prediction as pre-training for 3D object detection We pretrain the inverse graphics network weights, and then train a 3D object detector module supervised to map a 3D feature volume M to 3D object boxes, as described in Section 3.1 We are interested in seeing the benefit of this pre-training across different amounts of label supervision, so we first use the full CARLA train set for view prediction training (without using box labels), and then use a randomly sampled subset of the CARLA train set for box supervision; validation set. We varied the size the box supervision subset across the following range: 100, 200, 500, 1000, 10000, 80000. We show mean average precision (at an IoU of 0.75) for detection function of the number of annotated 3D bounding box examples in Figure We compare our model against version of our model that optimizes for RGB regression, similar to Tung et al. (2019) but with a 6 DoF camera motion as opposed to 2 DoF, as well as a model trained from random weight initialization (i.e., without pre-training). After pre-training, we freeze the feature layers after view predictive learning, and only supervise the detector module; for the fully supervised baseline (from random initialization), we train end-to-end.
evaluate the CARLA
As expected, the supervised model performs better with more labelled data. In the low-data regime, pre-training greatly improves results, and more so for view-contrastive learning than RGB learning- We could not compare against alternative view prediction models as the overwhelming majority of them consider pre-segmented scenes (single object setups; e.g., Kar et al. 2017) and cannot generalize beyond those settings. The same is the case for the model Eslami et al. (2018), which was greatly outperformed by GRNNs in the work of Tung et al. 2019
4.1.1 SIM-TO-REAL (CARLA-TO- -KITTI) TRANSFER
We evaluate whether the 3D predictive feature representations learned in the CARLA simulator are useful for learning 3D object detectors in the real world by testing on the real KITTI dataset (Geiger eet al. 2013). Specifically, we use view prediction pre-training in the CARLA train set, and box supervision from the KITTI train set, and evaluate 3D object detection in the KITTI validation set.
Existing real-world datasets do not provide enough camera viewpoints to support view view-predictive learning. Specifically, in KITTI, all the image sequences come from moving car and thus all viewpoints lie on a ntar-straight trajectory. Thus, simulation-to-real transferability features is especially important for view predictive learning.
We show sim-to-real transfer re- sults in Table D We compare the proposed view-contrastive prediction pre-training with view regression pre- training and random weight initial- ization (no pre-training). In all cases, we train a 3D object detector on the features, supervised using KITTI 3D box annotations. We also compare freezing versus finetuning the weights of the pretrained inverse graphics net- work. The results are consistent with the CARLA tests: view-contrastive pre-training is best, view regression pre-training is second, and learning from annotations alone is worst. Note that depth in KITTI is acquired by real LiDAR sensor, and therefore has lower density and more artifacts than CARLA, yet our model generalizes across this distribution shift.
mAP@ IOU
Method
0.33
0.50
0.75
No pre-training (random init.)
.59
.52
.17
View regression pret., frozen
.64
.54
.15
View regression pret., finetuned
.65
.55
.18
View-contrastive pret., frozen
.67
.58
.15
View-contrastive pret., finetuned
.70
.60
.19
Table I: CARLA-tO-KITTI feature transfer. We train 3D detector module on top of the inferred 3D feature maps M using KITTI 3D object box annotations
Prior work has evaluated supervidd 2D deep features in this way, by re-purposing them for 2D detection task, c.g., Doersch et al. (2015).
7

---

Published as a conference paper at ICLR 2020
Input pointelouds
Estimated ego- stabilized flow
Ground-truth object flow
Input RGBs
Moving object proposals
SON
Color map
Figure 3: 3D feature flow and object proposals, in dynamic scenes. Given the input frames on the 3D flow fields, and converts these into object
left, our model estimates dense proposals. We visualize colorized pointelouds and flow fields in a top-down (bird's eye) view.
4.2 UNSUPERVISED 3D MOVING OBJECT DETECTION
In this section, we our model's ability to detect moving objects in 3D with no 3D object anno- lations, simply by clustering 3D motion vectors. We use two-frame sequences of dynamic scenes from our CARLA validation set, and split it into two parts for evaluation: scenes where the camera is stationary, and scenes where the camera is moving. This splitting is based on the observation that moving object detection made substantially more challenging under a moving camera.
We show precision-recal curves for 3D moving object detection under stationary camera in Fig- We compare our model against a similar one trained with RGB view regression (Tung al.) 2019 and a 2.5D baseline. The 2.5D baseline computes 2D optical flow using PWC-Net (Sun al., 2018). then proposes object masks by thresholding and clustering 2D flow magnitudes; these 2D masks mapped to 3D boxes by segmenting the input pointcloud and fitting boxes to the points. Our model outperforms the baselines. Note that even with ground-trum -trun 2D flow, ground-trus -ruth depth, and oracle threshold, 2.5D baseline can at best only capture the currently-visible fragment of each object. As a result, 3D proposal computed from 2.5D often underestimates the extent of the object. Our model does not have this issue, since it imagines the full 3D scene at each timestep.
ure
We show precision-redd -rcall curves for 3D moving object detection under a moving camera in Figure We compare our model where egomotion predicted by our neural egomotion module, against our model with ground-truth egomotion, as well as a 2.5D baseline, and a stabilized 2.5D baseline. The 2.5D baseline uses optical flow estimated from PWC-Net as before. To stabilize the 2.5D flow, we subtract the ground-truth scene flow from the optical flow estimate before generating proposals. Our model's 's performance is similar to level in static scenes, suggesting that the egomotion module's stabilization mechanism effectively disentangles camera motion from the 3D) feature maps. The 2.5D baseline performs poorly in this setting, as expected. Surprisingly, performance drops further after stabilizing the 2D flows for egomotion. We confirmed this is due to the estimated scene flow being imperfect: subtracting ground-truth scene flow leaves many motion fragments in the back- ground. With ground-truth 2D flow, the baseline performs similar to its static-scene level.
We have attempted to compare against the unsupervised object segmentation methods proposed in Kosiorek et al. (2018) and Hsich et al. (2018) by adapting the publicly available code to the task. These models consume a video as input, and predict the locations 2D object bounding boxes, well as frame-to-frame displacements, in order minimize view regression not able to produce meaningful results from these models. The success of Hsieh et al. (2018) may partially depend on carefully selected priors for the 2D bounding box locations and sizes, match the statistics of the "Moving MNIST" dataset used in that work (as suggested in the official code). For our CARLA experiments, we do not assume knowledge of priors for box locations or sizes.
We were
4.3
3D MOTION ESTIMATION
In this section, we evaluate accuracy of our 3D imagination flow module. The previous section eval- uated this module indirectly since it plays a large part in unsupervised 3D moving object detection;
8

---

Published as a conference paper at ICLR 2020
1.0
3D flow, contrastive (.518 mAP)
1.0
GT-stab. 3D flow (.444 mAP)
Estim--stab. 3D flow (.439 mAP)
2.5D PWC-Net flow (.312 mAP)
0.8
0.8
Unstab. 2.5D flow (,036 mAP)
3D flow, regression (.184 mAP)
Precision
0.6
Procision
0.6
GT-stab. 2.5D flow (.001 mAP)
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Recall
Recall
Figure 4: Unsupervised 3D moving object de- tection with stationary camera.
Figure 5: Unsupervised 3D moving object de- tection with moving camera
here we evaluate its accuracy directly. We two-frame video sequences of dynamic scenes from our CARLA validation set.
Method
Full
Static
Moving
Zero-motion
0.19
0.0
6.97
View regression pret.
0.77
0.63
5.55
We compare 3D flow trained over frozen 3D feature representations obtained from the proposed view-contrastive prediction, against flow trained over frozen features from the baseline RGB regression model (Tun et al. 2019)) We also compare against zero-motion bascline that predicts zero motion everywhere. Since approximately 97% of the voxels belong to the static scene, a zero- motion baseline very competitive an overall average. We therefore evaluate error separately in static and moving parts of the scene. We show the motion estimation errors in Table 2 Our method achieves dramatically lower error than the RGB regression baseline, which suggests that proposed view-contrastive objectives in 3D and 2D result in learning features that are correspond- able across time, even for moving objects, despite the fact that the features were learned using only multi-view data of static scenes.
View-contrastive pret.
0.26
0.17
3.33
Table 2:
Mean endpoint error of the 3D flow in etaabiion-stabilized scenes. View-contrastive fea- tures lead to more accurate motion estimation. Error is L1 in voxel units.
4.4 LIMITATIONS
The proposed model has two important limitations. First, our work assumes an embodied agent that can move around at will. This is hard to realize in the real world, and indeed there are almost no existing datasets with enough camera views to approximate this. Second, our model architecture consumes a lot of GPU memory, due to its extra spatial dimension. This severely limits either the resolution or the metric span of the latent map M. On 12G Titan X GPUs we encode a space sized 32m x 32m 8m resolution of 128 x 128 x 32; with batch size of 4, iteration time is 0.2s/iter. Supervised 3D object detectors typically cover twice this metric range. Sparsifying our feature grid, using points instead of voxels, clear areas for future work.
CONCLUSION
We propose models that learn space-aware 3D feature abstractions of the world given 2.5D input, by minimizing 3D and 2D view-contrastive prediction objectives. We show that view-contrastive prediction leads to features useful for 3D object detection, both in simulation and the real world. We further show that the ability visually imagine full 3D egomotion-stable scenes allows to estimate dense 3D motion fields, where clustering non-zero motion allows objects to emerge without any human supervision. Our experiments suggest that the ability imagine 3D visual information can drive 3D object detection. Instead of learning from annotations, the model learns by moving and watching objects move (Gibson 1979).
ACGNOWLEDGEMENTS We thank Christopher G. Atkeson for providing the historical context of the debate on the utility and role of geometric 3DD models versus feature-based models, and for many other helpful discussions.

---

Published as a conference paper at ICLR 2020
REFERENCES
Simon Baker and lain Matthews. Lucas-kanade 20 years on: A unifying framework. International journal of computer vision, 56(3):221-22 2004.
Alla Brodski, Georg-Friedrich Friedrich Paasch, Saskia Helbling, and Michael Wibral. The faces of predictive coding. Journal of Neuroscience, 35(24))8997-99000 2015.
Rodney A Brooks. Intelligence without reason. In Proceedings of the 12th International Joint
Conference on Artificial Intelligence, 1991.
Thomas Brox and Jitendra Malik. Object segmentation by long term analysis point trajectories. In Kostas Danillidis, Petros Maragos, and Nikos Paragios (eds.), ECCV, pp. 282-295, 2010.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. 1rXiv:193.111000 2019.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rict 3D Model Repository. arXiv:1512.03012, 2015.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. In EMNLP, 2014.
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In /CCV, 2015.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In CORL, pp. 1-16, 2017.
S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, 360(6394)) 1204 1210, 2018. ISSN 0036-8075. doi: 10.1116/scienceeaarb//
Katerina Fragkiadaki, Pablo Arbelaez, Panna Felsen, and Jitendra Malik. Learning to segment moving objects videos. In CVPR, June 2015.
Karl Friston. Learning and inference in the brain. Neural Networks, 16(9):1325-133 1352, 2003.
Dashan Gao, Vijay Mahadevan, and Nuno Vasconcelos. On the plausibility of the discriminant center-surround hypothesis for visual saliency. Journal of vision, 8, 2008.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The KITTI dataset. International Journal of Robotics Research (IJRR), 2013.
James J. Gibson. The Ecological Approach to Visual Perception. Houghton Mifflin, 1979.
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In CVPR, pp. 2616-2625, 2017.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge
university press, 2003.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
F. Henriques and A. Vedaldi. MapNet: An allocentric spatial memory for mapping environments. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2018.
10

---

Published as a conference paper at ICLR 2020
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 17(1-3): 185-203, 1981.
Michael Hornacck, Andrew Fitzgibbon, and Carsten Rother. SphereFlow: 6 DoF scene flow from RGB-D pairs. In CVPR, 2014.
Jun-Ting Hsieh, Binghin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In NIPS, pp. 517-526, 2018.
Abhishek Kar, Christian Häne, and Jitendra Malik. Learning a multi-view stereo machine. In NIPS, 2017.
Anna Khoreva, Rodrigo Benenson, Eddy Ilg, Thomas Brox, and Bernt Schiele. Lucid data dreaming for object tracking. In CVPR Workshops, 2017.
Adam R. Kosiorek, Hyunjik Kim, Ingmar Posner, and Yee Whye Teh. Sequential attend, infer, repeat: Generative modelling moving objects. NIPS, 2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. Journal of Machine Learning Research, 17:1 2016.
Chen-Hsuan Lin and Simon Lucey. Inverse compositional spatial transformer networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
John C Loehlin. Latent variable models: An introduction to factor, path, and structural analysis. Lawrence Erlbaum Associates, Inc, 1987.
Vijay Mahadevan and Nuno Vasconcelos. Spatiotemporal saliency in dynamic scenes. TPAMI, 32, 2010.
James L. McClelland and David E Rumelhart. An interactive activation model of context effects in letter perception: an account of basic findings. Psychological review, 88(5):375, 1981.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen-
tations vector space. 301.71130 2013.
Raul Mur-Artal and Juan Tardós. ORB-SLAM2: An open-source slam system for monocular, stereo, and RGB-D cameras. IEEE Transactions Robotics, 3255:12255 2017.
P. Ochs and T. Brox. Object segmentation in video: A hierarchical variational approach for turning point trajectories into dense regions. In ICCV, 2011.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv:1807.03744 2018.
Aftab E Patla. Visual control of human locomotion. In Advances in psychology, volume 78, pp- 55-97. Elsevier, 1991.
Yair Pinto, Simon van Gaal, Floris P de Lange, Victor AF Lamme, and Anil K Seth. Expectations accelerate entry of visual stimuli into awareness. Journal of Vision, 15(8):13-13, 2015.
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpreta- tion of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79, 1999.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, pp. 91-99, 2015.
Lawrence Roberts. Machine perception of three-dimensional solids. PhD thesis, MIT, 1965.
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima. Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. arXiv:1955.051/ 2019.

---

Published as a conference paper at ICLR 2020
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A unified embedding for face recognition and clustering. In CVPR, 2013.
Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, (755330))1553-1559) 1997.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer. DeepVoxels: Learning persistent 3D feature embeddings. In CVPR, 2019.
Kihyuk Sohn. Improved deep metric learning with multi-class N-pair loss objective. In NIPS, pp- 1857-1865, 2016.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, 2016.
Kasey Soska and Scott Johnson. Development of three-dimensional object completion in- fancy. Child development, 7965):123--1230 2008.
Elizabeth S Spelke, J Mchler, M Garrett, and E Walker. Perceptual knowledge of objects in infancy. In NJ: Erlbaum Hillsdale (ed.), Perspectives on mental representation, chapter 22. Erlbaum, 1982.
D. Sun, S. Roth, and M. Black. Secrets optical flow estimation and their principles. In CVPR, 2010.
Deqing Sun, Xiaodong Yang. Ming- Yu Liu, and Jan Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In CVPR, 2018.
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Single-view to multi-view: Recon- structing unseen views with a convolutional network. ECCV, 2016.
Hsiao-Yo Yu Fish Tung, Ricson Cheng, and Katerina Fragkiadaki. Learning spatial common sense with geometry-avee recurrent networks. In CVPR, 2019.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Condi- tional image generation with pixelenn decoders. In NIPS, pp. 4790-4798, 2016.
Chao-Yuaa Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. ICCV, pp. 2840-2848, 2017.
Jason J. Yu, Adam W. Harley, and Konstantinos G. Derpanis. Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. In ECCV, 2016.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe, Unsupervised learning of depth and ego-motion from video. In CVPR, 2017.
12

---

Published as a conference paper at ICLR 2020
A APPENDIX OVERVIEW
B wC provide implementation details for our 3D-bottlenecked architecture, egomotion module, and 3D imagination flow module. In Section C provide additional experiments, and additional visualizations of our output. In Section we discuss additional related work.
In Section
B ARCHITECTURE DETAILS
Inputs Our input images are sized 128 384 pixels. We trim the input pointclouds to a maximum of 100,000 points, and to range of 80 meters, to simulate a Velodyne LiDAR sensor.
2D-1o-3D unprojection This module converts the input 2D image 1() and pointcloud D into 4D tensor U E Rwxxxdx4, by filling the 3D imagination grid with samples from the 2D image grid, using perspective (un)projection. Specifically. for each cell in the imagination grid, indexed by the coordinate [ k)), we compute the floating-point 2D pixel location [u, w) K A:]" that it projects to from the current camera viewpoint, using the pinhole camera model (Hartley & Zisserman 2003), where is the similarity transform that converts memory coordinates to camera coordinates and K is the camera intrinsics (transforming camera coordinates to pixel coordinates). A.j.k with the bilinearly interpolated pixel value f(9 We transform our depth map D, in similar way and obtain a binary occupancy grid (*)) Bxwxhxux1, by assigning each voxel value of or 0, depending on whether or not point lands in the voxel. We concatenate this to the unprojected RGB, making the tensor [U (() E Ruxhxdx4
We fill (
We pass the tensors ( (01 through a 3D encoder-decoder network. The 3D feature encoder- decoder has the following architecture, using the notation k-s-c for kernel-stride-channes. 4-2-64, 4-2-128, 4-2-256, 4-0.5-128. 128, 4-0.5-64, 1-1-F, where F is the feature dimension. We use 32. After each transposed convolution in the decoder, concatenate the same-resolution featuremap from the encoder. Every convolution layer (except the last in each net) is followed by leaky ReLU activation and batch normalization.
Egomotion estimation This module computes the relative 3D rotation and translation between the current camera viewpoint and the reference coordinate system of the map ( We significantly changed the module of Tung et al. (2019) which could only handle degrees of camera motion. We consider general camera with full 6-DoF motion. Our egomotion module is inspired by the state-of-the-as PWC-Net optical flow method (Sun et al., 2018): incorporates spatial pyramids, incremental warping, and cost volume estimation via cross coss-correlation.
While M and (t) can be used directly as input to the egomotion module, we find better per- formance can be obtained by allowing the egomotion module to learn its own feature space. Thus, we begin by passing the (unregistered) 3D inputs through 3D encoder-decoder, producing a ref- erence tensor (L)) E R=xxhxxxe, and a query tensor (()) E x=xxxxxe We wish to find the rigid transformation that aligns the two.
We use a coarse-to-fine architecture, which estimates a coarse 6D answer at the coarse scale, and refines this answer in finer scale. We iterate across scales in the following manner: First, downsample both feature tensors to the target scale (unless we are at the finest scale). Then, we generate several 3D rotations of the second tensor, representing "candidate rotations" making set 0, 10, e θ}. where θ is the discrete set of 3D rotations considered. We then use 3D axis-eligned
cross-correlations between (1)) and the F(t). which yields a cost volume of shape r w X d e, where is the total number of spatial positions explored by cross-correlation. We average across spatial dimensions, yielding a tensor shaped r x e, representing an average alignment score for each transform. We then apply small fully-connected network to convert these scores into 6D vector. We then warp F(t) according to the rigid transform specified by the 6D vector, to bring into (closer) alignment with F(1), We repeat this process at each scale, accumulating increasingly fine corrections to the initial 6D vector.
Similar to PWC-Net (Sun et al. 2018), since we compute egomotion in coarse-to-fine manner, need only consider small set of rotations and translations each scale (when generating the
13

---

Published as a conference paper at ICLR 2020
cost volumes); the final transform composes all incremental transforms together. However, unlike PWC-Net, we do not repeatedly warp our imput tensors, because this accumulates interpolation error. Instead, we follow the inverse compositional Lucas- Kanade algorithm (Baker & Matthews 2004 Lin & Lucey 2017), and at each scale warp the original input tensor with the composed transform.
3D-to-2D projection This module "renders" 2D feature maps given a desired viewpoint V(k) by projecting the 3D feature state ( We first appropriately orient the 3D featuremap by re- sampling (() into a view-aligned version M S s-'v(k) i ki]", where S is (as before) the similarity transform that brings imagination co- ordinates into camera coordinates, (A) is the transformation that relates the reference camera coor- dinates the viewing camera coordinates, [ k] are voxel indices in M and J'', k"] are voxel indices in (
views' The sampling is defined by ['',"'
view,, such that perspective viewing rays become axis-aligned. We implement this by sampling from the memory tensor with the correspon- dence [u, v, df K j", kT"" where the indices [u, u] span the image we wish to generate, and d spans the length of each ray. We use logarithmic spacing for the increments of d, finding it far more effective than linear spacing (used in Tung et al. 2019)), likely because our scenes cover a large metric space. We call the sespective-transfommee tensor M
viewe" We then warp the view-oriented tensor ())
proje To avoid repeated interpolation, we M() prijy from ( with a single trilinear sampling step. Finally, we pass the trrspectivettrasffrmne tensor through a CNN, converting it to a 2D feature map M view,, The CNN has the following architecture (using the notation k-s-c for kernel-stride- :: max-poo along the depth axis with x 8 x kernel and 1 x 8 x 1 stride, to coarsely aggregate along each camera ray, 3D convolution with 3-1-32, reshape to place rays together with the channel axis, 2D convolution with 3-1-32, and finally 2D convolution with 1-1-F, where E is the channel dimension. For predicting RGB, F 3; for metric learning, we use F= = 32. We find that with dimensionality E- = 16 or less, the model underfits.
actually compose the view transform with the perspective transform, and compute
3D imagination flow To train our 3D flow module, we generate supervised labels from synthetic transformations of single-timestep input, and an unsupervised loss based on the standard standard variational loss (Horn& & Schunck 1981; Yu et al. 2016). For the synthetic transformations, we randomly sample from three uniform distributions of rigid transformations: (i) large motion, with rotation angles in the range [-6, 6] (degrees) and translations in 1, (meters), (ii) small motion, with angles from [-1, and translations from 1- 0.1], (iii) zero motion. We found that without sampling (additional) small and zero motions, the model does not accurately learn these ranges. Still, since these synthetic transformations cause the entire tensor to move at once, flow module learned from this supervision alone tends to produce overly-smooth flow in scenes with real (non- rigid) motion. The variational loss Lwap. described in the main text, overcomes this issue.
We also apply a smoothness loss penalizing local flow changes: Lsmouth = Σnj l| VV kJ.all
where w(t)) is the estimated flow field and is the 3D spatial gradient. This is a standard technique to prevent the model from only learning motion edges (Horn& & Schunck 1981 Yu et al. (0166).
3D occupancy estimation The goal in this step is to estimate which voxels the imagination grid are "occupied" (i.e., have something visible inside) and which are "free" (i.e., have nothing visible inside). For supervision, we obtain (partial) labels for both "free" and "occupied" voxels using the input depth data. Sparse "occupied" voxel labels are given by the voxelized pointcloud O(t) To obtain labels of "free" voxels, we trace the source- camera ray to each occupied observed voxel, and mark all voxels intersected by this ray as "free".
reg
Our occupancy module takes the memory tensor M (1) as input, and produces new tensor p()) with a value in [o, 1] at each voxel, representing the probability of the voxel being occupied. This is achieved by single 3D convolution layer with x 1 1 filter (or, equivalently, fully-connected network applied at each grid location). followed by a sigmoid nonlinearity. We train this network
with the logistic loss, Lace = (1/Σ1 "1" log(1 + exp( (t) p())), where p (t) is the label
map, and (() is an indicator tensor, indicating which labels are valid. Since there are far more "free" voxels than "occupied", we balance this loss across classes within each minibatch.
14

---

Published as a conference paper at ICLR 2020
Figure 6: Dataset comparison for view prediction. From left to right: Shepard-Metzlee (Eslami al. 2018), Rooms-Roee -aee--Rinnn-aaen Camera (Eslami al. 2018), ShapeNet arrangements (Tung al. 2019). cars Tatarchenko et al. (016) and CARLA (used in this work) (Dosovitskiy et al. 2017). CARLA scenes are more realistic, and are not object-centric.
2D CNN The CNN that converts the target view into embedding image is residual network (He et al., 2016) with two residual blocks. with 3 convolutions in each block. The convolution layers' channel dimensions are 64, 64, 64, 128, 128, 128. Finally there one convolution layer with E' channels, where E is the embedding dimension. We use E' = 32.
Contrastive loss For both the 2D and the 3D contrastive loss, for each example in the minibatch. we randomly sample a set of 960 pixel/voxel coordinates S for supervision. Each coordinate E S gives a positive correspondence T,, B,, since the tensors are aligned. For each Tú, we sample a neg- ative B& from the samples acquired across the entire batch, using the distance- weighted sampling strategy of Wu et al. (2017). In this way, on every iteration we obtain an equal number of positive and negative samples, where the negative samples spread out in distance. We additionally apply an Le loss on the difference between the entire tensors, which penalizes distance at all positive cor- respondences (instead of merely the ones sampled for the metric loss). We find that this accelerates training. We use a coefficient of 0.1 for (D) Contmast 1.0 for C3D Contrast and 0.001 for the L2 losses.
Code and training details Our model is implemented in Python/Tensorfen kernels for the 3D cross correlation (used in the egomotion module and the flow module) and for the trilinear resampling (used in the 2D)-to-3D and -D-to-2D modules). The CUDA operations memory than native-tensorflow equivalents, which facilitates training with large imagination tensors (128 x 128 32 x 32). Training to convergence (approx. 200k iterations) takes 48 hours on single GPU. We use a learning rate of 0.001 for all modules except the 3D flow module, for which we use 0.0001. We use the Adam optimizer, with 31 0.9, 33g 0.999.
with custom CUDA
less
C ADDITIONAL EXPERIMENTS
C.I DATASETS
CARLA other datasets We test our method scenes we collected from the CARLA simu- lator (Dosovitskiy et al. (017]). an open-source driving simulator urban scenes. CARLA permits moving the camera to any desired viewpoint in the scene, which is necessary for view-based learning strategy. Previous view prediction works have considered highly synthetic datasets: The work Eslami et al. (2018) introduced the Shepard-Metzine dataset, which consists of seven colored cubes stuck together in random arrangements, and the Gooms-Ring-Camett dataset, which consists of random floor and wall colors and textures with variable numbers of shapes in each room of dif- ferent geometries and colors. The work of Tung et al. (2019) introduced a ShapeNet arrangements dataset, which consists of table arrangements of ShapeNet synthetic models (Chang et al.) 2015). The work of Tatarchenko et al. (2016) considers scenes with a single car. Such highly synthetic and limited-complexity datasets east doubt on the usefulness and generality of view prediction for visual feature learning. The CARLA simulation environments considered in this work have photo- realistic rendering, and depict diverse weather conditions, shadows, and objects, and arguably are much closer real world conditions,
shown in Figure 6 While there exist real-world datasets which are visually similar (Geiger et al. 2013) Caesar et al. 2019), they only contain a small number viewpoints, which makes niew-predictive training inapplicable.
Since occlusion is a major factor in a dataset's difficulty. we provide occlusion statistics collected from our CARLA data. Note that in 2D or unrealistic 3D world, most of the scene would be fully
15

---

Published as a conference paper at ICLR 2020
2D patch retrieval
3D patch retrieval
Method
P@1
P@5
P@ 10
P@1
P@5
p@ 10
2D-bottlenecked regression architectures
GQN with 2D L1 loss (Eslami et al. 2018
.00
.01
.02
3D-bottlenecked regression architectures
GRNN with 2D L1 loss (Tung et al. 2019)
.00
.00
.00
.42
.57
.60
GRNN-VAE VAE with 2D L1 loss & KL div.
.00
.00
.01
.34
.58
.66
3D bottlenecked contrastive architectures
GRNN with 2D contrastive loss
.14
.29
.39
.52
.73
.79
GRNN with 2D & 3D contrastive losses
.20
.39
.47
.80
.97
.99
Table 3: Retrieval precision at different recall thresholds. in 2D and in 3D. P@ K indicates precision the Kthh rank. Contrastive learning outperforms regression learning by substantial margin.
Input View
Target View
GRNN
VAE-GRNN
GQN
Figure 7: View prediction in CARLA. For each input, show the target view and view prediction attempts, by a GRNN-style architecture (Tung et al. 2019). VAE variant of it, and GQN (Eslami
al. 2018 The predictions are blurry and only coarsely correct, due the complexity of the RGB.
visible in every image. In CARLA, a single camera view reveals information on approximately 0.23 (+:0033) of all voxels in the model's 32m 32m 8m "in bounds" space, leaving 0.77 totally occludee/unobberved. This measure includes both "seene surface" voxels, and voxels intersecting rays that travel from the camera center the scene surface (i.e., "known free space"). The surface itself occupies only 0.01 (+:0.022) of the volume. Adding a random second view, the total volume revealed is 0.28 (=0005);; the surface revealed is 0.02 (+0.003). With all views, 0.42 (+0.04) of the volume is revealed; 0.03 (#00004) is revealed surface. These statistics illustrate that the vast majority of the scene must be "imagined" by the model to satisfy the contrastive prediction objectives.
C.2 RGB VIEW PREDICTION
Images from the CARLA simulator have complex textures and specularities and close to pho- torealism, which causes RGB view prediction methods to fail. We illustrate this in Figure given input image and target viewpoint (i.e., pose), show target views predicted by neural 3D mapper trained for RGB prediction (Tung et al. 2019), (ii) a VAE variant of that architecture, and (ii) Generative Query Networks (GQN; Eslami al. 2018 which does not have 3D representa- tion bottleneck, but rather concatenates 2D images and their poses in a 2D recurrent architecture. Unlike these works, our model does not use view prediction as the end task, but rather means of leamiing useful visual representation for 3D object detection, segmentation and motion estimation.
C.3 2D AND 3D CORRESPONDENCE
We evaluate our model's performance estimating visual correspondences in 2D and in 3D, using neighort retrieval task. In 2D, the task is as follows: extract one "query" patch from top-down render of a viewpoint, then extract 1000 candidate patches from bottom-up renders, with only true correspondence (i.e., 999 negatives). We then rank all bottom-up patches according
16

---

Published as a conference paper at ICLR 2020
Figure 8: 2D image patch retrievals acquired with the contrastive model (left) vs the regression model (right). Each row corresponds to a query. For each model, the query is shown on the far left, and the 10 nearest neighbors are shown in ascending order of distance. Correct retrievals are highlighted with green border. The neighbors of the contrastive model often have clear semantic relationships (e.g.. curbs, windows); the neighbors of the RGB model do not.
Ta distance from the query, and report the retrieval precision at several recall thresholds, averaged over 1000 queries. In 3D the task is similar, but patches are feature cubes extracted from the 3D imagination; we generate queries from one viewpoint, and retrievals from other viewpoints and other scenes. The 1000 samples are generated as follows: from 100 random test examples, we generate 10 samples from each, that each sample has negatives from the same viewpoint, and 990 others from different locations/scenes.
We compare the proposed model against (i) the RGB prediction baseline of Tung et al. (2019), (ii) Generative Query Networks (GQN) of EEslam et (2018), which do not have a 3D representation bottleneck, and (iii) a VAE alternative of the (deterministic) model of Tung et al. (2019).
Quantitative results are shown in Table For 2D correspondence, the models learned through the RGB prediction objectives obtain precision near zero at each recall threshold, illustrating that the model is not learning precise RGB predictions. The proposed new-contrastive losses perform better, and combining both the 2D and 3D contrastive losses is better than using only 2D. Interestingly, for 3D correspondence, the retrieval accuracy of the RGB- (GB-based models relatively high. Training neural 31) mappers as variational autoencoders, where stochasticity is added in the 3D bottleneck, improves its precision at lower ranks thresholds. Contrastive learning outperforms all baselines. Adding the 3D contrastive loss gives large boost over using the 2D contrastive loss alone. Note that DD-bottlenecked architectures (Eslami al.) 2018) cannot perform 3D patch retrieval. Qualitative retrieval results for our full model vs. Tung et al. (2019) are shown in Figure
C.4 UNSUPERVISED 3D OBJECT MOTION SEGMENTATION
Our method proposes 3D object segmentations, but labels are only available the form of oriented 3D boxes; we therefore convert our segmentations into boxes by fitting minimum-volume oriented 3D boxes to the segmentations. The precision-recall curves presented in the paper are computed with an intersection-over-une (IOU) threshold of 0.5. Figure Ishows sample visualizations of 3D) box proposals projected onto input images.
C.5 OCCUPANCY PREDICTION
We test our model's ability to estimate occupied and free space. Given a single view input, the model outputs an occupancy probability for each voxel the scene. Then, given the aggregated labels computed from this view and random next view, compute accuracy all voxels for which we have labels. Voxels that are not intersected by either view's 's camera rays are left unlabelled. shows the classification accuracy, evaluated independently for free and occupied voxels, and
17

---

Published as a conference paper at ICLR 2020
Proposal
Proposal
Proposal
Proposal
Proposel
Ground trath
Oleevve beights Evtimaned heights
10.055
32
36
S-00 030
123
Figure 9: Left: 3D object proposals and their center-surround scores (normalized to the range [0,1]). For each proposal, the inset displays the corresponding connected component of the 3D flow field, viewed from a top-down perspective. In each row, an asterisk marks the box with the highest center-surround score. Right: Observed and estimated heightmaps of the given frames, computed from 3D occupancy grids. Note that the observed (ground truth) heightmaps have view-dependent "shadows" due occlusions, while the estimated heightmaps dense and viewpoint-invariiat,
for all voxels aggregated together. Overall, accuracy is extremely high (97-98%) for both voxel types. Note that part of the occupied space (i.e., the voxelized pointcloud of the first frame) is an input to the network, so accuracy this metric is expected to be high.
We show visualization of the estimated oc- cupancy volumes in Higure(--righ We visual- ize the occupancy volumes by converting them to heightmaps. This is achieved by multiply- ing each voxel's occupancy value by its height coordinate in the grid, and then taking along the grid's height axis. The visualizations show that the occupancy module learns to fill the "holes" of the partial view.
3D area type
Accuracy
Occupied space
.97
Free space
All space
.98
Table 4: Occupancy classification accuracy. Ac-
curacy is nearly perfect each region type.
C.6 EGOMOTION ESTIMATION
We compare ego- against (Mur-
motion module
Method
R (rad)
t (m)
ARB-SLAM2
ORB-SLAM2 (Mur-Artal & Tardós 2017)
0.089
0.038
Artal & Tardós 2017). a
SIM-Net (Zhou et al. 2017) SIM-Net + GT depth
0.083
0,086
state-of-the-art geometric
0.100
0.078
simultaneous localization and mapping (SLAM) method, and against the al.] 2017) architecture, which is a 2D CNN that takes pairs of frames and outputs egomotion. We give ORB-SLAM2 access to mround-truthe pointclouds, but note that is being deployed merely as an egomotion module (rather than for SLAM).. We ran our own model and SfM-Net with images sized 128 x 384, but found that ORB-SIAM2 performs best at 256 x 768. SfM-Net is designed to estimate depth and egomotion unsupervised, but since egomotion module is supervised, supervise SIM-Net here as well. We evaluate two versions of one with RGB inputs (as designed), and one with RGB-D inputs (more similar to model). Table 5 shows the results. Overall the models all perform similarly, suggesting that our egomotion method performs on par with the rest.
Ours
0.120
0.036
Table Egomotion error. Our model is on par with the baselines.
SIM-Net (Zhou
Note that the egomotion module of Tung et al. (2019) is inapplicable to this task, since it assumes that the camera orbits about a fixed point, with 2 degrees of freedom. Here, the camera is free, with 6 degrees of freedom.
18

---

Published as a conference paper at ICLR 2020
D ADDITIONAL RELATED WORK
3D feature representations long-standing debate in Computer Vision is whether is worth pursuing 3D models in the form of binary voxel grids, meshes, or 3D pointclouds the output of visual recognition. The "blocks world" of Roberts (165) set as its goal to reconstruct the 3D scene depicted in the image in terms of 3D solids found in database. Pointing out that replicating the 3D world in one's s head is not enough to actually make decisions, Brooks (991) argued for feature- based representations, as done by recent work in end-to-end deep learning (Levine al. (016). Our work proposes learning-based 3D) feature representations in place of previous human-engineered ones, attempting to reconcile the two sides the dehate.
Some recent works have attempted various forms of map-building (Gupta et al. 2017 Henriques & Vedaldi 2018 as a form of gennetricalllyconniittn temporal integration of visual information, in long short-term memory (LSTM: Hochreiter & Schmidhuber 1997) or gated recurrent unit (GRU; Cho et al.) 2014) models. The closest to ours are Learned Stereo Ma- Recurrent Neural Nets (GRNNs; Tung et al.) 2019), which integrate images sampled from a viewing sphere into latent 3D feature memory tensor, in egomotion-stabiline manner, and predict views. All of these works consider very simplistic non oon-pootorcalistic environments and 2-degree-of-freee cameras (i.e., with the camera locked pre-defined viewing sphere). None of these works evalu- ate the suitability of their learned features for a downstream task. Rather, their main objective is to accurately predict views.
place of geometry-unavee
chines (LSMs;; Kar al. 2017), Deep Voxels (Sitzmann et al. (2019). and Geometry
Motion estimation and object segmentation Most motion segmentation methods operate in 2D image space, and use 2D optical flow segment moving objects. While earlier approaches tempted motion segmentation completely unsupervised, by exploiting motion trajectories and inte- grating motion information over time (Brox & 2010 20T0|Ochs Ochs Brox (201), recent works focus on learning segment objects in videos, supervised by annotated video benchmarks (Fragkiadaki ctt al. 2015 Khoreva et al., 2017), 7). Our work differs in the fact that we address object detection and segmentation in 3D opposed to 2D, by estimating 3D motion of the "imagined" (complete) scene, as opposed to 2D motion of the observed scene.
19


Reviews:
[{"review_id": "SkgYMyY65r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper show that view prediction learning to help 3D detection. They explored the link of view predictive learning and the emergence of 3D perception in computational models of perception, on mobile agents in static and dynamic scenes. The whole paper is very well written, and organized.  In general, the whole model is quite straightforward, and quite heavy, while most components come from existing papers. It’s really a good engineering work in term of integrating them together. I have several comments.\n\nhow’s the proposed model different from [1], and [2]? Only Tung et al. (2019) is employed as baselines; and it’s really hard to tell whether the proposed model is SOTA.\n\n(2) The model is still built upon Tung et al. (2019)  with several novel components, including handling more general camera motion beyond the a 2-degree-of-freedom sphere-locked camera. \nI would like to check how significant this point? Particularly, how’s the performance of the model variant without using this component (just 2-degree-of-freedom sphere-locked camera) ?\n\n(3) In  Fig.5, it is very interesting that, the results of  Estim.-stabilized 3D flow are even better than those of GT-stabilized 3D flow, when recall>0.5. Any insight here?\n\n(4) The setting of SEMI-SUPERVISED LEARNING OF 3D OBJECT DETECTION is quite unclear and sloppy. What dataset are used as the labeled, and unlabelled images? how many labeled images?   \nAlso not quite unclear about the settings in UNSUPERVISED 3D MOVING OBJECT DETECTION.\n\n(5) Any chance to give some evaluation about the significance of components introduced in the model? This may give us more insights about the model. I notice that some experimental results may reflect some perspective of this point, while it may be better to explicitly discuss it. \n\n\n[1]  Implicit 3D Orientation Learning for 6D Object Detection from RGB Images. ECCV 2018\n[2]  PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points. NeurPIS 2019", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJgeVwIWqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of visual representation learning from 2.5D video streams by exploring the 2D-3D geometry structures in the 3D visual world. Building upon the previous work GRNN (Tung et al. 2019), this paper introduced a novel view-contrast objective applied to its internal 2D and 3D feature space. To facilitate the 3D view-contrast learning, this paper proposed a novel 2D-3D inverse graphics networks with a 2D-to-3D un-projection encoder, a 2D encoder, a 3D bottlenecked RNNs, an ego-motion stabilization module, and a 3D-to-2D projection module. Compared to previous work (Tung et al. 2019), view-contrastive inverse graphics networks decode in the feature space rather than RGB space. Experimental evaluations are conducted using CARLA simulator (sim) and KITTI dataset (real). Results demonstrate the strengths of the proposed view-contrastive framework in feature learning, 3D moving object detection, and 3D motion estimation.\n\nOverall, this paper studies an important problem in computer vision with a novel solution using unsupervised feature learning. While the technical novelty is clear, reviewer has several questions regarding the implementation and experimental details.\n\n(1) For 3D box detection on KITTI (see Table 1), the comparisons to state-of-the-art models are currently missing. While the benefit of unsupervised feature learning has been demonstrated, it would be more convincing to compare against the following papers (at least with a paragraph of discussion).\n\n(2) The 3D-to-2D projection module seems very expensive. Can you possibly report the training and inference time compared to baselines? Also, the design of the projection module is a bit counter-intuitive as it has 8x8 convolutions. In principle, such projection should be learning-free or with only 1x1 convolutions (aggregation along depth channel). It would be good to consider such ablation studies in the final version.\n\n-- Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency. Tulsiani et al. In CVPR 2017.\n-- Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision. Yan et al. In NIPS 2016.\n-- MarrNet: 3D Shape Reconstruction via 2.5D Sketches. Wu et al. In NIPS 2017.\n\n(3) It seems that the proposed method assumes slow moving background across consecutive frames. In principle, the view-contrastive objective should mask out new pixels in frame T+1. Also, because the view-contrastive loss is applied at feature-level, reviewer would like to know performance on detecting small objects.\n\n(4) As the latent map update module uses an RNN, it would be good to consider consistency beyond 2 frames (given mask is applied to view-contrastive objective). Curriculum learning could be helpful for further improvements.\n\n-- Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis. Yang et al. In NIPS 2015.\n\n(5) How does the proposed method perform when applied to indoor environments?\n\n(6) Additional ablation study to consider: what if 2D/3D contrastive loss is turned off?\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgZnF9b9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a view-prediction inverse graphics model that takes input RGBD streams and produces a 3D feature map of the scene, including regions that are unobserved due to occlusion.  This model is used to learn a 3D visual representation that can be applied for semi-supervised 3D object detection, and for unsupervised 3D moving object detection.  The approach is similar to recent work on Geometry-aware RNNs (Tung et al. 2019), but is applied to more realistic scenes (urban landscapes datasets generated using the CARLA simulator as opposed to simple tabletop arrangements of ShapeNet objects) and makes fewer assumptions about the camera pose (6 DoF camera parameterization as opposed to 2 DoF cameras placed around the objects).  Moreover, the proposed model is evaluated on the downstream 3D object detection tasks to demonstrate the utility of the learned 3D visual representation.\n\nExperiments are performed to evaluate the proposed method against baselines on 3D object detection (both in the semi-supervised setting and unsupervised moving object detection), 3D motion estimation, as well as sim-to-real transfer results (training in CARLA and testing on the KITTI dataset).  The results show that the proposed method: 1) has higher 3D object detection mAP than a view regression-based baseline from prior work (Tung 2019) for settings with little available 3D boundign box supervision; 2) has better 3D detection transfer from CARLA to KITTI (evaluated by mAP) compared to view regression baseline; 3) outperforms the view regression baseline and a 3D motion flow-based baseline on 3D moving object detection (measured through precision-recall curves and mAP); 4) has lower 3D flow error for moving objects compared to zero-motion and view regressive baselines.\n\nI am positive with respect to accepting this work, but find that there are a few unclear points in the evaluation that should be clarified to strengthen the empirical results.  The data is generated from 50 frame sequences at 30fps (i.e. ~1.6 seconds of simulation) where each frame has 6 randomly sampled camera viewpoints in a 20m hemisphere in front of the car.  It would seem to me that the distances between the viewpoints in different frames from these sequences are likely to be quite small, so most of the viewpoint variance would come from random sampling within the hemisphere and randomly selecting one of the views as the target/unseen view. From this description, it is not clear how much variation of unobserved vs observed surfaces exists in the training and test data. It would have been informative to provide some statistics about observed vs occluded object surface area to elucidate the dataset construction.  This aspect of the dataset likely impacts the performance of the method significantly and should thus be addressed a bit more clearly.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgbmrE95S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with turning a 2.5D video representation into a 3D representation of an environment or a scene. The authors introduce self-supervised methods to pretrain the 2d-3d projection with a contrastive loss, which is the neural backbone for multiple other tasks. They then assess their approach on numerous tasks such as 3D-object detection, 3D-moving object detection, and 3D motion estimation. The authors also evaluate the transferability of the features in a challenging sim2real setting.\n\nIt is dense paper with multiple modules (2d-3d, ergomotion, memory, etc.) and concept. Still, the authors make it accessible by concise paragraphs, highlighting key equations (The enum + eq 1 and 2 are quite useful), and well-designed sketch (Figure1). I had some difficulties digging into the visual head component for each task as I was not familiar with this topic. However, the authors always explain their choices in a few lines and refer to the related papers for technical details in a meaningful way.\n\nI am pretty convinced with the experiments, especially Sim2Real, in Tab1, where the baselines are clears and make sense.\n\nI appreciated the limitation section, which is transparent and honest, and clearly states the strength and weaknesses (such as image downscaling) of the approach. Besides, the code and the data should be released, which is always a positive point.\n\nRemarks:\n - Latent map update: running average is a simple and efficient mechanism, it also makes sense as you are dealing with big 3D tensors. Yet, have you tried other update mechanisms?\n - A natural follow-up to this paper is Contrastive Predictive Losses (which had several successes in pure vision setting[1]). Did you already try this approach?\n - In visual CPC papers [1] (or since the early days of visual representation learning!), data transformation has been applied to improve model performance. Would it make sense to apply it to I_{n+1}, D_{n+1} ?\n - Although the authors assess their approach with RGB-D, the models were still trained on 2.5D video. It would have been useful also to assess a pretraining on pure RGB-D data\n\nHowever, I have two (somehow related) concerns. First of all, the machine learning novelties are rather small, contrastive losses are now widespread, and the models are closed to Tung et al. as mentioned by the authors.  However, I believe the paper to be a substantial contribution in vision, as they show the feasibility of their approach on large scale scenarios and over a highly diverse set of tasks. Again, the authors also release the code, making the paper a valuable baseline for the following work. On my side, I am impressed by the sim2real env. \nMy second concern is the following, it is a high quality vision paper, and I am curious why the authors chose ICLR over CVPR. Besides, the tasks are vision-oriented, and 2.5D vision is not common in the ML community. Having said that, the proposed approach is pretty generic, can be applied to RGB-D (more common in ML), and require few expert knowledge in vision (only the Egomotion module).\n\nAs a result, I would advocate for clear accept if we assess vision-based contribution for ICLR; otherwise, I would only recommend weak accept the paper is solely based on ML contributions (the paper is still sound, well-written, with numerous experiments and with a semi-generic architecture)", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to learn space-aware 3D feature abstractions of the world given 2.5D input, by minimizing 3D and 2D view contrastive prediction objectives. The work builds upon Tung et al. (2019) but extends it by removing some of the limitations, making it thus more general. To do so, they learn an inverse graphics network which takes as input 2.5D video and maps to a 3D feature maps of the scene. The authors present experiments on both real and simulation datasets  and their proposed approach is tested on feature learning, 3D moving object detection, and 3D motion estimation with good performance. All reviewers agree that this is an important problem in computer vision and the papers provides a working solution. The authors have done a good job with comparisons and make a clear case about their superiority of their model (large datasets, multiple tasks). Moreover, the rebuttal period has been quite productive, with the authors incorporating reviewers' comments in the manuscript, resulting thus in a stronger submission. Based in reviewer's comment and my own assessment, I think this paper should get accepted, as the experiments are solid with good results that the CV audience of ICLR would find relevant. 

Decision: Accept (Poster)