TransformerFusion: Monocular RGB Scene Reconstruction using Transformers

1 Introduction . Monocular 3D reconstruction is a core task in 3D computer vision , aiming to reconstruct a complete and accurate 3D geometry of an object or an environment from only 2D observations captured by an RGB camera . A geometric understanding is key to applications such as robotic or autonomous vehicle navigation or interaction , as well as model creation and scene editing for augmented and virtual reality . In addition , geometric scene reconstructions form the basis for 3D scene understanding , supporting tasks such as 3D object detection , semantic , and instance segmentation [ 34 , 35 , 36 , 29 , 7 , 43 , 15 , 16 ] . While state-of-the-art SLAM systems [ 3 , 41 ] achieve robust and scale-accurate camera tracking leveraging both visual and inertial measurements , dense and complete 3D reconstruction of largescale environments from monocular video remains a very challenging problem – particularly for interactive settings . Simultaneously , notable progress has been made on multi-view depth estimation , estimating depth from pairs of images by averaging features extracted from the images in a feature cost volume [ 42 , 17 , 19 , 38 , 13 ] . Unfortunately , averaging features across a full video sequence can lead to equal-weight treatment of each individual frame , despite some frames possibly containing less information in various regions ( e.g. , from motion blur , rolling shutter artifacts , very glancing or partial views of objects ) , making high-fidelity scene reconstruction challenging . Inspired by the recent advances in natural language processing ( NLP ) that leverage transformer-based models for sequence to sequence modelling [ 40 , 11 , 2 ] , we propose a transformer-based method that fuses a sequence of RGB input frames into a 3D representation of a scene at interactive rates . Key to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . our approach is a learned feature fusion of the video frames using a transformer-based architecture , which learns to attend to the most informative image features to reconstruct a local 3D region of the scene . A new observed RGB frame is encoded into a 2D feature map , and unprojected into a 3D volume , where our transformer learns a fused 3D feature for each location in the 3D volume from the image view features . This enables extraction of the most informative view features for each location in the 3D scene . The 3D features are fused in coarse-to-fine fashion , providing both improved reconstruction performance as well as interactive runtime . These features are then decoded into high-resolution scene geometry with an MLP-based surface occupancy prediction . In summary , our main contributions to achieve robust and accurate scene reconstructions are : • Learned multi-view feature fusion in the temporal domain using a transformer network that attends to only the most informative features of the image views for reconstructing each location in a scene . • A coarse-to-fine hierarchy of our transformer-based feature fusion that enables an online reconstruction approach running at interactive frame-rates . 2 Related Work . Multi-view depth estimation . Estimating depth from multi-view image observations has been long-studied in computer vision . COLMAP [ 37 ] introduced a patch matching based approach which achieves impressive accuracy and remains established as one of the most popular methods for multiview stereo . While COLMAP offers robust depth estimation for distinctive features in images , the patch matching struggles to densely reconstruct areas without many distinctive color features , such as floor and walls . Recently , learning-based approaches that build data-driven priors from large-scale datasets have improved depth estimation in these challenging scenarios . Some proposed methods rely only on a 2D network with multiple images concatenated as input [ 42 ] . Several recent approaches instead build a shared 3D feature cost volume in reference camera space using feature averaging [ 13 , 17 , 19 , 25 , 26 ] . These approaches estimate the reference frame ’ s depth within a local window of frames , but some also propagate information from previously estimated depth maps by using probabilistic filtering [ 25 ] , a Gaussian process [ 17 ] , or an LSTM bottleneck layer [ 13 ] . Such multi-view depth estimation approaches predict single-view depth maps , which must be fused together to construct a geometric 3D representation of the observed scene . 3D reconstruction from monocular RGB input . Multi-view depth estimation approaches can be combined with depth fusion approaches , such as volumetric fusion [ 6 ] , to obtain a volumetric reconstruction of the observed scene . MonoFusion [ 33 ] is one of the first methods using depth estimate from a real-time variant of PatchMatch stereo [ 1 ] . However , fusing noisy depth estimates causes artifacts in the 3D reconstruction , which lead to the development of recent approaches that directly predict the 3D surface reconstruction instead of per-frame depth estimates . One of the first approaches to predict 3D surface occupancy from two input RGB images is SurfaceNet [ 20 ] , which converts volumetrically averaged colors into 3D surface occupancies using a 3D convolutional network . Atlas [ 28 ] extends this approach to a multi-view setting , while also leveraging learned features instead of colors . Recently , NeuralRecon [ 39 ] proposed a real-time 3D reconstruction framework , adding GRU units distributed in 3D to fuse reconstructions from different local windows of frames . Our approach also fuses together learned features from RGB frame input in an online fashion , but our transformer-based multi-view feature fusion enables relying only on the most informative features from the observed frames for a particular spatial location in the reconstructed scene , producing more accurate 3D reconstructions . Transformers in computer vision . The transformer architecture [ 40 ] has achieved profound impact in many computer vision tasks in addition to its natural language processing origins . For a detailed survey , we refer the reader to [ 22 ] . In computer vision , transformers have been leveraged successfully for tasks such as object detection [ 4 ] , video classification [ 44 ] , image classification [ 12 ] , image generation [ 30 ] , and human reconstruction [ 45 ] . In this work , we propose transformer-based feature fusion for 3D scene reconstruction from a monocular video . Given a sequence of observed RGB frames , our approach learns to attend to the most informative features from each image to predict a dense occupancy field . 3 End-to-end 3D Reconstruction using Transformers . Given a set of N RGB images Ii ∈ RW×H×3 of a scene with corresponding camera intrinsic parameters Ki ∈ R3×3 and extrinsic poses Pi ∈ R4×4 , our method reconstructs the scene geometry by predicting occupancy values o ∈ [ 0 , 1 ] for every 3D point in the scene . Fig . 2 shows an overview of our approach . Each input image Ii is processed by a 2D convolutional encoder Θ , extracting coarse and fine image features ( Φci and Φ f i , respectively ) : Θ : Ii ∈ RW×H×3 7→ ( Φci , Φ f i ) From these 2D image features , we construct a 3D feature grid in world space . To this end , we regularly sample grid points in 3D at a coarse resolution of every vc = 30 cm and a fine resolution of vf = 10 cm . For these coarse and fine sample points , we query corresponding 2D features in all N images and predict fused coarse ψc and fine 3D features ψf using transformer networks [ 40 ] : Tc : ( Φc1 , . . . , ΦcN ) 7→ ( ψc , wc ) Tf : ( Φf1 , . . . , Φ f N ) 7→ ( ψ f , wf ) Note that we also store the intermediate attention weights wc and wf of the first transformer layers for efficient view selection , which is explained in Sec . 3.4 . To further improve the features in the 3D spatial domain , we apply 3D convolutional networks Cc and Cf , at the coarse and fine level , respectively : Cc : { ψc } C×C×C 7→ { ψ̃c } C×C×C Cf : { ( ψ̃c , ψf ) } F×F×F 7→ { ψ̃f } F×F×F Finally , to predict the scene geometry occupancy for a point p ∈ R3 , the coarse ψ̃c and fine features ψ̃f are trilinearly interpolated and a multi-layer perceptron S maps these features to occupancies : S : ( ψ̃c , ψ̃f ) 7→ o ∈ [ 0 , 1 ] This extraction of surface occupancies is inspired by convolutional occupancy networks [ 32 ] and IFNets [ 5 ] . From this occupancy field we extract a surface mesh with Marching cubes [ 27 ] . Note that in addition to surface occupancy , we also predict occupancy masks for near-surface locations at the coarse and fine levels . These masks are used for coarse-to-fine surface filtering ( see Sec . 3.2 ) , which improves reconstruction performance with a focus on the surface geometry prediction and enables interactive runtime . We train our approach in end-to-end fashion by supervising the surface occupancy predictions using the following loss : L = Lc + Lf + Lo , where Lc and Lf denote binary cross-entropy ( BCE ) losses on occupancy mask predictions for near-surface locations at the coarse and fine levels , respectively ( see Sec . 3.2 ) , and Lo denotes a BCE loss for surface occupancy prediction ( see Sec . 3.3 ) . 3.1 Learning Temporal Feature Fusion via Transformers . For a spatial location p ∈ R3 in the scene reconstruction , we learn to fuse coarse ψc and fine level features ψf from the N coarse and fine feature images ( Φci and Φ f i , respectively ) , which are extracted by the 2D encoder Θ . Specifically , we train two instances of a transformer model , one for fusing coarse-level features ψc and one for fusing fine-level features ψf . Both transformers Tc and Tf share the same architecture . Thus , for simplicity , we omit the coarse and fine notation in the following . Our transformer model T is independently applied to each sample point in world space . For a point p , the transformer network takes a series of 2D features φi as input that are bilinearly sampled from the feature maps Φi at the corresponding projective image location . The projective image location is computed via a full-perspective projection Πi ( p ) = π ( Ki ( Rip + ti ) ) , assuming known camera intrinsics Ki and extrinsics Pi = ( Ri , ti ) . To inform the transformer about invalid features ( i.e. , a sample point is projected outside an image ) , we also provide the pixel validity vi ∈ { 0 , 1 } as input . In addition to these 2D features φi , we concatenate the projected depth di = ( Rip + ti ) z , and the viewing ray ri = ( p− ci ) /||p− ci||2 to the input ( ci ∈ R3 denoting the camera center of view i ) . These input features are converted to an embedding vector θi ∈ RD using a linear layer θi = FCN ( φi , di , vi , ri ) , before feeding it into the transformer network that then predicts a fused feature ψ ∈ RD : T : ( θ1 , . . . , θN ) 7→ ( ψ , w ) As described above , w denotes the attention values of the initial attention layer , which are used for view selection to speed-up fusion ( see Sec . 3.4 ) . Transformer architecture . We followed [ 12 ] when designing the transformer architecture T . It consists of 8 modules of feed-forward and attention layers , using multi-head attention with 4 attention heads and embedding dimension D = 256 . Feed-forward layers process the temporal inputs independently , and contain ReLU activation , linear layers with residual connection , and layer norm . The model returns both fused feature ψ ∈ RD and attention weights w ∈ RN over all temporal inputs from the initial attention layer that are later used for selecting which views to maintain over longer sequences of input image views .

Summary: The paper presents a fully end-to-end trainable deep 3D reconstruction pipeline taking monocular RGB video sequences as input. The proposed pipeline works in an online fashion and at interactive frame rates. In the first stage, the method extracts 2D features from input RGB images. In a next stage, 3D features are generated by projecting a 3D location into the images, pooling the corresponding 2D features and fusing them through a transformer network. After refining the 3D features with 3D convolutions, the geometry is predicted through a neural implicit decoder and finally extracted through marching cubes. Notably, the method uses a two-level hierarchy of features for improved quality and runtime performance. Furthermore, the self-attention weights are reused to only keep the most important observations for each 3D point at any given time. Finally, the authors compare their pipeline to several state-of-the-art methods and perform an ablation study to validate the individual components.