TransformerFusion: Monocular RGB Scene Reconstruction using Transformers

1 Introduction . Monocular 3D reconstruction is a core task in 3D computer vision , aiming to reconstruct a complete and accurate 3D geometry of an object or an environment from only 2D observations captured by an RGB camera . A geometric understanding is key to applications such as robotic or autonomous vehicle navigation or interaction , as well as model creation and scene editing for augmented and virtual reality . In addition , geometric scene reconstructions form the basis for 3D scene understanding , supporting tasks such as 3D object detection , semantic , and instance segmentation [ 34 , 35 , 36 , 29 , 7 , 43 , 15 , 16 ] . While state-of-the-art SLAM systems [ 3 , 41 ] achieve robust and scale-accurate camera tracking leveraging both visual and inertial measurements , dense and complete 3D reconstruction of largescale environments from monocular video remains a very challenging problem – particularly for interactive settings . Simultaneously , notable progress has been made on multi-view depth estimation , estimating depth from pairs of images by averaging features extracted from the images in a feature cost volume [ 42 , 17 , 19 , 38 , 13 ] . Unfortunately , averaging features across a full video sequence can lead to equal-weight treatment of each individual frame , despite some frames possibly containing less information in various regions ( e.g. , from motion blur , rolling shutter artifacts , very glancing or partial views of objects ) , making high-fidelity scene reconstruction challenging . Inspired by the recent advances in natural language processing ( NLP ) that leverage transformer-based models for sequence to sequence modelling [ 40 , 11 , 2 ] , we propose a transformer-based method that fuses a sequence of RGB input frames into a 3D representation of a scene at interactive rates . Key to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . our approach is a learned feature fusion of the video frames using a transformer-based architecture , which learns to attend to the most informative image features to reconstruct a local 3D region of the scene . A new observed RGB frame is encoded into a 2D feature map , and unprojected into a 3D volume , where our transformer learns a fused 3D feature for each location in the 3D volume from the image view features . This enables extraction of the most informative view features for each location in the 3D scene . The 3D features are fused in coarse-to-fine fashion , providing both improved reconstruction performance as well as interactive runtime . These features are then decoded into high-resolution scene geometry with an MLP-based surface occupancy prediction . In summary , our main contributions to achieve robust and accurate scene reconstructions are : • Learned multi-view feature fusion in the temporal domain using a transformer network that attends to only the most informative features of the image views for reconstructing each location in a scene . • A coarse-to-fine hierarchy of our transformer-based feature fusion that enables an online reconstruction approach running at interactive frame-rates . 2 Related Work . Multi-view depth estimation . Estimating depth from multi-view image observations has been long-studied in computer vision . COLMAP [ 37 ] introduced a patch matching based approach which achieves impressive accuracy and remains established as one of the most popular methods for multiview stereo . While COLMAP offers robust depth estimation for distinctive features in images , the patch matching struggles to densely reconstruct areas without many distinctive color features , such as floor and walls . Recently , learning-based approaches that build data-driven priors from large-scale datasets have improved depth estimation in these challenging scenarios . Some proposed methods rely only on a 2D network with multiple images concatenated as input [ 42 ] . Several recent approaches instead build a shared 3D feature cost volume in reference camera space using feature averaging [ 13 , 17 , 19 , 25 , 26 ] . These approaches estimate the reference frame ’ s depth within a local window of frames , but some also propagate information from previously estimated depth maps by using probabilistic filtering [ 25 ] , a Gaussian process [ 17 ] , or an LSTM bottleneck layer [ 13 ] . Such multi-view depth estimation approaches predict single-view depth maps , which must be fused together to construct a geometric 3D representation of the observed scene . 3D reconstruction from monocular RGB input . Multi-view depth estimation approaches can be combined with depth fusion approaches , such as volumetric fusion [ 6 ] , to obtain a volumetric reconstruction of the observed scene . MonoFusion [ 33 ] is one of the first methods using depth estimate from a real-time variant of PatchMatch stereo [ 1 ] . However , fusing noisy depth estimates causes artifacts in the 3D reconstruction , which lead to the development of recent approaches that directly predict the 3D surface reconstruction instead of per-frame depth estimates . One of the first approaches to predict 3D surface occupancy from two input RGB images is SurfaceNet [ 20 ] , which converts volumetrically averaged colors into 3D surface occupancies using a 3D convolutional network . Atlas [ 28 ] extends this approach to a multi-view setting , while also leveraging learned features instead of colors . Recently , NeuralRecon [ 39 ] proposed a real-time 3D reconstruction framework , adding GRU units distributed in 3D to fuse reconstructions from different local windows of frames . Our approach also fuses together learned features from RGB frame input in an online fashion , but our transformer-based multi-view feature fusion enables relying only on the most informative features from the observed frames for a particular spatial location in the reconstructed scene , producing more accurate 3D reconstructions . Transformers in computer vision . The transformer architecture [ 40 ] has achieved profound impact in many computer vision tasks in addition to its natural language processing origins . For a detailed survey , we refer the reader to [ 22 ] . In computer vision , transformers have been leveraged successfully for tasks such as object detection [ 4 ] , video classification [ 44 ] , image classification [ 12 ] , image generation [ 30 ] , and human reconstruction [ 45 ] . In this work , we propose transformer-based feature fusion for 3D scene reconstruction from a monocular video . Given a sequence of observed RGB frames , our approach learns to attend to the most informative features from each image to predict a dense occupancy field . 3 End-to-end 3D Reconstruction using Transformers . Given a set of N RGB images Ii ∈ RW×H×3 of a scene with corresponding camera intrinsic parameters Ki ∈ R3×3 and extrinsic poses Pi ∈ R4×4 , our method reconstructs the scene geometry by predicting occupancy values o ∈ [ 0 , 1 ] for every 3D point in the scene . Fig . 2 shows an overview of our approach . Each input image Ii is processed by a 2D convolutional encoder Θ , extracting coarse and fine image features ( Φci and Φ f i , respectively ) : Θ : Ii ∈ RW×H×3 7→ ( Φci , Φ f i ) From these 2D image features , we construct a 3D feature grid in world space . To this end , we regularly sample grid points in 3D at a coarse resolution of every vc = 30 cm and a fine resolution of vf = 10 cm . For these coarse and fine sample points , we query corresponding 2D features in all N images and predict fused coarse ψc and fine 3D features ψf using transformer networks [ 40 ] : Tc : ( Φc1 , . . . , ΦcN ) 7→ ( ψc , wc ) Tf : ( Φf1 , . . . , Φ f N ) 7→ ( ψ f , wf ) Note that we also store the intermediate attention weights wc and wf of the first transformer layers for efficient view selection , which is explained in Sec . 3.4 . To further improve the features in the 3D spatial domain , we apply 3D convolutional networks Cc and Cf , at the coarse and fine level , respectively : Cc : { ψc } C×C×C 7→ { ψ̃c } C×C×C Cf : { ( ψ̃c , ψf ) } F×F×F 7→ { ψ̃f } F×F×F Finally , to predict the scene geometry occupancy for a point p ∈ R3 , the coarse ψ̃c and fine features ψ̃f are trilinearly interpolated and a multi-layer perceptron S maps these features to occupancies : S : ( ψ̃c , ψ̃f ) 7→ o ∈ [ 0 , 1 ] This extraction of surface occupancies is inspired by convolutional occupancy networks [ 32 ] and IFNets [ 5 ] . From this occupancy field we extract a surface mesh with Marching cubes [ 27 ] . Note that in addition to surface occupancy , we also predict occupancy masks for near-surface locations at the coarse and fine levels . These masks are used for coarse-to-fine surface filtering ( see Sec . 3.2 ) , which improves reconstruction performance with a focus on the surface geometry prediction and enables interactive runtime . We train our approach in end-to-end fashion by supervising the surface occupancy predictions using the following loss : L = Lc + Lf + Lo , where Lc and Lf denote binary cross-entropy ( BCE ) losses on occupancy mask predictions for near-surface locations at the coarse and fine levels , respectively ( see Sec . 3.2 ) , and Lo denotes a BCE loss for surface occupancy prediction ( see Sec . 3.3 ) . 3.1 Learning Temporal Feature Fusion via Transformers . For a spatial location p ∈ R3 in the scene reconstruction , we learn to fuse coarse ψc and fine level features ψf from the N coarse and fine feature images ( Φci and Φ f i , respectively ) , which are extracted by the 2D encoder Θ . Specifically , we train two instances of a transformer model , one for fusing coarse-level features ψc and one for fusing fine-level features ψf . Both transformers Tc and Tf share the same architecture . Thus , for simplicity , we omit the coarse and fine notation in the following . Our transformer model T is independently applied to each sample point in world space . For a point p , the transformer network takes a series of 2D features φi as input that are bilinearly sampled from the feature maps Φi at the corresponding projective image location . The projective image location is computed via a full-perspective projection Πi ( p ) = π ( Ki ( Rip + ti ) ) , assuming known camera intrinsics Ki and extrinsics Pi = ( Ri , ti ) . To inform the transformer about invalid features ( i.e. , a sample point is projected outside an image ) , we also provide the pixel validity vi ∈ { 0 , 1 } as input . In addition to these 2D features φi , we concatenate the projected depth di = ( Rip + ti ) z , and the viewing ray ri = ( p− ci ) /||p− ci||2 to the input ( ci ∈ R3 denoting the camera center of view i ) . These input features are converted to an embedding vector θi ∈ RD using a linear layer θi = FCN ( φi , di , vi , ri ) , before feeding it into the transformer network that then predicts a fused feature ψ ∈ RD : T : ( θ1 , . . . , θN ) 7→ ( ψ , w ) As described above , w denotes the attention values of the initial attention layer , which are used for view selection to speed-up fusion ( see Sec . 3.4 ) . Transformer architecture . We followed [ 12 ] when designing the transformer architecture T . It consists of 8 modules of feed-forward and attention layers , using multi-head attention with 4 attention heads and embedding dimension D = 256 . Feed-forward layers process the temporal inputs independently , and contain ReLU activation , linear layers with residual connection , and layer norm . The model returns both fused feature ψ ∈ RD and attention weights w ∈ RN over all temporal inputs from the initial attention layer that are later used for selecting which views to maintain over longer sequences of input image views .

Summary: This paper presents an RGB 3D reconstruction algorithm with monocular RGB input, using transformer network. The transformer architecture is utilized for temporal feature fusion from multi-view images. Coarse-to-fine hierarchical scheme was designed for efficient surface detection and real-time reconstruction. Also, attention weights from the transformer is used in order to select the most dominant (feature-abundant) frames so that more interactive scene reconstruction can be available. By doing so, the proposed algorithm shows comparable performance on the benchmark experiment, compared to CNN-based previous methods.