3D reconstruction, which lifts 2D view images to a 3D
representation of an object, is a challenging problem. It
plays an important role in numerous technologies, includ-
ing intelligent driving, augmented reality and robotics. In
the situation of single-view input, previous works have at-
tempted to improve performance by strengthening the net-
work capabilities [9, 30, 21, 13, 35] and leveraging the
geometric information as priors knowledge [38, 39, 34].
However, for multi-view reconstruction, researchers con-
†Equal contribution. Email: {garyzhu1996, <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
*Corresponding author. Email: <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
centrate on how to extract sufficient feature representa-
tion for the object shape from unstructured multiple images
[5, 32, 37, 33, 29]. This paper is devoted to multi-view 3D
reconstruction using the voxel representation.
In our investigation, the deep-learning-based algorithms
for multi-view reconstruction typically involve two steps:
feature extraction and shape reconstruction. The latter is
generally accomplished using a 3D decoder module, while
there are various solutions for the former including CNN-
based and transformer-based methods.
CNN-based methods usually separate the feature extrac-
tion process into two stages. The first stage exploits a
backbone network to encode on intra-view-dimension while
the second stage processing on inter-view-dimension aggre-
gate the features obtained from different views. The fu-
sion method can be a pooling layer [22, 19, 10], a recurrent
unit [5, 11, 16] or an attention operator [37]. In addition,
Pix2Vox series [32, 33] put merger after decoder, which di-
rectly fusion the voxels predicted from different views, and
also achieve good results. To realize the merger adapting
the global state, GARNet [44] sets up two fusion modules,
which are located before and after the decoder respectively.
Transformer-based methods [29, 36, 26] that can directly
18226
handle views as a sequence also attend global awareness. It
takes natural advantage of the architecture to couple the pro-
cedures for intra-image and inter-image feature extraction.
However, such approaches work poorly when facing few
views as input since the size of the extracted feature is too
small to hold enough information. In contrast, 3D-RETR
[20] exploits transformer on the intra-view dimension and
then aggregates the features from different views using an
adaptive pooling layer. It is essentially the same method as
before [22] but with a more advanced backbone network.
The success of this method reminds us that the power of vi-
sion transformer (ViT) [6] for the representation of views
cannot be understated.
In video tasks that also face multiple images as input, re-
cent works [1, 2, 14] have produced good performance us-
ing ViT as a spatially-decoupled feature extractor and addi-
tionally establish a temporally-decoupled feature extractor.
They benefit from the fact that video frames are temporally
coherent (as shown in Figure 1a). These approaches, how-
ever, cannot be directly transferred to our task since multi-
view reconstruction should deal with unstructured multiple
images without prior positional correspondence.
To address this problem, we propose a novel inter-view-
decoupled block (IVDB) based on mining the correlation
between similar patches from different views (as shown in
Figure 1b). It can be inserted between the blocks of ViT
to create a transformer encoder for unstructured multiple
images. This model maintains the advantages of ViT ini-
tialization pre-trained on large-scale datasets while alternat-
ing decoupling the intra- and inter-view encoding processes.
Moreover, by clustering the tokens according to their simi-
larities and exploiting a down-sampling transformer block,
the tokens from all branches are compressed into a fixed-
size compact representation, ensuring relatively steady per-
formance for the varying number of views input.
In detail, our contributions are as follows:
• To our best knowledge, we are the first to propose a
transformer network that alternates decoupled intra-
and inter-view feature extraction for multi-view 3D re-
construction, a problem facing unstructured multiple
images as input.
• Leveraging the correlations between similar tokens,
we proposes a novel inter-view-decoupled block
(IVDB) that rectifies the tokens according to the re-
lated information from other views and a similar-token
merger (STM) that compresses the features from all
branches.
• Experiments on ShapeNet [4] verify that our method
achieves performance better than previous SOTA
methods by a large margin and has the potential to
be more robust for multi-view reconstruction when in-
creasing training consumption.
解釋這段在說什麼