能否帮我解释一下以下这个方法的训练策略、模型架构和普通的lrm有什么区别？
3.4 ControLRM
In this section, we delve into the specific modules of
ControLRM. The design of the conditional generator was
detailed in Fig. 2 in Section 3.4.1. Depending on the chosen backbone for the conditional generator, ControLRM
manifests in two variants: 1) ControLRM-T featuring a
transformer-based conditional generator (Section 3.4.2); 2)
ControLRM-D integrating a diffusion-based conditional
generator (Section 3.4.3). Subsequently, we present the
condition-to-triplane transformer decoder in Section 3.4.6.
The training objectives encompassing adversarial loss, clip
loss, and rendering loss are expounded upon in Section
3.4.7.
3.4.1 Design of Conditional Generator
As depicted in Fig. 2, the conditional generator utilizes the
2D condition and the text embedding of CLIP [56] as input
to produce the 2D latents required for subsequent procedures. A naive design of this generator is a transformerbased backbone with cross-attention mechanism between
the feature sequence extracted from condition image and
the text feature. However, this design with only the crossattention mechanism fails to generate a regular results but
yielding meaningless results in the experiments. A similar
issue was observed in [57], indicating that the main reason
for this optimization failure stems from the notable disparity
between the 2D renderings and the ground truth images. As
noted by [58], the optimization gradient becomes unreliable
when the generated distribution and the target distribution
are disjoint. In contrast, the backward gradients to the 2D
latents in our model must traverse a series of modules,
including the condition encoder, triplane transformer, and
NeRF modules. This complexity of pathways may significantly impede the optimization process, consequently resulting in unexpected failures. A straightforward remedy
proposed in [57] involves the incorporation of randomness
(e.g., Gaussian noise) into the network architecture. By increasing the overlap between the rendered distribution and
the target distribution, the gradients during training become
more meaningful, promoting convergence. In summary, the
key considerations for designing the condition generator
in ControLRM are: 1) Incorporation of randomness for improved training outcomes. 2) Emphasis on the efficiency of
the generator for fast inference speed.
3.4.2 Transformer-based Conditional Generator
For ControLRM-T model, we have devised a lightweight
transformer-based generator, illustrated in Figure 3 (a).
Building upon the preceding discussion, we introduce randomness through a style injection module. Drawing inspiration from the original style injection concept in StyleGAN
[59], where style features and random noise are integrated
into the generator via Adaptive Instance Normalization
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 7
Fig. 3. The architecture of the 2D conditional generator in ControLRM.
(a) shows the transformer-based generator in ControLRM-T, and (b)
shows the diffusion-based generator in ControLRM-D.
(AdaIN), we adapt this approach by treating the text embedding as the style feature. This text embedding is concatenated with random Gaussian noise and passed through a 3-
layer MLP within our style injection module. The resulting
feature vector is then combined with the output of each
convolution layer to incorporate the text feature. In Figure
3 (a), the convolution blocks and transformer blocks are
stacked together, with residual connections applied to the
convolution blocks in a U-Net configuration.
3.4.3 Diffusion-based Conditional Generator
For the ControLRM-D model, we have intricately integrated LoRA adapters [60] into the original latent diffusion
model, incorporating small trainable weights. Leveraging
the inherent randomness within the diffusion model, and
aided by the pre-trained weights obtained from large-scale
datasets, we aim to address the discrepancy issue highlighted in Section 3.4.4. In addressing efficiency concerns,
we opt for the fast one-step diffusion model [61] as the foundational framework. Specifically, we initialize the Diffusionbased generator with the pre-trained weights of SD-Turbo
[62]. To form the 2D latents for subsequent procedures,
we concatenate the outputs of the last three layers of the
decoder depicted in Figure 3 (b).
3.4.4 Condition Encoder
In Figure 2 (a), the 2D latents are firstly interpolated to
match the resolution of the input condition image, and
then divided into the feature sequence {gi
|gi ∈ R
De }
Np
i
.
Similar to the feature sequence {hi}
Np
i
extracted from the
input image discussed in Sec. ??, De denotes the feature
dimension, while Np corresponds to the number of patches.
Within the condition encoder, the feature sequence {gi}
Np
i
is passed through a sequence of transformer layers, each
comprising a self-attention sub-layer and an MLP sub-layer.
g
self
i = Self(gi; gi) + gi (16)
g
out
i = MLP(g
self
i ) + g
self
i
(17)
where g
out
i
is the output feature.
To integrate the random sampling process, the output
g
out
i
of the final transformer layer is fed to another MLP to
regress the mean and variance results:
µgi
, σgi = MLP(g
out
i ) (18)
where µg is the mean feature and σg represents the
variance. Throughout training, the output feature sequence
{g˜i}
Np
i
is is stochastically sampled from a Gaussian distribution, where g˜i ∼ N (µgi
, σ2
gi
).
3.4.5 Auxiliary Decoder
To boost the performance, we further introduce an auxiliary
decoder for the 2D latents to enhance the training process.
The generated 2D latents from the conditional generator
(refer to Sections 3.4.2 and 3.4.3) are passed through a
lightweight three-layer convolutional neural network. The
resulting image xaux is combined with the 2D renderings
to compute the loss function for the generated images. The
inclusion of the auxiliary decoder offers direct guidance to
the 2D generator, aiding in overall network convergence.
3.4.6 Triplane Transformer Decoder
The condition-to-triplane decoder receives the condition feature sequence {g˜i}
Np
i
and the triplane feature sequence f
in
.
Analogous to the image-to-triplane decoder discussed in
Sec. 3.1, each transformer layer consists of a cross-attention
sub-layer, a self-attention sub-layer, and an MLP layer. The
input tokens for each sub-layer are influenced by the camera
features c˜. The operation of each transformer layer can be
described as follows:
f
cross-c
j = Cross-C(ModLN(f
in
j ); {g˜i}
Np
i
) + f
in
j
(19)
f
self
j = Self(ModLN(f
cross-c
j ); ModLN(f
cross-c
j )) + f
cross-c
j
(20)
f
out
j = MLP(ModLN(f
self
j )) + f
self