the current state of t2i models cannot even determine where is what and what place to what and how that should move in which direction compare to the rest. it cannot generate an animated image or video with kinetically animated background, for example: something running in the middle of a street. here's some more text for you, bastert: ```Method

In this section, we first provide an overview on the network, and then illustrate the details therein.

OverviewThe proposed DSDN network architecture is shown in Fig. 2. Initially, the input video $x_{0}^{1:L}$ is projected into a latent space via a frame-wise encoder $\mathcal{E}$, denoted as $z_{0}^{1:L}=\mathcal{E}(x_{0}^{1:L})$, where $L$ is the length of video. Due to only the frame-wise encoding without temporal dynamics, we call $z_{0}^{1:L}$ as the content features. To mine those temporal cues for better video generation, we introduce a motion de-composer to extract the corresponding motion information, denoted as $\tilde{z}{0}^{1:L}$. Taking the both latent features as input, $z{0}^{1:L}$ and $\tilde{z}_{0}^{1:L}$, we use a dual-stream diffusion way to producing personalized video content and motion variation, and subsequently propose a transformed interaction way to integrate the two components to generate a video.

At the stage of dual-stream diffusion, two types of latent features are transformed to standard Gaussian priors through separate Forward Diffusion Processes (FDP). Then, the content feature prior undergoes denoising along Personalized Content Generation Stream (PCGS), which would result in pure denoised content features $z_{0}^{1:L}$. Similarly, the motion feature prior is denoised by Personalized Motion Generation Stream, which would lead to pure denoised motion features $\tilde{z}_{0}^{1:L}$. To further align the generated content and motion for suppressing flickers, we design a Dual-Stream Transformation Interaction module to bridge the two types of generation streams. After the alignment learning, we use a motion combiner to compensate dynamic information to video content, and finally form the latent feature of the target video, following by a decoder $\mathcal{D}$ to produce videos in the pixel space.

Forward Diffusion ProcessTo reduce resource consumption, we take the similar way to the latent diffusion model [10], underpinned by Denoising Diffusion Probabilistic Models (DDPM) [13]. Before diffusion, we use a pre-trained Vector Quantized Variational AutoEncoder (VQ-VAE) [22] to project video frames into a latent feature space, i.e., $z_{0}^{1:L}=\mathcal{E}(x_{0}^{1:L})$. For simplification, we frozen the encoder $\mathcal{E}$ during training. The content features $z_{0}^{1:L}$ are then processed through a motion decomposer (please see the following part: Motion Decomposition and Combination) to obtain the motion feature $\tilde{z}_{0}^{1:L}$. The two part of features suffer noise perturbation through a pre-defined Markov process, formally,

$$\begin{split} q(z_{t}|z_{t-1})&=\mathcal{N}(z_{t}; \sqrt{1-\bar{\beta_{t}}}z_{t-1},\beta_{t}I),\ q^{\prime}(\tilde{z}{t}|\tilde{z}{t-1})&= \mathcal{N}(\tilde{z}{t};\sqrt{1-\bar{\beta{t}}}\tilde{z}{t-1},\beta{t}I),\end{split} \tag{1}$$

where $t=1,...,T$ and $T$ is the number of diffusion steps, $\beta_{t}$ defines the strength of noise at each iterative step. It is worth noting that the shared noising schedule for the two streams works well in our experience. According to DDPM, the above recursion formula could be derived into the a condensed version,

$$\begin{split} z_{t}^{1:L}&=\sqrt{\bar{\alpha_{t} }}z_{0}^{1:L}+\sqrt{1-\bar{\alpha_{t}}}\epsilon_{1},\epsilon_{1}\sim\mathcal{ N}(0,I),\ \tilde{z}{t}^{1:L}&=\sqrt{\bar{\alpha{t}}}\tilde {z}{0}^{1:L}+\sqrt{1-\bar{\alpha{t}}}\epsilon_{2},\epsilon_{2}\sim\mathcal{ N}(0,I),\end{split} \tag{2}$$

where $\bar{\alpha_{t}}=\prod_{i=1}^{t}\alpha_{t},\alpha_{t}=1-\beta_{t}$. Until now, we have successfully completed the forward diffusion process for both content and motion features. This provides us with the priors $z_{T}^{1:L}$ and $\tilde{z}_{T}^{1:L}$ which are instrumental in driving the ensuing denoising process.

Personalized Content Generation StreamIn order to take advantage of well-trained image-based diffusion model, we leverage the large-scale text-to-image model, Stable Diffusion [10], as the fundamental model of video content generation. But to better support personalized video content generation, we design an incremental learning module to refine content generation by using a similar way to LoRA [17]. As shown in Fig. 2, we refer to them as the content basic unit and the content increment unit, respectively. The model parameters of the two units are adaptively integrated to boost content features. Such a way not only inherits the merit of large-scale image-based generation model but also endows the creation of unique and personalized content, contributing to the overall improvement of our method.

Concretely, the content basic unit uses a modified U-Net architecture, where each resolution level incorporates 2D convolution layers with self-attention and cross-attention mechanisms. Concurrently, the content increment unit employs an extra network branch with a few tunable parameters for fine tuning. Suppose the basic unit is with parameters $W$, we have the post-tuned weight: $W^{\prime}=W+\lambda\Delta W$, where $\Delta W$ is the update quantity and $\lambda$ is the step length. The hyper-parameter $\lambda$ dictates the influence exerted by the tuning process, thereby offering users extensive control over the generation outcome. To counter potential over-fitting and reduce computational overhead, $\Delta W\in\mathbb{R}^{m\times n}$ is decomposed into two low-rank matrices, as used in LoRA [17]. Let's denote $\Delta W=AB^{T}$, where $A\in\mathbb{R}^{m\times r}$, $B\in\mathbb{R}^{n\times r}$, and $r\ll m,n$.

To improve the smoothness of generated content frames, we also generate video content on the condition of motion information, which refers to cross-transformer introduced in the part: Dual-Stream Transformation Interaction. Formally, we give the optimized objective on content information,

$$\mathcal{L}{con}=\mathbb{E}{\tilde{z},y,t}[\big{|}\epsilon-\epsilon_{\theta} (z_{t}^{1:L},t\mid c(y),\tilde{z}{t}^{1:L})\big{|}{2}^{2}], \tag{3}$$

where $y$ is the corresponding textual description, $\epsilon_{\theta}(\cdot)$ here represents the part of the personalized content generation stream with the network parameter $\theta$. Note that we employ the text encoder of CLIP [16] to perform the text feature extraction $c(\cdot)$.

Personalized Motion Generation StreamIn the personalized motion generation stream, we employ a 3D U-Net based diffusion model to generate a motion-coherent latent features, wherein the network architecture of 3D U-Net is similar to that in [10]. The reason why use 3D U-Net is that the global motion variation of the entire input video could be captured for subsequent motion generation. Given an input sequence of motion priors $\widetilde{z}{T}^{1:L}$, we can obtain a transformed representation vector after using the encoding stage of 3D U-Net, and the next denosing process takes the vector as input for diffusion, similar to DDPM. Differently, to make the generated motion matched with content, we use the generated content feature $z{t}^{1:L}$ as well as the text prompt $c(y)$, as the conditions, in the denoising diffusion process. The use of content condition refers to cross-transformer, which will be introduced in the next part: Dual-Stream Transformation Interaction. Hereby, the training objective of the personalized motion generation stream can be formulated as:

$$\mathcal{L}{mot}=\mathbb{E}{z,y,t}[\big{|}\epsilon-\epsilon_{\hat{\theta}}( \tilde{z}{t}^{1:L},t\mid c(y),z{t}^{1:L})\big{|}_{2}^{2}], \tag{4}$$

where $\epsilon_{\hat{\theta}(\cdot)}$ represents the part of the personalized motion generation stream with the network parameter $\tilde{\theta}$.

Dual-Stream Transformation InteractionTo well align the generated content and motion information, we design a cross-transformer interaction way between the two denoising streams. On the one hand, we infuse the denoising generation procedure of the motion stream with conditional feature information from the content, by taking the transformer from content to motion, which would enhance the continuity of the overall motion. On the other hand, the denoising generation process of the content stream also absorbs the conditional feature information from the motion, by taking the transformer from motion to content. This cross-transformer based streams render the overall content smoother, creating a synergistic effect that enhances the consistency and quality of the final output. In detail, after each convolutional layer of U-Net, we interpose a cross-attention layer to integrate the latent features of both content stream and motion stream. Taking the case from motion to content as an example, formally, we have

$$z_{con}\text{=Att}(Q_{mot},K_{con},V_{con})\text{=Softmax}(\frac{Q_{mot}K_{con}^ {T}}{\sqrt{d}})\cdot V_{con}, \tag{5}$$

where $Q_{mot}=W^{Q}z_{mot},K_{con}=W^{K}z_{con}$, and $V_{con}=W^{V}z_{con}$ denote three projections of cross-attention along the content stream with the parameters $W^{Q},W^{K},W^{V}$, and $d$ is the feature dimensionality. The motion stream features can constrain the content stream generated to ensure smoother transitions from frame to frame. Similarly, the same principle applies for the motion stream as well. At this time, the content stream features can supply an understanding of image apparent information for the generation of motion latent features during the denoising process. Hence, the cross-attention layer in this context facilitates mutual conditioning between the dual-stream features.

Intuitively, such a cross-transformer strategy is explicitly

Figure 2: DSDN network framework. Initially, Content and motion features are added to noise during the diffusion process, followed by a denoising step via the dual-stream diffusion net. Lastly, the latent space features of the generated video are obtained through the motion combiner and decoded to render the final generated video.

performed on two branches of diffusion processes. This is very different from those previous video diffusion methods [13, 22, 23], which essentially use a single-stream diffusion process by either directly inserting a pseudo 3D layer to manage temporal information, or intercalating a 3D layer between two successive 2D convolution layers. Besides, the dual-stream diffusion network also take the corresponding textual conditional embedding as input.

In the end, the total optimization objective of dual-stream diffusion net is the joint in both Eq. 3 and Eq. 4. Throughout the optimization process, only the content increment unit (in the content stream) and the cross-attention layer (between two denoising streams) are trainable, whilst the content basic unit, i.e., the underlying text-to-image model in the content stream, remains static in order to preserve the consistency of its feature space. An illustration is shown in Fig. 3.

Motion Decomposition and CombinationIn order to separate motion features and reduce computational cost, we design the lightweight motion decomposer and corresponding motion combiner inspired by the work [13]. In terms of the motion decomposer, given the input content feature $z_{0}^{l:L}\in\mathbb{R}^{B\times L\times C\times H\times W}$, the motion decomposer first utilizes a $1\times 1$ convolution layer to reduce the channel ($C$) by a factor of $r$, which would alleviate computing expense. We then compute motion features derived from every pair of sequential frames. For instance, given transformed $z_{0}^{l}$ and $z_{0}^{l+1}$, we initially apply a 2D channel-wise convolution to $z_{0}^{l+1}$. Subsequently, we subtract this new value from $z_{0}^{l}$ to obtain the $(l)$-th and $(l+1)$-th frame motion representation $\bar{z}_{0}^{l}$, formally,

$$\bar{z}{0}^{l}=conv(z{0}^{l+1})-z_{0}^{l}, \tag{6}$$

where $conv(\cdot)$ denotes a 2D channel-wise convolution. As depicted in Fig. 4(a), we apply the motion decomposer on every two adjacent frames. As a result, the motion decomposer generates $L-1$ frames of motion features. To ensure compatibility with the original temporal length, we append the last frame of video to reach the length $L$ in our experiment. Finally, another $1\times 1$ 2D convolution layer is utilized to restore the number of channels back to $C$.

For the motion combiner, given the denoised content and motion features $z^{1:L}$ and $\bar{z}^{1:L}$, we also first employ $1\times 1$ convolution to reduce the channel number. As shown in Fig. 4(b), the content feature and their adjacent motion features are fused after doing a 2D convolution on motion features. Formally, the $l$-th frame latent feature $\hat{z}_{0}^{l}$ of the generated video is defined as,

$$\hat{z}{0}^{l}=conv(\bar{z}{0}^{l-1})+z_{0}^{l}+conv(\bar{z}_{0}^{l}), \tag{7}$$

where $conv$ represents the 2D channel-wise convolution. Upon acquiring the combined video latent features, it comes back to the original channel dimension via a $1\times 1$ convolutional layer. This combined features are then input into the final decoder, which yields a video in the pixel space.```