ECHOPulse: ECG Controlled Echocardio-gram Video Generation

## Abstract

Abstract Echocardiography (ECHO) is essential for cardiac assessments, but its video quality and interpretation heavily relies on manual expertise, leading to inconsistent results from clinical and portable devices. ECHO video generation offers a solution by improving automated monitoring through synthetic data and generating high-quality videos from routine health data. However, existing models often face high computational costs, slow inference, and rely on complex conditional prompts that require experts’ annotations. To address these challenges, we propose ECHOPulse, an ECG-conditioned ECHO video generation model. ECHOPulse introduces two key advancements: (1) it accelerates ECHO video generation by leveraging VQ-VAE tokenization and masked visual token modeling for fast decoding, and (2) it conditions on readily accessible ECG signals, which are highly coherent with ECHO videos, bypassing complex conditional prompts. To the best of our knowledge, this is the first work to use time-series prompts like ECG signals for ECHO video generation. ECHOPulse not only enables controllable synthetic ECHO data generation but also provides updated cardiac function information for disease monitoring and prediction beyond ECG alone. Evaluations on three public and private datasets demonstrate state-of-the-art performance in ECHO video generation across both qualitative and quantitative measures. Additionally, ECHOPulse can be easily generalized to other modality generation tasks, such as cardiac MRI, fMRI, and 3D CT generation. Demo can seen from https://github.com/levyisthebest/ECHOPulse_Prelease .

## 1 Introduction

Echocardiography (ECHO) is widely used for evaluating heart function and structure due to its real-time, non-invasive, and radiation-free nature (SF, ; Otto, ; Omar et al., ). Recent advancements in deep learning have shown promising performance in computer-aided cardiac diagnosis (Oktay et al., ; Smistad et al., ; Kim et al., ). However, echocardiography depends on experienced operators, leading to insufficient data for training robust models. Moreover, annotating ECHO data demands expert knowledge, making the process time-consuming and sometimes unavailable (Leclerc et al., ; Lu et al., ).

One promising avenue is the use of video generation models to synthesize ECHO videos. In general domains, text-to-video (T2V) models (Yu et al., ; Zhang et al., ; Wang et al., ; Villegas et al., ) have been successful in creating realistic videos from text prompts. However, applying these models to the medical field remains challenging due to the necessity for precise control over anatomical structures, which cannot vary freely as in general domain. Previous controllable ECHO generation frameworks (Ashrafian et al., ; Zhou et al., ; Yu et al., ) rely on carefully curated conditional prompts such as segmentation masks, clinical text, or optical flow, which only experts can provide and fail to incorporate time-series information. Moreover, these diffusion-based models suffer from high computational costs and slow inference. Therefore, a fast and controllable ECHO generation framework without the need for expert prior conditions is highly desirable (Reynaud et al., ).

Inspired by these motivations, we propose ECHOPulse, an ECHO video generation model conditioned on electrocardiogram (ECG) signals. ECG data are readily accessible in both clinical and outpatient settings (Al-Khatib et al., ), and even from personal wearable devices (Liu et al., ). Since ECGs are typically the first step in evaluating cardiac issues before performing further echocardiography (Al-Khatib et al., ; Del Torto et al., ), they serve as a natural and intrinsic prior for coherent ECHO video generation. By leveraging ECG signals, we can generate ECHO videos without the need for complex conditional prompts or expert annotations. To the best of our knowledge, this is the first work to utilize time-series prompt for ECHO video generation. In clinical scenarios where prior ECHO images exist, ECHOPulse can be co-conditioned on both ECG and prior imaging to provide updated cardiac function information, such as left ventricular ejection fraction (LVEF/EF) for cardiac disease monitoring and prediction (Ponikowski et al., ), enhancing computer-aided diagnosis beyond ECG alone.

Our main contributions are:

- •
We propose ECHOPulse, a fast and controllable ECHO video generation framework conditioned on ECG for the first time, which also shows potential to generalize to other modality generation tasks, such as cardiac MRI, fMRI, and 3D CT generation.
- •
By utilizing readily available ECG data for controllable and precise ECHO generation personalized for each patient, we bypass the need for paired expert annotations, showing potential for scaling to larger video datasets.
- •
Evaluations on two public and one private dataset demonstrate that our framework achieves SOTA performance in both quantitative and qualitative metrics.
- •
We collect a synthetic ECHO video dataset consisting of ECHO videos paired with ECG signals and segmentation masks, which we plan to release to enable researchers to evaluate the practicality and reliability of synthetic ECHO video generation.

## 2 RELATED WORK

### 2.1 Visual Tokenization

Visual Tokenization is crucial for converting images or videos into discrete tokens that can be processed by language models. One of the foundational approaches to visual tokenization is VQ-VAE (Van Den Oord & Vinyals, ), which uses a convolutional neural network (CNN) to encode visual data, followed by a vector quantization (VQ) step. This method assigns each embedding to the closest entry in a codebook, converting continuous embeddings into discrete tokens. Video tokenization presents greater challenges, and VQGAN has been adapted to address these difficulties (Villegas et al., ; Yu et al., ). The current state-of-the-art in video tokenization is MAGVIT2 (Yu et al., ), which employs an enhanced 3D architecture, leverages an inflation technique for initialization based on image pre-training, and incorporates robust training losses to improve performance.

### 2.2 Video Generation

Recent video generation advances have increasingly focused on discrete representations to manage the high dimensionality of video data. Models like VQ-VAE (Van Den Oord & Vinyals, ) convert continuous video frames into discrete vector indices, reducing data complexity and enabling more efficient sequence modeling. Phenaki (Villegas et al., ) uses a causal Transformer with discrete temporal embeddings for variable-length video generation, while Masked Generative Transformer (Chang et al., ) leverages discrete techniques for sparse spatiotemporal relationships. These studies highlight the advantages of discrete representations in balancing complexity and efficiency, especially for long sequences. Moreover, autoregressive models with discrete tokens (Yu et al., ) generate diverse, high-quality content, and discrete diffusion models (Austin et al., ) extend the diffusion process to the discrete symbol space, enhancing scalability and long-range video modeling.

### 2.3 ECHO Video Generation

Ultrasound imaging is a non-invasive, real-time, and portable medical imaging modality that plays a crucial role in clinical diagnostics. However, the quality and interpretation of ultrasound images heavily depend on the operator’s expertise, and obtaining high-quality ultrasound datasets is challenging. Consequently, the generation of ultrasound images has attracted considerable attention in recent years. Recent works, such as HeartBeat (Zhou et al., ), have leveraged deep learning techniques like diffusion models (Ho et al., ) to achieve controllable synthesis of echocardiograms. By incorporating multimodal conditions—such as cardiac anatomical annotations and functional indices—and employing diffusion models, these methods can generate realistic ultrasound videos with specific cardiac morphology and function. This approach enriches training datasets and enhances algorithm robustness. Reynaud et al. () introduced the EchoNet-Dynamic model, which utilizes spatiotemporal three-dimensional convolutional neural networks to segment the left ventricle frame by frame and estimate the ejection fraction (EF), outperforming manual measurements. Such advancements hold promise for assisting clinicians in more precise and efficient cardiac function assessments. These techniques typically use images as conditional inputs to generate corresponding ultrasound images. Recent studies have also explored generating ultrasound video sequences from single-frame images (Reynaud et al., ), offering new possibilities for dynamic ultrasound applications like cardiac function evaluation. However, these studies are conditioned on carefully curated conditional prompts such as segmentation masks and clinical text but do not leverage time-series ECG data. In contrast, our work is the first to efficiently generate controllable ECHO videos conditioned on ECG data.

## 3 METHODS

ECHOPulse, as shown in Figure 1, consists of three key components: (1) a video tokenization model that encodes videos into temporal token sequences, (2) a masked generative transformer that learns the latent alignment between video tokens and ECG signal, and (3) a progressive video generation pipeline that enables fast, unlimited video generation. The video tokenization and transformer components are trained separately.

Figure: Figure 1: The pipeline of the ECHOPulse. ECHOPulse contains a two-step training procedure. a) The first step trains the video tokenizer on the natural video dataset first and fine-tunes it on the public ECHO video dataset. b) The second step trains the transformer via the input token produced by ECG foundation model and video tokenizer, pretrained in the first step. Followed by the video generation procedure (c), ECHOPulse accepts input with or without a conditional image. The empty tokens will be reconstructed via the frozen transformer, trained in the second step, through the guidance of ECG siganls. ECHOPulse is capable of generating continuous long videos by sequentially shifting the token sequence and integrating new ECG inputs.
Refer to caption: extracted/5921014/Images/pipeline_EchoPulse.png

### 3.1 Video Tokenization

The key to generating videos from ECG signals lies in capturing the temporal correlation between the video and ECG data. Unlike text (Touvron et al., ), ECG is a continuous temporal signal with variable sequence lengths. As shown in Figure 2, each ECG phase should correspond to different stages of the ECHO. Thus, encoding the entire video into a single embedding risks disrupting temporal alignment between ECG and video, also complicating autoregressive video generation and reducing generalizability. To temporally align ECHO video with the various phases of the ECG and support autoregressive video generation, an effective approach is to discretize the video into tokens while splitting the ECG into patches and embedding them for cross-modal alignment. Drawing inspiration from ViViT (Arnab et al., ) and C-ViViT (Villegas et al., ), we developed ECHOPulse’s video tokenization model to meet these specific requirements.

##### Model architecture:

The encoding starts with a video sequence of $T+1$ frames, each with resolution $H\times W$ and $C$ channels, denoted as $\mathbf{X}\in\mathbb{R}^{(T+1)\times H\times W\times C}$. The sequence is compressed into tokens of size $(T^{\prime}+1)\times H^{\prime}\times W^{\prime}$, where the first frame is tokenized independently, and subsequent frames are spatio-temporal tokens that autoregressive depending on the previous. Non-overlapping patches of size $P_{h}\times P_{w}\times C$ (for images) and $P_{t}\times P_{h}\times P_{w}\times C$ (for video) are extracted, resulting in $T^{\prime}=\frac{T}{P_{t}}$, $H^{\prime}=\frac{H}{P_{h}}$, and $W^{\prime}=\frac{W}{P_{w}}$. Each patch is flattened and projected into a $D$-dimensional space, forming a tensor of shape $(T^{\prime}+1)\times(H^{\prime}\times W^{\prime})\times D$. Spatial transformers are applied with self attention, followed by temporal transformers with causal attention. The patch embeddings $\mathbf{Z}\in\mathbb{R}^{(T^{\prime}+1)\times H^{\prime}\times W^{\prime}
\times D}$ are then quantized into codewords $\mathbf{C_{z}}$ via vector quantization. The decoder reverses the process by transforming quantized tokens into embeddings, applying temporal and spatial transformers, and mapping the tokens back to pixel space via linear projection to reconstruct the frames $\hat{\mathbf{X}}$.

Additionally, Our framework incorporates the Lookup-Free Quantization (LFQ) method, inspired by the advancements in MAGVIT2 (Yu et al., ), to streamline the quantization process. Unlike traditional vector quantization that relies on an explicit codebook lookup, LFQ parameterizes the quantization operation directly. Moreover, due to the lack of high-quality medical video datasets comparable in scale to natural video datasets, we introduce LoRA (Low-Rank Adaptation) (Hu et al., ) to enhance the learning efficiency and generalizability for medical video generation. This also lays the foundation for offering personalized video generation services for users or patients in future applications.

##### Loss design:

During video tokenization training phrase, the total loss function $\mathcal{L}_{\text{total}}$ is formulated as:

$$ $\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{recon}}+\mathcal{L}_{\text{ percep}}+\mathcal{L}_{\text{VQ}}+\lambda_{\text{adaptive}}\cdot\mathcal{L}_{ \text{GAN}},$ (1) $$

where $\mathcal{L}_{\text{recon}}$ is the reconstruction loss, $\mathcal{L}_{\text{percep}}$ is the perceptual loss, $\mathcal{L}_{\text{VQ}}$ represents the vector quantization loss, and $\mathcal{L}_{\text{GAN}}$ is the generator adversarial loss. The parameter $\lambda_{\text{adaptive}}$ dynamically adjusts the contribution of the adversarial loss relative to the perceptual loss during training.

Specifically, the reconstruction loss $\mathcal{L}_{\text{recon}}$ is computed using the Mean Squared Error (MSE) to ensure the reconstructed video $\hat{\mathbf{V}}$ to closely match the original video $\mathbf{V}$ at the pixel level:

$$ $\mathcal{L}_{\text{recon}}=\frac{1}{N}\left\|\mathbf{V}-\hat{\mathbf{V}}\right \|_{2}^{2},$ (2) $$

where $N$ is the total number of pixels across all frames, $\mathbf{V}_{i}$ and $\hat{\mathbf{V}}_{i}$ are the $i$-th pixel of the original video and the reconstructed video, respectively.

To improve the perceptual quality of the reconstructed video, perceptual loss $\mathcal{L}_{\text{percep}}$ is employed to measure the similarity (Johnson et al., ) between the original and reconstructed videos in the feature space of a pre-trained VGG16 (Qassim et al., ) network $\phi$. By extracting high-level features from randomly selected video frames $\mathbf{V}_{f_{j}}$ and $\hat{\mathbf{V}}_{f_{j}}$, the perceptual loss is defined as:

$$ $\mathcal{L}_{\text{percep}}=\frac{1}{M}\sum_{j=1}^{M}\left\|\phi(\mathbf{V}_{f _{j}})-\phi(\hat{\mathbf{V}}_{f_{j}})\right\|_{2}^{2},$ (3) $$

where $M$ is the number of feature elements, and $\phi(\cdot)$ denotes the VGG feature extraction operation.

The vector quantization loss (van den Oord et al., ) $\mathcal{L}_{\text{VQ}}$ regularizes the encoder outputs by ensuring that they are close to their nearest codebook entries, encouraging efficient and discrete representation learning. The VQ loss is defined as:

$$ $\mathcal{L}_{\text{VQ}}=\beta\cdot\left\|\text{sg}[\mathbf{z}_{e}]-\mathbf{e} \right\|_{2}^{2},$ (4) $$

where $\mathbf{z}_{e}$ is the encoder output, $\mathbf{e}$ is the closest codebook entry, and $\beta$ is a weighting parameter. The term $\text{sg}[\cdot]$ represents the stop-gradient operation, which ensures that the encoder is encouraged to commit to a discrete codebook entry without backpropagating gradients through this operation.

The generator adversarial loss $\mathcal{L}_{\text{GAN}}$ drives the model to produce videos that are indistinguishable from real videos, as evaluated by a discriminator $D$. We employ the hinge loss formulation for stable adversarial training:

$$ $\mathcal{L}_{\text{GAN}}=-\mathbb{E}_{\hat{\mathbf{V}}\sim P_{G}}\left[D(\hat{ \mathbf{H}})\right],$ (5) $$

where $\hat{\mathbf{H}}$ is a video generated by the model, sampled from the generator’s distribution $P_{G}$, and $D(\hat{\mathbf{H}})$ is the discriminator’s output for the generated video.

To balance the adversarial loss $\mathcal{L}_{\text{GAN}}$ and the perceptual loss $\mathcal{L}_{\text{percep}}$, we introduce an adaptive weight $\lambda_{\text{adaptive}}$. This weight is computed dynamically based on the gradients of both losses with respect to the parameters of the decoder’s last layer (Johnson et al., ). The adaptive weight is calculated as:

$$ $\lambda_{\text{adaptive}}=\frac{\left\|\nabla_{\theta}\mathcal{L}_{\text{ percep}}\right\|_{2}}{\left\|\nabla_{\theta}\mathcal{L}_{\text{GAN}}\right\|_{ 2}+\epsilon},$ (6) $$

where $\theta$ are the parameters of the decoder’s last layer, $\nabla_{\theta}\mathcal{L}_{\text{percep}}$ and $\nabla_{\theta}\mathcal{L}_{\text{GAN}}$ represent the gradients of the perceptual and adversarial losses with respect to $\theta$, and $\epsilon$ is a small constant to avoid division by zero.

### 3.2 Alignment between Video and ECG Tokens

Next, we embed the ECG signals and align them with the video tokens. To fully leverage the time-series ECG data, we implemented the following encoding and alignment design.

##### ECG encoder:

To better couple with echo video, the time-series ECG is first divided into non-overlapping temporal and spatial patches, which are then linearly projected into a high-dimensional space (Na et al., ). Self-attention mechanisms capture spatio-temporal relationships across time segments, and a masked autoencoder (MAE) (He et al., ) further refines the model’s understanding of ECG dynamics, transforming them into separate embedding features. In this work, we divide ECG signals into patches using the embedding layer of an ECG foundation model (ECG-FM), whose strong predictive performance ensures the effectiveness of this patch-based representation. The proposed patchify approach is as follows:
The raw ECG signal, denoted as $\mathbf{X}\in\mathbb{R}^{L\times T}$, where $L$ is the number of leads and $T$ is the time length, is split into non-overlapping spatio-temporal patches. Each lead $l$ produces patches $\mathbf{P}^{l}\in\mathbb{R}^{n\times p}$, where $n=\frac{T}{p}$ and $p$ is the patch size.

Each patch $\mathbf{P}_{i}^{l}$ is passed through the ECG-FM encoder, producing a $D$-dimensional embedding:

$$ $\mathbf{Z}_{i}^{l}=\text{ECG-FM}(\mathbf{P}_{i}^{l})\in\mathbb{R}^{D},$ $$

which results in a sequence of embeddings $\mathbf{Z}=\{\mathbf{Z}_{1}^{l},\mathbf{Z}_{2}^{l},\dots\}\in\mathbb{R}^{L
\times n\times D}$.

##### Aligning ECG and video token sequences for video prediction:

Our video generation framework employs a bidirectional transformer with MVTM (Chang et al., ) to predict video tokens. We mask tokens during training and use parallel decoding during inference to iteratively fill in missing tokens based on context from all directions. This design allows for faster and more efficient video synthesis, leveraging the bidirectional self-attention mechanism to condition video generation on both past and future frames. The masked token prediction is driven by minimizing the negative log-likelihood of the masked tokens:

$$ $\mathcal{L}_{\text{mask}}=-\mathbb{E}_{\mathbf{X}}\left[\sum_{i=1}^{N}m_{i} \log P(\mathbf{y}_{i}|\mathbf{Y}_{M})\right],$ (7) $$

where $\mathbf{y}_{i}$ are the masked tokens, $m_{i}$ indicates the mask for each token, and $\mathbf{Y}_{M}$ represents the partially masked token sequence.

At each iteration, the mask scheduling function $\gamma(t/T)$ determines the fraction of tokens to mask:

$$ $\gamma(t/T)=\cos\left(\frac{\pi t}{2T}\right),$ (8) $$

where $t$ is the current iteration and $T$ is the total number of iterations. The cosine schedule progressively reduces the mask ratio, enabling a gradual refinement of the generated video (Chang et al., ).

The bidirectional transformer architecture allows for flexible video generation guided by various conditioning inputs. We also adapted T5X (Roberts et al., ) following (Reynaud et al., ; Ashrafian et al., )’s text-based ECHO video generation method to create a text-to-ECHO video version for comparative experiments. What’s more, we introduce an ECG mask similar to a text mask. This mask ensures that certain portions of the ECG signal are masked out during training, forcing the model to learn to generate corresponding video frames based on the visible ECG features.

##### Optimization:

To ensure the transformer model fully learns the entire action of ECHO videos, we add a critic loss function (Lezama et al., ). The critic takes a sequence of predicted tokens $\hat{\mathbf{y}}$ and real tokens $\mathbf{y}$ and computes whether the predicted tokens are real or fake.
The critic’s objective is expressed as:

$$ $\mathcal{L}_{\text{critic}}=-\mathbb{E}_{\mathbf{y}}\left[\log D(\mathbf{y}) \right]-\mathbb{E}_{\hat{\mathbf{y}}}\left[\log(1-D(\hat{\mathbf{y}}))\right],$ (9) $$

where $D(\mathbf{y})$ is the critic’s prediction for the real tokens, and $D(\hat{\mathbf{y}})$ is its prediction for the fake tokens (predicted by transformer). In our framework, $\mathbf{y}$ represents the actual video tokens, and $\hat{\mathbf{y}}$ are the predicted tokens sampled during the transformer reconstruction process.

Figure: Figure 2: Video generation example of ECHOPulse. The inputs to ECHOPulse consist of a conditioning image and an ECG signal. Utilizing these inputs as constraints, ECHOPulse generates corresponding videos. The RSTP waves represent the four phases of the ECG. The R wave corresponds to end-diastole (ED), which is the frame with the largest ventricular segmentation area, while the T wave corresponds to end-systole (ES), the frame with the smallest ventricular area. In this case the EF of the generated ECHO video is 24.05.
Refer to caption: x1.png

In practice, the critic’s loss function is computed using the following binary cross-entropy (BCE) loss (Goodfellow, ):

$$ $\mathcal{L}_{\text{BCE}}=-\frac{1}{N}\sum_{i=1}^{N}\left[y_{i}\log(\hat{y_{i}} )+(1-y_{i})\log(1-\hat{y_{i}})\right],$ (10) $$

where $y_{i}$ are the true labels (1 for real tokens, 0 for fake tokens) and $\hat{y_{i}}$ are the critic’s predicted probabilities.

### 3.3 Video Generation

We first train a video tokenization model and an ECG-conditioned video token prediction transformer. To generate videos conditioned on ECG, we pass the empty video value through the video tokenization model to obtain empty tokens and use the ECG tokens as input to the pre-trained transformer. This mirrors the training setup where all video tokens are masked, enabling video prediction from ECG. Additionally, as the first frame is independently tokenized, using it as a condition mimics the training scenario where only the first frame is visible, allowing for video generation based on the initial frame. Similarly, by conditioning on the final segment of a previous video and another ECG inputs, the transformer can generate coherent subsequent frames.

##### Progressive video generation with auto-regressive extrapolation:

During inference, video tokens are sampled iteratively, similar to the process described in MaskGIT (Chang et al., ), using classifier-free guidance with a scaling factor $\lambda$ to balance the alignment between the generated video and the provided ECG condition. Once the first $t_{x}+1$ frames are generated in the latent space, we can further extrapolate additional frames in an auto-regressive manner by re-encoding the last $K$ generated frames using the video encoder. These tokens are then used to initialize the pre-trained transformer, which generates the remaining video tokens conditioned on the provided ECG. This approach enables rapid and length-unconstrained generation of ECHO videos, significantly enhancing its clinical applicability.

## 4 EXPERIMENTS

Figure: Figure 3: Video generation results from ECHOPulse under different ECG conditions. The first row displays the ground truth, while the second row shows the generated video using the same ECG. The third row illustrates results under flat ECG conditions. The fourth condition depicts a shifted R wave, and the final condition, collected from an Apple Watch, represents the most common scenario. The ECG for Condition 4 was collected directly from the Health app on Apple Watch v9.
Refer to caption: x2.png

##### Dataset and implementations:

In this experiment, the process is conducted in a distributed manner. Initially, we train the video tokenization model on the Webvim-10M dataset (Bain et al., ), followed by fine-tuning on the CAMUS dataset (Leclerc et al., ) to adapt the tokenization for medical imaging tasks. With the introduction of LFQ, the codebook size is set to $2^{13}$. All natural videos are resized to 128x128 resolution with 11 frames. The temporal patch size is set to 2, and the spatial patch size is set to 8, resulting in a token sequence with a length of 1536. More detailed information can be seen in Appendix Table 4 and Table 5.

For the subsequent training of ECHOPulse, we utilized a private dataset consisting of 94,078 video and ECG samples. A total of 473 videos were extracted from the dataset to serve as the test dataset, and all video generation evaluations were performed on this test dataset. During the training process, all ECHO videos are resized to a format of 128×128 with 11 frames in the dataloader to match the input requirements of the video encoder. Due to the large volume and diverse sources of our dataset, the overall quality is not as consistent as that of publicly available ECHO datasets. Therefore, each video undergoes random contrast augmentation during loading to enhance its visual quality. We used the pretrained weight of ST-MEM (Na et al., ) as ECG signal encoder. Similarly, the evaluation metrics were maintained to ensure a fair and direct comparison. This allowed us to evaluate the model’s performance and validate the effectiveness of the proposed method.

For evaluation, we use the CAMUS (Leclerc et al., ) and EchoNet-Dynamic (Ouyang et al., ) datasets. CAMUS, a public dataset with 500 ECHO videos with diverse spatial resolutions, is currently the highest-quality public ECHO dataset available. EchoNet-Dynamic includes 10,030 4-chamber cardiac ultrasound sequences with a resolution of 112×112 pixels.

All experiments were conducted on 8 NVIDIA H-100 GPUs, each with 80GB of memory. Due to potential privacy concerns, the private dataset used in this study cannot be made publicly available. However, we have released all the code, scripts, and the pretrained ECG2Video model weights to facilitate further research and replication of our work.

### 4.1 Video Tokenization Reconstruction Performance

In our experiments, we conduct a quantitative comparison between the performance of the diffusion-based EchoNet-Synthetic model and our VQ-VAE-based model, primarily evaluated using MSE (Mean Squared Error) (Gauss, ) and MAE (Mean Absolute Error) (Qi et al., ) on reconstruction results of different datasets. As shown in Table 4.1.

Specifically, the EchoNet-Synthetic model achieves an MSE of $3.00\times 10^{-3}$ and an MAE of $3.00\times 10^{-2}$ on the Echo-Dynamic dataset. In comparison, our VQ-VAE based model, initially trained on a natural video dataset, exhibits an MSE of $5.02\times 10^{-3}$ and an MAE of $4.08\times 10^{-2}$ on the same dataset. Although the initial errors are slightly higher, after only one epoch of fine-tuning, the MSE of our model significantly improves to $2.89\times 10^{-3}$, with the MAE reduced to $2.92\times 10^{-2}$, surpassing the performance of the diffusion-based model.

On the private dataset, the results followed a similar trend. Before fine-tuning, the VQ-VAE model had an MSE of $6.09\times 10^{-3}$ and an MAE of $4.41\times 10^{-2}$. However, after one epoch of domain transfer, the MSE was significantly reduced to $2.91\times 10^{-3}$, and the MAE to $2.97\times 10^{-2}$.

**Table 1: Reconstruction metrics: Videos reconstructed using video tokenization and diffusion-based models are evaluated across two datasets with MAE and MSE. The best results are highlighted.**
| Methods | Dataset | MSE$\downarrow$ | MAE$\downarrow$ |
| --- | --- | --- | --- |
| EchoNet-Synthetic (Reynaud et al., ) | Echo-Dynamic | $3.00\times 10^{-3}$ | $3.00\times 10^{-2}$ |
| ECHOPulse (Only natural videos) | Echo-Dynamic | $5.02\times 10^{-3}$ | $4.08\times 10^{-2}$ |
| ECHOPulse (Domain transfer) | Echo-Dynamic | $\pagecolor{orange}{2.89\times 10^{-3}}$ | $\pagecolor{orange}{2.92\times 10^{-2}}$ |
| ECHOPulse (Only natural videos) | Private data | $6.09\times 10^{-3}$ | $4.41\times 10^{-2}$ |
| ECHOPulse (Domain transfer) | Private data | $\pagecolor{orange}{2.91\times 10^{-3}}$ | $\pagecolor{orange}{2.97\times 10^{-2}}$ |

These results indicate that, despite the common expectation that diffusion-based models perform better in terms of reconstruction quality, our VQ-VAE model was able to quickly converge in a new domain. With only one epoch of fine-tuning, it achieved results comparable to or even surpassing those of the diffusion-based model. This further demonstrates the potential of our model for clinical applications, where it can quickly adapt to new tasks or datasets without the need for retraining from scratch, offering greater practical utility and flexibility.

### 4.2 Video Generation Performance

We evaluate our model on two objectives:
1) image quality (FID (Heusel et al., ), FVD (Unterthiner et al., ), SSIM (Wang et al., )) and left ventricular ejection fraction (LVEF/EF) accuracy. Since we are the first to explore ECG-guided ECHO video generation, and most prior works generating ECHO videos from text have not been open-sourced, we were unable to reproduce their results on our dataset. However, those related papers have clearly reported experimental parameters, datasets, and data split methods. Therefore, we trained ECHOPulse using T5X (Roberts et al., ) as the text encoder on both the CAMUS and EchoNet datasets. All experimental settings, including video resolution and parameters, were kept consistent with those reported in previous studies.

**Table 2: Image quality metrics: The videos generated by different methods under various conditions across three datasets are evaluated using three image quality metrics. The best and second results are highlighted. Note that EchoNet-Synthetic (Reynaud et al., ) did not report the SSIM metric and did not specify whether the results correspond to A2C or A4C views.**
| Methods | Dataset | Condition | A2C | A4C |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FID$\downarrow$ | FVD$\downarrow$ | SSIM$\uparrow$ | FID$\downarrow$ | FVD$\downarrow$ | SSIM$\uparrow$ |  |  |  |
| MoonShot (Zhang et al., ) | CAMUS | Text | 48.44 | 202.41 | 0.63 | 61.57 | 290.08 | 0.62 |
| VideoComposer (Wang et al., ) | CAMUS | Text | 37.68 | 164.96 | 0.60 | 35.04 | 180.32 | 0.61 |
| HeartBeat (Zhou et al., ) | CAMUS | Text | 107.66 | 305.12 | 0.53 | 76.46 | 381.28 | 0.53 |
| ECHOPulse (Only natural videos) | CAMUS | Text | 12.71 | 273.15 | 0.61 | 15.38 | 336.04 | 0.58 |
| ECHOPulse (Domain transfer) | CAMUS | Text | 5.65 | 211.85 | 0.79 | 8.17 | 283.32 | 0.75 |
| EchoDiffusion (Reynaud et al., ) | Echo-Dynamic | Text+EF | 24.00 | 228.00 | 0.48 | 24.00 | 228.00 | 0.48 |
| ECHOPulse (Only natural videos) | Echo-Dynamic | Text | 36.10 | 319.25 | 0.39 | 44.21 | 334.95 | 0.35 |
| ECHOPulse (Domain transfer) | Echo-Dynamic | Text | 27.50 | 249.46 | 0.46 | 29.83 | 312.31 | 0.41 |
| ECHOPulse (Only natural videos) | Private data | Text | 27.49 | 291.67 | 0.53 | 34.13 | 374.92 | 0.51 |
| ECHOPulse(Domain transfer) | Private data | Text | 25.44 | 224.90 | 0.54 | 31.21 | 334.09 | 0.54 |
| ECHOPulse (Only natural videos) | Private data | ECG | 18.73 | 200.45 | 0.56 | 27.37 | 302.89 | 0.55 |
| ECHOPulse (Domain transfer) | Private data | ECG | 15.50 | 82.44 | 0.67 | 20.82 | 107.40 | 0.66 |
| HeartBeat (Zhou et al., ) | CAMUS | Text+Sketch+Mask+… | 25.23 | 97.28 | 0.66 | 31.99 | 159.36 | 0.65 |
| EchoNet-Synthetic (Reynaud et al., ) | Echo-Dynamic | Video+EF | 16.90 | 87.40 | - | 16.90 | 87.40 | - |

Based on Table 2, ECHOPulse outperforms other models across several key metrics. In the ECG-conditioned setting, it achieves the lowest FID scores of 15.50 (A2C) and 20.82 (A4C), indicating highly realistic video generation. Its FVD scores of 82.44 (A2C) and 107.40 (A4C) further highlight its ability to capture temporal dynamics. Additionally, ECHOPulse’s SSIM scores of 0.67 (A2C) and 0.66 (A4C) are higher than those of competing models, showcasing its superior video quality and structural coherence.

ECHOPulse+Text significantly outperforms the HeartBeat model, which also uses text control, on the CAMUS dataset, achieving lower FID 5.65/8.17 and higher SSIM scores 0.79/0.75. As CAMUS is currently the highest-quality publicly available ECHO dataset, all models perform better on CAMUS, while the modest FVD scores 211.85/283.32 highlight the limitations of text-based control. ECHOPulse+Txt also exceeds EchoDiffusion on EchoNet-Dynamic. While VQ-VAE models may lag behind diffusion in video quality, their compatibility with ECG foundation models, coupled with the strong ECG-ECHO correlation, drives ECHOPulse’s superior performance.

After LoRA fine-tuning, ECHOPulse shows further improvement but still falls slightly short of HeartBeat, which uses six conditions, and EchoNet-Synthetic, which modifies entire videos based on a given LVEF value. However, ECHOPulse+ECG excels on the larger, more complex private dataset, demonstrating the effectiveness of ECG as a condition for video generation.

To validate that the generated videos are synchronized with the ECG signals, we compared the cardiac phases between the generated and original videos, focusing on the end-diastole (ED) and end-systole (ES) phases. ED, which corresponds to the R wave in the ECG, represents the moment when the chamber area is at its largest. Similarly, ES corresponds to the T wave, when the chamber area is at its smallest. As shown in Figure 2, the generated videos, conditioned on the input image and ECG, strictly follow the phase changes dictated by the ECG. A more quantitative evaluation is performed by assessing the LVEF, demonstrating the alignment of cardiac dynamics between the generated and original ECHO video.

To evaluate our model’s performance on LVEF accuracy, we applied SAM2 (Ravi et al., ) to the generated videos for direct segmentation of the endocardium, epicardium, and left atrium. After segmenting the cardiac chamber structures, we compared the LVEF between the original echocardiograms and the generated ones. $LVEF=\frac{LVED-LVES}{LVED}$.
This comparison ensured that the generated videos were controlled solely by the ECG signals—introducing temporal information without affecting the structural integrity of the heart images. As shown in Figure 3.

It can be observed that ECHOPulse is capable of generating realistic videos with relatively accurate EF values, regardless of whether the input is an original ECG or a flat ECG. In particular, Condition 4 demonstrates the model’s ability to generate videos using daily ECG signals collected directly from an Apple Watch. This highlights ECHOPulse’s zero-shot capability in handling general ECG signals, showcasing its potential for broad applicability with real-world ECG data.

**Table 3: Clinical and time-inference metrics: This table presents the EF (Ejection Fraction) differences between the generated videos and the target ones. The sampling time refers to the time required to generate 64 video frames. The best results are highlighted.**
| Methods | Condition | $\bf R^{2}$ $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$ | S. time$\downarrow$ | Parameters |
| --- | --- | --- | --- | --- | --- | --- |
| EchoDiffusion (Reynaud et al., ) | Image+EF | 0.89 | 4.81 | 6.69 | Gen. 146s | 381M |
| ECHOPulse | Image+ECG | 0.85 | 2.51 | 2.86 | Gen. 6.4s | 279M |

Quantitative results of $LVEF$ are provided in Table 3, where our model outperforms the other (Reynaud et al., ) in terms of MAE and RMSE. Although the R-square value is slightly lower, this is understandable given the higher difficulty of our task. Additionally, ECHOPulse achieves these results with fewer parameters and requires less sampling time to generate higher-quality videos. This not only underscores the validity of using ECG as a conditioning signal but also demonstrates the clinical potential of our model.

## 5 CONCLUSION

In this paper, we present ECHOPulse, an innovative framework for generating ECHO videos conditioned on ECG signals, marking a significant advancement in the integration of time-series data for video synthesis. By utilizing VQVAE-based tokenization and a transformer with MVTM-driven generation, ECHOPulse not only achieves superior performance in video quality but also addresses key challenge of ejection fraction estimation in cardiac video generation. Our extensive evaluation on both public and private datasets highlights the model’s robust potential for enhancing clinical workflows and enabling real-time cardiac monitoring. This work lays the foundation for further exploration in automated, data-driven cardiac assessment, with implications for both research and practical applications.

## 6 LIMITATION AND FUTURE WORK

There are several limitations of this study that have yet to be addressed. Although we have expanded the dataset through the generation of synthetic data, we are still unable to automatically ensure that the generated videos are free from potential personal privacy information, which limits our ability to fully release the whole private dataset. Additionally, while ECHOPulse can generate videos of unlimited length, it currently lacks the ability to improve video frame rates (FPS) and resolution. In the future, integrating. ECHOPulse, when combined with diffusion models and techniques like super-resolution and denoising, could further enhance the quality of generated videos. This approach addresses current limitations and pushes the boundaries of ECHO video generation.

## Appendix A Appendix

### A.1 Reproducibility Statement

To ensure the reproducibility of our results, we will make all relevant resources publicly available. This includes the complete codebase, training scripts, pre-trained model weights, and synthesized ECHO video data. By providing these materials, we encourage the research community to replicate and build upon our work. Detailed instructions on how to set up and run the experiments will also be included in our repository to facilitate easy reproduction of our results.

### A.2 MODEL SETUP AND HYPERPARAMETERS

#### A.2.1 Video Tokenizer

The configuration of the video tokenizer model is as follows:

**Table 4: Video tokenizer hyperparameters**
| Parameter | Value |
| --- | --- |
| Video input | 11 frames, frame stride 1, 128 $\times$ 128 resolution |
| Batch size | 128 |
| Embedding size | 512 |
| Codebook size | 8192 |
| Spatial patch size | 8 (non-overlapping) |
| Temporal patch size | 2 (along the temporal axis) |
| Spatial transformer depth | 4 layers |
| Temporal transformer depth | 4 layers |
| Transformer head dimension | 64 |
| Number of heads | 8 |
| Feed-forward multiplier | 4 (MLP size of 2048, i.e., $32\times 64$) |

#### A.2.2 Video GENERATION TRANSFORMER

The configuration of the transformer model in Figure 1b is as follows:

**Table 5: Video generation model hyperparameters**
| Parameter | Value |
| --- | --- |
| Number of tokens | 8192 (aligned with the codebook size) |
| Batch size | 128 |
| Embedding dimension | 512 |
| ECG signal dimension | 768 |
| Depth (number of layers) | 6 |
| Learning rate | 1e-4 |
| Training steps | 4,000,000 steps |
| Exponential Moving Average (EMA) | Updated every 10 steps with a decay rate of 0.995 |
| Optimizer | Adam optimizer with $\beta$ values set to (0.9, 0.99) |

### A.3 Dataset

Our private dataset consists of 94,078 ECHO videos, including both apical 2 view and apical 4 view. It contains ECHO recordings from both healthy individuals and patients with various cardiovascular diseases, with real-time ECG signals recorded for each video. All videos were anonymized, underwent random contrast enhancement, and were resized to 128x128 resolution. Additional video generation examples are shown in the Figure 4,5,6 and 7.

Figure: Figure 4: A2C video generation example from ECHOPulse.
Refer to caption: x3.png

.

Figure: Figure 5: A2C video generation example from ECHOPulse.
Refer to caption: x4.png

Figure: Figure 6: A4C video generation example from ECHOPulse.
Refer to caption: x5.png

Figure: Figure 7: A4C video generation example from ECHOPulse.
Refer to caption: x6.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. As mentioned in the weaknesses section, was there a specific reason the qualitative comparison for tokenization models (reconstructed video comparison) and video generation models (generated video comparison) were not conducted?
2. Instead of using non-overlapping patches for video tokenization, would the VQ-VAE based tokenization improve performance using overlapping patches? Was this explored experimentally?
soundness: 3
strengths: 1. The paper contains novel contribution in that it is the first paper to present Echo video generation conditioned on ECG signals, and also plan to release the ECHO videos paired with ECG dataset.

2. The paper is well-written and easy to follow, with the pipeline figure covering all three steps used in EchoPulse.

3. The experiments compare the tokenization performance and video generation performance, and compares to other related papers despite this paper being the first to explore ECG-guided ECHO video generation.
summary: The authors propose the first Echocardiography generation model conditioned in ECG signals named ECHOPulse. The overall pipeline ECHOPulse consists of three stages. The first stage is video tokenizer training, which utilizes a VQ-VAE based model to train the tokenizer for ECHO video quantization. The second stage is to align the ECG and video tokens, and the ECG-FM is used to encode the ECG and masked token prediction is used for alignment. The final stage is video generation, which is autoregressive video token generation conditioned on ECG tokens.
weaknesses: 1. The video tokenization performance results in Table 1 compare the reconstruction metrics of MSE and MAE between EchoNet-Synthetic and ECHOPulse. However, there should also be qualitative result comparison to compare the reconstruction results between the two tokenization models, similar to qualitative video generation results in Figure 3. 

2. Similar to the video tokenization weakness above, the video generation experiments also do not contain any qualitative comparison between other models such as MoonShot, VideoComposer, and HeartBeat. The figures only display the qualitative results of ECHOPulse under different ECG conditions. Therefore, qualitative results should be displayed for each models. 

3. As mentioned in the Limitation and Future Works section, instead of using GAN loss for ECHOPulse, it will be better to see combination of diffusion models in this pipeline for ECHO video generation.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Questions:
See Weaknesses
soundness: 3
strengths: Strengths: 
1) Significance: The most important contribution of this manuscript is the use of easily accessible ECG signals as control conditions to generate echo videos. The proposed method tries to combine these two modalities of video data and time series data (ECG), which is significant for medical image analysis and provides new research directions. The synthesized video data can be applied to many downstream tasks and the methods used can be generalized to more tasks.
2) Originality: Although this paper uses a several of the published techniques, the work centers on synthesizing echo videos using more temporally informative time-series data, rather than using text or otherwise.
3) Quality: The manuscript is clearly structured and accurately presented, and the figures and tables are of high quality.
4) Reproducibility: the authors provide detailed experimental procedures and details in the main text and appendices with corresponding code, which greatly helps in the reproduction of this work.
summary: This manuscript propose ECHOPULSE, a method to synthesize echo videos from easily accessible ECG signals. The method creatively uses the ECG signal, which is highly consistent with the ultrasound video, as a condition to generate the echo video. Experiments on multiple datasets and provided demos show excellent performance.
weaknesses: Weaknesses:
1) Although ECGs can be easily obtained from wearable devices, how should condition images be obtained when the method is applied in practice? How to ensure that the condition image is reasonable?
2) Compared with the previous method, the generation of real-time has been greatly improved, but still can not reach the real-time generation requirements. It is not clear how much inference resources are consumed by this method (Flops, GPU Mem et al). This determines whether it can be deployed on wearable devices.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - How would such method actually have clinical impact ? 
- How much do the generated data contribute in training downstream tasks ? 
- How would we ensure that the model doesn't hallucinate and provide clinically irrelevant  or harmful information ?
- Why there is no thought given to potential confounding factors that would effect the generating process ?

Overall the method is over engineered, with limited guardrails and questionable clinical significance
soundness: 3
strengths: - The method shows potential generalizability to other medical video modalities, like cardiac MRI and CT, indicating versatility beyond ECHO generation.
- The paper is novel and interesting
- The language and presentation is proper
summary: The paper introduces ECHOPulse, an ECG-controlled ECHO video generation model that creates echocardiography (ECHO) videos using only ECG signals as input, claiming that it allows bypassing the need for complex conditional annotations. ECHOPulse uses a VQ-VAE-based tokenization and a transformer model to align ECG with video data, achieving efficient, realistic video generation suitable for scalable clinical applications.
weaknesses: - The use of comic sans font on the figures should re-evaluated 
- High model complexity and memory demands may restrict usability on devices with limited processing power, potentially limiting real-time deployment in resource-constrained settings, such as medical ones.
- The method is over-engineered
- The premise of the paper and the approach taken actually have semantic issues. The difficulty in ECHO data comes from the poor quality of the data as the authors correctly point out, however , to the best of the reviewers knowledge, the solution is not to create synthetic to monitor patients when only ECGs are provided. Details in the videos and images have clinical significance and the proposed method makes no attempt to ensure fidelity and the trustworthiness of the generated videos, making clinical significance limited 
- There is no mention of medical professionals evaluating the quality of the videos and how realistic they are 
- There is no discussion about confounding factors that could influence  the generation of the videos and hence lead the model to hallucinate 
- There is insufficient discussion about the use of such a video generation tool to train downstream models
- Going from an ECG to a video is an information creating process and the method makes no attempt to constrain this so the model does not generate implausible or unrelated to the patient information. Especially when there is no conditioning image/video

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Do you have details about how good the models are without pretraining on natural data?
soundness: 2
strengths: I believe the idea of aligning the representation of frames with ECG data is very good and has a large potential to work for other domains as well. 

The writing is very clear and from beginning to end it is easy to understand what the authors are doing. 

I believe that conditioning videos on time-series data could have a strong impact even outside of ultrasound videos.
summary: The authors propose Echopulse, a video diffusion model conditioned on time-series data. Their pipeline is split into different parts including a tokenization step for efficient generation, a feature alignment step to align visual features with time series data, and a generation step for video generation. 
In general, the paper is well written and easy to follow. 
While their contribution is clear, I am not convinced that the proposed method is better than what was previously proposed in the literature. The main experiments lack exhaustive comparison (eg. Tab.1 private data, tab2 private data) and multiple things proposed in the paper are not properly validated. Overall I give a weak reject.
weaknesses: There is not enough evidence supporting the claim that the generated videos follow the ECG signal. The LVEF estimation presented in Table 3 only considers ED and ES phases which are only a part of the ECG cycle. Furthermore, the results on EF estimation are inferior compared to previous methods in terms of R2 score which is the gold standard metric for estimating EF. Figure 3 only shows one example and is difficult to understand. Maybe it would help to add LVED and LVES values to the frames. 

Table 3 is missing references. Why is EchoNet-Synthetic not mentioned as a baseline in Table 3? Their inference time is below 3 seconds. 

It is unclear how much influence reconstruction has on the final results. While the reconstruction results of the method presented are best, I am not convinced that the difference will be visible when using the method for downstream tasks such as video generation or EF estimation. I think it would be helpful to provide metrics such as rFID or rLVEF (reconstruction LVEF) to get an estimate to how much information is lost through encoding and how much better EchoPulse is compared to previous methods. 

Inconsequent comparison: Why are there no generative results for EchoDiffusion or EchoNetSynthetic on Private data? Also, Table 1 misses the EchoNet-Synthetic baseline. 

Motivation: While I think the application is exciting I believe there is a lack of motivation for ECG to video generation in particular. It would help if you could further motivate. Especially on what is the motivation of doing ECG conditioned generation over EF condtioined video generation. 

Minor: 
Table 2 should be separated into all datasets to make comparison easier.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - Can you provide specific details or comparisons that validate the claims in the abstract regarding the computational cost, slow inference, and complex conditioning of existing methods? Why was EchoNet-Synthetic not included when discussing generation times?

- There appears to be some confusion between the works of Reynaud et al. (2024) and Ouyang et al. (2020) in Section 2.3. Could you clarify the differences between these works and how they relate to your methodology?

- The paper suggests that prior works such as EchoDiffusion and EchoNet-Synthetic use complex conditional prompts like segmentation masks and clinical text, but these models actually use image and EF for conditioning. Could you revise this section to more accurately reflect the conditioning methods used in these studies?

- The loss function presented in Equation 1 lacks references to foundational works. Could you cite the relevant literature that inspired or derived this loss, such as Gu et al. (2022), Rombach et al. (2022), or Esser et al. (2021)?

- The comparison between VQ-VAE and diffusion models seems incomplete, as the EchoNet-Synthetic metrics presented relate only to the VAE model. Could you clarify how these models were compared and whether it is appropriate to conclude that VQ-VAEs surpass diffusion models based on these metrics?

- How were MoonShot and VideoComposer evaluated for this specific task? Did you replicate their results, or were the metrics taken from the HeartBeat paper? Additionally, can you clarify how SSIM was computed, considering it typically requires paired input/output data, which is not evident for ECHOPulse?

- EchoNet-Dynamic contains only A4C views. How was ECHOPulse evaluated on A2C views, given that these metrics are not reported in EchoDiffusion and EchoNet-Synthetic?

- The comparison with baselines such as MoonShot and VideoComposer appears unfair, given that these are general-purpose models. Could you explain how the experimental setup ensures fair comparison with domain-optimized models, and why only the ablation results of HeartBeat were compared?

- EchoNet-Synthetic is conspicuously absent from Table 3. Why was this model excluded from the comparison, and how would including it impact the conclusions?

- The method for calculating LVEF in this paper differs significantly from that used in the EchoNet-Dynamic dataset. How do you account for this discrepancy, and does it affect the validity of the comparisons?

- While the novelty of using ECG signals is appreciated, the claim that your model achieves state-of-the-art results seems overstated. Can you clarify the specific advantages of your model compared to EchoNet-Synthetic and HeartBeat, without overstating the performance? 

- The qualitative results suggest that the visual quality of ECHOPulse samples is lower than that of competing models, but this is not explicitly acknowledged in the text. How do you address this issue, and why were FID and FVD metrics chosen for evaluation, given their limitations for this task?
soundness: 3
strengths: - The use of ECG signals for controlling cardiac motion in echocardiogram generation is a novel and under-explored area, making the concept innovative and potentially impactful.
- The methodology is well-grounded in leveraging VQVAE tokenization for fast decoding, which addresses some of the computational bottlenecks in generative models.
summary: The authors propose a Transformer and VQVAE-based generative model for echocardiogram generation, conditioned on ECG signals. This is a novel approach that seeks to explore ECG-based conditioning for controlling cardiac motion during echocardiogram generation, a relatively under-explored area.
weaknesses: several concerns and inaccuracies in the paper need to be addressed for clarity and rigor.

First, the abstract claims that existing methods have high computational costs, slow inference, and rely on complex conditioning prompts. This statement is not entirely accurate. For example, the paper does not provide a comparison with EchoNet-Synthetic when discussing generation times, which is essential for validating such claims. Neglecting such a comparison is poor practice and can mislead the reader about the novelty and efficiency of the proposed method.

In Section 2.3, there is a reference to "Reynaud et al. (2024) introducing EchoNet-Dynamic." However, this passage is unclear, and it appears there might be confusion between the works of Reynaud et al. (2024) and Ouyang et al. (2020). Clarifying this would help avoid misattribution of prior work.

Additionally, the statement that these studies are conditioned on carefully curated prompts such as segmentation masks and clinical text is not fully correct. Only HeartBeat (Zhou et al., 2024) uses these forms of conditioning. EchoDiffusion (Reynaud et al., 2023) and EchoNet-Synthetic (Reynaud et al., 2024) use image and EF as conditioning inputs. This oversimplification of related work needs to be corrected.

Furthermore, Eq. 1 lacks proper citations. The loss function used in this equation appears to be inspired by previous works, yet no references are given. The authors should consider citing relevant works such as Gu et al. (2022) [1], Rombach et al. (2022) [2], and Esser et al. (2021) [3] to give due credit.

Section 4.1 presents an incorrect conclusion. The numbers extracted from the EchoNet-Synthetic paper reflect only the VAE model metrics, without involving the diffusion model. Similarly, the metrics for ECHOPulse involve only the VQVAE, not the token-prediction model. Therefore, the comparison between VQ-VAEs and diffusion models is not valid, and the conclusion that VQ-VAEs outperform diffusion models cannot be drawn from this comparison.

In Table 2, there are several points that need further explanation. It is unclear how MoonShot and VideoComposer were evaluated for this task, as the numbers seem to have been taken from the HeartBeat paper without further elaboration. Additionally, EchoNet-Dynamic includes only A4C views, so it is not clear how ECHOPulse was evaluated on A2C views, given that these metrics are not reported in EchoDiffusion and EchoNet-Synthetic. The SSIM computation is also problematic, as SSIM generally requires paired input/output data, and it is not clear how ECHOPulse reconstructs a video. Moreover, how the text conditionings for CAMUS and EchoNet-Dynamic were generated should be clarified. Lastly, the table construction raises questions, particularly why HeartBeat and EchoNet-Synthetic are singled out at the end. 

Section 4.2 discusses the comparison with baselines, but the experimental setup lacks sufficient explanation, making the comparisons appear unfair. MoonShot and VideoComposer are general -- purpose generative models, so it is unsurprising that they perform worse than domain -- specific models. HeartBeat’s use of multiple conditions emphasizes the importance of using as many conditioning variables as possible. However, only the ablation results of HeartBeat are compared to other methods, which limits the value of the comparison. Furthermore, the conclusion of this section is poorly formulated. While the qualitative results clearly indicate that the visual quality of ECHOPulse samples lags behind, the conclusion only hints at this issue without directly acknowledging it. Moreover, the FID and FVD metrics are poor indicators of quality in this specific task, and this should be discussed more critically.

In Table 3, EchoNet-Synthetic is conspicuously absent. Including this model would provide a more complete comparative analysis.

In the paragraph following Table 3, the method for calculating LVEF in this paper is substantially different from that used for the EchoNet-Dynamic dataset, making direct comparisons problematic. This discrepancy is not adequately addressed, and the conclusion is again misleading, as EchoNet-Synthetic is not included in this analysis, even though it shows superior performance in all reported metrics.

Section 5 makes some strong claims about the performance of the proposed approach. While the novelty of the idea is clear, the assertion that it achieves the best results is far-fetched. Previous works such as EchoNet-Synthetic and HeartBeat have demonstrated better performance. The authors should focus on highlighting the innovative aspects of their method rather than making overstated claims about its superiority.

Overall, using ECG signals for ECHO video generation is an intriguing approach. The idea of using ECG is interesting, the method (VQVAE + Transformer) is less common in the field if image/video genetation, similar to Phenaki [4]. However, the literature review and baseline comparisons are inadequate, and the paper is riddled with oversights and false conclusions. A more thorough review and revision of the comparisons and claims would significantly improve the work. Additionally, proper citations for key components of the methodology are necessary.

References:
[1] Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., and Guo, B., 2022. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10696-10706).  
[2] Rombach, R., Blattmann, A., Lorenz, D., Esser, P. and Ommer, B., 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).  
[3] Esser, P., Rombach, R. and Ommer, B., 2021. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 12873-12883).  
[4] Villegas, R., Babaeizadeh, M., Kindermans, P.J., Moraldo, H., Zhang, H., Saffar, M.T., Castro, S., Kunze, J., and Erhan, D., 2022, October. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations.

Meta-review: This paper introduces ECHOPulse, an ECG-conditioned echocardiogram video generation model leveraging VQ-VAE tokenization for efficient synthesis, enabling controllable, high-quality medical video generation and advancing multimodal cardiac monitoring applications.

Strengths
- Reviewers agree that the paper introduces a novel approach by conditioning ECHO video generation on ECG signals.
- Echopulse demonstrates state-of-the-art performance in both quantitative and qualitative measures for video generation and shows potential generalizability to other medical imaging modalities.
- Echopulse is implemented in an efficient way to support real-time applications, including resource usage metrics and potential deployment on wearable devices.

Weaknesses
- Reviewers are not entirely convinced of the clinical significance and reliability of the generated videos.
- Some of the metrics (FID, SSIM) used for evaluation didn’t seem suitable, and baseline models could be stronger.
- Some of the descriptions regarding the limitations of previous work are inaccurate and oversimplified.

Decision: Accept (Poster)