ECHOPulse: ECG Controlled Echocardio-gram Video Generation
Authors: Yiwei Li
Sekeun Kim
Zihao Wu
Hanqi Jiang
Yi Pan
Pengfei Jin
Sifan Song
Yucheng Shi
Xiaowei Yu
Tianze Yang
Tianming Liu
Quanzheng Li
Xiang Li
Venue: ICLR 2025 Poster (2025)
Keywords: Medical video generation
ECHO synthesis
Multimodality
Wearable device
Medical foundation model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Echocardiography (ECHO) is essential for cardiac assessments, but its video quality and interpretation heavily relies on manual expertise, leading to inconsistent results from clinical and portable devices. ECHO video generation offers a solution by improving automated monitoring through synthetic data and generating high-quality videos from routine health data. However, existing models often face high computational costs, slow inference, and rely on complex conditional prompts that require experts' annotations. To address these challenges, we propose ECHOPulse, an ECG-conditioned ECHO video generation model. ECHOPulse introduces two key advancements: (1) it accelerates ECHO video generation by leveraging VQ-VAE tokenization and masked visual token modeling for fast decoding, and (2) it conditions on readily accessible ECG signals, which are highly coherent with ECHO videos, bypassing complex conditional prompts. To the best of our knowledge, this is the first work to use time-series prompts like ECG signals for ECHO video generation. ECHOPulse not only enables controllable synthetic ECHO data generation but also provides updated cardiac function information for disease monitoring and prediction beyond ECG alone. Evaluations on three public and private datasets demonstrate state-of-the-art performance in ECHO video generation across both qualitative and quantitative measures. Additionally, ECHOPulse can be easily generalized to other modality generation tasks, such as cardiac MRI, fMRI, and 3D CT generation. We will make the synthetic ECHO dataset, along with the code and model, publicly available upon acceptance.

Published as a conference paper at ICLR 2025
ECHOPULSE: ECG CONTROLLED ECHOCARDIO- GRAMS VIDEO GENERATION
Yiwei Li Sekeun Kim Zihao Wu Hanqi Jiang Yi Pan": Pengfei Jin', Sifan Song', Yucheng Shi Xiaowei Yu², Tianze Yang". Tianming Liu'? Quanzheng Li Xiang Lif !Department of Radiology, Massachusetts General Hospital and Harvard Medical School "School of Computing, University of Georgia 3 "Department of Computer Science and Engineering, University of Texas at Arlington
ABSTRACT
Echocardiography (ECHO) is essential for cardiac assessments, but its video qual- ity and interpretation heavily relies manual expertise, leading inconsistent re- sults from clinical and portable devices. ECHO video generation offers a solution by improving automated monitoring through synthetic data and generating high- quality videos from routine health data. However, existing models often face high computational costs, slow inference, and rely complex conditional prompts that require experts' annotations. To address these challenges, we propose ECHOPulse, an ECG-conditioned ECHO video generation model. ECHOPulse introduces two key advancements: (1) it accelerates ECHO video generation by leveraging VQ- IQ-V tokenization and masked visual token modeling for fast de- coding, and (2) it conditions on readily accessible ECG signals, which are highly coherent with ECHO videos, bypassing complex conditional prompts. the best of our knowledge, this is the first work for ECHO video generation. ECHOPulse not only enables controllable synthetic ECHO data generation but also provides updated cardiac function information for disease monitoring and prediction beyond ECG alone. Evaluations on three public and private datasets demonstrate state-of-the-ard performance in ECHO video generation across both qualitative and quantitative measures. Additionally, ECHOPulse can be easily generalized to other modality generation tasks, such cardiac MRI, fMRI, and 3D CT generation. Codes and examples can seen from Prelease
time-series prompts like ECG signals
INTRODUCTION
Echocardiography (ECHO) is widely used for evaluating heart function and structure due to its real-
time, non-invasive, and radiation-free nature (SF) 2009 Otto 2013 Omar al.] 2016). Recent advancements in deep learning have shown promising performance in computer-aided cardiac di- agnosis (Oktay al. 2018 (Smistad al. 2020 Kim al.) 2021). However, echocardiography depends on experienced operators, leading to insufficient data for training robust models. More- over, annotating ECHO data demands expert knowledge, making the process time-consuming and
sometimes unavailable (Leclerc el al. 2019a Lu el al., 202T).
One promising avenue is the use of video generation models synthesize ECHO videos. In general domains, text-to-vided (T2V) models (Yu et al. [2023] Zhang et al. 2024 Wang et al. 2024 Villegas al. 2022) have been successful creating realistic videos from text prompts. However, applying these models to the medical field remains challenging due to the necessity for precise control over
anatomical structures, which cannot vary freely as general domain. Previous controllable BCHO generation frameworks (Ashrafian al. 20244 Zhou et al.) [0244] Yu al. (024a) rely carefully curated conditional prompts such as segmentation masks, clinical text, or optical flow, which only
"Co-first Authors.
1 CCo-eecond Authors.
1 'Corresoonding author email: xli60 wwe.l@@gh..haaar...oom

---

Published as a conference paper at ICLR 2025
experts can provide and fail to incorporate time-series information. Moreover, these diffusion- basedd models suffer from high computational costs and slow inference. Therefore, fast and controllable ECHO generation framework without the need for expert prior conditions is highly desirable (Rey- naud et al. 2024).
Inspired by these motivations, we propose ECHOPulse, ECHO video generation model condi- tioned on electrocardiogram (ECG) signals. ECG data are readily accessible in both clinical and outpatient settings (AI-Khatib et al. 2018), and even from personal wearable devices (Liu et al. 2013). Since ECGs are typically the first step in evaluating cardiac issues before performing further echocardiography (Al KhatKb al. 2018 Del Torto al. 2022), they serve as natural and intrin- sic prior for coherent ECHO video generation. By leveraging ECG signals, we can generate ECHO videos without the need for complex conditional prompts or expert annotations. To the best of our knowledge, this is the first work to utilize time-series prompt for ECHO video generation. In clini- cal scenarios where prior ECHO images exist, ECHOPulse can be co-conditioned on both ECG and prior imaging to provide updated cardiac function information, such as left ventricular ejection frac- tion (LVEF/EF) for cardiac disease monitoring and prediction (Ponikowski et al. 2016), enhancing computer-aided diagnosis beyond ECG alone.
Our main contributions are:
We propose ECHOPulse, a fast and controllable ECHO video generation framework condi- tioned on ECG for the first time, which also shows potential to generalize to other modality generation tasks, such as cardiac MRI, fMRI, and 3D CT generation.
By utilizing readily available ECG data for controllable and precise ECHO generation per- sonalized for each patient, we bypass the need for paired expert annotations, showing po- tential for scaling to larger video datasets.
Evaluations on two public and one private dataset demonstrate that our framework achieves SOTA performance in both quantitative and qualitative metrics.
We collect synthetic ECHO video dataset consisting of ECHO videos paired with ECG signals and segmentation masks, which we plan to release to enable researchers to evaluate the practicality and reliability of synthetic ECHO video generation.
RELATED WORK
2.1 VISUAL TOKENIZATION
Visual Tokenization is crucial for converting images videos into discrete tokens that can be pro- cessed by language models. One of the foundational approaches to visual tokenization VQ-VAE (Van Den Oord & Vinyals 2017), which uses convolutional neural network (CNN) to encode visual data, followed by a vector quantization (VQ) step. This method assigns each embedding the closest entry codebook, converting continuous embeddings into discrete tokens. Video tokenization presents greater challenges, and VQGAN has been adapted to address these difficul- ties (Villegas et al., 2022; Yu et al., (203).. The current state-of-the-an the-ar in video tokenization MAGVIT2 (Yu et al.) (2024b), which employs an enhanced 3D architecture, leverages an inflation technique for initialization based on image pre-training, and incorporates robust training losses to improve performance.
2.2 VIDEO GENERATION
Recent video generation advances have increasingly focused on discrete representations to manage the high dimensionality of video data. Models like VQ-VAE (Van Den Oord & Vinyals (017) con- vert continuous video frames into discrete vector indices, reducing data complexity and enabling more efficient sequence modeling. Phenaki (Villegas et al., 2022) uses a causal Transformer with discrete temporal embeddings for variable-length video generation, while Masked Generative Trans- former (Chang et al. 2022) leverages discrete techniques for sparse spatiotemporal relationships. These studies highlight the advantages of discrete representations in balancing complexity and ef- ficiency, especially for long sequences. Moreover, autoregressive models with discrete tokens (Yu al. 2022) generate diverse, high-quality content, and discrete diffusion models (Austin et al.
2

---

Published as a conference paper at ICLR 2025
2021 extend the diffusion process the discrete symbol space, enhancing scalability and long- range video modeling.
2.3 ECHO VIDEO GENERATION
Ultrasound imaging is a non-invasive, real-time, and portable medical imaging modality that plays a crucial role in clinical diagnostics. However, the quality and interpretation of ultrasound images heavily depend on the operator's expertise, and obtaining high-quality ultrasound datasets is chal- lenging. Consequently, the generation of ultrasound images has attracted considerable attention in recent years. Recent works, such as HeartBeat (Zhou et al., 2024), have leveraged deep learning techniques like diffusion models (Ho et al. 2020) to achieve controllable synthesis of echocardio- grams. By incorporating multimodal conditions- as cardiac anatomical annotations and func- tional indices and employing diffusion models, these methods can generate realistic ultrasound videos with specific cardiac morphology and function. This approach enriches training datasets and enhances algorithm robustness. Ouyang et al. (2020) introduced the SehhoNet-Dnnamic model, which utilizes spatiotemporal three-dimensional convolutional neural networks segment the left ventricle frame by frame and estimate the ejection fraction (EF), outperforming manual measure- ments. Such advancements hold promise for assisting clinicians in more precise and efficient car- diac function assessments. These techniques typically use images as conditional inputs to generate corresponding ultrasound images. Recent studies have also explored generating ultrasound video sequences from single-frame images (Reynaud et al.) (023). offering new possibilities for dynamic ultrasound applications like cardiac function evaluation. However, these studies rely on carefully curated conditional prompts, such as segmentation masks and clinical text (Zhou et al. (2024) or require complex procedures and expert annotations for data acquisition, as seen with Ejection Frac- tion (EF) (Reynaud et al. 2023 2024), yet they do not leverage time-series ECG data. In contrast, our work is the first to efficintlyy generate controllable ECHO videos conditioned on ECG data.
3 METHODS
ECHOPulse, as shown in Figure I consists of three key components: (1) video tokenization model that encodes videos into temporal token sequences, (2) a masked generative transformer that learns the latent alignment between video tokens and ECG signal, and (3) progressive video genera- tion pipeline that enables fast, unlimited video generation. The video tokenization and transformer components are trained separately.
3.1 VIDEO TOKENIZATION
The key to generating videos from ECG signals lies in capturing the temporal correlation between the video and ECG data. Unlike text (Touvron et al., (023), ECG is continuous temporal signal with variable sequence lengths. As shown in Figure 2 each ECG phase should correspond to dif- ferent stages of the ECHO. Thus, encoding the entire video into single embedding risks disrupting temporal alignment between ECG and video, also complicating autoregressive video generation and reducing generalizability. To temporally align ECHO video with the various phases of the ECG and support autoregressive video generation, an effective approach is to discretize the video into tokens while splitting the ECG into patches and embedding them for cross-modal alignment. Draw- ing inspiration from ViViT (Arnab et al.) 202T) and C-ViVIT (Villegas et al., 2022), we developed ECHOPulse's video tokenization model to meet these specific requirements.
Model architecture: The encoding starts with video sequence of T + frames, each with reso- lution H W and C' channels, denoted as X E R [T+1)) x xC' The sequence compressed into tokens of size (T" 11) x H' x W", where the first frame is tokenized independently, and subsequent frames are apatio-temporat tokens that autoregressive depending on the previous. Non-overlapping patches of size PA Pw x C' (for images) and Pt x PA x Pu x C' (for video) are extracted, resulting T = :. H' p and W" P. Each patch is flattened and projected into a D-dimensional space, forming a tensor of shape (T" + 1) x (H' x W') x D. Spatial transformers are applied with self attention, followed by temporal transformers with causal attention. The patch embeddings Z (TT'' +1)xH' Hw''''''Xx are then quantized into codewords C₂ via vector quantization. The decoder
3

---

Published as a conference paper at ICLR 2025
Trilkig
betwoes Sigulls ECHO Vidne Tokeus
As
Craad Tranineee
BCG EM
Signr
Tobmiin
Linan tranccdeng
LetA
Raadovs Madiing
Aoomvitataal Tikera
laferoci
Visel Token
Tokesion
Masked Takem
Condioca Tragg
Exgpty Talovs
ECG Ensknklnngs
Fir Frune
Scooel Frone
Last Prnnn
Aalling Coesigteeet oo oo
DCG Madhin
Saapk
ELG
Warobl Device
The pipeline of the ECHOPulse. ECHOPulse contains two-step training procedure. a) The first step trains the video tokenizer on the natural video dataset first and fine-tunes it on the public ECHO video dataset. b) The second step trains the transformer via the input token produced by ECG foundation model and video tokenizer, pretrained in the first step. Followed by the video generation procedure (c), ECHOPulse accepts input with or without a conditional image. The empty tokens will be reconstructed via the frozen transformer, trained in the second step, through the guidance of ECG siganls. ECHOPulse is capable generating continuous long videos by sequentially shifting the token sequence and integrating
Figure 1:
ECG inputs.
reverses the process by transforming quantized tokens into embeddings, applying temporal and spa- tial transformers, and mapping the tokens back pixel space via linear projection to reconstruct the frames X.
Additionally, Our framework incorporates the Lookup-Free Quantization (LFQ) method, inspired by the advancements MAGVIT2 (Yu et al., 2024b), to streamline the quantization process. Unlike traditional vector quantization that relies quantization operation directly. Morcover, due to the lack high-quality medical video datasets comparable in scale natural video datasets, introduce LoRA (Low-Rank Adaptation) (Hu al., 2021) to enhance the learning efficiency and generalizability for medical video generation. This also lays the foundation for offering personalized video generation services for users patients in future applications.
explicit codebook lookup. LFQ parameterizes the
Loss design: During video tokenization training phrase, the total loss function Liotal is formulated as:
Cistll Lreson Epercep + LvQ + Andapiive LGAN-
where Leeen is the reconstruction loss, Lperepp the perceptual loss, Lvo represents the vector quantization loss, and LGAN Gu et al. (2022) is the generator adversarial loss. The parameter Aadaptive dynamically adjusts the contribution of the adversarial loss relative to the perceptual loss during training.
Specifically, the reconstruction loss Lreom is computed using the Mean Squared Error (MSE) to ensure the reconstructed video V to closely match the original video V at the pixel level:
Lregon
2
where is the total number of pixels across all frames, V, and V, are the i-th pixel the original video and the reconstructed video, respectively.
To improve the perceptual quality of the reconstructed video, perceptual loss Epeccep is employed to measure the similarity (Johnson al. 2016) between the original and reconstructed videos in the
4

---

Published as a conference paper at ICLR 2025
feature space of pre-trained VGG16 (Qasiim al., 2018 network 6. By extracting high-level features from randomly selected video frames V 5z and fz- the perceptual loss is defined as:
1 M M
percep
Σ ==1
o(V,,) - -
(3)
where M is the number of feature elements, and (6-) denotes the VGG feature extraction operation.
The vector quantization loss (van den Oord al. 2017) Lvo regularizes the encoder outputs by ensuring that they are close to their nearest codebook entries, encouraging efficient and discrete representation learning. The VQ loss is defined as:
Lvo [|s[[z]] - ell2.
where is the encoder output, is the closest codebook entry, and is weighting parameter. The term sg[:] represents the stop-gradient operation, which ensures that the encoder is encouraged to commit to discrete codebook entry without backpropagating gradients through this operation.
The generator adversarial loss LUAN drives the model to produce videos that are indistinguishable from real videos, as evaluated by discriminator D. We employ the hinge loss formulation for stable adversarial training:
CGAN = -EEuPPE [D(in]
(5)
where H video generated by the model, sampled from the generator's distribution Po, and (fH) the discriminator's output for the generated video.
To balance the adversarial loss LGAN and the perceptual loss peecep, we introduce an adaptive weight Addaptive This weight is computed dynamically based on the gradients of both losses with respect to the parameters of the decoder's 's last layer (Johnson et al. 2016). The adaptive weight is calculated as:
V Cpeccpll2 Vo COANN c
andaptive
6
where 8 are the parameters of the decoder's 's last layer, Voopercp and VaLGAN represent the gradients of the perceptual and adversarial losses with respect to 0, and e a small constant to avoid division by zero.
3.2 ALIGNMENT BETWEEN VIDEO AND ECG TOKENS
Next, embed the ECG signals and align them with the video tokens. To fully leverage the time- series ECG data, we implemented the following encoding and alignment design.
ECG encoder: To better couple with echo video, the time-series ECG is first divided into non- overlapping temporal and spatial patches, which are then linearly projected into a high-dimensional space (Na et al.] (024). Self-attention mechanisms capture spatio-temporal relationships across time segments, and a masked autoencoder (MAE) (He et al. 2022) further refines the model's 's understand- ing of ECG dynamics, transforming them into separate embedding features. In this work, we divide ECG signals into patches using the embedding layer of an ECG foundation model (ECG-FM), whose strong predictive performance ensures the effectiveness of this patch-based representation. The pro- posed patchify approach is as follows: The raw ECG signal, denoted as X E LRT,,, where L is the number of leads and the time length, split into non-overlapping spatio-temporal patches. Fach lead produces patches pl E RRp where I and is the patch size.
Each patch P is passed through the ECG-FM encoder, producing a D-dimensional embedding:
z ECG-FM(P!) E RRD
which results in sequence of emheddings Z (Z) Z .... LXxxD
5

---

Published as a conference paper at ICLR 2025
Aligning ECG and video token sequences for video prediction: Our video generation frame- work employs a bidirectional transformer with MVTM (Chang et al. 2022 to predict video tokens. We mask tokens during training and use parallel decoding during inference to iteratively fill in miss- ing tokens based on context from all directions. This design allows for faster and more efficient video synthesis, leveraging the bidirectional self-attention mechanism to condition video generation on both past and future frames. The masked token prediction is driven by minimizing the negative log-likelihood of the masked tokens:
maakk
Exx
Σ m. log Ply: Ya))
where yi are the masked tokens, my indicates the mask for each token, and Y represents the partially masked token sequence.
At each iteration, the mask scheduling function 7t/TT') determines the fraction of tokens to mask:
π 27
T(ITT) = cos
where / is the current iteration and T is the total number of iterations. The cosine schedule progres- sively reduces the mask ratio, enabling a gradual refinement of the generated video (Chang et al.] 2022).
The bidirectional transformer architecture allows for flexible video generation guided by various conditioning inputs. We also adapted T5X (Roberts et al. 2023) following (Reynaud et al. 2024 Ashrafian et al. 2024)'s text-based ECHO video generation method to create a text-to-ECHO video version for comparative experiments. What's more, introduce ECG mask similar to text mask. This mask ensures that certain portions of the ECG signal are masked out during training, forcing the model to learn to generate corresponding video frames based the visible ECG features.
Optimization: To ensure the transformer model fully learns the entire action of ECHO videos, we add critic loss function (Lezama et al.. 20022). The critic takes a sequence of predicted tokens ý and real tokens and computes whether the predicted tokens are real or fake. The critic's objective expressed as:
(9) where D(y) is the critic's prediction for the real tokens, and D(y) is its prediction for the fake tokens (predicted by transformer). In our framework, y represents the actual video tokens, and ý are the predicted tokens sampled during the transformer reconstruction process.
Letti = -y [log D(y)] - Ey [log(1 - D(}))),
In practice, the critic's loss function is computed using the following binary cross-entropy (BCE) loss (Goodfellow) 2016):
LACF
[y log(t)) (1 - y:) log(1 1/))
(10)
where y4 are the true labels (1 for real tokens, 0 for fake tokens) and ý the critic's predicted probabilities.
3.3 VIDRO GENERATION
We first train a video tokenization model and an ECG-conditioned video token prediction trans- former. To generate videos conditioned on ECG, we pass the empty video value through the video tokenization model to obtain empty tokens and use the ECG tokens as input to the pre-trained trans- former. This mirrors the training setup where all video tokens are masked, enabling video prediction from ECG. Additionally, as the first frame is independently tokenized, using it as a condition mimics the training scenario where only the first frame is visible, allowing for video generation based on the initial frame. Similarly, by conditioning on the final segment previous video and another ECG inputs, the transformer generate coherent subsequent frames.

---

Published as a conference paper at ICLR 2025
Condition ECG Signal
R1
R2
R: R Wave (ED)
R3
T: Wave (ES)
P: Wave
T1
P1
T2
S: S Wave
S2
Condition Image
S1
24.05
$2
T2
R3
Video generation example of ECHOPulse. The inputs to ECHOPulse consist of a ditioning image and an ECG signal. Utilizing these inputs as constraints, ECHOPulse generates corresponding videos. The RSTP waves represent the four phases of the ECG. The R wave responds to end-diastole (ED), which is the frame with the largest ventricular segmentation area, while the T wave corresponds end-systole (ES), the frame with the smallest ventrieular area. In this case the EF of the generated ECHO video is 24.05.
Figure 2:
Progressive video generation with auto-regressive extrapolation: During inference, video to- kens are sampled iteratively, similar to the process described MaskGIT (Chang et al. (2022). using classifier- free guidance with scaling factor to balance the alignment between the generated video and the provided ECG condition. Once the first tu frames generated in the latent space, we can further extrapolate additional frames in an auto-regressive manner by re-encoding the last K generated frames using the video encoder. These tokens are then used to initialize the pre-trained transformer, which generates the remaining video tokens conditioned on the provided ECG. This ap- proach enables rapid and mangth-uncoostrained generation of ECHO videos, significantly enhancing its clinical applicability.
EXPERIMENTS
Dataset and implementations: In this experiment, the process is conducted in a distributed man- ner. Initially, train the video tokenization model the Webvim-10M, dataset (Bain al. (021), followed by fine-tuning on the CAMUS dataset (Leclerc et al. 2019h) to adapt the tokenization for medical imaging tasks. With the introduction of LFQ, the codebook size videos are resized to 128x122 128 resolution with 11 frames. The temporal patch size is set to 2, and the spatial patch size is set 8, resulting token sequence with length of 1536. More detailed information can be seen in Appendix Table and Table 5
2 All natural
For the subsequent training of ECHOPulse, we utilized a private dataset consisting 94,078 video and ECG samples. A total of 473 videos were extracted from the dataset to serve as the test dataset, dataset. During the training process,
and all video generation evaluations were performed on this all ECHO videos are resized to format of 128x128 28 with 11 frames in the dataloader to match the input requirements of the video encoder. Due the large volume and diverse sources our dataset, the overall quality is not as consistent as that of publicly available ECHO datasets. Therefore, each video undergoes random contrast augmentation during loading to enhance its visual quality. We used the pretrained weight of ST-MEM (Na al. 2024) ECG signal encoder. Similarly, the evaluation metrics were maintained to ensure a fair and direct comparison. This allowed us to evaluate the model's performance and validate the effectiveness of the proposed method.

---

Published as a conference paper at ICLR 2025
Chamber
=70.53
[Biggest Segmentation)
(Smallest Segmentation)
GT
+
pred1
68.68
Condition1
pred2
7.97
Condition2
pred3
66.90
Condition3
pred4
Average EF 74.18
Condition
ECG collected from pple Watch
Figure 3: Video generation results from ECHOPulse under different ECG conditions. The first row displays the ground truth, while the second row shows the generated video using the same ECG. The third row illustrates results under flat ECG conditions. The fourth condition depicts a shifted R wave, and the final condition, collected from Apple Watch, represents the most common scenario. The ECG for Condition was collected directly from the Health app on Apple Watch v9.
For evaluation, we use the CAMUS (Leclerc et al. 2019a) and EchoNet-Dynamic (Ouyang al., 2020) datasets. CAMUS, a public dataset with 500 ECHO videos with diverse spatial resolutions, currently the highest-quality public ECHO dataset available. SchoNet-Dynamn includes 10,030 4-chamber cardiac ultrasound sequences with a resolution of 112x1 12 pixels.
All experiments potential privacy concerns, the private dataset used in this study cannot be made publicly available. However, have released all the code, scripts, and the pretrained ECG2Video model weights to facilitate further research and replication of our work.
conducted on NVIDIA H-100 GPUs, each with 80GB of memory. Due to
4.1 VIDEO TOKENIZATION RECONSTRUCTION PERFORMANCE
In our experiments, we conduct a quantitative comparison between the performance of the diffusion- based FchoNer-Syntheite model and our VQ-VAE-based model, primarily evaluated using MSE
(Mean Squared Error) (Gauss] (809) and MAE (Mean Absolute Error) (Qi al. 2020 on recon- struction results of different datasets. As shown in Table
Specifically, the EchoNet- model achieves an MSE of 3.00 x 10 and an MAE 3.00 x 10-2 on the Echo-Dynamic dataset. In comparison, our VQ-VAE based model, initially trained on a natural video dataset, exhibits an MSE of 5.02 x 10-3 and an MAE of 4.08 x 1 on the same dataset. Although the initial errors slightly higher, after only epoch of fine-tuning, the MSE our model significantly improves to 2.89 x 10-3 with the MAE reduced to 2.92 x surpassing the performance of the diffusion-based model.
8

---

Published as a conference paper at ICLR 2025
Table I: Reconstruction metrics: Videos reconstructed using video tokenization and diffusion- based models are evaluated across two datasets with MAE and MSE. The best results are high- lighted.
Methods
Dataset
MSEJ
MAE!
EchoNet-Synthetic (Reynaud al. 2024
Echo-Dynamic
3.00 x 10
3.00 10
ECHOPulse (Only natural videos)
Echo- Seho-Dynamic
5.02 x 10-3
4.08 10-2
ECHOPulse (Domain transfer)
Echo-Dynen Echo-Dynamic
2.89 10
2.92 10
Syytheett (Reynaud et al. 2024 ECHOPulse (Only natural videos)
Mhn--Dynamic
3.12 x 10
3.37 x 10-2
Private data
6.09 x 10
4.41 x 10-2
ECHOPulse (Domain transfer)
Private data
2.91 x 10
2.97 x 10
The videos generated by different methods under various condi- tions across three datasets are evaluated using three image quality metrics. The best results are highlighted. Note that EchoNet-Synthetic (Reynaud et al. 2024) did not report the SSIM metric and did not specify whether the results correspond to A2C or A4C views.
Table 2: Image quality metrics:
Methods
Dataset
Condition
A2C
A4C
FID.
FVD.
SSIM1
FID
FYD.
SSIM
MoonShor Zhang al.
Text
Vidoooomnss Wangg Cl 11 2024
CAMUS
Text
37.68
164.96
0.60
35.04
180.32
0.61
HeartReat Zhen
CAMUS
Text
107.66
305.12
0.53
76.46
381.28
0.53
HeartBeal Zixu ct al 2124
CAMUS
Toxt-Slccch-Mask......
25.23
97.28
0.66
31.99
159.36
0.65
ECHOPulse (Oaly natural vileos)
CAMUS
TextEF)
12.71
273.15
0.61
15.38
336.04
0.58
BCHOPalse (Domain transfer)
CAMUS
Text(FF)
5.65
211.85
0.79
8.17
283.32
0.75
Echaliffuusio Reynand et
2023
Fcho-Dynamic
EF EF TestEF) Text(F))
24.00
228.00
0.48
SyhthNet-Satthhie KKyyaan 0 2024
Echo-Dynamic
16.90
87,40
ECHOPulse (Only naueetvooess
Sato-Dyymmic
ECHOPalse (Domain trunsfer)
Echo-Dynan Dynamic
28.25
296.50
0.43
EchaDiffusion Reenaud et al.
2023
Private data
EF
20.71
379.43
0.55
23.20
390.17
0.53
BohoNet-Syntheln: Keyyaanee ef
Private
18.39
91.29
26.13
120.91
0.55
ECHOPalse (Oaly natural vi02os)
Privati data
TestEF)
27.49
291.67
0.53
34.13
374.92
0.51
ECHHOPuls(Dommin trunsfer)
Private data
TexHET)
25.44
224.90
0.54
31.21
334.09
0.54
ECHOPulse (Oaly natural videos)
Private data
ECG
18.73
200.45
0.56
27.37
302 002.89
0.55
ECHOPPlse (Domain tunsserr
Private data
BCG
15,50
82.44
0.67
20.82
107.40
0.66
On the private dataset, the results followed similar trend. Before fine-tuning, the VQ-VAE model
had an MSE of 6.09 10 and MAE of 4.41 x 10-2. However, after one epoch of domain transfer, the MSE was significantly reduced to 2.91 10-3, and the MAE to 2.97 x 10-2
These results indicate that, despite the common expectation that diffusion- based models perform better in terms of reconstruction quality, our VQ- VAE model was able to quickly converge in new domain. With only one epoch of fine-tuning, it achieved results comparable to or even surpassing those of the diffusion- model's VAE in this task. This further demonstrates the potential of our model for clinical applications, where it can quickly adapt to new tasks or datasets without the need for retraining from scratch, offering greater practical utility and flexibility.
4.2 VIDEO GENERATION PERFORMANCE
We evaluate our model on two objectives: 1) image quality (FID) (Heusel et al. 2017). FVD (nn- terthiner al., 2018). SSIM (Wang et al. 2004)) and left ventricular ejection fraction (LVEF/EF) accuracy. Since we are the first to explore ECG-guided ECHO video generation, and some prior works generating ECHO videos from text have not been open-sourced, their results on our dataset. However, those related papers have clearly reported experimental param- eters, datasets, and data split methods. Therefore, we trained ECHOPulse using T5X (Roberts al.) 2023) the text encoder on both the CAMUS and EchoNet datasets. All experimental settings, including video resolution and parameters, were kept consistent with those reported in previous studies. Details can be seen in Appendix
were unable reproduce
A.2.3 A.2.3
Based on Table 2 ECHOPulse outperforms other models across several key metrics. In the ECG- conditioned setting, it achieves the lowest FID scores of 15.50 (A2C) and 20.82 (A4C), indicating highly realistic video generation. Its FVD scores of 82.44 (A2C) and 107.40 (A4C) further highlight its ability to capture temporal dynamics. Additionally, ECHOPulse's SSIM scores of 0.67 (A2C) and 0.66 (A4C) are higher than those of competing models, showcasing its superior video quality and structural coherence.

---

Published as a conference paper at ICLR 2025
Table Clinical and time-inference metries: This table presents the EF (Ejection Fraction) (Mur- phy et al. 2020) differences between the generated videos and the target ones. The sampling time refers to the time required to generate 64 video frames. The best results are highlighted.
Methods
Condition
R²²
MAE
RMSE +
S. time.
Parameters
EchoDiffusion [Reynaud et al. 2023 EchoNet-Synne Synthetic (Reynaud et at. 2024 ECHOPulse (Our Model)
Image+EF
0.83
5.14
7.21
Gen. 147s
402M
Image+EF
0.85
2.71
3.13
Gen. 20.2s
283M
Image+ECG
0.85
2.51
2.86
Gen. 6.48 279M
ECHOPulse+Text significantly outperforms the HeartBeat model, which also uses text control, on the CAMUS dataset, achieving lower FID 5.65/8.17 17 and higher SSIM scores 0.79/0.75, As CAMUS currently the highest-quality publicly available ECHO dataset, all models perform better on CA- MUS, while the modest FVD scores 211 1.85/283.33 highlight the limitations of text-based control. ECHOPulse+ also exceeds EchoDiffusion on RchoNet-Dynamite While VQ-VAE models may lag behind diffusion in video quality, their compatibility with ECG foundation models, coupled with the strong ECG-ECHO correlation, drives ECHOPulse's superior performance.
After LoRA fine-tuning, ECHOPulse shows further improvement but still falls slightly short of HeartBeat, which uses six conditions, and EchoNet-Synthetic which generates entire videos based on a given LVEF value and condition image. However, ECHOPuls++EGG excels on the larger, more complex private dataset, demonstrating the effectiveness of ECG as a condition for video generation. Comparison examples can be found in Appendix Figure x and
To validate that the generated videos are synchronized with the ECG signals, we compared the cardiac phases between the generated and original videos, focusing the end-diastole (ED) and end-systole (ES) phases. ED, which corresponds to the wave in the ECG, represents the moment when the chamber area is at its largest. Similarly, ES corresponds to the T wave, when the chamber area is at its smallest. As shown in Figure [ the generated videos, conditioned on the input image and ECG, strictly follow the phase changes dictated by the ECG. A more quantitative evaluation performed by assessing the LVEF, demonstrating the alignment of cardiac dynamics between the generated and original ECHO video.
To evaluate our model's performance on LVEF accuracy, we applied SAM2 (Ravi et al., 2024) to the generated videos for direct segmentation of the endocardium, epicardium, and left atrium. After segmenting the cardiac chamber structures, we compared the LVEF between the original echocar- diograms and the generated ones. LV LVED-IV L.D--VVE ES This comparison ensured that the gen- erated videos were controlled solely by the ECG gooall--ntrrduuiing temporal information without affecting the structural integrity of the heart images. As shown in Figure 3 Expert evaluation results and more examples be seen in
LVED
Relevant information
can be found in the Appendix A.3 Appendix A.4
It can be observed that ECHOPulse is capable of generating realistic videos with relatively accurate EF values, regardless of whether the input is an original ECG or a flat ECG. In particular, Condition 4 demonstrates the model's ability generate videos using daily ECG signals collected directly from an Apple Watch. This highlights ECHOPulse's s zero-shot capability in handling general ECG signals, showcasing its potential for broad applicability with real-world ECG data.
Quantitative results of LV EF are provided in Table 3 where our model outperforms others (Rey- naud et al.) 20023 2024 terms of square MAE and RMSE. ECHOPulse achieves these results using less sampling time to generate higher-quality videos. This not only underscores the validity of using ECG as a conditioning signal but also demonstrates the clinical potential of our model.
5 CONCLUSION
In this paper, we present ECHOPulse, an innovative framework for generating ECHO videos con- ditioned on ECG signals, marking a significant advancement in the integration of time-series data for video synthesis. By utilizing VQVAE- based tokenization and a transformer with MVTM-driven generation, ECHOPulse not only achieves superior performance in video quality but also addresses key challenge of ejection fraction estimation in cardiac video generation. Our extensive evaluation on both public and private datasets highlights the model's robust potential for enhancing clinical
10

---

Published as a conference paper at ICLR 2025
workflows and enabling real-time cardiac monitoring. This work lays the foundation for further exploration in automated, data-driven cardiac assessment, with implications for both research and practical applications.
LIMITATION AND FUTURE WORK
There are several limitations of this study that have yet to addressed. Although we have expanded the dataset through the generation of synthetic data, we are still unable to automatically ensure that the generated videos are free from potential personal privacy information, which limits our ability to fully release the whole private dataset. Additionally, while ECHOPulse can generate vidcos of unlimited length, it currently lacks the ability to improve video frame rates (FPS) and resolution. In the future, integrating. ECHOPulse, when combined with diffusion models and techniques like super- resouutinn and denoising, could further enhance the quality of generated videos. This approach addresses current limitations and pushes the boundaries of ECHO video generation.
REFERENCES
Sana M Al-Khatib, William G Stevenson, Michael J Ackerman, William Bryant, David Callans, Anne B Curtis, Barbara J Deal, Timm Dickfeld, Michael E Field, Gregg C Fonarow, et al. 2017 aha/acc/hrs guideline for management of patients with ventricular arrhythmias and the prevention of sudden cardiac death: report of the american college of cardiology/ameee hoael////mmmmcaan heart association task force on clinical practice guidelines and the heart rhythm society. Journal of the American College of Cardiology, 72(14):e1-e222) 2018.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. Proceedings of the IEEE/CVF international conference on computer vision, pp. 6836-6846, 2021.
Pooria Ashrafian, Milad Yazdani, Moein Heidari, Dena Shahriari, and Ilker Hacihaliloglu. Vision-language synthetic data enhances echocardiography downstream tasks. arXiv preprint arXiv:2403.19880 2024.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 14417981-17993, 2021.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
Huiwen Chang, Han Zhang, Lu Jiang, et al. Maskgit: Masked generative image transformer. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315- 11325, 2022.
Alberico Del Torto, Andrea Igoren Guaricci, Francesca Pomarico, Marco Guglielmo, Laura Fusini, Francesco Monitillo, Daniela Santoro, Monica Vannini, Alexia Rossi, Giuseppe Muscogiuri, et al. Advances in multimodality cardiovascular imaging in the diagnosis of heart failure with preserved ejection fraction. Frontiers Cardiovascular Medicine, 9:758975, 2022.
Carl Gauss. Theoria motus corporum coelestium sectionibus conicis solem mabientium [theory of the motion of the heavenly bodies moving about the sun in conic sections). Perthes and Besser, Hamburg, Germany, 1809.
Ian Goodfellow. Deep learning, 2016.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10696-10706, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.

---

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Published as a conference paper at ICLR 2025
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Neural Information Processing Systems, 33:6840-6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 00600985, 2021.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pp. 694-711. Springer, 2016.
Taeouk Kim, Mohammadali Hedayat, Veronica V Vaitkus, Marek Belohlavek, Vinayak Krishna- murthy, and Iman Borazjami. Automatic segmentation of the left ventricle in echocardiographic images using convolutional neural networks. Quantitative Imaging in Medicine and Surgery, 11 (5):1763, 2021.
Sarah Leclere, Erik Smistad, Joao Pedrosa, Andreas Østvik, Frederic Cervenansky, Florian Es- pinosa, Torvald Espeland, Erik Andreas Rye Berg, Pierre-Marc Jodoin, Thomas Grenier, et al. Deep learning for segmentation using an open large-scale dataset in 2d echocardiography, IEEE transactions medical imaging, 38(9):2198 2210, 2019a.
Sébastien Leclerc, Erik Smistad, Jorge Pedrosa, Andre Ostvik, et al. Deep learning for segmenta- tion using an open large-scale dataset 2d echocardiography. IEEE Transactions on Medical Imaging, 38(9): 2000 Sept. 2019b.
José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with token-critic. In European Conference on Computer Vision. pp. 70-86. Springer, 2022.
Jun Liu, Fei Xie, Yaqi Zhou, Qian Zou, and Jianfeng Wu. A wearable health monitoring system with multi-parameters. In 2013 6th International Conference on Biomedical Engineering and Informatics, pp. 332-336. IEEE, 2013.
Tong Lu, Tingting Chen, Feng Gao, Biao Sun, Vasilis Nuziachristos, and Jiao Li. Lv-gan: A deep learning approach for limited-view optoacoustic imaging based on hybrid datasets. Journal of biophotonics, (4220:202000325, 2021.
Sean P Murphy, Nasrien E Ibrahim, and James L Januzzi. Heart failure with reduced ejection fraction: a review. Jama, 324(5):488550) 2020.
Yeongycon Na, Minje Park, Yunwon Tae, and Sunghoon Joo. Guiding masked representa- tion learning to capture spatio-temporal relationship of electrocardiogram. arXiv preprint 241:::2002.09450, 2024.
Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv: 1804.0399 2018.
Alaa Mabrouk Salem Omar, Manish Bansal, and Partho P Sengupta. Advances in echocardiographic imaging in heart failure with reduced and preserved ejection fraction. Circulation research, 119 (2):357-374, 2016.
Catherine M Otto. Textbook of clinical echocardiography. Elsevier Health Sciences, 2013.
David Ouyang, Bryan He, Amirata Ghorbani, Neal Yuan, Joseph Ebinger, Curtis P Langlotz, Paul A Heidenreich, Robert Harrington, David H Liang, Fuan A Ashley, al. Video-based ai for beat-to-beat assessment of cardiac function. Nature, 580(7802):22-2256 2020.
Piotr Ponikowski, Adriaan A Voors, Stefan D Anker, Héctor Bueno, John GF Cleland, Andrew JS Coats, Volkmar Falk, José Ramón Gonzälez-Junaatey, Veli-Pekka Harjola, Ewa A Jankowska, et al. 2016 ese guidelines for the diagnosis and treatment of acute and chronic heart failure. Kardiologia Polska (Polish Heart Journal), 14310):1077-1147, 2016.
12

---

Published as a conference paper at ICLR 2025
Hussam Qassim, Abhishek Verma, and David Feinzimer. Compressed residual-vgg16 cnn model for big data places image recognition. In 2018 JEEE 8th annual computing and communication workshop and conference (CCWC), pp. 169-175. IEEE, 2018.
Jun Qi, Jun Du, Sabato Marco Siniscalchi, Xiaoli Ma, and Chin-Hui Lee. On mean absolute error for deep neural network based vector-to-vectee regression. IEEE Signal Processing Letters, 27: 1485-1489, 2020.
Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint
14, 2024.
Hadrien Reynaud, Mengyun Qiao, Mischa Dombrowski, Thomas Day, Reza Razavi, Alberto Gomez, Paul Leeson, and Berhhard Kainz. Feature cascaded video diffusion models for precise echocardiogram synthesis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 142-152. Springer, 2023.
Hadrien Reynaud, Qingjie Meng, Mischa Dombrowski, Arijit Ghosh, Thomas Day, Alberto Gomez, Paul Leeson, and Bernhard Kainz. Echonet-synthetic: Privacy-preservin video generation for safe medical data sharing. arXiv preprint 2xXxx:2406.00808 2024.
Adam Roberts, Hyung Won Chung, Gaurav Mishra, Anselm Levskaya, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. Scaling up models and data with t5x and seqio. Journal of Machine Learning Research, 24(377):1-8, 2023.
NAGUEH SF. Recommendations for the evaluation of left ventricular diastolic function by echocar- diography. J Am Soc Echocardiogr, 22:107-133, 2009.
Erik Smistad, Andreas Østvik, Ivar Mjäland Salte, Danicla Melichova, Thuy Mi Nguyen, Kristina Haugaa, Harald Brunvand, Thor Edvardsen, Sarah Leclerc, Olivier Bernard, et al. Real-time automatic ejection fraction and foreshortening detection using deep learning. JEEE transactions on ultrasonics, ferroelectrics, and frequency control, 67(12):2595-2604 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint MxXi:2337.002888 2023.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprini arXiv: /8/2.01717, 2018
Aaron Van Den Oord and Oriol Vinyals. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. Advances in neural information processing systems, 30, 2017.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, et al. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022.
Xiang Wang, Hangjie Yuan, Shiwei Zhang. Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion con- trollability. Advances in Neural Information Processing Systems, 36, 2024.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Junxuan Yu, Rusi Chen, Yongsong Zhou, Yanlin Chen, Yaofei Duan, Yuhao Huang, Han Zhou, Tan Tao, Xin Yang, and Dong Ni. Explainable and controllable motion curve guided cardiac ultrasound video generation. arXiv preprint 2xXvv:240..214900 2024a.
13

---

Published as a conference paper at ICLR 2025
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuar Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 10459-10469, 2023.
Luming Yu, Jose Lezama, Nikhil B Gundavarapu, et al. Language model beats diffusion-tokenizze key to visual generation. In The Twelfth International Conference on Learning Representattons, 2024b.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. International Conference on Learning Representations, 2022.
David Junhao Zhang, Dongxu Li, Hung Le, Mike Zheng Shou, Caiming Xiong, and Doyen Sahoo. Moonshot: Towards controllable video generation and editing with multimodal conditions. arXiv preprint arXiv:2401.01827 2024.
Xinrui Zhou, Yuhao Huang, Wufeng Xue, Haoran Dou, Jun Cheng, Han Zhou, and Dong Ni. Heart- beat: Towards controllable echocardiography video synthesis with multimodal conditions-guided diffusion models. arXiv preprint arXiv:2406. 14098, 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we will make all relevant resources publicly available. This includes the complete codebase, training scripts, pre-trained model weights, and synthesized ECHO video data. By providing these materials, we encourage the research community to replicate and build upon our work. Detailed instructions on how to set up and run the experiments will also be included in our repository to facilitate easy reproduction of our results.
A.2
MODEL SETUP AND HYPERPARAMETERS
A.2.1
VIDEO TOKENIZER
The configuration of the video tokenizer model is as follows:
Parameter
Value
Video input
11 frames, frame stride 1, 128 x 128 resolution
Batch size
128
Embedding size
512
Codebook size
8192
Spatial patch size
8 (non-overlapping)
'emmporal patch size
2 (along the temporal axis)
Spatial transformer depth
4 layers
Temporal transformer depth
4 layers
Transformer head dimension
64
Number of heads
8
Feed-forward multiplier
4 (MLP size of 2048, i.c., 32 64)
Table 4: Video tokenizer hyperparameters
A.2.2
VIDEO GENERATION TRANSFORMER
The configuration of the transformer model in Figure Pp is as follows:
Parameter
Value
Number of tokens
8192 (aligned with the codebook size)
Batch size
128
Embedding dimension
512
ECG signal dimension
768
Depth (number of layers)
6
Learning rate
le-4
Training steps
4,000,000 steps
Exponential Moving Average (EMA)
Updated every 10 steps with decay rate of 0.995
Optimizer
Adam optimizer with 3 values set to (0.9, 0.99)
Table 5; Video generation model hyperparameters
A.2.3 EXPERIMENTAL DETAILS
In the comparative experiments with Heartbeat, we encountered challenges in directly reproducing its results due to the lack of publicly available source code and detailed parameter specifications. Similarly, other generic methods compared in the Heartbeat paper, such as MoonShot and Video- Composer, were fine-tuned sufficient implementation details to facilitate replication. However, Heartbeat explicitly specifies key aspects such the video resolution, certain parameters, and the use of the CAMUS dataset with
retrained the CAMUS dataset, but the paper does not provide
15

---

Published as a conference paper at ICLR 2025
Table 6: Video reconstruction metries: This table presents the (reconstruction FID) rFID and
(reconstruction LVEF) rLVEF differences between the reconstructed videos and the target ones.
Methods
rFID
rLVEF
EchoNet-Syntheiiee Synthetioeeeennaadddd et al. 2024
10.93
0.33
ECHOPulse (Our Model)
9.16
1 + 0.26
its defined train, test, and validation splits. To ensure fair comparison, we aligned the experimen- tal conditions by training EchoPulse under the same settings as Heartbeat. Additionally, the text conditions for EchoPulse were configured match those described in the Heartbeat study.
For the comparative experiments with EchoDiffusion and EchoNet-Syythert on 8 NVIDIA H100s, we first trained EchoPulse on the Dehname- dataset, adhering to the same video format used in these methods. Specifically, the videos were resized to the 112x112 12 resolution standard of the EchoNet-Dynane condition to represent EF values in textual format, allowing EF data serve input to the model. However, converting numerical values into textual representations may have introduced some impact on the training effectiveness.
dataset. To ensure consistency with these approaches, we adapted our text
For experiments on the private dataset, we replicated the results of EchoDiffusion and EchoNet- Synthetic by following the training procedures described in their respective open-sourg codebases and publications. Subsequently, we compared the models' performance using the test set of the private dataset.
For the computation of metrics such as FVD, FID, and SSIM, we followed the approach outlined in BchoNet-Synthetic to enable comparison between generated videos and Ground-Truth videos. Specifically, we used the first frame of the Ground-Truth video as the condition image input. For EchoDiffusion and SchoNet-Synthetic, we provided the EF value of the Ground-Thue video as input, whereas for EchoPulse, the ECG signal of the Ground-Trute video was used. Quantitative evaluation metrics were then calculated for the generated and Ground-Truth video pairs. Addition- ally, we compared the EF values extracted from the generated videos with those of the Ground- Truth videos. As it is shown Table 3 Table 3 further demonstrates the advantage of using ECG a condition in our model by calculating the deviations of ED and ES key frames between the generated videos and the ground-truth videos.
Regarding sampling time, we adhered to the original EchoDiffusion setup and calculated the time re- quired to generate 64 frames, corresponding to 8 complete ECG eycles, or eight heartbeats. EchoD- iffusson and EchoNet-Syntheticc were similarly adjusted to generate 64 frames, ensuring the output covered heartbeats. It is worth noting that EchoNet-Synthetic employs a video editing-like ap- proach, where the 64-frame video is generated by randomly repeating single heartbeat cycle. This differs fundamentally from EchoPulse, which generates videos with higher information content. Di- modified to match that of EchoPulse by generating eight heartbeat cycles separate steps using the same EF value. This adjustment allowed for a fair comparison of generation speed.
rect comparisons under these conditions would be unfair. Therefore, in this experiment, the generation length of EchoNet-Synthine
A.3 DATASET
Our private dataset consists of 94,078 ECHO videos, including both apical 2 view and apical 4 view. It contains ECHO recordings from both healthy individuals and patients with various cardiovascu- lar diseases, with real-time ECG signals recorded for each video. All videos were anonymized, underwent random contrast enhancement, and were resized to 128x 128 resolution.
Electrocardiography (ECG) the most direct and reliable signal for capturing and analyzing the electrical activity of the heart, which is essential for understanding cardiac function. The ECG signal is generated by the depolarization and repolarization of myocardial cells during the cardiac cycle. This electrical activity originates in the sinoatrial (SA) node, often referred to the heart's natural pacemaker, which initiates an electrical impulse. The impulse propagates through the atria, causing atrial contraction, and is then delayed the atrioventricular (AV) node to allow proper
16

---

Published as a conference paper at ICLR 2025
Table Prediction errors for ED and ES frames, calculated by comparing the segmented frames from the ground-truth video and the generated video. These enrors validate the consistency of cardiac motion the generated video and provide insights into whether the model accurately follows ECG- driven dynamics.
Methods
Average Error (ED).
Average Error (ES).
EchoDiffusion Reynaudd et al. 2023
1,55
1.25
BchoNee-Symbheteekeeee et 2024 ECHOPulse (Our Model)
1.06
0.97
0.18
0.21
ventricular filling. It subsequently travels through the bundle of His, bundle branches, and Purkinje fibers, leading to coordinated ventricular contraction. The resulting electrical activity produces the characteristic waveforms observed in ECG: the P wave (atrial depolarization), the QRS complex (ventricular depolarization), and the wave (ventricular repolarization).
Ejection Fraction (EF) is critical parameter for evaluating cardiac function, representing the per- centage of blood volume ejected from the left ventricle during systole relative to the total volume at end-diastole. It calculated using the left ventricular volumes at two key phases of the cardiac cycle: end-diastole (EDV/LVED) and end-systole (ESV/LVES). The general formula for EF is expressed as:
EDV ESV
EF
x 100%
EDV
or
LVED LVES EF
100%
LVED
A.4 ADDITIONAL GENERATED SAMPLES AND ANALYSIS
We asked one experienced ECHO clinician with over 10 years of expertise and a postdoctoral re- searcher with years of experience in ECHO engineering manually evaluate 100 video samples. Among these, 50 were generated samples, and 50 were true samples from the original dataset. The evaluators were tasked with classifying each video as real or fake. The expert accuracies were 51.17% and 49.39%, respectively. Both experts reported that the generated videos were nearly in- distinguishable from the real ones.
Additional video generation examples are shown in the Figure 45 and 7 The comparison among ECHOPulse, EchoDiffusion and SchoNet--Snttheie examples can seen in Figure g and | Both EchoDiffusion and EcchNNet-Snthhetic exhibit issues such as missing details, inaccuracies in valve structures, and incorrect valve motion phases. In contrast, ECHOPulse demonstrates significant improvements in these aspects, highlighting the advantages of using ECG control condition. However, ECHOPulse still has failure cases under extreme conditions, such as poor-quality condi- tion images and ECG signals that deviate from normal pattern, shown in Figure T0 Visualization results of model's reconstruction results are Figure
17

---

Published as a conference paper at ICLR 2025
Prediction
Conditions
+
Ground-Trre Truth
Figure 4: A2C video generation example from ECHOPulse.
Prediction
Conditions
+
Ground-Trus Truth
Figure 5: A2C video generation example from ECHOPulse.
18

---

Published as a conference paper at ICLR 2025
Prediction
Conditions
+
Ground Truth
Figure 6: A4C video generation example from ECHOPulse.
Prediction
Conditions
+
Cround-Trulli
Figure 7: A4C video generation example from ECHOPulse.
19

---

Published as a conference paper at ICLR 2025
Condition Image
Ground-Tnud
EchoDiffusion
Loss details
Blur caused by eroor accunulation
=0
EchoNet-Syuule Syythetic
Valvular structural eror
OR
Detail blur
ECHOPulse
MWm wWm
Figure 8: A2C video generation comparison results among ECHOPulse, EchoNer-Synthetic.
EchoDiffusion and
Condition Image
Ground-Trut
EchoDiffusion
EF
EchoNet Synthetic
Valvular missing
Valvular motion corrsponding phase emor
OR
Valvular structural emor
Detail blur
PCse
Figure 9: A4C video generation comparison results among ECHOPulse, EchoDiffusion and Syhthee--Snthettic.
20

---

Published as a conference paper at ICLR 2025
Condition Image
Ground-Truth
ECHOPulse
Figure 10: ECHOPulse's failure case occurs when encountering extremely poor-quality condition images and abnormal ECG patterns that deviate from typical physiological norms.
Original video
Reconstruction results of ECHOPulse
Reconstruction results EchoNet-Synthetic
Deñais missing
Details missing
Figure Reconstruction results comparison between ECHOPulse and Schthee--ynthhetic.
21


Reviews:
[{"review_id": "jspDXcoG9I", "reviewer": "Reviewer_A3Vb", "summary": "The authors propose the first Echocardiography generation model conditioned in ECG signals named ECHOPulse. The overall pipeline ECHOPulse consists of three stages. The first stage is video tokenizer training, which utilizes a VQ-VAE based model to train the tokenizer for ECHO video quantization. The second stage is to align the ECG and video tokens, and the ECG-FM is used to encode the ECG and masked token prediction is used for alignment. The final stage is video generation, which is autoregressive video token generation conditioned on ECG tokens.", "questions": "1. As mentioned in the weaknesses section, was there a specific reason the qualitative comparison for tokenization models (reconstructed video comparison) and video generation models (generated video comparison) were not conducted?\n2. Instead of using non-overlapping patches for video tokenization, would the VQ-VAE based tokenization improve performance using overlapping patches? Was this explored experimentally?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper contains novel contribution in that it is the first paper to present Echo video generation conditioned on ECG signals, and also plan to release the ECHO videos paired with ECG dataset.\n\n2. The paper is well-written and easy to follow, with the pipeline figure covering all three steps used in EchoPulse.\n\n3. The experiments compare the tokenization performance and video generation performance, and compares to other related papers despite this paper being the first to explore ECG-guided ECHO video generation.", "weaknesses": "1. The video tokenization performance results in Table 1 compare the reconstruction metrics of MSE and MAE between EchoNet-Synthetic and ECHOPulse. However, there should also be qualitative result comparison to compare the reconstruction results between the two tokenization models, similar to qualitative video generation results in Figure 3. \n\n2. Similar to the video tokenization weakness above, the video generation experiments also do not contain any qualitative comparison between other models such as MoonShot, VideoComposer, and HeartBeat. The figures only display the qualitative results of ECHOPulse under different ECG conditions. Therefore, qualitative results should be displayed for each models. \n\n3. As mentioned in the Limitation and Future Works section, instead of using GAN loss for ECHOPulse, it will be better to see combination of diffusion models in this pipeline for ECHO video generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UyuNSXyt9n", "reviewer": "Reviewer_5Znn", "summary": "This manuscript propose ECHOPULSE, a method to synthesize echo videos from easily accessible ECG signals. The method creatively uses the ECG signal, which is highly consistent with the ultrasound video, as a condition to generate the echo video. Experiments on multiple datasets and provided demos show excellent performance.", "questions": "Questions:\nSee Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths: \n1) Significance: The most important contribution of this manuscript is the use of easily accessible ECG signals as control conditions to generate echo videos. The proposed method tries to combine these two modalities of video data and time series data (ECG), which is significant for medical image analysis and provides new research directions. The synthesized video data can be applied to many downstream tasks and the methods used can be generalized to more tasks.\n2) Originality: Although this paper uses a several of the published techniques, the work centers on synthesizing echo videos using more temporally informative time-series data, rather than using text or otherwise.\n3) Quality: The manuscript is clearly structured and accurately presented, and the figures and tables are of high quality.\n4) Reproducibility: the authors provide detailed experimental procedures and details in the main text and appendices with corresponding code, which greatly helps in the reproduction of this work.", "weaknesses": "Weaknesses:\n1) Although ECGs can be easily obtained from wearable devices, how should condition images be obtained when the method is applied in practice? How to ensure that the condition image is reasonable?\n2) Compared with the previous method, the generation of real-time has been greatly improved, but still can not reach the real-time generation requirements. It is not clear how much inference resources are consumed by this method (Flops, GPU Mem et al). This determines whether it can be deployed on wearable devices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RXEwJDVGbU", "reviewer": "Reviewer_31ri", "summary": "The paper introduces ECHOPulse, an ECG-controlled ECHO video generation model that creates echocardiography (ECHO) videos using only ECG signals as input, claiming that it allows bypassing the need for complex conditional annotations. ECHOPulse uses a VQ-VAE-based tokenization and a transformer model to align ECG with video data, achieving efficient, realistic video generation suitable for scalable clinical applications.", "questions": "- How would such method actually have clinical impact ? \n- How much do the generated data contribute in training downstream tasks ? \n- How would we ensure that the model doesn't hallucinate and provide clinically irrelevant  or harmful information ?\n- Why there is no thought given to potential confounding factors that would effect the generating process ?\n\nOverall the method is over engineered, with limited guardrails and questionable clinical significance", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The method shows potential generalizability to other medical video modalities, like cardiac MRI and CT, indicating versatility beyond ECHO generation.\n- The paper is novel and interesting\n- The language and presentation is proper", "weaknesses": "- The use of comic sans font on the figures should re-evaluated \n- High model complexity and memory demands may restrict usability on devices with limited processing power, potentially limiting real-time deployment in resource-constrained settings, such as medical ones.\n- The method is over-engineered\n- The premise of the paper and the approach taken actually have semantic issues. The difficulty in ECHO data comes from the poor quality of the data as the authors correctly point out, however , to the best of the reviewers knowledge, the solution is not to create synthetic to monitor patients when only ECGs are provided. Details in the videos and images have clinical significance and the proposed method makes no attempt to ensure fidelity and the trustworthiness of the generated videos, making clinical significance limited \n- There is no mention of medical professionals evaluating the quality of the videos and how realistic they are \n- There is no discussion about confounding factors that could influence  the generation of the videos and hence lead the model to hallucinate \n- There is insufficient discussion about the use of such a video generation tool to train downstream models\n- Going from an ECG to a video is an information creating process and the method makes no attempt to constrain this so the model does not generate implausible or unrelated to the patient information. Especially when there is no conditioning image/video", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wpb1mvuAIT", "reviewer": "Reviewer_vh23", "summary": "The authors propose Echopulse, a video diffusion model conditioned on time-series data. Their pipeline is split into different parts including a tokenization step for efficient generation, a feature alignment step to align visual features with time series data, and a generation step for video generation. \nIn general, the paper is well written and easy to follow. \nWhile their contribution is clear, I am not convinced that the proposed method is better than what was previously proposed in the literature. The main experiments lack exhaustive comparison (eg. Tab.1 private data, tab2 private data) and multiple things proposed in the paper are not properly validated. Overall I give a weak reject.", "questions": "Do you have details about how good the models are without pretraining on natural data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I believe the idea of aligning the representation of frames with ECG data is very good and has a large potential to work for other domains as well. \n\nThe writing is very clear and from beginning to end it is easy to understand what the authors are doing. \n\nI believe that conditioning videos on time-series data could have a strong impact even outside of ultrasound videos.", "weaknesses": "There is not enough evidence supporting the claim that the generated videos follow the ECG signal. The LVEF estimation presented in Table 3 only considers ED and ES phases which are only a part of the ECG cycle. Furthermore, the results on EF estimation are inferior compared to previous methods in terms of R2 score which is the gold standard metric for estimating EF. Figure 3 only shows one example and is difficult to understand. Maybe it would help to add LVED and LVES values to the frames. \n\nTable 3 is missing references. Why is EchoNet-Synthetic not mentioned as a baseline in Table 3? Their inference time is below 3 seconds. \n\nIt is unclear how much influence reconstruction has on the final results. While the reconstruction results of the method presented are best, I am not convinced that the difference will be visible when using the method for downstream tasks such as video generation or EF estimation. I think it would be helpful to provide metrics such as rFID or rLVEF (reconstruction LVEF) to get an estimate to how much information is lost through encoding and how much better EchoPulse is compared to previous methods. \n\nInconsequent comparison: Why are there no generative results for EchoDiffusion or EchoNetSynthetic on Private data? Also, Table 1 misses the EchoNet-Synthetic baseline. \n\nMotivation: While I think the application is exciting I believe there is a lack of motivation for ECG to video generation in particular. It would help if you could further motivate. Especially on what is the motivation of doing ECG conditioned generation over EF condtioined video generation. \n\nMinor: \nTable 2 should be separated into all datasets to make comparison easier.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6cxNJzdkw5", "reviewer": "Reviewer_F9gf", "summary": "The authors propose a Transformer and VQVAE-based generative model for echocardiogram generation, conditioned on ECG signals. This is a novel approach that seeks to explore ECG-based conditioning for controlling cardiac motion during echocardiogram generation, a relatively under-explored area.", "questions": "- Can you provide specific details or comparisons that validate the claims in the abstract regarding the computational cost, slow inference, and complex conditioning of existing methods? Why was EchoNet-Synthetic not included when discussing generation times?\n\n- There appears to be some confusion between the works of Reynaud et al. (2024) and Ouyang et al. (2020) in Section 2.3. Could you clarify the differences between these works and how they relate to your methodology?\n\n- The paper suggests that prior works such as EchoDiffusion and EchoNet-Synthetic use complex conditional prompts like segmentation masks and clinical text, but these models actually use image and EF for conditioning. Could you revise this section to more accurately reflect the conditioning methods used in these studies?\n\n- The loss function presented in Equation 1 lacks references to foundational works. Could you cite the relevant literature that inspired or derived this loss, such as Gu et al. (2022), Rombach et al. (2022), or Esser et al. (2021)?\n\n- The comparison between VQ-VAE and diffusion models seems incomplete, as the EchoNet-Synthetic metrics presented relate only to the VAE model. Could you clarify how these models were compared and whether it is appropriate to conclude that VQ-VAEs surpass diffusion models based on these metrics?\n\n- How were MoonShot and VideoComposer evaluated for this specific task? Did you replicate their results, or were the metrics taken from the HeartBeat paper? Additionally, can you clarify how SSIM was computed, considering it typically requires paired input/output data, which is not evident for ECHOPulse?\n\n- EchoNet-Dynamic contains only A4C views. How was ECHOPulse evaluated on A2C views, given that these metrics are not reported in EchoDiffusion and EchoNet-Synthetic?\n\n- The comparison with baselines such as MoonShot and VideoComposer appears unfair, given that these are general-purpose models. Could you explain how the experimental setup ensures fair comparison with domain-optimized models, and why only the ablation results of HeartBeat were compared?\n\n- EchoNet-Synthetic is conspicuously absent from Table 3. Why was this model excluded from the comparison, and how would including it impact the conclusions?\n\n- The method for calculating LVEF in this paper differs significantly from that used in the EchoNet-Dynamic dataset. How do you account for this discrepancy, and does it affect the validity of the comparisons?\n\n- While the novelty of using ECG signals is appreciated, the claim that your model achieves state-of-the-art results seems overstated. Can you clarify the specific advantages of your model compared to EchoNet-Synthetic and HeartBeat, without overstating the performance? \n\n- The qualitative results suggest that the visual quality of ECHOPulse samples is lower than that of competing models, but this is not explicitly acknowledged in the text. How do you address this issue, and why were FID and FVD metrics chosen for evaluation, given their limitations for this task?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The use of ECG signals for controlling cardiac motion in echocardiogram generation is a novel and under-explored area, making the concept innovative and potentially impactful.\n- The methodology is well-grounded in leveraging VQVAE tokenization for fast decoding, which addresses some of the computational bottlenecks in generative models.", "weaknesses": "several concerns and inaccuracies in the paper need to be addressed for clarity and rigor.\n\nFirst, the abstract claims that existing methods have high computational costs, slow inference, and rely on complex conditioning prompts. This statement is not entirely accurate. For example, the paper does not provide a comparison with EchoNet-Synthetic when discussing generation times, which is essential for validating such claims. Neglecting such a comparison is poor practice and can mislead the reader about the novelty and efficiency of the proposed method.\n\nIn Section 2.3, there is a reference to \"Reynaud et al. (2024) introducing EchoNet-Dynamic.\" However, this passage is unclear, and it appears there might be confusion between the works of Reynaud et al. (2024) and Ouyang et al. (2020). Clarifying this would help avoid misattribution of prior work.\n\nAdditionally, the statement that these studies are conditioned on carefully curated prompts such as segmentation masks and clinical text is not fully correct. Only HeartBeat (Zhou et al., 2024) uses these forms of conditioning. EchoDiffusion (Reynaud et al., 2023) and EchoNet-Synthetic (Reynaud et al., 2024) use image and EF as conditioning inputs. This oversimplification of related work needs to be corrected.\n\nFurthermore, Eq. 1 lacks proper citations. The loss function used in this equation appears to be inspired by previous works, yet no references are given. The authors should consider citing relevant works such as Gu et al. (2022) [1], Rombach et al. (2022) [2], and Esser et al. (2021) [3] to give due credit.\n\nSection 4.1 presents an incorrect conclusion. The numbers extracted from the EchoNet-Synthetic paper reflect only the VAE model metrics, without involving the diffusion model. Similarly, the metrics for ECHOPulse involve only the VQVAE, not the token-prediction model. Therefore, the comparison between VQ-VAEs and diffusion models is not valid, and the conclusion that VQ-VAEs outperform diffusion models cannot be drawn from this comparison.\n\nIn Table 2, there are several points that need further explanation. It is unclear how MoonShot and VideoComposer were evaluated for this task, as the numbers seem to have been taken from the HeartBeat paper without further elaboration. Additionally, EchoNet-Dynamic includes only A4C views, so it is not clear how ECHOPulse was evaluated on A2C views, given that these metrics are not reported in EchoDiffusion and EchoNet-Synthetic. The SSIM computation is also problematic, as SSIM generally requires paired input/output data, and it is not clear how ECHOPulse reconstructs a video. Moreover, how the text conditionings for CAMUS and EchoNet-Dynamic were generated should be clarified. Lastly, the table construction raises questions, particularly why HeartBeat and EchoNet-Synthetic are singled out at the end. \n\nSection 4.2 discusses the comparison with baselines, but the experimental setup lacks sufficient explanation, making the comparisons appear unfair. MoonShot and VideoComposer are general -- purpose generative models, so it is unsurprising that they perform worse than domain -- specific models. HeartBeat’s use of multiple conditions emphasizes the importance of using as many conditioning variables as possible. However, only the ablation results of HeartBeat are compared to other methods, which limits the value of the comparison. Furthermore, the conclusion of this section is poorly formulated. While the qualitative results clearly indicate that the visual quality of ECHOPulse samples lags behind, the conclusion only hints at this issue without directly acknowledging it. Moreover, the FID and FVD metrics are poor indicators of quality in this specific task, and this should be discussed more critically.\n\nIn Table 3, EchoNet-Synthetic is conspicuously absent. Including this model would provide a more complete comparative analysis.\n\nIn the paragraph following Table 3, the method for calculating LVEF in this paper is substantially different from that used for the EchoNet-Dynamic dataset, making direct comparisons problematic. This discrepancy is not adequately addressed, and the conclusion is again misleading, as EchoNet-Synthetic is not included in this analysis, even though it shows superior performance in all reported metrics.\n\nSection 5 makes some strong claims about the performance of the proposed approach. While the novelty of the idea is clear, the assertion that it achieves the best results is far-fetched. Previous works such as EchoNet-Synthetic and HeartBeat have demonstrated better performance. The authors should focus on highlighting the innovative aspects of their method rather than making overstated claims about its superiority.\n\nOverall, using ECG signals for ECHO video generation is an intriguing approach. The idea of using ECG is interesting, the method (VQVAE + Transformer) is less common in the field if image/video genetation, similar to Phenaki [4]. However, the literature review and baseline comparisons are inadequate, and the paper is riddled with oversights and false conclusions. A more thorough review and revision of the comparisons and claims would significantly improve the work. Additionally, proper citations for key components of the methodology are necessary.\n\nReferences:\n[1] Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., and Guo, B., 2022. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10696-10706).  \n[2] Rombach, R., Blattmann, A., Lorenz, D., Esser, P. and Ommer, B., 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).  \n[3] Esser, P., Rombach, R. and Ommer, B., 2021. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 12873-12883).  \n[4] Villegas, R., Babaeizadeh, M., Kindermans, P.J., Moraldo, H., Zhang, H., Saffar, M.T., Castro, S., Kunze, J., and Erhan, D., 2022, October. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)