Bidirectional Recurrence for Cardiac Motion Tracking with Gaussian Process Latent Coding
Authors: Jiewen Yang
Yiqun Lin
Bin Pu
Xiaomeng Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Medical Image Analysis
Cardiac Motion Tracking
Primary area: machine_learning_for_healthcare
TL;DR: Use bidirectional Recurrent Manner Equip with Gaussian Process for Cardiac Motion Tracking

Abstract: Quantitative analysis of cardiac motion is crucial for assessing cardiac function. This analysis typically uses imaging modalities such as MRI and Echocardiograms that capture detailed image sequences throughout the heartbeat cycle. Previous methods predominantly focused on the analysis of image pairs lacking consideration of the motion dynamics and spatial variability. Consequently, these methods often overlook the long-term relationships and regional motion characteristic of cardiac. To overcome these limitations, we introduce the GPTrack, a novel unsupervised framework crafted to fully explore the temporal and spatial dynamics of cardiac motion. The GPTrack enhances motion tracking by employing the sequential Gaussian Process in the latent space and encoding statistics by spatial information at each time stamp, which robustly promotes temporal consistency and spatial variability of cardiac dynamics. Also, we innovatively aggregate sequential information in a bidirectional recursive manner, mimicking the behavior of diffeomorphic registration to better capture consistent long-term relationships of motions across cardiac regions such as the ventricles and atria. Our GPTrack significantly improves the precision of motion tracking in both 3D and 4D medical images while maintaining computational efficiency. The code is available at: https://github.com/xmed-lab/GPTrack.

Bidirectional Recurrence for Cardiac Motion Tracking with Gaussian Process Latent Coding
Jiewen Yang Yiqun Lin
Bin Pu Xiaomeng Li
The Hong Kong University of Science and Technology (jyangeu,
y ylindd))cconect. ust.hk {eebinpu, eexmli]
Abstract
Quantitative analysis of cardiac motion is crucial for assessing cardiac function. This analysis typically uses imaging modalities such as MRI and Echocardiograms that capture detailed image sequences throughout the heartbeat cycle. Previous methods predominantly focused on the analysis of image pairs lacking considera- tion of the motion dynamics and spatial variability. Consequently, these methods often overlook the long-term relationships and regional motion characteristic cardiac. To overcome these limitations, we introduce the GPTTrack, novel un- supervised framework crafted to fully explore the temporal and spatial dynamics of cardiac motion. The GPTrack enhances motion tracking by employing the sequential Gaussian Process in the latent space and encoding statistics by spatial information each time stamp, which robustly promotes temporal consistency and spatial variability of cardiac dynamics. Also, we innovatively aggregate se- quential information bidirectional recursive manner, mimicking the behavior of diffeomorphic registration to better capture consistent long-term relationships of motions across cardiac regions such as the ventricles and atria. Our GPTrack significantly improves the precision of motion tracking in both 3D and 4D medi- cal images while maintaining computational efficiency. The code available https//ggituu..com/mmeedaaaaa
1 Introduction
Cardiac motion tracking from Cardiac Magnetic Resonance Imaging (MRI) and Echocardiograms is crucial in quantitative cardiac image processing. These imaging techniques provide comprehensive image sequences that cover an entire heartbeat cycle, allowing for detailed analysis of cardiac dynamics. Conventional non-parametric cardiac motion tracking approaches, such as B-splines [1], Demons algorithms [2] and optical-flow based methods [3, 4, 5], are commonly utilized due to their flexibility and ability align detailed structures within images. However, these methods constraints
face significant challenges in motion tracking because they lack and temporal coherence. The diffeomorphic registration method [6. 7, 8). which formulates the registration process as group of diffeomorphisms in Lagrangian dynamics, is nice candidate for ahpology-preeerving motion tracking. However, traditional optimization-bnn based diffeomorphic registration methods are computationally intensive and sensitive to noises, hindering their applications in efficient cardiac motion tracking-
Current deep learning-bass based techniques [9, 10, 11, 12, 13, 14, 15, 16) employ these advanced
imaging modalities register image pairs within the same patient, and some [14, 15, 16] adopt the diffeomorphic routine and learn the Lagrangian strain to describe the motion relationship between the reference frame and subsequent frames, aggregating dynamic information into consecutive cardiac motions as Lagrangian displacements, Although the above diffeomorphic methods better model the dynamic and continuous nature of cardiac motion, they still have room for improvement in handling the long-term temporal relationship in videos. For example, the approach [14] requires segmentation
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Right trium Myocardium
Ead dizatol
Frane
Frane?
Eroeigieoe
Trane
Frame
Eud disande
Frame335
Cadaa:
oi Maki Trijeoyy
Regional Deformation Dimecton and Intensity Between Right Arrum and Myocardium
End-dlastale
End-dlastole
End diastale
End-syytole
End-systole
Cantac Traestion Nodeted 25 Prss Krowdedg: VS Staedinn Frocess
End-s
End-systole
End-diastole
End-dlastele
Fnd-dlastale
Figure 1:
Regional Motions in Cardiac: The left sequential MRI frames within a heartbeat cycle
illustrate that motion direction and intensity completely different between the right atrium and myocardium during End-diastole and End-systole. Formulate Cardiac Motion as Prior Knowledge: The right figure depicts the regions of motion trajectory across the heartbeat cycle, alongside the probability distributions of motion trajectory. Curves (Middle) are the motion trajectory changes of different MRI sequences (Left). Highlighting the cardiac motion trajectory that follows a certain pattern can be modelled as prior knowledge via the Gaussian Process (Right).
annotation to generate dense motion trajectories and calculate Lagrangian strain. Additionally, the methods outlined in [15, 16] are prone to error accumulation as Lagrangian displacements are integrated without regularizing temporal variations. In spatial views, while the global trajectory flow may follow specific pattern, significant variations exist within each region regarding the phases, amplitudes, and intensities of the motion. For example, Figure shows regions of the Right Atrium (red) and Myocardium (green) performing the opposite trajectories during the heartbeat cycle. Conversely, similar regions across different cases exhibit consistent motions. Hence, ignoring the regional scale may lead to fragmented understanding of cardiac motion, underscoring the need for more nuanced analytical approaches. Furthermore, as shown in the right of Figure 1, the deformation bounded in the space of periodically specific human cardiac motion variation.
To leverage discussed temporal and spatial information for cardiac motion tracking, we propose novel framework named GPTrack. Our GPTrack has several appealing facets: 1) GPTrack employs the Gaussian Process (GP) to formulate the consistent temporal patterns in the latent space of diffeomorphic frameworks, promoting the consistency of cardiac motion; 2) GPTrack utilizes position information in the latent space to encode the statistics of the sequential Gaussian process, by which model the region-specifts motion and obtain a more precise estimation related cardiac motion; 3) GPTrack leverages the inherent temporal continuity in cardiac motion by aggregating long-term relationships through forward and backward video flows, which mimics the Borward-backwerr manner of classical diffeomorphic registration framework [6]. To evaluate the performance of GPTrrakk in cardiac motion tracking, conduct experiments based 3D Echocardiogram videos [17, 18] and 4D temporal MRI image [19]. Results in Tables 1, 2 and 3, show the GPTrack enhance the accuracy of motion tracking performance in clear margin, without substantially increasing the computational cost in comparison to other state-of-the-ar methods. Our contributions are summarized as follows:
1. We propose novel cardiac motion tracking framework named the GPTrack. This framework employs the Gaussian Process (GP) to promote temporal consistency and regional variability in compact latent space, establishing robust regularizer to enhance cardiac motion tracking accuracy.
The GPTrack framework is designed to capture the long-term relationship of cardiac motion via a bidirectional recursive manner, and its forward-backwant manner mimics the workflows of the classical diffeomorphic registration framework. By this approach, our method provides a more accurate and reliable estimation of cardiac motion.
Our GPTrack framework achieves state-of-thes performance on both 3D Echocardiogram videos and 4D temporal MRI datasets, maintaining comparable computational efficiency. The results demonstrate that our method adapts effectively across different medical imaging modalities, proving its utility in different clinical settings.
Related Work
2.1 Cardiac Motion Tracking via Non-parametric Registration Approach
Extensive works have been proposed to address registration by optimising within the space of displacement vector fields. Models related to elastic matching were proposed by [20, 21]. [22] utilized

---

statistical parametric mapping for improvement. Techniques incorporating free-form deformations with B-splines and Maxwell demons were adapted by [1] and [2], respectively. The Harmonic phase-based method, which utilizes spectral peaks in the Fourier domain for cardiac motion tracking, was introduced by [23]. This method calculates phase images from the inverse Fourier transforms and is specifically exploited in the analysis of tagged MRI. Popular formulations [6, 8, 24, 7] introduce diffeomorphic transforms. In the realm of diffeomorphic registration, inverse consistent diffeomorphic deformations have been estimated by [6] and [7]. Syn [24] proposed standard symmetric normalization. RDMM [8] considered regional parameterization based on Large Deformation Diffeomorphic Metric Mapping [6]. Despite their remarkable success in computational anatomy studies, these approaches are also time-consuming and susceptible to noise.
2.2 Cardiac Motion Tracking with Deep-Learning based Registration Method
Recent advancements in medical image registration have increasingly leveraged Deep Learning technologies. Pioneering studies [14, 25, 26, 27, 28] utilize ground truth displacement fields obtained by simulating deformations and deformed images, typically estimated using non conammmmeriic methods. These approaches, however, may be limited by the types of deformations they can effectively model, which can affect both the quality and accuracy of the registration. Unsupervised methods, as discussed by [9, 10, 15, 29, 30, 31, 32] have shown promise by learning deformation through the warping of fixed image to a moving image using spatial transformation functions [33]. These methods have been extended to include deformable models for single directional deformation field tracking [9, 26, 34] and diffeomorphic models for stationary velocity fields [32, 35, 36J. Further application of diffeomorphic models to cardiac motion tracking has been explored by [13, 15, 16, 30, 31j. These models predict motion fields that are both differentiable and invertible, ensuring one-to-one mappings and topology preservation. Recent studies in denoising diffusion probabilistic models (DDPM), such [11] and [12], have achieved considerable success in registration tasks. However, DDPM- based methods face challenges in building temporal connections and demand substantial computational resources. The DL-based optical flow (OF) methods [37, 38, 39, 40] apply widely in nature image motion tracking. However, illustrated in [7, 15, 16), due annotations requirements and photometric constraints, they cannot be adopted in unsupervised cardiac motion tracking in medical image domains (See Section Al in Appendix for detailed discussion).
Methodology
3.1 Diffeomorphic Tracking of Cardiac Motion
Diffeomorphic motion tracking and registration techniques are widely used in medical image analysis be- cause they seek topology topology-preserving preserving mapping between source and target images [6. 8]. Formally, given the source image za and target image .1, the diffeomorphic registration aims at family of differentiable and in- vertible mappings ( (dthe with the boundary condition <n = Id and di (zo) = I1, where Id is the iden- tity mapping. The diffeomorphism
X
X2
GPTodi
Lave
Posicional
Enoouiig
Encoder
Encodar
GP
GP
Decoder
Decoder
Dexoder
Drander
$
db
$
(a), Our GPTrack Framework
(b). Conventional Registration Framework
Figure 2: Comparsion between our GPTrack (a) and conven- tional registration framework (b).
&e can be parameterized as its derivatives (velocity field) v, as follows:
dø: dt
v.(()) ve Φ = Φb -o + S vo(d))ds, SE [0,1),
(I)
where o is the composition operator. For numerical implementation, the associative property of diffeomorphism group indicates &i = Φi, o Φi.. and the integral of Equation can be approximated by Φt+8 R Φ, + v,d for ( E [0, 1) and small enough d. In this study, we follow
the parameter settings of [7, 15, 16, 35, 36) and take å 2w. N 7 to discretize the path of diffeomorphic deformation.
3

---

GPTrack Pipeline
GPTrack Cell
1t
X
GPTradk
Spatial Postioonal Knonding
Rg-1
Linear Self ttettonn
GPTrack Cell
Layer
R
LayerNorm
LayerNorm
Ro h-1
Queyy
Key
Value
GPTrack Cell
Linear Self A Ateentonn
hi
h hha
Ar
50:00 +
Zo Mt-1
GP
fig
LayerNorn
A
Temporai Fosttiina Enrnding
Feed Forward
fe
Linear
Decoder
ses Positional Encode
Duta flow
+ Flement wise A ddittio
x Cruss grodict
Figure 3:
The overview pipeline of GPTrack (one layer). The æ h, h and denote input, forward hidden states, backward hidden states and latent coordinates. Feature St with probabilistic prior on the latent space via Gaussian Process then enters the decoder to predict the motion field ø. Subscript t denotes the 1-th position in total moments. elu(.) represent the exponential linear units [42].
3.2 Motion Tracking with Gaussian Process Latent Coding
Our proposed method adopts the generative variational autoencoder (VAE) framework as the backbone of a diffcomorphic tracking network, as suggested by previous methods [15, 16, 36]. However, as shown in Figure 2, the first difference between our method and other methods that ours allows the registration network to aggregate the spatial information temporally. both forward and backward (see Figure 2(a)). The conventional approaches [9, 10, 15, 16, 24, 31] only conduct between two adjacent frames, which ignore the relationship of long-term dependency of the cardiac motion (see Figure 2(b)). Secondly, despite the large deformation through the path of diffeomorphism, the space of periodically specific human cardiac motion variation is bounded. Though methods [14, 15, 16] use Lagrangian strain to formulate the continuous dynamic of cardiac motion, however, without considering the motion consistency between two adjacent state spaces, the Lagrangian strain is prone to error accumulation and degrade the tracking performance. To address this problem, we take the simple yet efficient Bayesian approach, which employs the Gaussian Process (GP) to model cardiac motion dynamics in the compact feature space, predicting more consistent motion fields over dynamics parameters. Our proposed GPTrack can also be easily extended to other modalities or motion-teee motion-tracking tasks, such as 3D Echocardiogram videos and 4D cardiac MRI.
In this paper, we follow the research [41] that employs the recursive manner in transformer for sequential data. As shown in Figure 3 left, the GPTrack pipeline comprised the GPTrack layer for feature extraction, the Gaussian Process (GP) layer for modelling the cardiac motion dynamics, and the Decoder for motion field estimation. Given the sequential 4D inputs {x1]T=1 23 E I2 xW Dx1 where H, W, D. I denote the height, width, depth, length of the input. For each It, we first decompose it to P non-overlapping patches of shape p X p X p. where P 2 W. x and p, H W. E zt p
We then embed each patch as a feature with C' channels via embedding layers and disentangle patches with the dimension of R from de. The GPTrack layer then takes the x. both forward and backward hidden states h, h E PxC as the input, then predicts the motion field ó via the decoder after the GP layer. Note that the initial hidden states of forward hg and backward hg are set as zero
3.3 Bidirectional -arcaa-rd Recursive Cell
The GPTrack layer consists of two independent GPTrack cells that respond to forward and backward computation. Similar to the [41, 43], adapting the hidden state maintain and aggregate the sequential information allows the input with variable length. Meanwhile, the conventional convolutional neural network or vision in the wranformer-baseed method limited by the fixed input length. Furthermore, medical images such as Echocardiogram videos usually consist of hundreds of frames that cover multiple heartbeat cycles. Hence, parallel computing all frames requires a large amount of computational consumption, which hinders the application in real scenarios limited by low- computational devices. To this end, as shown in Tables and 2, our GPTrack is able to formulate the variable temporal information while maintaining the comparable computational cost.

---

Using the forward GPTrack cell shown in the right of Figuer 3 as example, the and h followed by the addition of learnable position encoding pos, E R PxC are respectively normalized by Layer Normalization. The linear self-attention then computes the attentive weight A, PRC of combined 2y and 1 The above operations can formulated as follows:
A = (6(W@=)++++++ +)(((()==+++))
x LN(x, + pos,) @ LN + pos,) E R Px22 (2)
where Wq. WK. WV E RCxC are learnable weights of different projection layers named query, key and value, 8(-) represent the exponential linear units elu(-) [42], LN(.) and are the layer normalization and the concatenation operation, respectively.
In order to raise the descendant hidden state hy that aggregates information before t + moment. The attentive weight A, then conducts the element- wise addition with ancestral positional encoded h4-1. Additionally, the A, takes the positional encoded ze as the residual connection and applies the addition operation. The Feed Forward Network denoted as FFN(-), is then introduced to output the feature 1-th moment. The formulation can be written as follows:
h2 = A, + LN( E PxC, J, = FFN(LN((A, A, + LN(2,, + pos_))) E E PXC.
(3)
In the Bidirectional Forward- -ackwaar Recursive Cell, both forward and backward share the same computation processes through the GPTrack cell. The only difference between the two directions is that the forward process starts from the first moment zo of input while the backward starts from the last moment Zy. Hence, the feature in t-th moment is formulated ft fi + Fe, fe R where St aggregate the forward information from 0 to I, while the Je aggregate the backward information from the last frame to the (-th frame
3.4 Gaussian Process in Cardiac Motion Tracking
The primary objective of integrating the Gaussian Process (GP) is to establish probabilistic prior in the latent space that incorporates prior knowledge. Specifically, it posits that cardiac motion across different individuals within the same region should yield similar motion fields in latent space encodings. Furthermore, as illustrated in Figure cardiac structures within an individual that are spatially distant or exhibit motions in opposite directions should consistently adhere to the periodic pattern of motion.
Initially, we define a covariance (kernel) function for the GP layer as depicted in Figure 3. We design the prior for the latent space processes to be stationary, mean square continuous, and differentiable in sequential motion fields. This design stems from our expectation that the latent functions should anticipate the latent
model cardiac motion more prominently than visual features. Consequently, space to manifest continuous and relatively smooth behaviour. To this end, we employ the isotropic and stationary Matern kernel (refer to Equation 4) to fulfil the required covariance function structure:
21-p K(0,, xT-1) = T(v) (V2v D(51, K'K D(11,
1
1
where a, > O are the smoothness, magnitude and length scale parameters, K, is the modified Bessel function, and D(...) denotes the distance metric between features of two consecutive motion fields. Our goal is to formulate cardiac motion robust prior knowledge applicable to unseen data. To address this, we propose position- related distance measurement.
As outlined in section 3.3 and referenced in [44], we utilize a learnable parameter pôs E RPxC the spatial positional encoding for each region, which provides relative or absolute positional information about the decomposed patches. To capture the periodic temporal positional information of cardiac motion. distinct from the spatial encoding pos, we apply sine and cosine functions of various frequencies as temporal encoding {pōs, ) pos, E R. 1'C' for each moment. The overall positional encoding at moment is formulated as:
pos, = pós pos, E PxC. where (t, i) L(i=zaysin(t n #) (1i22411)00((t )) (5)
The i denotes the i-th position of channels, and n the scaling factor. We assign independent GP priors to all values in { Rt BBxx to disseminate temporal information between frames
5

---

in the sequence. During this process, we regard the sequential output {ft of GPTrack as noise- corrupted versions of the ideal latent space encodings, formulating the inference as the following GP regression model with noise observations:
2x GP (u(pos,),* x. pos,)) ft 2t + ct, &t (O
where o² is the noise variance of the likelihood model set as the learnable parameter in GPTrack. The above Gaussian process can be treated as the temporal sequence with intrinsic Markov property, and we adopt the methodology of connecting the Gaussian process with state space model [45] to decrease its computational complexity from O(T*) to O(T), where T is the number of time step. Concretely, the Gaussian process Equation 6 corresponds to the following linear stochastic differential equation:
di zt(() - Az(t) + bwt(t, f(t) = h z(t) + c(t), e(t) ~ <(t)
~N0,00))
(6)
with the solution as:
ztt) = exp( z(r) + L exp( bu(ssss,
buu)))), Vr <rtt, <t,
((1) - h22(() + E(), <(l) ~ ~(0,00))
where z(() == (=(),, =()). (() is the zero-mean Gaussian random process, and h = (0, 1) is used for modelling observation model. The state transition matrix (vector) A and b can be calculated from the covariance function x of the Gaussian process. For the Matern kemel shown in Equation 4, we take v = 3. and corresponding state transition matrix A and vector b [46] read as:
2.
A 2
0,
2/3 b b= 10,
(8)
Then we can discretize Equation 7 and get its weakly equivalent state-space model of Equation as: Zt D. Z + ny. ft hT te, e(t) ~ (0, of), t t= I...,,,
where c exp Diposs,pos A) n2 ~ No0 F,bQu 1, t)b T and Qw, t - 1) is the co- o²
variance of (t). Given the initial value Z ~ (No, En) with μ4 = 0 and Eg diag( 2 30 we can sequentially calculate the posterior distribution z fi----1- ~ _N Σ,) and x, fia ~ NN(,, E,) using update criterion of Kalman filter for state space model [47] as:
P1 11 μr f pe (ff hTm)). E, + E, k,h H1 t t= =1,1.... I....,T,
E, 2 E
T.- + E. E. $
(10)
where K, := hE,h+²² is the optimal Kalman gain at time t. The output of the GP layer in f-th moment thus can be formulated as 2, ReL,U(k,2)) where ReLU the activation function. In the final, the t-th motion field 0, is obtained by decoder from the 2.P
E,h
3.5 Overall Loss of Tracking a Time Sequence of Cardiac Motion
The decoder takes the Gaussian- process coding {2, Jr=1 as velocity filed to composite the diffeo- morphic motion field & according to the criterion of Section 3.1. Here, we adopt the training loss of [15, 16], which minimizes the integration of four components summarized as follows: a) Dissimilari- ties of tracking results between adjacent states from both forward and backward; b) Smoothness of motion fields between adjacent states from both forward and backward: e) Dissimilarities of tracking results between the start state and each state; d) Smoothness of motion fields between the start state and each state. The overall loss function L is formulated as:
T-1
Σ (ulla)) (aal + f.mm (0411)) + or (nc(E1-1, o @01-1) +as Lom((ee (+1+11)
where 01 a2 and are loss weights. and 61.4, is the motion field from state ti to tz. [EI Fe+1) ) = [KL(g(2 ( ( ( is the summation of forward and backward VAE losses with latent coding 'GT, posterior distribution q and conditional distribution p. Tine is the negative normalized local cross-correlation metric, and Com (Φ) || 2 is the total variation metric.
| xt))

---

4 Experiment
4.1 Datasets
CardiacUDA [17]. The CardiacUDA dataset collected from two medical centers consists of 314 echocardiogram videos from patients. The video is scanned from the apical four-chamber heart (A4C) view. In this paper, we conduct training and validation in the A4C view that consists of 314 videos with 5 frame annotations in the Left/Right Ventricle and Atrium (LV, LA, RV, RA). For testing, we report our results in 10 videos with full annotation provided by the CardiacUDA.
CAMUS [18]. The CAMUS dataset provides pixel-level annotations for the left ventricle, my- ocardium, and left atrium in the Apical two-chamber view, which consists of 500 echocardiogram videos in total. There are 450 subjects in the training set with 2 frames annotated in the Left Ventricle (LV), Left Atrium (LA) and Myocardium (Myo) in the end-diastole (ED) and end-systole (ES) of the heartbeat cycle. The remaining 50 subjects without any annotation masks belong the testing set.
ACDC [19]. The ACDC dataset consists of 100 4D temporal cardiac MRI cases. All data provide the segmentation annotations corresponding with the Left Ventricle (LV), Left Atrium (LA) and Myocardium (Myo) in the end-diastole (ED) and end-systole (ES) during the heartbeat cycle.
4.2 Implementation Details
Training. We trained the model using the Adam optimizer with betas equal to 0.9 and 0.99. The training batch size of the model was set to I. We trained for a total of 1000 epochs with an initial learning rate of 5e and decay by factor of 0.5 in every 50 epochs. During training, for UaA [17] and CAMUS [18], we resized each frame to 384 x 384 and then randomly cropped them to 256 x 256. All frames were normalized to [0,1] during training. In temporal augmentation of datasets [17, 18]. we randomly selected 32 frames from an echocardiogram video with a sampling ratio of either or 2. For ACDC [19], we resampled all scans with a voxel spacing of 1.5 x 1.5 3. 3.15mm and cropped them to 128 x 128 x 32, normalized the intensity of all images to [-1, 1J. For spatial data augmentation of all datasets, we randomly applied flipping. rotation and Gaussian blurring. In CardiacUDA, we split the dataset into 8 2 for training and validation. During testing, we reported results in 10 fully annotated videos. In the CAMUS [18] dataset, videos without annotation are used for only training. while we randomly split the remaining 450 annotated videos into 500/50/100 for training, validation and testing. In the ACDC [19]. following the [11, 12], we split the training set in the ratio of 90 and 10 for training and testing. The reproduced methods strictly follow the official code and the description the paper. For all experiments, We use Intel(R) Xeon(R) Platinum 8375C with 1 x RTX3090 for both training and inference. All reproduced methods strictly followed the training settings with their original paper in the same experimental environment.
Inference. For CardiacUDa and CAMUS, we resized videos to 384 x 384, cropped to 256 x 256 in central and normalized to [0.1]. We sample 32 frames that cover the segmentation annotation. When the sequence has more than 32 frames, the extra frames will be removed from the sequence, except for the first and the last one. The ACDC dataset remains the same sampling strategy as training in the inference stage, without any argumentation except for normalizing intensity to [-1,1].
Evaluation Metrics. For the evaluation of the quality of registered target frames, we follow [12] to use the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) [48] to measure whether the Lagrangian motion field is accurately estimated between the first frame and the following wrapped frames. We also use the Dice [49] score to measure the discrepancy between tracked and ground-truth cardiac segmentation. For CardiacUDA [17], only the first frame and corresponding segmentation are provided for tracking the following 32 frames, then report the averaged results by the above metries in these 32 frames. For CAMUS [18] and ACDC [19], frame and segmentation of the ED stage are used to track the go-after frames, and we report all the metrics in the ES stage. We evaluate diffeomorphic property by computing the percentage of pon-iositive values of the Jacobian determinant (Jo( < 0 (%) on the Lagrangian motion field. In order to access the evaluation of comparing the physiological plausibility following the [50, 51], we also compute the mean absolute difference between the 1 and Jacobian determinant (I]Jol - 1l over the tracking areas. For a fair comparison, we evaluate the computational efficiency and report the computational time in seconds (Times), the parameter quantities in millions (Params), and the tera-floating point operations per second (TFlops). We also provide the result evaluated by Hussdorf Distance (HD) in Tables B1, B2 and B3 of Appendix Section B.
7

---

Table 1: The performance of different registration methods in Cardiac-UDA dataset [17]. Results
reported in structures (RV, RA, LV, LA) and the overall averaged Dice score (Avg. %).
21 Metliess
IVT
RV
LAT
RAT
Ave.
PSNR
SSIM Times (S) Params (M) THops
(256x256)
Nowviigi Registaaion
LODMM
69.44 10.6 57.03. 70.78.1 69.22.s. 13.12au 25.6742u4
26.45.17
76.44124
177.9
RDMM [8]
70.50
21.12
57.
10
22
70.84
5.102gr00 8.6024+ v
26.8027
76.92a1
2241
ANTs (SyN) [24]
73.51as 74.12u55
60.49.14 74.69
73.71
16.09_s0m)
40,06
27.96.24
76.52ssa
4156.4241
Deep Leurning Based Registration
VM-SSD [10
74,262ss 7488
66.78..
76.24 75.86242 0.374
0.262:13x
29.01a2s 15.89
0.011aaa
0.010
VM-NCC [10]
74,04
76.20 67.54
77.36 44 76.51 0.685 cooss
0.905
28.53
75.77
0.01lase
0.118
0.010
SYMNe1
75,21 75.33 69.67
77.78 1s 76.60:42
0.454
0.63
28.56.25
76.87
0.101:33
0.419
0.125
VM-DIF [9]
73.53415 76.37 68.10-1: 78.55asi 76.83asa 0.387
0.437
28.80+a3
76.87a1s
0.011429
0.109
0.010
Abo 55, [3]]
15.66 77.24
=1 79.20ga> 77.04241
2.6642
29,86 77.59
0.017g3a
7.783
DifluscMorph [12]
77.02xe: 80.45as:
72.50-1
80.81
79.27
0.319
0.339
29.48.2a
77.02x2
0.103:23
90.67
0.227
DeepTagg [15, 16]
76,83ars 80. 135s -2.87-1 80.98247
79.41a3s
0.273gmr
mm
28.83235
76 26
0.011:14
0.107
0.010
GPTmck-- (Ours)
16.94grs 81.72 -3.13-u 80.85
61.64g5x
0.256
0.
$1,28.23a
22
0.013
0.467
0.015
GPTack-L (Ours)
77.07.x0
82.57em
73.11-1
81.24:aa
22.1
0.250
0.019
$1.57422
78
70
0.016.39
5.161
0.041
GPTmck-XI. (Ours)
78.51g*
82,48:04 73.43.1 81.20ga 82.37gm7 0.279ggnres
0.027
32,00kg
80.04
0.0226279
7,536
0.053
Table 2: The performance of different registration methods in ACDC [19] dataset. Results reported
in structures (RV, LV, Myo) and overall averaged Dice score (Avg. %).
313 Methods (128x x32)
RV
TVT
1yo
Ave.
1 [ II ( PSNKR
SSIM
Times (8) Params (Vi TPlpps I
Not-rigi Registatiin
LDDMM [6]
73.6lass 65.62
56,44
72.39
451.81112
653.5amm
31.20
84.59:
11535s44
RDMM[8] [8]
76.43ara 69.50...
62.19aa
45.5
144.2sca
266.0auss
31.66
84.36.54
+1715
ANTs (SyN) [24)
75.30
66.92.36
58.03
74.64.m
15.82
57,26ga
30.92
84.26
*1166a
Doep Learning Based Repistation
VM-SSD [10]]
m9
74.27.m
64.44:::
77.56.12
3.144:330
4.602
12.61
03.88..
0.015
0.767
VM-NCC [10]
a1.60
77.00
67.90
79.90..1
0.260
0.079
34.68
85.01
0.015
0.327
0.767
VM.DIF [9]
81.50
75.50
65.902
78.
28.90am
0.286aane
0,083gmm
33.48245
84,22
0.015
0.327
0.267
SYMNet [36]
80.46
77.81
66,22..
79.47a15
0.341
0.121
32,91
83,55:43
NICE-Trans [52]
79.97
78.55
67.02
79.66.m
0.278
0.093
33.
83.88
0,486
5.619
0.280
DiffuseMorph [12]
10 78.30
67.80s
80.50.n
0,237
0,061
34,73ass
84.30ss
0,458
0.327
0.642
CorMLP [53]
80.33
80.07.7x
70.51.a
20.44
0.248
0.059
34.90azs
84.27:44
0.070ane
0.303
DeepTag [15, 16]
81.89..>
79.10.nz
10.37
80.83
0.185
cooe
0,044
33.64
83.09 c43
0.015 co
0.362
0.113
Transmatch [54]
81.22am0
80.34_ss
81.35
0,226ssm
0.077
33,89
84.78
0.325
70.71
0.603
ESDDifReg [11]
82.70aa1
80.90
72.40
82.30
0.214
0,054gmmm
35.34a31
85.85
1,106
1,320
GPTrack-M (Ours)
81.65270
80.77
71.53am
0.208a
goos
34,82
85.78
0.022
04488
0.201
GPTrack-L. (Ours)
82,78
16.06
a1.71am
K2.38.m
1822mmm
0.035
34.9921s
85.62249
0,023
0.942
0.204
GPTrac-XLL (Ours)
72.86
0,178 rears
0,032
33.52
:6.19.aa
0.034
4.3 Results
Result of 3D Echocardiogram Video. In Table 1,
Table 3: The segmentation performance' of dif-
we compare our method with state-of-the-arts 2D
ferent cardiac structures in the CAMUS [18].
registration [9, 10, 12, 15, 31, 52] and non deep
learning [6, 8, 24] methods in CardiacUDA dataset. In comparison to DeepTag [15], our GPTrack-XL reach 82.37% and 12.75 in DICE and HD scores with the best non-positive Jacobian determinant value, which denotes the learned motion field is smooth. The registration quality of our method r
Methods
CAMUS (Dice%)
(256x256)
T.V
T.A
Myn
Avg.
VM-NCC [10]
79.50=aa
80.10
67.70+93
75.80-m
SYMNet [36]
85.24=n0
82.77ass
76.36a62
84.84
VM RM-SSD| [10]
86.301s
85.20 77.90144
83.10
Ahn SS, et al. [3]]
86.42.a2
83.95
75.68 22.33
DiffuseMorh [11]
15.76
84.49
76.65141 83.57
VM-DIF [9]
87,70=sa
85.40+eo
80.40
84,50-37
also achieved the best with 32.03 (3.50 1) and 80.04 (3.64 t) in PSNR and SSIM compared to the second-best method, respectively. Though GP- Track introduces more learnable parameters and requires a small amount of additional computation, slightly increases inference time and TFlops compared to DeepTag [15] and VM-DIF [9]. GPTrack surpasses all other methods in the registration and verifies the necessity of formulating a strong temporal relationship among frames by using the recursive manner. Tables I and 3 show registration results of cardiac structures (LV, RV, LA, RA, Myo). Compared to other methods, our GPTrack can outperform existing baseline methods by substantial margin. In areas such as the left atrium (LA) and left ventricular (LV), which usually cause larger deformation, our method can also provide better alignment than other approaches.
DeepTag [15]
87,60a45
87.90asa 79.00ass
84,80=si
GPTrrak- (Ours)
88.63
89.13 S0.37 85,29_42
'Segmentation results are reported in the Dice score (%). Bold, underline denote the best results and the second best performance. The superscript indicates computational time reported in only CPU implementation. We use the t-test for statistical significance analysis, where the p-value between the two methods is p < 0.05, indicating statistically significant improvements
8

---

Syn [2]]
VM-SSD
VM-DIF
Abn,S, al. (33]
DeppTag [15]
Our GPTrack
Ground
Figure 4: The visualization 3D Echocardiogram video of motion tracking error. We visualised the last frame of tracking result and ground truth from 32 consecutive frames in CardiacUDA [17]. Colours Red, Blue, Green and Orange denote cardiac structures RA, RV, LVV and LA, respectively.
VM-IIF
DiffueMMophh
DepTaag [15
ESDImTee
Our GPTrack
Figure 5: The visualization in 4D Cardiac MRI of motion tracking error. We visualised the result of the last frame tracking from ED ES and corresponding ground truth in ACDC [17]. Colours Red, Blue, and Green denote cardiac structures MYO, LA, and LV, respectively.
As shown in Figure 4, the tracking error of our GPTrack and other methods show a significant difference LA (Labelled by Orange Colour) and L.V (Labelled by Green Colour) when compared to the ground truth. The methods [24, 10, 9, 31] present inaccuracy tracking results due to lack of the constraints on the consecutive motion and ignore the long-term temporal information. These results further verify that our specifically designed GPTrack for modelling motion patterns is more suitable for cardiac motion tracking on echocardiography.
Result of 4D Temporal Cardiac MRI Dataset. We compare our GPTrack method against with the sfate-of-the-ar deep learning based methods [9, 10, 11, 12, 15] and different Non-rigid approaches [6, 8, 24]. As illustrated in Table 2. our @PTrack-XL. achieves the best average DICE score of 82.65 compared to FSDiffReg [11] with 82.30. registration quality, our GPTrack-Xi. reaches the highest scores, 31.52 and 86. 19, in PSNR and SSIM, respectively. Moreover, our Jacobian determinant on deformation fields shows numbers comparable to other methods with the diffeomorphic constraint. All results are based on our fast and lightweight model, reducing around 96.93% inference time, 17.2% model parameters and 76.02% computational consumption (TFlops) compared to the second- best performance. In comparison to the diffusion-baat based method [12, 11], which requires enormous computation that hinders real-time inference and is nearly impossible to deploy real scenarios, the GPTrack preserve light-weight and considerable performance by formulating cardiac motion patterns as the Gaussian process latent coding and bidirectionally understand the cardiac motion. The visualization result in Figure 5 also indicates our GPTrack achieve better tracking accuracy.
4.4 Ablation Study
The Scale of Model Hyper-parameters. Table 4 shows the settings of the 2D/3D GPTrack- M/LL/LL For the 3D echocardiogram video dataset and 4D cardiac MRI dataset, the GPTrack with different scales has different patch sizes and dimension numbers. Referring to the result performed by Tables 3, I and 2, the registration result can be boosted by increasing the layers number and dimension size of GPTracks according to different requirements.

---

Table 4: The ablation study the different config- urations of our 2D 3D GPTracks (M, L, XL).
Table 5: Ablations of Bi-Directional (Bi-direct.)
and Gaussian Process (GP) of GPTrack-XI.
Model
T Layer
Patch Size
Dim
GiPlops
Param (M)
Bi direa.
GP
Dice.Avg
del(Ja]
x
79.83
0.048
GPTrack-M
2/2
16/8
64//22
1,54/ 20.1
0.467 0.418
X
v
g0.8
0.052
GPTrack-I.
2/2
0.942
x
81.21
=.039
GPTrcc-X1.
4/4
256/64
20.5
7,536/ 1.094
v
82.37
0.027
0.90-
0.85-
0.80-
Our GPTrack
0.75
Without Gaussian
0.70-
Without Bidirection
Deep Tag
10
20
25
30
Frame Number
Figure 6: The Tracking Error performed by different methods in 32 consecutive frames of Car- diacUDA [17] full annotated set.
The Ablation of Bidirectional Recursive Manner and Gaussian Process. Table illustrates the Bi-directional layer in GPTracks can better formulate both forward and backward motion fields by aggregating the temporal information of the cardiac heartbeat cycle. The Gaussian Process models cardiac motion with strong prior knowledge from data, which makes
accurate predictions of the deformation field. The Figure illustrate the tracking error from 1-st to 32-th frame in CardiacUDA [17] full annotated set. Our GPTrack and the DeepTag [15] both the Lagrangian strain. However, the accuracy degrades significantly without aggregating the temporal information and GP when the input length increases. As shown in Figure 6, the tracking error after the 20-th frame becomes larger by using only Lagrangian strain, while introducing GP and bidirectional recursive methods can efficiently eliminate the tracking error by predicting more accurate motion fields.
Conclusion and Limitation
In this paper, we proposed a new framework named GPTrack improve cardiac motion tracking accuracy. GPTrack innovatively aggregates both forward and backward temporal information by using the bidirectional recurrent transformer. Furthermore, the variability and predictability of cardiac motion. In experiments, our framework demonstrates the state-of-the the-artt in 3D echocardiogram and 4D cardiac MRI datasets. The limitation of our framework is that we use positional encoding as the prior knowledge of the cardiac motion, which may degrade tracking performance out-of-domain datasets. our future work, we will build more robust representation of cardiac motion and further our work across different medical domains. We also provide the illustration of Broader Impacts, please see Section A2 in Appendix.
introduce the Gaussian Process to model
Acknowledgements
This work was partially supported by grants from the National Natural Science Foundation of China (Grant No. 62306254), the Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone (Project No. (18--KCZYB-22000 Hong Kong Special Administrative Region, China (Project Reference Number: T45-401/22-N).
and the Research Grants Council of the
References
[1] Daniel Rueckert, Luke Sonoda, Carmel Hayes, Derek LG Hill, Martin Leach, and David Hawkes. Nourigid registration using free-form deformations: application to breast mr images. IEEE transactions on medical imaging, 18(8):712 721, 1999.
10

---

[2] J-P Thirion. Image matching as diffusion process: analogy with maxwell's demons. Medical image analysis, 2(3):243-260, 1998.
[3] Alessandro Becciu, Hans van Assen, Luc Florack, Sebastian Kozerke, Vivian Roode, and Bart M ter Haar Romeny. A multi-scale feature based optic flow method for 3d cardiac motion estimation. In Scale Space and Variational Methods in Computer Vision: Second International Conference, SSVM 2009, Voss, Norway, June 1-5, 2009. Proceedings 2. pages 588-599, Springer, 2009.
[4] Liang Wang, Patrick Clarysse, Zhengjun Liu, Bin Gao, Wanyu Liu, Pierre Croisille, and Philippe Delachartre. A gradient-hased optical-flo cardiac motion estimation method for cine and tagged mr images. Medical image analysis, 17:136-148, 2019.
[5] KY Esther Leung, Mikhail G Danilouchkine, Marijn van Stralen, Nico de Jong, Antonius FW van der Steen, and Johan G Bosch. Left ventricular horder tracking using cardiac motion models and optical flow. Ultrasound in medicine & biology, 37(4):605- 616, 2011.
[6] M Faisal Beg, Michael Miller, Alain Trouvé, and Laurent Younes. Computing large deformation metric mappings via geodesic flows of diffeomorphisms. International journal of computer vision, 61:139 1577,,-157, 2005.
journal
[7] John Ashburner. A fast diffcomorphic image registration algorithm. Neuroimage, 38( 1):95 113, 2007.
[8] Zhengyang Shen, François-Xaviss Xavier Vialard, and Mare Niethammer. Region-specific diffeomorphic metric mapping. Advances in Neural Information Processing Systems, 32, 2019.
[9] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. An unsupervised learning model for deformable medical image registration. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9252-9260, 2018
[10] Guha Balakrishnan, Amy 7hao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Voxelmorph: a learning framework for deformable medical image registration. IEFE transactions on medical imaging. 3888):1788-180M 2019.
[11] Yi Qin and Xiaomeng Li. Fsdiffreg: Feature-wise and score-wise diffusion-guided unsupervised deformable image registration for cardiac images. In International Conference on Medical Image Computing and Cosiuter-Assisted Intervention, pages 655-665 Springer, 2023.
[12] Boah Kim, Inhwa Han, and Jong Chul Yc. Diffuusmorphh: Unsupervised deformable image registration using diffusion model. In European conference on computer vision, pages 347-364. Springer, 2022.
[13] Hanchao Yu, Shanhui Sun, Haichao Yu, Xiao Chen, Honghui Shi, Thomas Huang, and Terrence Chen. Foal: Fast online adaptive learning for cardiac motion estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4313 1113-433,2220 2020.
[14] Nripesh Parajuli, Allen Lu, Kevinminh Ta, John Stendahl, Nabil Boutagy, Imran Alkhalil, Melissa Eberle, Geng-Shi Shi Jeng, Maria Zontak, Matthew O'Donnell, et al. Flow network tracking for spatiotemporal and periodic point matching: Applied to cardiac motion analysis. Medical image analysis, 55:1 16-135, 2019.
[15] Meng Ye, Mikael Kanski, Dong Yang, Qi Chang. Zhennan Yan, Qiaoying Huang, Leon Axel, and Dimitris Metaxas. Deeptag: An unsupervised deep learning method for motion tracking on cardiac tagging magnetic resonance images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
pages 7261-7271, 2021.
[16] Meng Ye, Dong Yang, Qiaoying Huang, Mikael Kanski, Leon Axel, and Dimitris N Metaxas. Sequence- morph: A unified unsupervised learning framework for motion tracking on cardiac image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[17] Jiewen Yang. Xinpeng Ding, Ziyang Zheng, Xiaowei Xu, Xiaomeng Li. Graphecho: Graph-driver unsupervised domain adaptation for cchocardiogram video segmentation. Proceedings of the IEFF//VF International Conference on Computer Vision, pages 1188--1 1187, 2023.
[18] Sarah Leclere, Erik Smistad, Joao Pedrosa, Andreas Østvik, Frederic Cervenansky, Florian Espinosa, Torvald Espeland, Erik Andreas Rye Berg, Pierre-Mare Jodoin, Thomas Grenier, et al. Deep learning for segmentation using an open large-scale dataset in 2d echocardiography. IEEE transactions on medical imaging. 38(9):2198-22 2210, 2019.
[19] Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-An Heng. Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, et al. Deep learnigg techniques for automatic mri cardiac multi-structure, segmentation and diagnosis: is the problem solved? IEEE transactions medical imaging, 37(11):2514 2525, 2018

---

[20] Ruzena Bajesy and Stane Kovačič. Multiresolution elastic matching, Computer vision, graphics, and image processing, 46(1):1221, 1989.
[21] Christos Davatzikos. Spatial transformation and registraation of hrain images using elastically deformable models. Computer Vision and Image Understanding, 66(2):207-22,, 1997.
[22] John Ashburner and Karl J Friston. Voxel-based morphomelry- the methods. Neuroimage, 1 1166)805--21, 2000.
[23] Nael F Osman, William S Kerwin, Elliot R MeVeigh, and Jerry I. Prince. Cardiac motion tracking using cine harmonic phase (harp) magnetic resonance imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonunce in Medicine, 42(6),1048- 1060, 1999.
[24] Brian B Avants, Charles I. Epstein, Murray Grossman, and James C. Gee. Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerrativ brain. Medical image analysis. 12(1):26-4) 41, 2008.
[25] Julian Krebs, Tommaso Mansi, Hervé Delingelte, Li Zhang. Florin Ghesu, Shun Miao, Andreas Maier, Nicholas Ayache, Rui Liao, and Ali Kamen. Robust non-rigid registration through agent-based action learning. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11 11-13, 2017, Proceedings, Part I 20, pages 344 352. Springer, 2017.
[26] Xiaohuan Cao. Jianhua Yang. Jun Zhang, Dong Nie, Minjeong Kim, Qian Wang, and Dinggang Shen. Deformable image registration based on similarity-steered cnn regression. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th Interaatinnal Conference, Quebec City, QC. Canada, September 1 11-13, 2017, Proceedings, Part I 20, pages 300-308. Springer, 2017.
[27] Marc-Michel Rohé, Manasi Datar, Tobias Heimann, Maxime Sermesant, and Xavier Pennec. Svf-net: learning deformable image registration using shape matching. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part 20, pages 266-274. Springer, 2017
[28] Hessam Sokooti, Bob De Vos, Floris Berendsen, Boudewijn PF Lelieveldt, Ivana Isgum, and Marius Staring. Nonrigid image registration using multi-scale 3d convolutional neural networks. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Port I 20, pages 232-239.. Springer, 2017.
[29] Bob D De Vos, Floris F Berendsen, Max A Viergever, Marius Staring, and Ivana Iggum. End-to-end unsupervised deformable image registration with a convolutional neural network. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 3. pages 904-212. Springer, 2017.
[30] Kevinminh Ta, Shawn S Ahn, Allen Lu, John C Stendahl, Albert Sinusas, and James Duncan. A semi- supervised joint learning approach left ventricular segmentation and motion tracking in echocardiography. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 1734-1737. IEEE, 2020.
[31] Shawn S Ahn, Kevinminh Ta. Allen Lu, John C Stendahl, Albert Sinusas, and James S Duncan. Unsupervised motion tracking of left ventricle in echocardiography. In Medical imaging 2020: Ultrasonic imaging and tomography, volume 11319, pages 196-202. SPIE, 2020.
[32] Tony CW Mok and Albert CS Chung. Unsupervised deformable image registration with absent corre- spondences in pre-operative and post-recurrence brain tumor mri scans. In International Conference on Medical Image Computing and Computer-Assisee Assistee Intervention, pages 25-35. Springer, 2022.
[33] Max Jaderberg. Karen Simonyan, Andrew Zisserman, al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
[34] Xiaohuan Cao,, Jianhua Yang. Jun Zhang, Qian Wang. Pew-Thian Yap, and Dinggang Shen. Deformable im- age registration using cue-aware deep regression network. IEEE Transactions on Biomedical Engineering. 65(9):1900-191) 2018
[35] Adrian V Dalca, Guha Balakrishnan, John Guttag, and Mert R Sabuncu. Unsupervised learning for fast probabilistic diffeomorphic registration. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings. Part I. pages 729-738. Springer, 2018
12

---

[36] Tony CW Mok and Albert Chung. Fast symmetric diffeomorphic image registration with convolutional neural networks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition. pages 0664-4653, 2020.
[37] Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg. Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758-2766, 2015.
[38] Pengpeng Liu, Michael Lyu, Irwin King, and Jia Xu. Selflow: Self-supervised leamning of optical flow. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4571-4580, 2019.
[39] Junhwa Hur and Stefan Roth. Iterative residual refinement for joint optical flow and occlusion estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5754-5761
2019.
[40] Eddy 1lg. Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462 220022247,2222 2017.
[41] Jiewen Yang, Xingbo Dong. Liujun Liu, Chao Zhang, Jiajun Shen, and Dahai Yu. Recurring the transformer for video action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14063-1407 14073, 2022.
[42] Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.0222 2015.
[43] Xingho Dong, Jiewen Yang, Andrew Beng Jin Teoh, Dahai Yu, Xiaomeng Li, and Zhe Jin. Video-based face outline recognition. Pattern Recognition, 152: 152:110488 10482, 2024.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[45] Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Camhridge University Press, 2019.
[46] Simo Särkkä, Arno Solin, and Jouni Hartikainen. Spatiotemporal learning via filtering and smoothing; A look at gaussian process regression through kalman filtering. IEEE Signal Processing Maguzine, 30(4):51-61, 2013.
bayesian
[47] Simo Sarkka and Jouni Hartikainen. Infnite-dimenssonnal kalman filtering approach to spatio-temporal gaussian process regression. In Artificial intelligence and statistics, pages 993-1001. PMLR, 2012.
[48] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4))600-61 -100000-1122200 2004.
[49] Abdel Aziz Taha and Allan Hanbury. Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC medical imaging,
2015.
[50] Chen Qin, Shuo Wang. Chen Chen, Huaqi Qiu. Wenjia Bai, and Daniel Rueckert. maaeecchanis-inffrmee neural networks for myocardial motion tracking in mri. In Medical Image Computing and Computer Assisted Intervention- MMCCAAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part III 23, pages 296-306. Springer, 2020.
[51] Chen Qin, Shuo Wang. Chen Chen, Wenjia Bai, and Daniel Rueckert. Generative myocardial motion track- ing via latent space exploration with mameechanic-inffrmeen prior. Medical Image Analysis, 83:102682, 2023.
[52] Mingyuan Meng, Lei Bi, Michael Fulham, Dagan Feng, and Jinman Kim. Non-iterative coarse-to-fine transformer networks for joint affine and deformable image registration. In International Conference on Medical Image Computing and Computer-Assisterr Intervention, pages 750-760. Springer, 2023.
[53] Mingyuan Meng, Dagan Feng. Lei Bi, and Jinman Kim. Correlation- aware coarse-to marse-to-finn fine mips for deformable medical image registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9645-9654, 2024.
13

---

[54] Zeyuan Chen, Yuanjie Zheng, and James C Gee. Transmatch: A transformer-basee multilevel dual-stream feature matching network for unsupervised deformable image registration. IEEE transactions on medical imaging, 43(1))11-27, 2023.
[55] Noemi Hernzzaa-Hrrrezelooo Ana Bajo. Filip Sroubek, Cristina Santamarta, Gabriel Cristóbal, Andrés Santos, and María Ledesma Ledesma-Carbayo. Motion estimation of tagged cardiac magnetic resonance images using variational techniques. Computerized Medical Imaging and Gruphics, 34(6):514-522, 2010.
[56] Simon Meister, Junhwa Hur, and Stefan Roth. Unflow: Unsupervised learning of optical flow with bidirectional census loss. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[57] Xiaoyu Shi, Zhaoyang Huang, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1599-1669 1610, 2023.
[58] Zhaoyang Huang, Xiaoyu Shi, Chao Zhang. Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: transformer architecture for optical flow. In European conference on computer vision, pages 685. Springer, 2022.
[59] Rico Jonschkowski. Austin Stone, Jonathan Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What matters in unsupervised optical flow. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 557-572. Springer, 2020.
[60] Ziyang Zheng, Jiewen Yang, Xinpeng Ding. Xiaowei Xu, and Xiaomeng Li. Gl-fusion: Global-local fusion network for multi-view echocardiogram video segmentation. In International Conference on Medical Image Computing and Computer-Asisted Intervention, pages 78-88. Springer, 2023.
[61] Jiewen Yang, Yiqun Lin, Bin Pu, Jiarong Guo, Xiaowei Xu, and Xiaomeng Li. Cardiacnet: Learning to reconstruct abnormalities for cardiac disease assessment from echocardiogram videos. In European Conference Computer Vision. Springer, 2024.
[62] Bin Pu, Kenli Li, Jianguo Chen, Yuhuan Lu, Qing Zeng. Jiewen Yang, and Shengli Li. Hfsced: hybrid neural network for fetal standard cardiac cycle detection ultrasound videos. IEEE Journal of Biomedical and Health Informaties, 2024.
14

---

Appendix
A1 Difference Between Optical Flow and Diffeomorphism Mapping
Optical flow (OF) based methods, often applied in object tracking of video sequences, have been explored by [3, 4, 5]. However, their effectiieness in medical imaging limited due challenges in accommodating large deformations and the inherent low quality of certain medical imaging modalities [4, 55], such as echocardiogram videos.
With the progression of deep learning, neural networks have also been employed to predict optical flow, which erucial for predicting dynamic motion trajectories in video sequences. Notable implementations include FlowNet [37], iterative methods by [39], and self-supervised learning approaches by [38] and [56]. However, while supervised methods require ground truth annotation for training cost fuucction [37, 39, 40, 57, 58], unsupervised approaches depend on photometric loss ensure motion consistency [38, 56, 59], which can be challenging to obtain in medical images.
Last but not least, OF-based methods do not necessarily preserve topology, non-globally one-to-one (objective) smooth and continuous mapping with derivatives that are invertible. In cardiac motion tracking, we consider the deformation in each point of the adjacent frame to remain one-on-one mapping and be invertible for forward and backward deformation fields. Directly using the OF-baseh method to predict the motion field of cardiac motion may lead to incorrect estimation.
A2 Broader Impacts
Our work focuses on cardiac motion tracking with an unsupervised framework named GPTrack. The GPTrack framework has the potential to support medical imaging physicians, such as radiologists and sonographers, observing the cardiac motion of patients. This a fundamental task for assessing cardiac function, and we are able to provide decision support and improve analysis efficiency and analysis reproducibility in clinical scenarios.
Moreover, our new framework illustrates that cardiac motion can be formulated as strong prior knowledge, which is able be utilised enhance tracking accuracy. Also, our work presents several advantages that help us make progress towards these benefits, which improve the performance of automated motion field estimation algorithms. The method not only improves the precision of motion tracking and segmentation in both 3D and 4D medical image modailites [17, 18, 19, 60, 61, 62] but also provides comprehensive observation of motion information radiologists and sonographers to facilitate human assessment. However, this work may still remain gaps between real-world clinical utilization due medical image analysis being low failure tolerance application. The meaning of this work is to present a new direction for cardiac motion tracking, which is different from the conventional approach. In the current stage, the trained model in public datasets and the results presented are not specific to provide support for clinical use.
15

---

B Additional Quantitative Results and Visualization
We provide the experiment reported on Hussdorf Distance (HD) for datasets UDA [17], CAMUS [18] and ACDC [19] in Tables B1, B2 and B3, respectively. Furthermore, the consequent tracking results of 3D echocardiogram and 4D Cardiac MRI are presented in Figures B1 and B2.
Table B1: The performance of different registration methods in CardiacUDA [17]. Results report in Structures (LV, RV, LA, RV) and overall averaged Dice score (Avg. %). Segmentation results reported in the Hussdorf Distance (HD). Bold, underline denote the best results and the second-best performance, respectively.
CardiacUDA
Methods 2256x256)
LV
RV
LA
RA
Avg.
Non-rigid Registration
LDDMM [6]
18.12a39
17.33
16.94
16.77438
17.27
RDMM [8]
17.47
17.69
16.36a32
15.97429
17.01
ANTs (SyN) [24]
17.02a75
16.44
15.81 #46
16.35
16.42s25
Deep Learning Based Registration
VM-SSD
17.09
16.11
15.60
15.46x32
16.11
VM-NCC [10]
16.84
15.78
15.93
15.16#25
15,88
SYMNet [36]
15.63
15.24
16.18
15.52+24
15.67226
VM-DIF [9]
16.16211
15.11
16.02
15.68191
15.73223
Ahn.S.S, ct al. [31]
16.63+27
15.13#30
16,45+31
14.91
15,48+26
DiffuseMorph [12]
16.26=27
15.28+25
14.60+32
14.77+34
15,54
DeepTTag [15]
15.81
15.68
14.3942a
13.70
14,94
G(TrrackM(OOus)
14.63
14.77
12.19a2s
13.94
13.87
Table B2: The performance of different registration methods in CAMUS [18] dataset. Results report in Structures (LV, RV, Myo) and overall averaged Dice score (Avg. %). Segmentation results are reported in the Hussdorf Distance (HD). Bold, underline denote the best results and the second-best performance, respectively.
CAMUS
Methods (256x256)
LV
RV
Myo
Avg.
Non-rigid Registration
LDDMM [6]
7.210
10.65
6,592.23
7.305
RDMM [8]
6,307=34
9.584267
6,91 1
6,831=26
ANTs (SyN) [24]
6.644
10.29
6.134a23
7.166a23
Deep Learning Based Registration
VM-SSD [10]
5.769
8.755
5.231
6.240
SYMNeI [36]
5.544
9.131
5.466:2s
6.499=20
VM-NCC [10]
5,454
9,094475
5.190
6.521
VM-DIF [9]
5.382=28
8.749
5.137
6.304-16
Ahn.S.S, et al. [31]
5.628
8.701
5.478
6.072
DiffuseMorph [12]
5,396
8.358
5,066
5,854
DeepTag [15]
5.207
7.651
4.870
5.388
GPTrack-M(Ours)
4.72221.
6.857
4,904
4.945
16

---

Table B3: The performance different registration methods in ACDC [19] dataset. Results report in Structures (LV, RV, Myo) and overall averaged Dice score (Avg. %). Segmentation results are reported in the Hussdorf Distance (HD). Bold, underline denote the best results and the second-best performance, respectively.
ACDC
Methods
IN
LA
Myo
Avg.
x256x256)
Non-rigid Registration
LDDMM [6]
5.817424
6.662429
6.337
6.562+21
RDMM [8]
5.430
6.268
5.953
5.728
ANTs (SyN) [24]
5.676+23
6.547+26
6.211+27
6.242+16
Deep Learning Based Registration
VM-SSD [10]
4.708
4.814
5,647
4.942
VM-NCC [10]
4.745
5.153219
5,231
5.336-113
VM-DIF [9]
4.466221
4.782219
5.365
4.802215
SYMNet [36]
4.864+23
5.149+21
5,552+27
5,254+199
NICE-Tram [52]
4.626119
4.805
5.096124
4,993
CorMLL [53]
3.850
4.061
3.653+24
3,812
DorpuseMorph [12]
4.102
4,054
4,184
3,977
DeepTag [15,, 16]
3.336+16
3.651
3,284
3,552+13
Transmatch [54]
3,904
3,855
3.770
3.716
GPTrack-M(Ours)
3.285
3.170
3.030
3.361
FSDiffieg [11]
2.970
3,298+20
2.862+18
3,283+12
GPTrack-XL (L(Our))
3,147+15
3,028
2.844:18
3.145
Frame 1 (ED)
Frame 2
Frame
Frame 4
Frame
Frame
Frame 7
Frame 8 (ES)
Figure B1: The visualization in 4D Cardiac MRI of estimated motion field and motion tracking results. We visualised the tracking result of the first frame (ED) to the last frame (ES) in ACDC [17].
Colours Red, Blue, and Green denote cardiac structures MYO, LA, and LV, respectively.
17

---

Frame 1
Frame 2
Frame 3
Frame
Frame
Frame 6
Frame 7
Frame
Figure B2: The visualization in 3D Echocardiogram video estimated motion field and motion tracking eиror. We visualised tracking results from the first frame to the last frame, with ground truth from 8 consecutive frames in CardiacUDA [17]. Colours Red, Blue, Green and Orange denote cardiac structures RA, RV, LV and LA, respectively.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main claims are presented in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We already discuss the limiations the last section.
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
We prove the full set of assumptions and complete (and correct) proof in our paper.
Justification:
19

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the experimental results be reproduced and promise that all code will be made publicly.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer:
[Yes]
Justification:
All datasets in experiments are public datasets, all code will be made publicly,
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Subllic/guidesCodeeeee
cassonnnlccyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ httpppscccppublcc//eccc httpubllcuuuudssssseeeepppeeaael /eggudddeeooeedmmiiiiii /ot////oodd/ssssseeeeeeee sssiisssionn assssnoooicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have described all the training and testing details.
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars, standard deviation and all details of result are presented in paper.
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
21

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see the implementation details.
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:://///e/pp//eeeeeee http:p/ss//oorssssscpcppptttt/ta Seppulic/Ethissuuiiii
Question:
Answer: [Yes]
Justification: We promise that all research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appendix A.2.
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
22

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: No dataset or generative data.
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: For all datasets, have acquired the license and permission for dataset usage.
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: N/A
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
23

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not involve crowdsourcing nor research with human subjects.
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not involve crowdsourcing nor research with human subjects.
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "E4MZMFp6Bg", "reviewer": "Reviewer_ssxr", "summary": "The authors proposed a latent modeling framework for cardiac motion tracking. They introduced the GPTrack module for image encoding, which considers both forward and backward information flow. A Gaussian process was integrated to describe the motion prior. Extensive experiments were performed on both echocardiography and cardiac MRI datasets, demonstrating good results.", "questions": "1.Are there any results showing the impact of varying the number of frames?\n2.The myocardium is the tissue of interest for motion tracking, as the blood pool of the left/right ventricle involves flow filling. Could there be more focus on comparing the myocardium?", "limitations": "The authors described the limitations appropriately.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.A transformer-based image encoder that integrates bidirectional information flow, considering long-range relationships.\n2.A Gaussian process latent modeling approach to describe the dynamics of cardiac motion.\n3.Competitive results on both echocardiography and cardiac MRI (CMR) datasets.", "weaknesses": "1.It appears that the authors misunderstood the cardiac MRI in Figure 2. The short-axis CINE image shows the myocardium of the left ventricle (green) and right ventricle (red) rather than the left atrium.\n2.The authors used PSNR, SSIM, and DICE to evaluate motion tracking performance, which is reasonable. However, they did not evaluate the physics/physiological plausibility. For the myocardium, which is considered an incompressible material, the determinant of the Jacobian |J| should be around 1. I suggest the authors add some evaluation comparing the physiological plausibility [1].\n[1] doi: 10.1016/j.media.2022.102682", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Dear reviewer ssxr:***  Thank you for your efforts and valuable feedback, we would like to address your questions point by point in the following.\n\n&nbsp;\n\n**Q1.Are there any results showing the impact of varying the number of frames?**\n\nTable R.4 shows the ablation study of varying frame numbers on the CardiacUDA dataset. Methods that do not consider the temporal information such as VM-DIF [9] and FSDiffReg [11] would not be affected by the frame length significantly. The DeepTag employs the Lagrangian displacements affected by the frame length, whose performance decreases when reducing the number of frames. Our GPTrack also sees degradation when reducing the number of frames, especially when the input only contains two frames, which leads the motion-tracking task to a pair-wise registration task. However, with the contribution of our designed framework and Gaussian Process, the result of GPTrack can also outperform other methods.\n\n**Q2. The myocardium is the tissue of interest for motion tracking, as the blood pool of the left/right ventricle involves flow filling. Could there be more focus on comparing the myocardium?**\n\nThank you for your kind suggestion, we highlight the myocardium in Tables R.1 and R.2, in our final version and following research, we will also emphasize the importance of myocardial in cardiac motion tracking.\n\n**Q3. The authors add some evaluation comparing the physiological plausibility.**\n\nThank you for your helpful suggestions on how to strengthen our paper. We have reported the related metric in Tables R.1 and R.2 and highlighted the results with blue colour (Please see Rebuttal pdf). Related research mentioned by the reviewer [54, 55] for motion tracking with biomechanics-informed prior uses supervised learning and our method employs unsupervised learning for motion-tracking tasks. Hence, in our rebuttal, we may not be able to compare these two works in our experiments.  However, we also consider that physiological plausibility is an important index that can help evaluate the LV myocardial strain estimates and the learnt biomechanical properties. Following the research [54, 55], we compute the mean absolute difference between the Jacobian determinant and 1 ($||J|-1|$) over the tracking areas. In our final version, we will also add these metrics and cite related works in our final revision. \n\n**Q4. It appears that the authors misunderstood the cardiac MRI in Figure 2. The short-axis CINE image shows the myocardium of the left ventricle (green) and right ventricle (red) rather than the left atrium.**\n\nThank you for pointing out our mistake and misunderstanding, we will be really carefully revising all Figures, Tables and Descriptions. Make sure all content in our paper is presented correctly in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sjYUBCoG12", "reviewer": "Reviewer_bQtH", "summary": "1. This paper presents GPTrack, an unsupervised framework designed to thoroughly investigate the temporal and spatial dynamics of cardiac movement. \n2. GPTrack refines motion tracking by utilizing sequential Gaussian Processes within the latent space and encoding statistical data with spatial information at each time point, thereby robustly fostering temporal consistency and accommodating spatial variations in cardiac dynamics. \n3. This paper aggregates sequential data in a bidirectional, recursive fashion, emulating the principles of diffeomorphic registration to more effectively capture the enduring motion relationships across cardiac areas, including the ventricles and atria. \n4. The method enhances the accuracy of motion tracking, all while preserving computational efficiency.", "questions": "Included in the Strengths and Weaknesses.", "limitations": "Included in the Strengths and Weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a new framework named GPTrack to improve cardiac motion tracking accuracy. \n2. GPTrack innovatively aggregates both forward and backward temporal information by using the bidirectional recurrent transformer. \n3. This paper leverages the Gaussian Process to model the variability and predictability of cardiac motion. \n4. This paper demonstrates the state-of-the-art in 3D echocardiogram and 4D cardiac MRI datasets. \n5. The methodology employed in this study is robust and rigorous.\n6.The implications of the findings are clearly articulated and show practical relevance.", "weaknesses": "1. Line 26, Can optical flow not capture temporal coherence? I doubt that's the case.\n2. Line 31, The citation is not formatted correctly.\n3. Line 61, \"contribuion\" repeats the idea of \"facets\" mentioned earlier, which is redundant.\n4. There is no experimental comparison with methods such as denoising diffusion probabilistic models (DDPM).\n5. Line 111, How is the relationship between the diffeomorphism \\phi_t and the method proposed in the paper demonstrated?\n6. The experiment does not include a comparison with the method in reference [36].\n7. The paper does not explain, demonstrate, or theoretically analyze long-term dependencies, nor does it clarify the duration of \"long-term.\"\n8. Many acronyms are repeated unnecessarily.\n9. Line 135, What is meant by \"motion consistency,\" and how do you define and assess its consistency?\n10. Line 148, Here, x_{t} represents features; it should not use the same mathematical symbol as the raw data to avoid confusion.\n11. Figure 3, in the right subplot, what is meant by \"elu(x)\"?\n12. Line 206, the formula below this line is not numbered. But the author refers to this equation.\n13. How is the multidimensional z_{t} output generated? Multidimensional z_{t} involves multitask Gaussian processes, which are not mentioned in this paper.\n14. Line 221, sometimes \\mu_{t} is a scalar, and sometimes it is a vector.\n15. Line 258 and Line 272, there is a conflict. Why is this done? Please explain the motivation.\n16. Figure 6, Why does the error decrease and then increase? Moreover, this trend is similar even across different methods.\n17. There should be a comparison with optical flow-based methods.\n18. Why is FSDiffReg only included in Table B3?\n19. The discussion of results is somewhat superficial. A deeper analysis linking the findings back to the research questions and theoretical framework would enrich the paper's contribution to the field.\n20. There are inconsistencies in the terminology used throughout the paper. Consistently using the same terms would improve clarity and reduce confusion for the reader.\n21. The paper would benefit from a stronger connection between the theoretical framework and the empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Dear Reviewer bQtH:***\nThank you for your valuable feedback, we would like to address your questions point by point in the following. \n\n&nbsp;\n\n**Q1. Line 26, Can optical flow ...**\n\nThe optical flow (OF) is also able to capture the temporal coherence. In *section Appendix A1*, we discussed the difference between OF-based and Diffeomorphism Mapping methods. We consider that OF-based methods do not necessarily preserve topology, non-globally one-to-one (objective) smooth and continuous mapping with invertible derivatives. These attributes do not match the description of human cardiac which is considered as an incompressible material. We also add a new experiment compared to optical flow-based methods in Table R.3 of our rebuttal pdf.\n\n**Q4. No experimental comparison with DDPM ...**\n\nIn our experiments, we compared two state-of-the-art DDPM methods [11, 12]. As shown in Tables R.1 and R.2. Our GPTrack can reach more accurate results compared to DDPM methods with faster inference speed and less computational consumption.\n\n**Q5. The relationship between the $\\phi_t$...**\n\nThe diffeomorphism $\\phi_t$ is the $t$-th tracking output of our method, which is differentiable and invertible mappings generated by the $t$-th and ($t$-1)-th frames.\n\n**Q6. Comparison with method [36]...**\n\nWe have added the corresponding experiment in Tables R.1 and R.2.\n\n**Q7 . Clarify of \"long-term.\"...**\n\nAs illustrated in Figure 2 of our manuscript, the \"long-term\" indicates that the historical information of the cardiac motion will be maintained and encoded as features by our framework. Previous conventional methods only consider the relationship between two adjacent frames, while ours can aggregate the features across long intervals bidirectionally.\n\n**Q9. Line 135, the meaning of \"motion consistency\", define and assess consistency**\n\nSorry that our description of \"motion consistency\" makes you confused. Motion consistency is the consistency of movements between two adjacent state spaces across different individuals. The state space is the set of all possible states of the human cardiac motion. We consider that cardiac motion usually has a certain principle. For example, as shown in Figure 2 in our manuscript, the Diastole and Systole always occur in each heartbeat cycle of every person. The movement of each organ (Ventricle, Atrium, Myocardium, \\emph{et al.}) always has its fixed direction, speed and range. Those motions can be considered as \"consistency\" and formulated as temporal patterns for learning. In our approach, we utilize the recursive manner to aggregate the long-term temporal feature without increasing the time complexity. Also, we employ the Gaussian Process to capture the temporal correlation and heterogeneity in cardiac motion, and provide predictions and interpolation. These designs help promote temporal consistency and regional variability in compact latent space.\n\n**Q11. Figure 3, what is meant by \"elu(x)\"...**\n\nAs described in line 166, the $elu(\\cdot)$ is the exponential linear unit [42]. We will make it clearer in our final version.\n\n**Q12. Line 206, the formula below this line is not numbered...**\n\nThanks for your kind reminder. We have corrected this typo. \n\n**Q13. How is $z_{t}$ generated ...**\n\nFirstly, as mentioned in Line 203, we assign independent GP priors to all values in the latent coding of $\\mathbb{R}^{P \\times C}$  dimension, indicating $R \\times C$ scalar Gaussian processes are independently assigned across $T$ time stamps. Due to our independent modelling, we take one scalar Gaussian process $\\\\{z_t\\\\} _{t=1}^T$  for illustration, whose corresponding observation is $\\\\{f_t\\\\} _{t=1}^T$. Then for each scalar sequential Gaussian process, within the framework of representing sequential Gaussian process with Kalman filtering, we use the initial condition $\\textbf{z}_0$ calculated in Line 220 and mean-variance calculated by Equation 10 to update $\\textbf{z}_t := \\boldsymbol{\\mu}_t,~t=1,...,T$. To sum up, the multidimensional $z_t$ comes from aggregating totally $R \\times C$ scalar Gaussian process together due to our independent modelling, and each individual is sequentially updated by the Kalman filtering of Equation 10.   \n\n**Q14. Line 221, sometimes $\\mu_{t}$ is a scalar, and sometimes it is a vector.**\n\nThanks for your kind reminder. Here $\\boldsymbol{\\mu}_{t}$ together with Equation 10 below are all vectors, and we have corrected this typo. \n\n**Q15. Conflict in Line 258 and Line 272...**\n\nThe $368 \\times 368$ in Line 272 is a typo, which should be the same size $384 \\times 384$ as Line 258. We will correct them in the final version.\n\n**Q16. The error decreases and then increases in Figure 6...**\n\nIn Figure 6, the error is calculated between the $t$-th tracking result tracked from the initial (reference) frame and the ground truth of the $t$-th frame. Since the human heartbeat is a reciprocating motion (refer to Figure 1), while the heartbeat state of the current frame is close to the initial frame, the registration between these two frames is easier, and therefore the smaller the error (the higher the Dice value). \n\n**Q17. Comparison with OF-based methods...**\n\nPlease see the Table R.3 for more details.\n\n**Q18.  FSDiffReg is only included in Table B3.**\n\nSorry that we only report FSDiffReg in Table B3 and Table 3 since FSDiffReg [11] only provides a 3D network for MRI images and reports the experiment in 4D MRI images. As shown in Table R2, we further the FSDiffReg in 3D echocardiogram video and report its motion tracking results.\n\n**Q2,3,8,10,19,20 and 21. Typos / Citation not formatted correctly / Redundant word / Acronyms repeated unnecessarily / Superficial discussion of results /  Inconsistencies terminology and structures.**\n\nThank you very much for pointing out the drawbacks and helping further improve our work. We will carefully check the paper to avoid typos and reorganize the connection and details to make it more coherent and easy to read.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5906ONsArC", "reviewer": "Reviewer_Tej6", "summary": "To capture the long-term relationship in cardiac motion, the authors proposed GPTrack, a novel unsupervised framework crafted to fully explore the temporal and spatial dynamics of cardiac motion. They proposed employing the sequential Gaussian Process in the latent space and aggregating sequential information in a bidirectional recursive manner. Through experiments on 3D Echocardiogram videos and 4D temporal MRI datasets, they demonstrate that they achieve state-of-the-art performance.", "questions": "1.\tHow is the proposed method applied to the complete cardiac cycle?  What's the difference between the proposed method and the traditional DL-based registration network? Is the comparison to previous pairwise registration approaches a fair comparison?\n2.\tDoes the author use a pair of images or a complete cycle of images as the input? If using the complete cardiac cycle, the incorporation of additional temporal information would significantly increase the computational cost and registration parameters, what is the potential application of the proposed methodology?\n3.\tThe cardiac cycle of different subjects contains different frames, how did the authors process the dataset and train the proposed method?\n4.\tWhy separate the original images into several patches? It would lead to significant unalignment if the motion is large.\n5.     Are the improvements significant?", "limitations": "The authors only mentioned one sentence of limitation in the conclusion, which needs further discussion.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\temploys the Gaussian Process (GP) to promote temporal consistency and regional variability\n2.\tcapture the long-term relationship of cardiac motion via a bidirectional recursive manner", "weaknesses": "1. Some details are lacking or not clear in the manuscript, regarding the data processing, how the proposed method is applied to the complete cardiac cycle, and the detailed difference in training and architecture between the proposed method and traditional methods.\n2. Lack of comparison to the latest registration methods.\n3. No statistical significance analysis is provided.\n4. More detailed discussions about the potential application and limitations are needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Dear reviewer Tej6***: Thank you for your kind comments and suggestions, which help us improve our paper's quality. Here are our responses to weaknesses and questions. New experiments are included in the uploaded PDF file to better illustrate your questions.\n\n&nbsp;\n\n**Q1. Method applied to complete cardiac cycle**\n\nThe video frames of the complete cardiac cycle can be continuously fed into our model like the recurrent neural networks (RNNs). The outputs are the deformation between two adjacent frames. Specifically, motion tracking can be regarded as a task of continuous registration and computing the diffeomorphism between every adjacent frame of a video. The motion tracking task is we give the video with $t$ frames ($x_0, x_1, \\cdots, x_{t-1}, x_t$) of the complete cardiac cycle, and we output the diffeomorphism ($\\phi_1, \\cdots, \\phi_{t-1}, \\phi_t$), where $\\phi_i$ indicates the movements from $(i-1)$-th frame to $i$-th frame. In this paper, we strictly follow the previous research [15, 16] and ensure that all the input and output are consistent in experiments. \n\n**Q2. The difference between... Is it a fair comparison?**\n\nAs the response in **Q1**, the input of our approach and the traditional DL-based registration network remains the same, i.e., $\\phi_1=\\text{net}(x_0, x_1), \\phi_2=\\text{net}(x_1, x_2), \\cdots, \\phi_t=\\text{net}(x_{t-1}, x_t)$.  The traditional DL-based registration network computes the diffeomorphism between only two adjacent frames. In comparison, by employing the recursive manner, our approach can aggregate and maintain the historical temporal information of cardiac motion by hidden state. The introduction of the Gaussian Process can also promote temporal consistency and regional variability in compact latent space. As shown in experimental results, our method reaches higher tracking accuracy while maintaining lightweight inference speed, computational consumption and model parameters. We ensure that the higher performance can be conducted by our approach under the same computational resource compared to those traditional DL-based registration network. \n\n**Q3. A pair of images or a complete cycle of images as the input?**\n\nPlease see our posed comment below.\n\n**Q4. Comparison to the latest registration methods.**\n\nWe have added three state-of-the-art registration/tracking methods [56, 57, 58] in our experiments on the ACDC dataset. Please see Table R.1.\n\n**Q5. Significantly increases the computational cost and parameters...**\n\nAs the response in **Q2**, our experiments illustrate that our approach will not significantly increase computational cost and model parameters. This is because our method can recursively accept the pair-wise frame as input step by step and each input shares the same network as the RNNs. This indicated that our method has a similar inference speed, computational consumption and model parameters as those other methods. All result showcases the significant improvement in tracking and registration accuracy.\n\n**Q6. Potential applications**\n\n As we discussed in the section *Appendix A2. Broader Impacts*, the cardiac motion tracking can help estimate and quantify myocardial motion within a cardiac cycle. It is a cost-efficient and effective approach for assessing cardiac function. [59] has utilized cardiac motion to efficiently predict human survival. [60] apply motion tracking for physiological motion compensation systems. Motion tracking has also been used for the identification of dilated cardiomyopathy patients in [61]. Our approach can improve the performance of tracking algorithms, and improve the precision of the myocardial motion tracking in model evaluation. Also, we provide motion information to radiologists and sonographers to facilitate human cardiac assessment. \n\n**Q7. How do authors process the dataset and train the proposed method?**\n\nInitially, we select 32 frames from each video, crop and resize all frames to $256\\times 256$ in weight and height. For an input video, the input size is $256\\times 256\\times 32$. During training, the batch size will be set as 1, and our network will take the first frame $x_0$ and initial hidden state $h_0$ (zero vector) as inputs, generating the feature $f_0$ and hidden state $h_1$. For the next time step, the second frame $x_1$ and $h_1$ will be input to the network and generate the $f_2$ and $h_2$. With such a recursive manner, the feature $f_t$ is always generated by $x_{t}$ and $h_{t}$, where the $h_{t}$ help maintain and aggregate the temporal feature information of cardiac motion from $0$ to $t$ moment. The bidirectional process shares the same pipeline as the above description. For the optimization of the training, we use the loss in section 3.5.\n\n**Q8. Why separate the original images into several patches? ...**\n\n1). Based on our observations of cardiac motion, we first found that cardiac motion has only a small displacement in Echocardiography/MRI images. The position of cardiac main structures remains roughly unchanged throughout images of the same person *(See Appendix Figures B1 and B2)*. 2). We do not simply decompose images into patches, we adapt multiple convolutional layers to convert the images into small patches. Through the receptive field brought by the convolution kernel, we can build the relationship within patches and all patches are not isolated.\n\n**Q9. Are the improvements significant...**\n\nOur method aims to provide a new approach by integrating temporal information via the recursive manner and Gaussian Process to make more accurate velocity field estimation. Compared to other methods, [36] reports a 1%-1.5% increase compared to [10], and [16] demonstrates only 1% improvement compared to the best method. Our method surpasses the SOTA method with 1%-3% accuracy in 3D and 4D datasets, which illustrates our improvements are significant. Furthermore, these improvements may also contribute to other tasks that may related to cardiac function assessment as illustrated in **Q6**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **We first thank all reviewers for their valuable feedback to help us improve our work**. 
### **Below, we will address the concerns of reviewers about the experiments and details of our proposed method. In our uploaded one-page PDF, we provide more experiments and ablation studies to better illustrate the questions.**
### **For point-to-point question answering, we left our rebuttal in the corresponding position for each reviewer.** *All Tables for rebuttal are included in our uploaded one-page PDF.* 

&nbsp;
### References:

[54] Qin, C., Wang, S., Chen, C., Qiu, H., Bai, W. and Rueckert, D., 2020. Biomechanics-informed neural networks for myocardial motion tracking in MRI. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23 (pp. 296-306). Springer International Publishing.

[55] Qin, C., Wang, S., Chen, C., Bai, W. and Rueckert, D., 2023. Generative myocardial motion tracking via latent space exploration with biomechanics-informed prior. Medical Image Analysis, 83, p.102682.

[56] Mingyuan Meng, Dagan Feng, Lei Bi, and Jinman Kim, "Correlation-aware Coarse-to-fine MLPs for Deformable Medical Image Registration," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9645-9654, 2024. 

[57] Chen, Z., Zheng, Y. and Gee, J.C., 2023. Transmatch: A transformer-based multilevel dual-stream feature matching network for unsupervised deformable image registration. IEEE transactions on medical imaging, 43(1), pp.15-27.

[58] Meng, M., Bi, L., Fulham, M., Feng, D. and Kim, J., 2023, October. Non-iterative coarse-to-fine transformer networks for joint affine and deformable image registration. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 750-760). Cham: Springer Nature Switzerland.

[59] Bello, G.A., Dawes, T.J., Duan, J., Biffi, C., De Marvao, A., Howard, L.S., Gibbs, J.S.R., Wilkins, M.R., Cook, S.A., Rueckert, D. and O’regan, D.P., 2019. Deep-learning cardiac motion analysis for human survival prediction. Nature machine intelligence, 1(2), pp.95-104.

[60] Richa, R., Poignet, P. and Liu, C., 2008. Efficient 3D tracking for motion compensation in beating heart surgery. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2008: 11th International Conference, New York, NY, USA, September 6-10, 2008, Proceedings, Part II 11 (pp. 684-691). Springer Berlin Heidelberg.

[61] Puyol-Antón, E., Ruijsink, B., Gerber, B., Amzulescu, M.S., Langet, H., De Craene, M., Schnabel, J.A., Piro, P. and King, A.P., 2018. Regional multi-view learning for cardiac motion analysis: Application to identification of dilated cardiomyopathy patients. IEEE Transactions on Biomedical Engineering, 66(4), pp.956-966.

[62] O'Briain, T., Uribe, C., Yi, K.M., Teuwen, J., Sechopoulos, I. and Bazalova-Carter, M., 2022. FlowNet-PET: unsupervised learning to perform respiratory motion correction in PET imaging. arXiv preprint arXiv:2205.14147.

[63] S. Mocanu, A. Moody, and A. Khademi, “FlowReg: Fast Deformable Unsupervised Medical Image Registration using Optical Flow,” 
Machine Learning for Biomedical Imaging, pp. 1–40, Sep. 2021.

Meta-review: This work proposes an unsupervised framework for cardiac motion tracking by considering the temporal and spatial dynamics of cardiac motion. Reviewers recognise that this work has merits in the innovation of model design, the state-of-the-art results, robust and rigorous methodology, and the clear articulation of findings. At the same time, the reviewers raise issues related to more evaluation and comparison, the impact of the number of frames, deeper analysis and stronger connection, and the statistical significant analysis. The authors have made efforts to provide responses to the raised issues. Two of the three reviewers indicate that the rebuttal has addressed their concerns and give positive final ratings. The third reviewer does not respond to the rebuttal and the rating is at borderline reject. The AC agrees that this work has its novelty and values and that the authors have well responded to the reviews. Meanwhile, it is recommended that this work apply statistical significant test (say, paired samples t test) to the results in Tables 2 and 3 to better assess the improvement. Considering all the factors, AC recommends this paper for acceptance. The authors shall effectively incorporate the key information in the rebuttal to further strengthen this work.
Decision: Accept (poster)