OFELIA: Optical Flow-based Electrode LocalIzAtion

Authors: xinyi  wang
zikang  xu
qingsong  yao
yiyong  sun
s kevin zhou
Venue: MIDL_io_2024_Conference
Published: 2024-01-29 02:41:55

Abstract: Catheter ablation is one of the most common cardiac ablation procedures for atrial fibrillation, which is mainly based on catheters with electrodes collecting electrophysiology signals.
Catheter electrode localization facilitates intraoperative catheter positioning, surgical planning, and other applications such as 3D model reconstruction.
In this paper, we propose a novel deep network for automatic electrode localization in an X-ray sequence, which integrates spatiotemporal features between adjacent frames, aided by optical flow maps.
To improve the utility and robustness of the proposed method, we first design a saturation-based optical flow dataset construction pipeline, then finetune the optical flow estimation to obtain more realistic and contrasting optical flow maps for electrode localization.
The extensive results on clinical-challenging test sequences reveal the effectiveness of our method, with a mean radial error (MRE) of 0.95 mm for radiofrequency catheters and an MRE of 0.71 mm for coronary sinus catheters, outperforming several state-of-the-art landmark detection methods.

Main paper: 
 introduction: 
 Atrial fibrillation (AFib), atrial flutter, and premature ventricular contractions are prevalent manifestations of cardiac arrhythmias. Frequent cardiac arrhythmias may give rise to serious consequences, for instance, AFib can lead to blood clots in the heart (Staerk et al., 2017) . Compared with pharmaceutical interventions, catheter-based radiofrequency ablation techniques in cardiac electrophysiology (EP) stand as the standard surgical intervention for the definitive treatment of rapid cardiac arrhythmias, characterized by immediate therapeutic effects and high success rates (Mark et al., 2019; Parameswaran et al., 2021) . The electrode is one of the most crucial components of the catheters, which is used for EP signal collection and catheter localization. Catheter electrode localization can facilitate intraoperative catheter positioning, surgical planning, 3D model reconstruction and so on. However, due to the unstable imaging quality and the intersections among multiple catheters throughout the clinical surgical procedure, it is hard for physicians to locate the electrodes precisely in real-time X-ray images. Thus, it is necessary to develop accurate catheter placement detection methods not only to alleviate burdens for clinicians in the surgery but help novice doctors get familiar with this surgery.There are research works (Ambrosini et al., 2017; Yang et al., 2019; Nguyen et al., 2020 ) that formulate this task as a segmentation task, which locates the catheters by the center of the mask of electrodes. Other studies adopt single frame landmark detection (LD) methods to solve the problem. Catheter segmentation information is indeed helpful, but the labeling is time-consuming. Experimentally, we observe that the optical flow map can to some extent provide shape and boundary information of catheter electrode without the need for specific annotations (Demoustier et al., 2023) . Besides, optical flow maps can provide temporal context in X-ray videos, which could make full use of correlation between successive frames and the label-free shape context.To solve the electrode localization problem and motivated by the the above empirical findings, we go beyond a single frame and propose an effective and easy-to-implement network, Optical Flow-based Electrode LocalIzAtion (OFELIA) for electrode localization in an X-ray video sequence.Specifically, we introduce the optical flow map between consecutive frames as the input to the LD network, which not only presents the position changes over time but also provides estimated shape information of the electrodes (As shown in Fig. 3 ). Besides, as the ground truth of optical flow is difficult to acquire in our task, we construct a simulated dataset, the Flying-Catheter Dataset, based on several pre-trained RAFT (Teed and Deng, 2020) models, to train the optical flow estimator.This paper offers the following contributions:1. We propose an OFELIA network, which integrates the spatiotemporal information in an X-ray sequence for precise electrode localization. To the best of our knowledge, it is the first to introduce optical flow into electrode localization in an X-ray sequence; 2. To bridge the gap between natural images and X-ray images, we construct a Flying-Catheter dataset and fine-tune RAFT for accurate optical flow estimation.3. Extensive experiments on test datasets illustrate that the OFELIA method outperforms the state-of-the-art electrod detection methods on two commonly used catheters. 
related works: 
 Optical Flows in MedIA. Optical flow maps are occasionally used in medical image areas. The FW-Net (Nguyen et al., 2020) introduces an end-to-end framework, which combines a segmentation network, an optical flow network, and a flow-guided warping function to learn temporal continuity for real-time catheter segmentation in a 2D X-ray fluoroscopy sequence. Optical flow maps are also utilized in (Xue et al., 2022) to achieve echocardiography segmentation. FlowReg (Mocanu et al., 2021) introduces a two-part deep learning system for unsupervised neuroimaging registration, combining 3D affine adjustments and 2D deformable fine-tuning based on the optical flow network to enhance global and local alignment of medical imaging volumes. Single-Image Landmark Detection. In (Yao et al., 2020) , a multi-task U-Net is implemented to predict both heatmap and offset maps of landmarks simultaneously. In (McCouat and Voiculescu, 2022) , an efficient contour-hugging landmark detection method with uncertainty estimation is depicted. In (Zhu et al., 2022) , a light-weighted universal anatomical landmark detection model has been developed. Video Landmark Detection. Compared to single-image landmark detection, video landmark detection utilizes the information between frames. In (Ullah et al., 2019) , a tracker is implemented to extract the tip detection results in the last frame as a reference for segmenting the tip in successive frames. U-LanD (Jafari et al., 2022) capitalizes on the uncertainty inherent in landmark prediction to achieve automatic detection of landmarks in key frames of videos. The most similar work to ours is ConTrack (Demoustier et al., 2023) , which uses both spatial and temporal context for tip detection and tracking. It incorporates multiple template frames and a search frame for catheter segmentation and initial tip detection. Subsequently, it uses successive segmentation to refine tips with optical flow maps. However, it relies on catheter segmentation masks, necessitating extensive annotations. In contrast to this, our OFELIA only requires point annotations, which is much easier to obtain. 
method: 
 Problem Definition Let D = {(X t , Y t )} N t=1 represents an X-ray video sequence with N frames, where X t ∈ R w×h is the t-th video frame with a shape of (w, h), and Y t ∈ R Ne×w×h denotes the position of N e landmarks in frame t. Specifically, suppose that the k-th landmark of the t-th frame is at (x, y). Y k t is defined:EQUATIONOFELIA aims to train a network f (•), which takes the {X t } N t=1 as input and predicts the locations of electrodes in each frame, i.e., { Ŷt } N t=1 . OFELIA Network The architecture of OFELIA is shown in Fig. 1 , which aims to predict the landmark positions Y t using both X t and X t+1 . Particularly, we try to solve this problem by utilizing the information between the two frames, i.e., the optical flow map.Optical flow, a concept to measure the motion of objects in continuous images, is widely used for tracking cells in fluoroscopy (Guo et al., 2013) . By computing the direction and magnitude of the velocity, the optical flow map can be used to describe the temporal-spatial information of the electrodes in an X-ray video.Specifically, we first predict the optical flow map F t→t+1 between X t and X t+1 using an optical flow estimator φ * (•), which takes two frames as input and predicts the pixel movement between them, i.e., F t→t+1 = φ * (X t , X t+1 ). Then, the raw image X t and optical flow map F t→t+1 are concatenated and sent to a modified U-Net (Ronneberger et al., 2015) . The encoder of the U-Net is a pre-trained ResNet-34, while the decoder consists of 5 upsampling layers with 512, 256, 128, 64, and 32 channels respectively. As there are N e electrodes that need to be localized in each frame, we add a convolution layer after the last layer of the decoder to squash the number of channels to N e . The n-th channel represents the predicted localization probability map of the n-th electrode and the position with the highest probability is regarded as the final prediction. The loss function L is defined as the average channel-wise cross-entropy loss between the predicted probability map Ŷ Ne t and ground truth Y Ne t , given asL = 1 Ne Ne k=1 L CE ( Ŷ k t , Y k t ). The information captured by the optical flow map illustrates the catheter movement along with time and provides additional spatial shape information of the electrodes, which usually requires manual annotations. Combining X-ray images and optical flow maps can drive the neural network to pay more attention to the electrode part of the field of view. This contributes to the precise localization of the electrodes.Optical Flow Estimator Estimating the optical flow map is essential for the final prediction of our OFELIA. However, the ground truth optical flow is inaccessible in our task. Besides, most applicable optical flow estimators are trained on natural image datasets, and the large domain gap results in poor prediction. Thus, we simultaneously construct a simulated X-ray optical flow dataset (called Flying-Catheter) and train a task-specific flow estimator on it. The pipeline is shown in Algorithm. 1.First, we adopt four publicly available pre-trained optical flow estimators on our catheter dataset, including raft-chairs, raft-kitti, raft-sintel and raft-things which are variations of RAFT (Teed and Deng, 2020) trained different natural RGB datasets. For each frame, X t , the four estimators predict four flows, F p t→t+1 , p ∈ {c, k, s, t}. Then, we propose a saturation-channel-based selection algorithm to decide the final pseudo optical flow F * t→t+1 for each frame. Specifically, the predicted optical flow map, which is coded in the RGB format following RAFT, is first converted to the HSV format. We empirically find that the saturation channel in HSV is good at partically capturing the eletrode boudaries. Then the mean saturation factor (SF) of each map is calculated using the below:EQUATIONAlgorithm 1 Optical Flow Estimator Input: Original Dataset:D ori = {X t } N t=1 , Pre-trained RAFTs: φ c , φ k , φ s , φ t , Original RAFT: φ, Quality Control Threshold: α, Flying-Catheter Dataset: D f -c = ∅ t ← 1 repeatPredict Optical Flow between X t and X t+1 : F p t→t+1 ← φ p (X t , X t+1 ), p ∈ {c, k, s, t}; Convert F p t→t+1 into HSV color space; Compute Saturation Factor SF p for each flow map using Eq. ( 2); Find the optical flow map with the highest SF using Eq. (3); if SF p * >= α then Add sample pair to the Flying-Catheter Dataset:D f -c = D f -c ∪{X t , X t+1 , F p * t→t+1 }; end if until t = N -1 Update φ * on D f -c using gradient descent; Output: Fine-tuned RAFT model: φ *where (x i , y i ) is the coordinate of the i-th electrode landmark in frame X t . The pseudo optical flow map is defined as the predicted optical flow map with the highest SF:EQUATIONHowever, due to the large gap between natural images and X-ray images, even the best of the four predictions may have low quality. Thus, we conduct a quality control procedure on the constructed dataset by discarding samples with an SF smaller than a threshold of α. Finally, we finetune the original RAFT on the remaining dataset, denoted as the Flying-Catheter Dataset, for task-specific optical flow estimation. Compared to the original RAFT and RAFT trained on other natural image datasets, RAFT trained on the Flying-Catheter dataset can better capture spatial information of the electrodes, which serves as a strong prior knowledge for landmark detection using OFELIA (as shown in Fig. 3 ).Dataset. This study uses an in-house multi-center dataset of fluoroscopic sequences captured during cardiac ablation procedures and animal experiments. Most of the frames include two types of commonly used catheters, Coronary Sinus (CS) and Radio-Frequency (RF) catheters. All the landmarks are defined as the center point of the electrodes except for the first landmark of the RF catheter, which is defined as the tip of the RF catheter. This results in 14 landmarks (4 for RF and 10 for CS) in each frame. The dataset is annotated by two skilled engineers using LabelMe (Russell et al., 2008) and reviewed by three professional clinical experts. The training and test sets consist of 560 sequences(14,768 frames) and 346 sequences(7,711 frames), respectively. To evaluate the stabilization and generalization of our proposed method, we extract two clinical-challenging (CCA) subsets, which consist of frames of a special scene in the operation (53 sequences, 575 frames, denoted as Test-DSA Subset) and frames where the catheters are partially obstructed (145 sequences, 2,266 We compare OFELIA with several commonly used algorithms for medical landmark detection (Ronneberger et al., 2015; Yao et al., 2020; McCouat and Voiculescu, 2022; Zhu et al., 2022) , and the quantitative results are shown in Table 1 . We observe that OFELIA outperforms the baseline methods on most of the metrics on the test sets. This might result from the temporal information introduced from the flow map, as other methods focus on spatial features only. Besides, our OFELIA presents good generalization on the CCA sequences as the SDR drop is much lower compared to other methods, which is justifiable as bringing in extra knowledge improves the robustness of the network and provides an aid to deal with difficult situations. We also present qualitative results of different detection methods in Fig. 2 , where the MRE of each image is on the top of the image. Our method outperforms other methods significantly. More results are in the Appendix. To evaluate the efficiency of our proposed method, we conduct several ablation studies and present the results below and in the Appendix.RAFT trained on Flying-Catheter. We use four pre-trained RAFT and FlyingCath RAFT, to predict the optical flow map, and the result is shown in Fig. 3 . From Fig. 3 we can find that the prediction of FlyingCath RAFT contains more spatial information of the electrodes, and the boundary is clearer, which brings a strong prior knowledge for landmark detection.The introduction of optical flow. The proposed OFELIA takes frame X t and the corresponding optical flow F t→t+1 as input. Here we replace F t→t+1 with (1) The segmentation component of X t with highest probability from Segment Anything Model (SAM (Kirillov et al., 2023) ) without prompt;(2) The subsequent frame X t+1 ; (3) Estimated optical flow map using RAFT trained on natural dataset F s t→t+1 ; (4) OFELIA without extra information. The result in Table 2 illustrates that, although aggregating extra information can improve the utility of landmark detection, the usage of optical flow tends to be more efficient. Besides, using optical flow maps generated by Flying-Cath RAFT exhibits better performance than the original RAFT model, which also proves the necessity of fine-tuning RAFT on the constructed Dataset.  
conclusion and future work: 
 Accurate and efficient electrode detection in real-time fluoroscopy holds a paramount significance. In this work, we propose OFELIA, which introduces optical flow features to the pipeline for precise electrode localization in X-ray series. To improve the model's utility and generalizability, we propose a saturation-based optical flow dataset construction algorithm and fine-tune the optical flow estimator on the synthetic dataset. The results on the test set and two CCA subsets illustrate the efficiency of our proposed OFELIA compared with several SOTA methods. It's worth noting that this approach may not be limited solely to catheter ablation but can be generalized to other tasks such as motion object detection. In terms of electrode landmark detection tasks, further research could be conducted on the usage of different combinations of loss functions, and the exploration of one-shot or few-shot methods to alleviate the burden of electrode annotation. In our study, the highest value in the heatmap can be used to determine whether the predicted landmark is reliable. As shown in Fig 7(a) , if the maximum values of the heatmaps corresponding to all electrodes of the CS catheter are less than the conventional threshold, we will conclude that there is no CS catheter in the current frame. This threshold, determined through our statistical analysis, is 30(before normalization). some extent. This saturation map was derived by converting the optical flow images in RGB into the HSV format. Our analysis further reveals that sufficient shape information from optical flow images was accessible when α is greater than 0.5. However, an excessively high α value led to a drastic reduction in the amount of data available for finetuning(α=0.5 6212 frames, α=0.75 315 frames), while too low an α would introduce noisy data. Therefore, we establish the flying catheter dataset with 0.5 as the threshold value and perform finetuning based on RAFT sintel model because the RAFT sintel model reveals a higher mean SF compared with other three raft models. 
 
 Appendix:  


Decision: This work was a true borderline case with many remarks from the reviewers. During the rebuttal phase, authors were able to explain and address many points, such that all reviewers increased their ratings to "borderline accept". This is certainly not a strong acceptance vote (one of the reviewers wrote that his/her "overall impression of the work did not fundamentally change"), but I think it can be acknowledged that all reviewers increased their initial ratings.

Reviews: review: paper_summary: paper_summary: This paper presents an approach for the localisation of electrodes over a series of temporal X-ray images. The authors propose using optical flow over two frames to gather temporal information which is then used to predict the locations of electrodes in the first of the two images. They demonstrate results on an in-house dataset, where the proposed approach performs better than three competing ones. Two subsets of the dataset, which consist of particularly difficult examples, are also evaluated and the proposed method is shown to be better in terms of two metrics. The paper also proposed an algorithm to construct a dataset on-the-fly to improve optical flow on the type of image observed in the task.
main_review: main_review: I am changing my rating to borderline accept as the authors have answered questions raised in the rebuttal well. However, I would encourage the authors to add a short discussion paragraph on how this approach is not specific to catheter ablation but can be generalised to other types of data too.
strength_weakness: strength_weakness: 1. The application of optical flow seems to be incomplete. The paper claims that it views the problem as a video landmark detection network, yet optical flow is used only for two consecutive frames, with the resulting localisation information not being used for further detections. The authors could explore using longer frame stacks, adding smoothing constraints to landmark locations over time, etc. As it is presented, the methodology seems to lack several possible explorations.
2. The authors mention Demoustier et al. 2023 as the closest work to their paper, yet it is not compared/evaluated against the proposed version. Indeed, Demoustier et al. 2023 also propose using temporal context to refine detections.
3. It is unclear if the index of each landmark has semantic identity. For example, in Figure 2, what is the semantic difference between landmarks appearing on the two catheters? This is important because the training loss optimises for the detection of each landmark separately, and there is no consensus between these detections (i.e., each landmark is detected individually).
4. The paper claims to evaluate the approach on three datasets, but it is the same dataset with two additional subsets from it.
questions: questions: I would appreciate if the authors could clarify some minor things.

1. In algorithm 1, F^hat is converted to the HSV colour space. What does this mean? Does this refer to the visualisation of the optical flow using the HSV colour space? If so, the saturation in this visualisation is set to 255, meaning that the saturation factor computed in Equation 3 will always be the same. Furthermore, what is the importance of the HSV colour space? This needs to be motivated better.
2. There is a missing comparison with a simple baseline U-Net which regresses landmarks from X_t. As the dataset used is in-house, establishing such a baseline is important in order to demonstrate to what extent simple and straightforward approaches work on it.
3. What is the purpose of training \phi* if it is not used in Algorithm 1? Are the demonstrated results obtained with \phi* or with \phi? If yes, is there a comparison with the other (i.e., if results are with \phi, are there results with \phi*, and vice versa)? Also, why is image X_{t+1} not included in D_{f-c} as it will also be needed to fine-tune the optical flow network?
paper_summary: paper_summary: The authors propose a deep neural networks-based approach for the localization of catheter electrodes in fluoroscopic x-ray sequences. The proposed approach integrates temporal information in each frame via optical flow maps obtained from the previous frame in the sequence. To obtain high-quality optical flow maps for training, outputs of four pre-trained models for optical flow estimation in natural images are used to produce refined realistic optical flow maps, which are then used as ground truths for the optical flow map estimation network. The proposed approach outperforms state-of-the-art localization networks.
main_review: main_review: I believe most of my concerns were addressed. However, I am not completely satisfied with the hyperparameter selection procedure, and the held-out site experiments were not performed for the competing methods. 
Overall, I'm making it a borderline accept.
strength_weakness: strength_weakness: Although the paper is well written, there are some missing details both in terms of results and methodologies, which I am listing in the 'detailed comments' and 'questions to be addressed in the rebuttal'. Furthermore, the proposed approach is demonstrated on a single dataset. It is difficult to tell how generalizable the proposed approach is.
questions: questions: 1 - In addition to the mean values across sequences (in tables), the metrics should also include information regarding the distribution of the metrics in the form of standard error or bar plots. Furthermore, statistical significance tests should also be performed.
2 - Please add more figures (like figure 2) in the appendix/supplementary materials, and also show failure cases and some cases from the CCA subset.
3 - Please explain the criterion to select CCA subset, and how was the distribution of CCA subsets in the training and testing splits.
4 - Please explain how alpha was selected, and how having SF above a specific threshold can help in producing better optical flow maps. What is the effect of alpha on the optical flow estimation networks?
5 - The proposed approach is demonstrated only on a single dataset (although it is multi-center). How would the proposed approach perform on other datasets? Did the author perform any held-out center experiments? It is crucial to evaluate the effectiveness of the proposed approach.
paper_summary: paper_summary: This work proposes a method for locating and tracking ablation catheters in real-time fluoroscopy. The main novelty (in my opinion) is a method to select training data for fine-tuning the RAFT optical flow method for this kind of data, so that it can serve as additional input for the catheter localization model. The latter is modeled as landmark prediction task, with a fixed number of landmarks encoded in separate output channels of U-net like CNN (with pretrained ResNet-34 as encoder). The dataset employed is an in-house dataset.
main_review: main_review: First of all, I want to highlight that the authors really took time to discuss with the reviewers and to revise their paper, which is great! Maybe the replies would have been more readable if they would have cited the respective review parts directly. Unfortunately, I still wonder what the fire symbol in Figure 1 is about – is it an easter egg? Probably, my (repeated) questions in OpenReview to this point were not clear enough due to the way they got displayed. But this is not a critical point I think.

To conclude, the paper was revised and I will raise my rating, but only very slightly because my overall impression of the work did not fundamentally change.
strength_weakness: strength_weakness: The motivation looks a little off to me; AFAIK, it is not the case that such a method is necessary to help humans because "it is difficult for clinicians to localize catheters in real-time fluoroscopy". The merit of such a method should rather be that the resulting geometry can be used for interventional support systems based on models from prior imaging, planning, and 3D models. I am fairly sure that doctors would not perform such procedures frequently if they could not see the catheters.

The prior works used for comparison are probably(?) implemented by the authors themselves (this is not stated explicitly, however). So it is not entirely clear how reliable that part of the evaluation is.

The in-house dataset is used for training and evaluation, and the derived "Flying Catheters" dataset is therefore also not published.  Although it is a multi-center dataset, from the description of the method I understand that it can only work if there are exactly the two catheters (of RF / CS type) with the given 14 landmarks (4 for RF and 10 for CS) visible in each processed frame.
questions: questions: Have you performed statistical significance tests when determining the "best" and "second best" values in the tables?

The description of the CCA test sets is incomplete:
1) Does the training data also contain such CCA frames?
2) Are the three test sets "Test", "DSA-Test" and "OBS-Test" disjoint? Or what are their (subset) relations?

Are the radii Euclidean distances or measured orthogonally to the catheter?

Does the training procedure really only have a single 1 in every output channel? No smoothing? How is the preprocessing (geometric transformations) implemented in order not to lose that 1?

Where is "This attention-based mechanism" that you refer to?

Comments: title: New Respond to Reviewer 62n7
New Respond to Reviewer 5zr1
New Respond to Reviewer 62n7
Responds to Reviewer sc8B
Responds to Reviewer 62n7
Responds to Reviewer 5zr1
comment: Thank you for your review, which greatly enhances the completeness of our work and experiments. I will run the experiments based on your suggestions and make modifications in the main manuscript accordingly.
Thank you again.
About the answers to my questions:
1 - It's not a good idea to use training loss to select hyperparameters
2 - Please add them to the final version of the main manuscript, and If possible please also calculate the results for the competing methods.
3 - Please calculate across sequences not frames.
I'll be changing my decision to borderline accept. However, I trust that you will add 2 and 3 to the final version of the manuscript.
Thank you for your review.

Respond to Question 1: The ablation study on longer stacks uses the X-ray images only, we didn’t incorporate optical flow maps. In X-ray sequences, the catheters occupy only a small portion of the frame most of the time, and the landmarks are concentrated at a specific point within it. Using longer stacks adds spatial information but lacks concentration, potentially introducing noise instead. Incorporating additional constraints or templates might yield better results. 

Frame12$\rightarrow$ 1 is different from OFELIA, OFELIA takes frame $X_t$ and optical flow map $F_{t\rightarrow t+1}$ generated by $\varphi^{*}$ as input while Frame12$\rightarrow$ 1 takes frame $X_t$ and $X_{t+1}$ as input, $\textbf{without optical flow map}$. The first and second frames of the X-ray sequence often appear very similar, sometimes providing redundant information, but optical flow map detects pixel differences, offering temporal information and boundary information to some extent, which finally leads to a better result.

Respond to Question 2: We agree that this approach is not specific to cardiac ablation procedures, and OFELIA could be used for other tasks involving video detection as well. OFELIA uses inter-frame information in sequences by using the optical flow maps generated by fine-tuned RAFT network ,the optical flow maps also contain certain boundary information and reduce noise or redundacy between consecutive frames in the X-ray sequence or other videos. 

OFILIA requires information between frames, whereas the dataset used by Zhu et al. consists of $\textbf{single images, lacking sequential information}$. We believe that OFELIA can achieve good results in handling problems other than Catheter Ablation. In the future, we will construct datasets with various types of landmark sequences and test the generalization of OFELIA.

Respond to Question 3: For the convenience of visualization, utilizing the code provided by RAFT, we can convert an HxWx2 optical flow image into an HxWx3 RGB image based on the standard flow color wheel. During our experiments, we observe background noise when directly visualizing with RGB. We find that converting to the saturation channel $\textbf{helps to suppress this noise}$ and constrain its values within the range of 0 to 1. Therefore, we employ the saturation channel for filtering and finally building the flyingcath dataset.

Respond to Question 4: The fine-tuned $\varphi^*$ in Algorithm 1 is used in OFELIA. We have corrected the typos in $\textbf{Method section}$. The $\varphi^*$ is used for the optical flow prediction between frames: $F_{t\rightarrow t+1} = \varphi^*(X_t, X_{t+1})$.

We hope these answers address your questions.
Thank you for your review.

Respond to Question 1: We train for 20 epochs, saving both the final model and the best model in terms of training loss, then evaluate the performance using the best model.

Respond to Question 2: We mix all the data from multicenters and randomly divide the mixed data into training and testing. There is no heldout center in our study. For this question, we conduct a held-out test. We select one group from the multicenter data solely as the test set and re-train on the remaining training set. Due to time constraints, we only train OFELIA. The results are as follows: RF Catheter: 0.95$\pm$3.95 93.42 96.41 97.40, CS Catheter: 1.31$\pm$2.43 94.99 96.90 98.01. These new results do not differ significantly from original OFELIA in Table 1.

Respond to Question 3: The standard errors and statistical significance tests are evaluated across frames.

I hope these answers address your questions.
I thank the authors for their response.

1. It seems taking longer stacks for estimating landmark positions greatly increases error, almost doubles to quadruples it (Table 6 in updated PDF). This seems quite counter-intuitive. Why do you think this is? Also, isn't Frame12->1 in Table 6 the same as OFELIA? What I understood is OFELIA uses frames at t=t, and t=t+1 to predict landmark locations at t=t, which would be the same as Frame12->1. Also, how are multiple frames implemented in the model? Optical flow consumes only two images, so details on how multiple frames were used should be included.

2. Comparison against ConTrack seems difficult on the in-house dataset because of lack of segmentation masks. However, I would like the authors to comment on my other observation, which was that the proposed algorithm does not seem to be specific to cardiac ablation procedures. Is it possible to include performance on public datasets (Cephalogram, Hand, Chest) from Zhu et al. 2022, Learning to localize cross-anatomy landmarks in x-ray images with a universal model? 

3. It is still not clear, what the reasoning behind choosing the saturation channel in the HSV space is. Optical flow on two (H x W x 3) images is one (H x W x 2) image, so an image with two channels. If RAFT has a conversion process to visualise this as an RGB image, and this image viewed in HSV space is somehow helpful in providing shape and boundary locations, then this part of the methodology is not evident at all. 

4. Could the authors please comment on Question 3 in the original review (what is the use of \phi* when it is never used)?
Thank you for addressing most of the comments. I have a few questions:
1 - How were hyperparameters selected? Is there validation data?
2 - The data is multi-centered. Do you have information regarding each center that could be used as a held-out center(during training) and only used during testing? 
3 - Are standard errors and statistical significance tests across frames or sequences?
Thanks for your review, which greatly enhances the completeness of our work and experiments.

Respond to Weakness: We have modified the length of the paper and the main motivation of our method, the future applications based on electrode localization are surely of concern to us. As you mentioned, if doctors cannot see the catheter, they would not perform such surgery. However, in some complex intraoperative scenarios, such as contrast agent injections and catheter intersections during cardiac ablation procedure, doctors may also experience discontinuity or difficulty in observing electrode positions. Hence, this is why we extract these two kinds of special scenario cases as the CCA dataset. 

The prior works are evaluated using the official code and we will add footnotes in the result table. We have added visualizations where only part of the landmarks are visible in the images, our OFELIA can still tackle this problem correctly under this condition.

Respond to Comments:
When tackling the issue of video landmark detection, to better observe video quality, we store the optical flow maps in RGB video format, following the conversion process provided by RAFT. During our analysis,  we observe that the optical flow map, especially its saturation channel,  provides shape and boundary information of catheter electrode to some extent. 
This saturation map is derived by converting the optical flow images in RGB into the HSV format. Our analysis further reveals that these shape information from optical flow images are accessible when alpha was greater than 0.5. However, an excessively high alpha value lead to a drastic reduction in the amount of data available for finetuning($\alpha$=0.5 6212 frames, $\alpha$=0.75 315 frames), while too low an alpha would introduce noisy data. Therefore, we establish the flying catheter dataset with 0.5 as the threshold and perform finetuning based on raft_sintel model because the raft_sintel model reveals a higher mean SF compared with other three raft models.

We have corrected the typos in the manuscript. To explain the order of landmarks and the position of the tip, we add a schematic figure of the catheter structure in the appendix A.1.

Our loss is computed by averaging $L_{CE}$ of each landmark instead of the difference of the whole output, please see R1W3 for details.

As shown in the appendix E, the optical flow map generated by our method($\alpha=0.5$) shows similarities with the SAM generated segmentation maps (Dice > 0.5). And concatenating the image and flow map before sent to the U-Net will make the network focus more on these areas.

We conduct an ablation study on all four pre-trained optical flow estimators in the appendix D.3, and sintel model reveals better MRE compare to the others.

Respond to Q1: We have added statistical significance tests in the appendix C., our OFELIA outperforms other methods with a significance of p < 0.01.

Respond to Q2: The training data contains a total of 560 sequences(14768 frames), within the training dataset, we have 61 DSA sequences (827 frames) and 174 OBS sequences (4342 frames). Test-DSA Subset contains 53 sequences (575 frames) and Test-OBS Subset 145 sequences ( 2266 frames), they are disjoint test sets manually split by experienced clinicians, please refer to R2C4 and the appendix F for the detailed dataset construction method.

Respond to Q3: We calculate the Euclidean distances between landmarks and their corresponding groundtruth.

Respond to Q4: Yes, we do not conduct smoothing. Even without smoothing we can train successfully. 

Respond to Q5: Sorry for confusion, we now delete the phrase.
Thanks for your review, which greatly enhances the completeness of our work and experiments.

Respond to Comment 1: We have merged the contributions in the revised version.

Respond to Comment 2: It takes the engineers about 15 min to annotate the landmarks of an X-ray video with 100 frames, while annotating the segmentation mask requires nearly 1.5h, about 5 times longer.

Respond to Comment 3: There are 560 sequences (14,768 frames) for training 346 sequences (7,711) test set, without any overlap. Besides, there are 53 sequences (575 frames) Test-DSA Subset, and 145 sequences (2,266 frames) Test-OBS Subset are seected from test set, there is no overlap between DSA subset  and OBS subset. 

Respond to Comment 4: Two CCA subsets are manually selected from entire test set by clinical experts, like ConTrack, they split their private test dataset into several types according to different scenarios. In our study, the DSA Set encompasses specific clinical scenarios: X-rays under angiography, where the injection of contrast agents leads to non-uniformed dark shadows moving with the bloodstream in the X-rays, potentially affecting the field of vision. The OBS Set includes situations in X-rays where catheters obscure each other or are obscured by external wires, patches, etc.. Both of the two subsets present augmented complexity for the detection of catheter electrodes.

Respond to Comment 5: We have added an ablation study on $\alpha$ in the appendix D.1. From the table, we can find that $\alpha=0.5$ has the best performance. This is because a high $\alpha$ gives strict quality control on the Fly-Catheter Dataset, resulting in an insufficient number of train samples. A low $\alpha$ introduces too many noisy samples to the dataset, thus hurting the final utility.

Respond to Comment 6: Figure 2(b) is a sample from DSA subset and Figure 2(c) is from OBS subset, we have added more caption under Figure2. To further address this issue, we have added more samples from CCA subsets in the appendix A.2 and appendix A.3.

Respond to Comment 7: We have added standard errors in the Tables, and added bar plots in the appendixes C.1, C.2 and C.3.

Respond to Q1: We have added standard errors in the Tables, and added bar plots in the appendixes C.1, C.2 and C.3..

Respond to Q2: We have added figures showing failure cases and cases from CCA, please see the appendix B.

Respond to Q3: Please refer to Respond to Comment 4.

Respond to Q4: Please refer to Respond to Comment 5, we also show 5 optical flows with $\alpha=0.25, \alpha=0.5, \alpha=0.75, \alpha=0.9$ in the appendix E. As shown in the Figure, optical flow maps with low $\alpha$ cannot provide enough accurate shape information for landmark detection, and a high $\alpha$ leads to a rapid decrease of the number of samples ($\alpha$=0.25 10345 frames, $\alpha$=0.5 6212 frames, $\alpha$=0.75 315 frames) for estimator finetuning.

Respond to Q5: As the electrode landmark detection in catheter ablation is a less popular but important direction, to the best of our knowledge, there is no publicly available dataset. However, we have added a simple baseline – the origin U-Net to the result Table 1 for better understanding of the difficulty of this task.
Thanks for your review, which greatly enhances the completeness of our work and experiments.

Respond to Weakness 1: We have added an ablation study on using longer frame stacks in appendix D.4.

Respond to Weakness 2: We are unable to compare with ConTrack because we do not have segmentation masks.

Respond to Weakness 3: The landmarks in our dataset are arranged in a specific order, with all landmarks positioned at the center of the electrodes, except for the first landmark, which is located at the tip of the RF catheter. To better illustrate the positions and order of landmarks, we have chosen an image annotated with the positions of the landmarks and their corresponding order, which is included in the appendix A.1.

Respond to Weakness 4: Two CCA subsets are manually selected from entire test set by clinical experts, like ConTrack, they split their private test dataset into several types according to different scenarios. We now change the phrases Test-DSA Set and Test-OBS Set to Test-DSA Subset and Test-OBS Subset to avoid confusion.

Respond to Question 1: When tackling the issue of video landmark detection, to better observe video quality, we store the optical flow maps in RGB video format, following the conversion process provided by RAFT. During our analysis,  we observe that the optical flow map, especially its saturation channel,  provides shape and boundary information of catheter electrode to some extent. This saturation map is derived by converting the optical flow images in RGB into the HSV format. Our analysis further reveals that these shape information from optical flow images are accessible when alpha was greater than 0.5. However, an excessively high alpha value lead to a drastic reduction in the amount of data available for finetuning($\alpha$=0.5 6212 frames, $\alpha$=0.75 315 frames), while too low an alpha would introduce noisy data. Therefore, we establish the flying catheter dataset with 0.5 as the threshold and perform finetuning based on raft_sintel model because the raft_sintel model reveals a higher mean SF compared with other three raft models.

Respond to Question 2: Thanks for your suggestion, we now add a baseline U-Net in Table 1. 

Respond to Question 3: We have corrected the typos in the manuscript.