SAM2Flow: Interactive Optical Flow Estimation with Dual Memory for in vivo Microcirculation Analysis
Authors: Luojie Huang
Ryan Zhang
Marisa M Morakis
Michaela Taylor-Williams
Gregory N. McKay
Nicholas J. Durr
Venue: NeurIPS 2025 poster ()
Keywords: Optical Flow Estimation
Microscopic Blood Cell Analysis
Segment Anything Model
Primary area: machine_learning_for_sciences
TL;DR: This work presents SAM2Flow, an interactive optical flow model for analyzing microvascular flow in OBM videos, leveraging SAM2-inspired prompting and a novel dual motion+context memory mechanism for efficient long-sequence flow estimation.

Abstract: Analysis of noninvasive microvascular blood flow can improve the diagnosis, prognosis, and management of many medical conditions, including cardiovascular, peripheral vascular, and sickle cell disease. This paper introduces SAM2Flow, an interactive optical flow estimation model to analyze long Oblique Back-illumination Microscopy (OBM) videos of *in vivo* microvascular flow. Inspired by the Segment Anything Model (SAM2), SAM2Flow enables users to specify regions of interest through user prompts for focused flow estimation. 
SAM2Flow also incorporates a dual memory attention mechanism, comprising both motion and context memory, to achieve efficient and stable flow estimations over extended video sequences. According to our experiments, SAM2Flow achieves SOTA accuracy in foreground optical flow estimation on both microvascular flow and public datasets, with a fast inference speed of over $20$ fps on $512\times512$ inputs. Based on the temporally robust flow estimation, SAM2Flow demonstrated superior performance in downstream physiological applications compared to existing models. The code and dataset will be published with this paper.

SAM2Flow: Interactive Optical Flow Estimation with Dual Memory for in vivo Microcirculation Analysis
Luojie Huang'
Ryan Zhang'
Marisa M. Morakis' Michaela Paylor-Wiliians
Gregory N. McKay' Nicholas J. Durr
Department of Biomedical Engineering, Johns Hopkins University, Baltimore, USA
ndurr@jhu j edu
Abstract
Analysis of noninvasive microvascular blood flow can improve the diagnosis, prognosis, and management of many medical conditions, including cardiovascular, peripheral vascular, and sickle cell disease. This paper introduces SAM221ow, an interactive optical flow estimation model to analyze long Oblique allmmillumnnation Microscopy (OBM) videos of in vivo microvascular flow. Inspired by the Segment Anything Model (SAM2). SAM2Flow enables users to specify regions of interest through user prompts for focused flow estimation. SAM2Flow also incorporates a dual memory attention mechanism, comprising both motion and context memory, achieve efficient and stable flow estimations over extended video sequences. According
experiments, SAM2Flow achieves SOTA accuracy in foreground optical flow estimation on both microvascular flow and public datasets, with a fast inference speed of over 20 fps on 512 x 512 inputs. Based on the temporally robust flow estimation, SAM2Flow demonstrated superior performance in downstream physiological applications compared existing models. The code is available at: https: //gitub.. com com/DurrLab/SAMPrr
1 Introduction
Mierovascular blood flow parameters, such as flow velocity, provide critical insight to understand and manage diseases that affect blood rheology and vascular mechanics. Noninvasive microvascular imaging allows visualization of vessel and blood cell dynamics at the single-cell level in vivo, providing a window into the early progression of discases and the real-time rheological status of the patient. Some applications include measuring microvascular elasticity to diagnose coronary microvascular dysfunction 1. monitoring tumor angiogenesis assess cancer progression and treatment response 4 and evaluating blood cell function in sickle cell disease [ 6].
Established techniques for measuring vascular flow include Laser Doppler Flowmetry (LDF) ( and Laser Speckle Contrast Imaging (LSCI) [8] But these techniques acquire low low-resolution data and are most sensitive arterioles and venules, which much larger and deeper beneath the skin compared to capillaries. Orthogonal Polarization Spectral (OPS) imaging 9 and Side-stream Dark Field (SDF) imaging 110 have also been explored for capillary-resllution microvaseular measurement of the human oral cavity and finger nailfold. Recently, Oblique Back- mack-illumination Microscopy (OBM) [Н] has demonstrated the potential for non-invasive microvascular measurements with high-speed imaging and subcellular resolution. Green-light OBM can simultaneously record phase and absorption contrast from individual blood cells flowing through superficial capillaries [
For all microvascular imaging techniques, guantitative, accurate, and efficient video analysis is critical to enable clinical impact. Current analysis approaches often use semiquantitative metrics
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

as the Microvascular Flow Index (MFI) [3]. or automated spatiotemporal diagram analysis [14]. which are labor-intensive requiring extensive pre- processsin, including background correction, video stabilization, vessel segmentation, and manual refinement. While cell tracking model [15] has been explored for flow characterization, the computational cost increases drastically for longer videos or large vessels packed with more cells, such as arterioles and venules. To achieve meaningful clinical impact with microvascular flow technologies, efficient, fully automatic flow estimation is critical.
Optical flow (OF) models predict the motion of objects in a video by estimating 2D vector fields that represent pixel-wise displacements between consecutive frames. Despite the impressive performance of deep learning-basse -baen OF models in various upplications, their application to in vivo microvascular flow estimation remains unexplored. Existing models, such as RAFT [16] and GMA [17), are mostly constrained to the analysis of two frames or short segments, More recent models incorporate mechanisms for longer sequences achieving robust flow estimation over long videos. Moreover, conventional OF models estimate optical flow maps across all pixels, which unnecessary when the background movements are not relevant. Lastly, the limited availability of high-quality datasets, especially non-synthetic videos, also hinders the adaptation of OF neural networks to various domains.
[8 19. such as memory [0]. However, challenges persist in
The Segment Anything Model (SAM) 21 and its successor SAM2 22 were introduced as founda- tion models for instance segmentation images and videos. Pre-trained with an unprecedentedly large dataset, SAM2 demonstrated superior scene and object understanding, as well as object memory across frames. Although these models are trained on natural scene datasets, impressive generalizable performance has been achieved by initializing with these pre-trained weights and fine-tuning with task- and modality-specific datasets, particularly in the medical field 25).
To bridge the gap between the latest deep learning research for video analysis and non-invasive blood flow estimation, we introduce SAM2Foow, an optical flow estimation model for vessel-emphasized blood flow analysis of long in vivo OBM videos. Inspired by SAM2, SAM2Flow accepts user prompts to select the regions of interest (ROI), such as target capillary segments specific branches of a complex vascular structure, for fine aie-ggrined flow estimation. To ensure robust flow estimation across long videos, SAM2Flow also incorporates dual memory mechanism, comprising both motion memory and context memory. Fig. [ssowcases the improved clinical workflow by incorporating SAM2Flow as the efficient end-to-end microvaseular flow estimation model.
SAM2Flow represents five major contributions to the field of optical flow estimation and microvascu- lar flow analysis: 1) Introduction of SAM2Flow: The first optical flow neural network specifically designed for blood flow estimation. 2) Interactive ROI optical flow estimation: Enables user-guided selection of regions of interest for efficient flow analysis. 3) Dual memory mechanism: Incorporates motion and context memories to ensure efficient and stable flow estimation in long video sequences. 4) In vivo blood flow dataset for optical flow estimation: Establishes a large human capillary flow video dataset with paired optical flow maps to facilitate future research. 5) General-pur foreground optical flow model: The experiment on the public benchmark. Spring, demonstrates the promising performance of SAM2Flow on joint motion foreground detection and ROI-centric optical flow estimation, beyond the mieroscopic domain.
2 Related Works
2.1 In Vivo Blood Flow Estimation
Superficial microvascular measurement is critical study skin perfusion
26
wound healing 27).
peripheral vascular diseases (28), and neurological blood flow changes 29 LDF 7 is a widely
adopted non-invasive technique to estimate blood flow by measuring the Doppler shift of low-power laser beam source caused by moving RBCs. The result is typically reported in units of perfusion, a combination of RBC concentration and velocity. Another popular imaging-based technique, LSCI (8) maps perfusion by the blur of coherent laser speckle grains from light-scatter of flowing red blood cells. Both methods qualitatively measure flow and typically display in units of relative blood flow instead of absolute flow velocity. In addition, the limited spatial or temporal resolutions of these techniques hinder their ability to localize blood flow changes in specific microvessels. Lastly, both techniques are very sensitive to patient or probe motions. Alternative techniques for flow perception use special sensors, such as event cameras [30] and spike cameras [31,, but the limited quality of the reconstructed frames hampers the visualization of clinically relevant anatomical details.

---

More recent flow estimation techniques using OPS [9] and SDF [10] image superficial capillaries at various sites, including oral cavity, nailfold, and retina. With these techniques, microvascular flow can be differentiated from the background tissue due to the strong green-light absorption of hemoglobin in red blood cells. Previous flow estimation work relied on the movement of absorption gaps from transparent white blood cells (WBCs) or plasma [32 33]- A semiquantitative metric, MFI (13), characterizes microcirculation status in OPS or SDF capillary videos. Estimations are achieved by subjectively classifying each flow absent (0), intermittent (1), sluggish (2), or smooth (3). The video-based MFI is calculated as the average score over all labeled FoV quadrants or vessels. Spatiotemporal(S77 diagrams [4] were applied to absorption-
blood flow videos as an automatic evaluation method. The flow can be quantified by the tilted angle of lines in the ST diagram. Bourquard et al. estimated flow speed and detected WBC from nailfold flows using this method [34]. However, continuous blood flow estimation is hard due to the scarcity of absorption gaps in blood. WBCs typically only account for 0.1 % to 0.2% of total blood cells (35). Moreover, WBC motion is unreliable in representing the net blood flow, due to their unique behaviors, such as rolling and adhesion along the endothelium for leukocyte recruitment during inflammation (36).
OBM has recently been applied to in vivo microcirculation measurement [2]. In addition to absorption contrast, OBM introduces phase contrast, resulting in enhanced visualization of the boundary membranes of both red and white blood cells. Deep learning-base based models have been explored to achieve cytometry and flow estimation OBM videos. CycleTrack [13]. multi-object tracker (MOT), showed promising performance detecting and tracking individual cells throughout the video. One major limitation is that the computational cost increases drastically the vidco gets longer or more cells are presented in the FoV. Therefore, an efficient end-to-end algorithm is needed for imaging-basee based in-vivo blood flow estimation.
2.2 Optical Flow Estimation
Two-frame optical flow is traditionally done through the use of optimizing energy functions to maximize the similarity between two images (3/ Most current OF research uses deep learning techniques to predict pixel level movement from one image to the next. FlowNet, one of the pioneering works in applying convolutional neural networks (CNNs) to OF, introduced an end-to-end trainable framework that demonstrated the viability of learning- baased motion prediction (42 Building upon the use of CNNs, RAFT utilizes a Convolutional Gated Recurrent Unit (ConvGRU), which allows iterative refinement of the flow output from multi-scale 4D cross-correlation volume [16]. Since RAFT was published, works such as GMA, Flowformer++. MatchFlow, and SEA-RAFT have all built upon and improved either the training or architecture of the RAFT base [1 46) Our method, SAM2Flow, harnesses the idea of iterative refinement but does so under explicit segmentation--wwar cues as well as dual memory mechanism to ensure flow consistency even across a wide range of time steps.
Video-based optical flow is a method in which multiple frames are used the input to predict the optical flow at a time point. PWC-Fusion fuses the past flow estimates to the current frame by warping them via small network [47]. This backward-flor floow fusion of past frames provides additional longer-term motion cues but yields only modest accuracy gains (0.65% improvement over two-frame PWC-Net). In contrast, TransFlow and VideoFlow explicitly leverage a wider temporal window by processing a five-frame window centered on the flow prediction [8 48). TransFlow employs a purely transformer-basee architecture with a spatio-temporal encoder that attends across patches of all input frames, capturing long-range correlations, and a decoder that uses the combined feature maps from multiple frames to predict the flow [48]. VideoFlow, on the other hand, uses TRi-frame Optical Flow (TROF) module that jointly estimates bi-directional flows from a center frame to its previous and next frames. A Motion Propagation (MOP) module then links these tri-frame units, propagating motion features so that the effective temporal receptive field grows to cover long sequences. The benefits of higher performance come at a computational cost, jointly modeling multiple frame incurs a large memory footprint and computational overhead. Both TransFlow and VideoFlow require access to future frames and run significantly slower than comparable two-frame models. StreamFllow [19] improves computation efficiency by eliminating redundant processing through the non-overlapping Streamlined In-batch Multi-frame (SIM) pipeline. However, the fixed temporal window remains limited to only a few frames, restricting its ability to maintain stable long-term estimations.
3

---

vovovlv Data
SAM2Flow
Microvascular now-bbaee
Collection
Gomplelee
Hiler Exceee
physiology
Hamory
Excode
Atseniie
Teurture
OBG Imaginng
Uptrn
Prampia
Central
Prompix
CHin prtertion
Giter
Norery
Heart rate, HIV
Vascular classicitt
Frozer SAHD/wighh wiights
Trons 6 inference doselow
Vaso ooc--occcliionnn
Parfusiio
Protecoi 115002641132
Sigrriad
becstive updans.
Figure 1: Clinical workflow of non-invasive microvascular flow evaluation integrating SAM2Flow, the end-to-end optical flow estimation model. Prompted by sparse point labels, SAM2Flow aims for robust flow estimation in detected ROIs, over long in-vivo microvascular flow sequences with the dual-memory mechanism, including context and motion memories. The flow estimation output is critical for various physiological measurements.
To address these drawbacks, MemFlow introduces an efficient memory-based design that processes videos in an online fashion [20). Instead of stacking multiple frames into the network once, MemFlow maintains a learned memory bank of frame-wise motion features and context embeddings, which queried (via attention) to provide relevant past motion cues for the current frame. This design effectively captures long-range temporal information without needing to explicitly feed in long frame sequence, dramatically reducing computation per flow frame. While all of these methods highlight the benefits of ablation studies MemFlow found that models do not meaningfully utilize extended histories, with less historical frame data leading to higher performance (0)) In the blood flow estimation task, objects move much faster across frames compared to traditional optical flow tasks, even when captured at 200 FPS. This results in fast-moving, optically ambiguous data that requires memory of previous flow patterns to maintain consistency. SAM2Flow seeks to solve this problem through a dual memory mechanism that ensures stable optical flow over long sequences.
input optical flow for maintaining consistency across time,
3 SAM2Flow
3.1 Definition and Overview
Problem Setup. Forward optical flow is per-pixel 2D displacement vector field: f (") E TR. Hxxx2 mapping the location (u, v) from the current frame I E R to the location (u + ("(,,)), f eu), in the next frame Ie+1-
Mainstream optical flow networks comprise three major modules: a feature encoder, 81 (L)), E hx that extracts pair of low-level textural features to match across frames; a context encoder, {I())) E R that encodes high-level current context to ensure meaningful and smooth optical flow field; and a flow update module, Aʃ = (E m4), which fuses context understanding with encoded motion features and iteratively refines optical flow estima- tion with an RNN-based model. The image feature pair is used to construct a series of 4D correlation volumes, C. = (T,) @ AvgPool(E' (T++)) E IR. hxwx 2F x " where ( represents the correlation operator which computes similarities (dot products) between each pixel in the image feature pair. is then used to calculate motion features, m, e (L)) E xx The LookUp operator is some function that returns a motion feature vector for each pixel in Ie-
The overview of SAM2Flow is presented in Fig. [ We used the SEA-RAFT feature encoder, ResNet-based encoder pre-trained on 6 different optical flow datasets, as the feature encoder backbone gd(( As the context encoder backbone (-).,, we integrate the SAM2 ViT image encoder for its strong semantic feature encoding and generalizability from its large-scale pretraining. To adapt the SAM2 encoder to the OBM domain with limited GPU resources, we apply the SAM2 adapter [49], which injects fine-tuning weights into the frozen encoding trunk layers. We choose Con-NeXt-RNN from DEA-RAFT as flow updates backbone for its superior efficiency in flow
Overview.

---

regression. We will elaborate on our proposed prompt-guided flow estimation in section 3.2 and dual memory module for optical flow estimation in section 3.3 We apply two efficient flow initialization methods, ROI Registration and Warm Start, to further facilitate flow estimation. Check section A for a detailed description.
3.2 Prompt-guided Flow Estimation
When predicting an optical flow map from pair of input frames, the feature encoder extracts textural features be matched pixel-wise during flow updates, while the context encoder focuses on semantic information to ensure the final flow outputs are meaningful and smooth within object-wise regions. Therefore, to achieve accurate flow estimation for specific ROIs, SAM2Flow conditions encoded context with foreground information.
Prompt-conditioned Context. Inspired by SAM, SAM2Flow takes input prompts that specify ROIs that are are more diagnostically relevant, such as different vessels or branches of a complex ves- sel structure, to condition focused flow estimation from the context encoder, in cases where complex overlapping vessel structures or irrelevant background motion (e.g., heartbeat, tissue movement) can mislead outputs.
Our model takes several sets of points and corresponding labels 4 E [1,03 (foreground-1 / background-O) as prompts P. = {(r; Ve: "" Each point prompt set representing an ROI is encoded into ID prompt vector with prompt encoder, E"(p,) R IIx,, Whenever there are user prompt inputs at a certain frame, the encoded context is augmented by the prompts via two-layer cross-attention:
(T)) = Cross-AAttention (T"
(1)
where (T)) E R.xxwDD the context prompted by the 1th ROI point set. During flow estimation, the prompted context would guide the flow update module, U1 to focus on the refinement for the foregrounds and output flow map for each region:
A = ((P(E mm))
(2)
If there is no user prompt input as in default, the model only predicts one unprompted flow map for the whole frame.
ROI-guided Correlation Lookup. One of the major computational bottlenecks in the current optical estimation model is the correlation search. Cg E /xx A/2 x x is the A.th layer of downsampled correlation volume in the pyramid. In RAFT-like architectures, at the beginning of each flow update iteration, the correlation map Corr E 1xx= K is generated by retrieving the correlation values for all pixels at all levels of the pyramid, based on the current flow prediction, resulting in complexity of O (hhw)).
As flows in OBM videos highly localized, occupying only small portion of the entire field of view, SAM2Flow speeds up the lookup operation and suppresses the background noise by retrieving only values for foreground regions. To guide the correlation lookup. SAM2Flow incorporates an ROI decoder, similar to the mask decoder in SAM2, that makes a prediction for an ROI It' E R based each prompted context. The ROI decoder also outputs 1D object pointer vector O R1xDMM that is used for context memory in the following section:
R' R², = D (P)(E())))
Therefore, ROI-guided correlation lookups have complexity of O(NrK), where NR is the total pixel number of ROIs
3.3 Dual Memory Module
For robust performance over an extended video sequence (e.g., 12,000 frames for 60 seconds of OBM videos), SAM2Flow incorporates memory from previous time points enhance current flow estimation. As optical flow is predicted by combining motion and context features, we propose the dual-memory mechanism, consisting of both motion and context memories. The motion memory ensures the long-term estimation smoothness for constantly pulsing flows, which is essential for downstream physiological analyses, while context memory helps to keep track of the identities of target vessels.
5

---

Memory Encoding, Previous flow predictions are encoded into motion memory and context memory by two separate memory encoders. We adopt memory encoders from SAM2 as our backbones. For motion memory at lo. the memory encoder directly takes the flow map of each ROI as input and fuses with the corresponding motion feature map:
Mr = -.MM. (fty mc,) E IR Ax
For context memory, the flow map will be binarized into the flow mask B(fta) =xxxxx before being fed into the context memory encoder, where it is combined with unprompted context by the context memory encoder:
Mi. = =BMM(((()... R hXwxDD
(5)
Memory Bank. The encoded motion and context memories, along with object pointers, are then stored in FIFO queue, named the memory bank. To save spaee for long-video inference, the memory bank is limited to store memories up to N recent frames. When the memory bank is full, the earliest memories discarded. When there memories from n recent frames (0 < N) in the memory bank at a time point, tp. the model retrieves all memories from the memory bank and stacks them into M byr-ns Mi E BaxhkxxDMM
Motion Memory Attention. The microvascular flow pattern within specific ROI usually temporally smooth and predictable. For example, the blood in certain vessel tends to flow in the same direction throughout the whole video. Therefore, SAM2Flow introduces motion memory that stabilizes flow regression. We utilize a stack of vanilla attention blocks of alternating self- and cross-attention [50] to condition the current motion feature with motion memories:
Mem(m Att" (m,, M (
(6)
Context Memory Attention. For reliable flow estimation time, the context branch of SAM2Flow should provide stable semantic information and segmentation of ROIs. Context memory propagates user-defined ROIs across frames. As result, SAM2Flow only requires sparse user inputs in the first few frames of a long video. To achieve this, we condition the current context from context encoder with the stacked context memory and object pointers, using the same attention operation as eq. (6)
Me,,E((I,,)) (E (E Mi Otpmm)
(7)
When comes a frame with user prompt inputs, SAM2Flow prioritizes the user prompts and skips context memory attention. With the memory-augmented motion and context features, the flow update module is able to generate temporally smooth flow estimations with the same iterative refinements as eq. for each ROI:
A u[Meem(E" (I,,)") Mem(m)))
(8)
4 Experiments
4.1 Datasets
Microvascular Dataset. We establish an in-vivo microvascular flow dataset that is larger than most of the existing public optical flow datasets, including 75 videos, with the paired ground truth flow maps of 306,800 frames in total. The grayscale videos are collected by imaging the superficial capillaries in the oral cavity of 15 healthy volunteers using the OBM system [6]. at 200FPS with frame size of 512 x 512. All participants gave written informed consent, and experiments were conducted under Johns Hopkins University Institutional Review Board-approved protocol (RB002641500) We split the dataset into training, validation, and testing subsets, containing 45, 15, and 15 videos, respectively. The flow map ground truths are generated using the spatiotemporal diagram [E] with manual refinement.
Public Datasets. We use two public datasets to test the generalization ability of SAM2Flow outside of the microscopic domain. Sintel [5]] and Spring (52) are both popular optical flow datasets with long animation sequences and pixel-accurate flow ground truth. Since this paper proposes novel challenge of joint motion ROI detection and optical flow estimation, we apply the SAM2 model to generate panoptic ROI masks these two dataset. We utilize the Spring training dataset, consisting

---

Table I: Comparative study of optical flow estimation performance on Microvascular test set.
Whole Image
Foreground
Speed
Model
EPEL
1pxt
3px
5px
FEPE,
5pxt
10px
15pxt
mspf
RAFT [6
3.18 (2.61)
0.86
0.89
0.91
27.73 (24.79)
0.39
0.52
0.56
51.48
GMA
17
3.22 (3.66)
0.87
0.89
0.91
28.34 (26.82)
0.38
0.54
0.58
43.66
SEA-RAFTHES
1.28 (1.03)
0.88
0.92
0.94
6.60 (5.47)
0.69
0.86
0.91
21.14
FlowFormert++
1.72 (1.38)
0.88
0.91
0.93
10.89 (9.28)
0.60
0.78
133.95
VideoFlow BOF
18
3.28 (2.51)
0.86
0.87
0.88
28.16 (26.64)
0.15
0.32
0.41
112.67
MemFloww(Mi
20
1.79 (1.40)
0.88
0.91
0.93
12.47 (10.23)
0.58
0.74
0.80
43,98
StreamFFlow (MF)
19
1.43 (1.02)
0.88
0.90
0.93
10.13 (8.36)
0.49
0.74
0.84
60.07
SAM2Flow (MF)
1.14 (0.92)
0.88
0.93
0.96
5.84 (4.86)
0.66
0.86
0.93
48.78
(MF) indicates multi-frames optical flow models; best performance is highlighted, while second hest performance is underlined; EPE & FEPE: Mcan(Standart Deviation, SD); mspÉ: milliseconds per frame.
37 videos, 10,000 paired flow GTs, and then create training, validation, and test splits with 25, 5, and videos, respectively. For Sintel dataset, we split the total of 23 scenes into training, validation, and test with 14, 3 and 6 videos. The Sintel training and validation combine both Clean and Final videos. For evaluation, we report the performance on the Clean and Final videos separately. More details about datasets, video pre-processing, and ground truth generation are described in section B
4.2 Implementation
SAM2Flow Settings. All the models trained and tested on two NVIDIA RTX A5500 GPUs. The feature and context encoders downscale inputs 8 times to 64x64. SAM2Flow runs 8 flow update iterations for training, and 4 for inference. We set the number of correlation pyramid layers K to and the memory bank limit N to 7. For SAM2Flow training, we input an 8-frame video at each step with MoL loss from SEA-RAFT, using the Adam optimizer and global learning rate starting 1c-4, with weight decay and scheduler. We use a multi-stage training strategy detailed in section C.I
Comparative Experiments. To demonstrate SAM2Flow's performance, we conduct the comparative study with six SOTA baseline models, including Two-frame Models: RAFT(16), GMA[T7], SEA- 46] FlowFormer++
and Multi-frame Models: VideoFlow [18], MemFlow [20], and FtreamFlow 19 on both Microvascular and Spring datasets. For fair comparison, we empirically choose the pretrained checkpoints and fine-tune the model the corresponding datasets following the configurations from the papers. (Details at section C.2
Evaluation Metrics. For microvascular videos. we evaluate accuracy with end-point errors (EPE) and percentages of pixel errors within [1, 3, 5px] for the whole frames. Since the SAM2Flow focuses the informative ROIs, we also report foreground EPE (FEPE) and larger pixel error rates [5, 10, 15px] within foregrounds, due to the rapid blood flow (~ 30pp00pppammmmeeee ffrme), shown in fig. 2 For microvascular videos, foregrounds are defined as in-focus vessels with active blood flow. For Sintel video, we also report FEPEs on both Clean and Final sets. Following previous works on Spring dataset, we report FEPE, foreground 1px errors, foreground flow outlier rate (F1) that is defined as > 3pxs and > 5% of GT flow magnitude), the average EPE of flow outliers (Fl-epe), as well as foreground weighted AUC (WAUC). The foreground ROIs in Sintel and Spring videos are defined as the objects or regions with motions that are distinctive from the background scenes.
5 Results
5.1
Microvascular Flow Prediction Performance
The baseline models and SAM2Flow are evaluated on the Microvascular test split. The quantitative results are illustrated in [a The best single-frame baseline performance comes from SEA-RAFT, achieving an EPE = 1.28(SDD 1.03) and a foreground EPE 6. (6.45)))) with the fastest inference speed among all models (21.1 14ms), thanks to the eight-weighted encoders and fewer flow update iterations. FlowFormer++ also achieves strong performance FEPE = 10.89(9.28) but at much higher computational cost (133.95ms due to its large cransformrr-based cost-volume encoder.
7

---

Table 2: Comparative study of foreground optical flow estimation on Sintel and Spring datasets.
Sintel-FEPE:
Spring-Fone Foreground
Model
Clean
Final
FEPE
Ipx (%)t
Fl (%))
Fl-epel
WAUC
RAFT
[6
5.21 (9.30)
5.47 (10.12)
2.25 (6.97)
74.25
9.87
7.98 (8.98)
74.76
GMA
17
4.65 (7.45)
5.14 (8.17)
2.17 (6.30)
76.21
9.08
7.67 (7.72)
79.67
SEA-RAFT
3,26 (7.54)
4.08 (8.94)
1.45
86.32
5,18
8.18 (10.85)
83.85
MemFlow (MF)
(MF)
20
3.77 (5.82)
4.27 (7.09)
1.56 (7.27)
86.49
7.24
8.82 (12.72)
83.54
StreamFlow
19
4.06 (5.37)
4,43 (5.66)
1.54 (7.23)
85.16
5.82
7.74 (10.16)
82.51
SAM2Flow
3.17 (6.89)
3.39 (5.97)
1.23 (4.21)
87.13
5.29
7.30 (5.29)
84.57
Multi-frame models generally infer more slowly than single-frame approaches as models process additional temporal information. MemFlow achieves reasonable flow estimation accuracy FEPE 12.47(10.25 with fast inference speed (43.98s), with only one frame of memory 20).. SAM22Flww outperforms all other models whole-image EPE 1.14(0.92) and FEPE 5.84 4.86), indicating its superior flow estimation accuracy. Compared to single-frame models and MemFlow SAM2Flow maintains a competitive inference speed (48.78m 78ms) while incorporating long-term dual memories of 7 frames. Overall, these results highlight the effectiveness of SAM2Flow in balancing high accuracy with efficient computation, outperforming both single-frame and other multi-frame models in critical performance metrics.
We note that our test set ground truths do not have pixel-level accuracy, as they are derived from ST diagrams, a spatially and temporally smoothed estimation. As well-trained models achieve lower errors, quantitative comparisons with the current ground truth may become less indicative of true performance. Consequently, in section 5.6 we conduct more in-depth analysis to assess whether the predicted flow maps accurately reflect the actual flow patterns from the videos.
5.2 Public Benchmark: Sintel
On the Sintel benchmark, SAM2Flow demonstrates superior performance in foreground flow esti- mation compared to all baselines. As shown in table 2 SAM2Flow achieves the lowest errors on both Clean (FEPE 3.17) and Final (FEPE 3.39) videos. While SEA-RAFT (FEPE 3.26) and MemFlow (FEPE = 3.77) show good accuracy in Clean videos that contains simpler scenes with less texture, their performances deteriorate when adding more textures and illumination effects the scene, as in Final videos (SEA-RAFT AFEPE = 0.82 and MemFlow AFEPE = 0.5); on the contrary, SAM2Flow shows strong robustness (AFEPE 0.22) to these challenges for optical flow estimation within the motion ROIs.
5.3 Public Benchmark: Spring
The quantitative results on Springs are shown in the second half of table [ For foreground flow estimation, SAM2Flow demonstrates the strongest performance FEPE 1.23 1.23(4.21) (4.21), with a 15% improvement compared to the best baseline model, SEA-RAFT FEPE 1.45(..61). With the context- guided focus motion foregrounds, SAM2Flow achieves finer estimation. especially for fast and complex motions within the ROIs, resulting in low flow outliers: F1 5.29%, Fl-epe 7.30. The visual comparison can be found in the fig. These results, combined with the strong performance on the Sintel dataset, highlight that SAM2Flow achieves impressive generalizability and the of-te--ftth-rrtt ROI-centric optical flow estimation beyond the microscopie domain.
5.4 Ablation Study
The results of the ablation study shown in table We conduct ablation studies test the efficacy of memory modules and ROI-guided lookup operation to flow estimation accuracy and efficiency.
Backbone Encoders. Backbones-Only is simple combination of pre-trained SAM2 con- text encoder and the SEA-RAFT feature encoder backbone with limited fine-tuning on our dataset. Even though FEPE 9.75 of this basic setup is among the SOTA performance, does not outperform the original SEA-RAFT model (table []. indicating that this back- bone configuration has the potential to be further improved by leveraging public datasets.
8

---

le
le
le
Vetocity Estimation
Frequency Spectrum
Inputs
GT
SAM2Flow SEA-RAFT MemFlow
Qualitative comparison on the microvascular test video. Left: Five representative frames, GT flow maps, and predictions from TOP3 models, on challenging video where two blood flows cross. Arrows highlight regions where MemFlow and SEA-RAFT mistakenly merge the two streams, while SAM2Flow maintains distinct ROIs throughout the sequence. Right: Velocity estimations at two fixed ROIs (dashed boxes) with corresponding frequency spectra over 5000 frames. SAM2Flow generates the most robust and clinically meaningful velocity estimations from both flows.
Figure 2:
Dual Memory Mechanism. We tested the motion and context modules sep- arately. M-Mem Only adds motion memory module to the backhones, while C-Mem Only contains only the context memory module. Based on the Fore- ground accuracy results, the context memory module better boosts the flow estimation accuracy (AFEPE = -3.43), the context encoder from the SAM ViT backbone is tailored to work with the memory mechanism. On the con- trary, motion features encoded by the CNN-based encoders benefit less from the memory mech- anism (AFEPE -2.34). Meanwhile, motion memory yields heavier computational overhead (AT = 16.22m.... as its attention is embedded within the iterative flow updates. No ROI consists of the complete dual memory module and encoder backbones. As the results suggest, the dual memory module outperforms either of the context/motion-only memories, achieving the best FEPE of all experiments. Therefore. the final SAM2Flow integrates both context and motion memories to achieve the most robust flow estimation.
Table 3: Ablation study on Microvascular validation set.
Foreground
Speed
Method
FEPE,
5pxt
10pxt
15pxt
mspf]
Backbones
9.75
0.54
0.77
32.76
M-Mem Only
7.41
0.63
0.81
48.98
C-Mem Only
6.32
0.64
0.84
0.91
38.63
No ROI
5.12
0.69
0.89
0.94
55.25
SAM2Flow
5.46
0.72
0.87
0.92
48.78
M-Mem: Motion memory; C-Mem: Context memory.
ROI-guided Lookup. No ROI does not include ROI-guided correlation lookup. To achieve higher efficiency, the final SAM2Flow uses ROI segmentation to guide correlation lookup. avoiding indexing irrelevant background pixels. speeds up inference by 12% without significantly affecting accuracy.
5.5 Qualitative Comparison
We visualize and compare SAM2Flow flow estimates with the best- dest-performing single-frame model, SEA-RAFT, and multi-frame model, MemFlow, on sample frames fig. [ The sample video is challenging case, two separate flows cross each other. The results suggest that both MemFlow and

---

SEA-RAFT get confused and merge two flows by mistake. On the other hand, with the help of sparse prompts in the first few frames, SAM2Flow outputs robust flow estimations as two separate flows. The visualization suggests that SAM2Flow achieves stronger flow estimation in more complex flow structures, especially when flows are closely tangled in the VoV.
5.6 Physiological Applications
We analyzed velocity estimation from the Top3 models throughout 5000 frames (25s) on the sample test video. Two sllocit--vver-imng plots are generated by averaging the flow estimation within the fixed subregions within two vessels. We also plot the frequency spectra to verify the extracted pulsatile patterns. SAM2Flow yields the strongest heart rate signals (50.4 BPM) from both flows. With temporal memory, SAM2Flow is less susceptible to noise. Taking a closer look at the waveforms, the pattern within each cardiac cycle from SAM2Flow closest to a meaningful clinical waveform from other means of measurement 7 53 including central peak (peak systolic velocity), subpeak, and a central trough (end-diastolic velocity). More clinically relevant biomarkers, such as peak ratio and resistive index [53], could be characterized based on SAM2Flow velocity estimation. Across long videos, SAM2Flow achieves more robust and accurate velocity estimation.
5.7 Limitations
Due to the focus on foreground estimation and ROI-guided correlation lookup, SAM2Flow relies on robust ROI detection from the context branch. Therefore, the flow estimation would deteriorate due to failed or incomplete ROI detection in some complex scenes. However, our model design mitigates the effect on flow estimation from different types of ROI errors:
Over segmentation. When ROI is overly segmented, some background pixels are also indexed in correlation lookup. However, this wouldn't directly degrade flow estimations, the trained motion encoder and RNN further delineate the motion boundary through iterative flow updates. Full-frame flow regression could be considered as an extreme case of over-segmentation.
Transient frames drop or incomplete ROL. Motion memory module provides redundancy against occasional failures by maintaining consistency with prior memories (section 3.3). This feature stabilizes predictions when one frame has incomplete or no ROI detection. This is demonstrated by the ablation study of the motion memory module, where FEPE improves from 6.32 to 5.46 (table B)) SAM2Flow incorporates an additional trick like warm start (section A). to further stabilize the flow estimation in cases of occasional ROI instability.
Missed/incomplete ROI over an extended period. Failed ROI detection over more consecutive frames would result in failed optical flow estimation, since the model fails to identify any target. However, as an interactive pipeline. SAM2Flow offers the ficxibility to add/correct user prompts for these challenging frames during inference. We recommend using multiple positive points spaced over the ROI for larger targets to mitigate prompt ambiguity.
Conclusion
In this paper, we first identified new challenge of ROI-centric optical flow estimation over long sequences. Therefore, we introduce SAM2Flow, an interactive optical flow estimation model for in vivo microvascular flow from OBM videos. Our technique enables user-specified ROIs through sparse point prompts for ROI-specific flow estimation. To ensure that flows remain temporally consistent, even fast and optically ambiguous data, we propose a novel dual memory attention mechanism. comprising both motion and context memory. We demonstrate the effectiveness of SAM2Flow by testing against other baseline models on both microvascular videos. The proposed model achieves the best EPE and foreground EPE among all baselines on the test set. Additional experiment on the public Spring dataset indicates the promising generalizability videos. For future work, we aim to further boost model speed and flow estimation accuracy by exploring more compact encoder backbones, such as SAM2-B+ and SAM2-S, and on diverse datasets. Moreover, we would adapt SAM2Flow to microvascular flow with various conditions, such as sickle cell and sepsis patient data. In summary, SAM2Flow introduces innovative techniques to address the unique challenge of region-speciff optical flow. This approach empowers physicians extract microcireulation biomarkers from OBM video data non- non- aon-invasvvely.
SAM2Flow beyond microscopic
10

---

Acknowledgement.
This work was supported in part by research funding from the Gates Foundation (INV-006005).
References
[1] Nocl Bairey Merz. Testing for coronary microvascular dysfunction. Jama, 322((3):2338-222 2358,, 2019.
[2] Lee Bockus and Francis Kim. Coronary endothelial dysfunction: from pathogenesis to clinical implications. Open Heart, 9(2). 2022.
[3] Allison G Hays, Sebastian Kelle, Glenn Hirsch, Sahar Soleimanifard, Jing Yu, Harsh K Agarwal, Gary Gerstenblith, Michael Schär, Matthias Stuber, and Robert G Weiss. Regional coronary endothelial function is closely related to local early coronary atherosclerosis in patients with mild coronary artery disease: pilot study. Circulation: Cardiovascular Imaging, 5(3):341-348, 2012.
[4] Chengwu Huang, Matthew Lowerison, Fabrice Lucien, Ping Gong, Diping Wang. Pengfei Song, and Shigao Chen. Noninvasive contrast-free 3d evaluation of tumor angiogenesis with ultrasensitive ultrasound microvessel imaging. Scientific reports, 9(1 (1):4907, 2019.
[5] Marisa Morakis, Luojie Huang, Gregory N McKay, Sophie Lanzkron, Lydia II Pecker, and Nicholas Durr. In vivo sickle cell blood rheology in humans with ventral tongue microscopy. Blood, 144:2481, 2024.
[6] Marisa M Morakis, Luojie Huang, Gregory N McKay, Sophie Lanzkron, Lydia H Pecker, and Nicholas J Durr. Sickle cell visualization in vivo in humans: microvascular occlusion formation and hemorheological indices. medRxiv, pages 2025-09, 2025.
[7] MD Stern. In vivo evaluation of microcirculation by coherent light scattering. Nature, 254(5495):56-58, 1975.
[8] David Briers and Sian Webster. Laser speckle contrast analysis (lasca): nonscanning, full-field technique for monitoring capillary blood flow. Journal of biomedical optics, 1(2):174 179, 1996.
Warren Groner, James W Winkelman, Anthony G Harris, Can Ince, Gerrit Bouma, Konrad Messmer, and Richard G Nadeau. Orthogonal polarization spectral imaging: a new method for study the microcirculation. Nature medicine, 5(10):1209-1113 1999.
[10] PT Goedhart, M Khalilzada, Rick Bezemer, J Merza, and Can Ince. Sidestream dark field (sdf) imaging: a novel stroboscopic led ring-based imaging modality for clinical assessment of the microcireulation. Optics express, 15(23):15101 15114, 2007.
[11] Tim Ford, Kengyeh K Chu, and Jerome Mertz. Phase graddeet microscopy in thick tissue with oblique back-illumination. Nature Methods, 9112) 2012.
[12] Gregory N McKay, Nela Mohan, and Nicholas J Durr. Imaging human blood cells in vivo with oblique back-illuminttion capillaroscopy. Biomedical optics express, ((5))237322382, 2020.
[13] E Christiaan Boerma, Keshen R Mathura, Peter HJ van der Voort, Peter E Spronk, and Can Ince. Quantifying bedside-derived imaging microcirculatory abnormalities in septic patients: a prospective validation study. Critical care, 9:1-6, 2005.
[14] Yoshinobu Sato, Jian Chen, Reza A Zoroofi, Noboru Harada, Shinichi Tamura, and Takeshi Shiga. Automatic extraction and measurement of leukocyte motion in microvessels using spatiotemporal image analysis. IEEE Transactions on Biomedical Engineering. 44(4):225-236, 1997.
[15] Luojie Huang, Gregory N McKay, and Nicholas Durr. A deep learning bidirectional temporal tracking algorithm for automated blood cell counting from non-invasive capillaroscopy videos. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 415-424. Springer, 2021.
[16] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow, In Computer 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part 16, pages 402-419. Springer, 2020.
[17] Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning to estimate hidden motions with global motion aggregation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9772-978. 2021.
11

---

[18] Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung. Simon See. Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12469- 12480, 2023.
[19] Shangkun Sun, Jiaming Liu, Huaxia Li, Guoqing Liu, Thomas Li, and Wei Gao. Streamflow: streamlined multi-frame optical flow estimation for video sequences. Advances in neural information processing systems, 37:9205- 9228, 2024.
[20] Qiaole Dong and Yanwei Fu. Memflow: Optical flow estimation and prediction with memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19068- 19078, 2024.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao. Spencer Whitchead, Alexander C. Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEF/CVF international conference computer vision, pages 4015-4026, 2023.
[22] Nikhila Ravi, Valentin Gabeur, Than-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloc Rolland, Laura Gustafson, al. Sam Segment anything in images and videos. arXiv preprint arXiv:2408.007/1 007144, 2024.
[23] Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1): 15(1),654, 2024.
[24] Jiayuan Zhu, Abdullah Hamdi, Yunli Qi. Yueming Jin, and Junde Wu. Medical sam 2: Segment medical images as vidco via segment anything model 2. arXiv preprint arXiv:2408.00874 2024.
[25] Wenxi Yue, Jing Zhang, Kun Hu, Yong Xia, Jiebo Luo, and Zhiyong Wang. Surgicalsam: Efficient class promptahle surgical instrument segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6890-6898, 2024.
[26] Guillaume Mahé, Anne Humeaa-Heurtterr Sylvain Durand, Georges Leftheriotis, and Pierre Abraham. Assessment of skin microvascular function and dysfunction with laser speckle contrast imaging. Circulation: Cardiovascular Imaging, 5(1):155-163, 2012.
[27] William W Li, Marissa J Carter, Flad Mashiach, and Stephen D Guthrie. Vascular assessment of wound healing: a clinical review. International Wound Journal, 14(3):460-469, 2017.
[28] Stefano Masi, Damiano Rizzoni, Stefano Taddei, Robert Jay Widmer, Augusto Montezano, Thomas F Lüssherr, Ernesto Schiffrin, Rhian M Touyz, Francesco Paneni, Amir Lerman, et al. Assessment and pathophysiology of microvascular disease: recent progress and clinical implications. European heart journal, 22(26))250--2044 2021.
[29] Maud van Dinther, Paulien HM Voorter, Jacobus FA Jansen, Elizabeth AV Jones, Robert J van Oost- enbrugge, Julie Staals, and Walter H Backes. Assessment of microvascular rarefaction human brain disorders using physiological magnetic resonance imaging. Journal of Cerebral Blood Flow & Metabolism. 42(5):718-737,200 2022.
[30] Xinglong Luo, Ao Luo, Zhengning Wang, Chunyu Lin, Bing Zeng, and Shuaicheng Liu. Efficient meshflow and optical flow estimation from event cameras. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pottern Recognition. pages 19198-1999 19207, 2024.
[31] Rui Zhao, Ruiqin Xiong, Jing Zhao, Zhaofei Yu, Xiaopeng Fan, and Tiejun Huang. Learning optical flow from continuous spike streams. Advances in Neural Information Processing Systems, 35:7905-7920. 2022.
[32] Linxi Zhao, Jiankai Tang, Dongyu Chen, Xiaohong Liu, Yong Zhou, Yuanchun Shi, Guangyu Wang, and Yuntao Wang. A comprehensive dataset and automated pipeline for nailfold capillary analysis. In 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pages 1 1-5, IEEE, 2024.
[33] Michael Berks, Graham Dinsdale. Andrea Murray. Tonia Moore. Joanne Manning, Chris Taylor, and Ariane L Herrick. Automated structure and flow measurement-2 promising tool in nailfold capillaroscopy. Microvascular Research, 118:173-177, 2018.
[34] Aurélien Bourquard, Alberto Trinida,, Ian Butterworth, Álvaro Sânchez-Ferro, Carolina Cerrato, Karem Humala. Marta Fabra Urdiola, Candice Del Rio, Betsy Valles, Jason M Tucker Schwart,, et al. Non-invasive detection of severe neutropenia in chemotherapy patients by optical imaging of nailfold microcireulation. Scientific reports, 8(1):5301, 2018
12

---

[35] Martin S Blumenreich. The white blood cell and differential count. Clinical Methods: The History, Physical, and Laboratory Examinations. 3rd edition, 1990.
[36] Marion Leick, Veronica Azcutia, Gail Newton, and Francis Luscinskas. Leukocyte recruitment in inflammation: basic concepts and new mechanistic insights based on new models and microscopic imaging technologies. Cell and tissue research, 355(3);647-66) 656, 2014.
[37] Michael J Black and Padmanabhan Anandan. A framework for the robust estimation of optical flow. In 1993 (4th) International Conference on Computer Vision, pages 23 231-236. IEEE, 1993.
[38] Michael Black and Paul Anandan. The robust estimation of multiple motions: Parametric and piccewise- smooth flow fields. Computer vision and image understanding, 63(1):75-100 104, 1996.
[39] Thomas Brox, Andrés Bruhn, Nils Papenberg. and Joachim Weickert. High accuracy optical flow estimation based theory for warping. Computer Vision-ECCV 2004; 8th European Conference Computer Vision, Prague, Czech Republic, May 11-14, 2004. Proceedings, Part IV 8, pages 25-36. Springer, 2004.
[40] Andrés Bruhn, Joachim Weickert, and Christoph Schnörr. Lucas/kanade meets hom/schunck; Combining local and global optic flow methods. International journal of computer vision, 61:211-231, 2005.
[41] Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 17(1-3):15-2033, 1981.
[42] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 2758 2766. 2015.
[43] Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin. Jifeng Dai. and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In European conference on computer vision, pages 668 685. Springer, 2022.
[44] Xiaoyu Shi, Zhaoyang Huang, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See. Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformert+:: Masked cost volume autoencoding for pretraining optical flow estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1.599-1610, 2023.
[45] Qiaole Dong, Chenjie Can, and Yanwei Fu. Rethinking optical flow from geometric matching consistent perspective. Proceedings of the IEEE/CVE Conference on computer vision and puttern recognition. pages 1337-1347, 2023.
[46] Yihan Wang. Lahav Lipson, and Jia Deng. Sea-raft: Simple, efficient, accurate raft for optical flow. In European Conference on Computer Vision, pages 36 54. Springer, 2024.
[47] Zhile Ren, Orazio Gallo, Deqing Sun, Ming-Hsuan Hsuan Yang. Erik Sudderth, and Kautz. A fusion approach for multi- frame optical flow estimation. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2077-2086. IERE, 2019.
[48] Yawen Lu, Qifan Wang, Siqi Ma, Tong Geng, Yingjie Victor Chen, Huaijin Chen, and Dongfang Liu. Transflow: Transformer flow learner. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18063-18073, 2023.
[49] Tianrun Chen, Ankang Lu, Lanyun Zhu, Chaotao Ding, Chunan Yu, Deyi Ji, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang. Sam2-adapter: Evaluating & adapting segment anything 2 in downstream tasks: Camouflage, shadow, medical image segmentation, and more. arXiv preprint Sii::24080045799 2024.
[50] Tri Dao. Flashattention-2:: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08611 2023.
[51] Daniel J Butler. Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for optical flow evaluation. In Computer Visi 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12, pages 6116611166666 1625. Springer, 2012.
[52] Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nalivayko, and Andrés Bruhn. Spring: A high- resolution high-detail dataset and benchmark for scene flow, optical flow and stereo. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 498 1-991, 2023.
13

---

[53] Christopher Lockhart, Andrew J Gamble, Derrick Rea, Sinead Hughes, R Canice McGivemn Clive Wolsley, Michael Stevenson, Mark T Harbinson, Richard D Plumb, and Gary E VeVgh... Nitric oxide modulation ophthalmic artery blood flow velocity waveform morphology in healthy volunteers. Clinical Science, 111(1):47-55 52, 2006.
[54] Moritz Menze. Christian Heipke, and Andreas Geiger. Object scene flow, ISPRS Journal of Photogranme- try and Remote Sensing (JPRS), 2018.
[55] Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg. Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758 2766, 2015.
[56] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafci Hu, Ashish Kapoor, and Scbastian Scherer. Tartanair: dataset to push the limits of visual slam, In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1909-4916. IEEE, 2020.
[57] Daniel Kondermann, Rahul Nair, Katrin Honauer, Karsten Krispin, Jonas Andrulis, Alexander Brock, Burkhard Gussefeld, Mohsen Rahimimoghaddam, Sabine Hofmann, Claus Brenner, al. The hei benchmark suite: Sterco and flow ground truth with uncertaintics for urban autonomous driving, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 19-28, 2016.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The SAM2Flow proposed in this paper is mainly designed to bridge the gap between deep learning-baan optical flow estimation and clinical in-vivo blood flow measurement. And the public benchmark experiments showed that the model has the generalizability to tackle a novel challenge, joint motion ROI detection and focused optical flow estimation, in other domains. All the above points summarized in the abstract and introduction sections, and further demonstrated in the experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The limitations, including the model dependence on robust ROT detection and the speed, are discussed in the Results 5.7 and the Conclusion 6 section.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattio, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The model design is detailed in the SAM2Flow section section. The dataset and experiment details, including data processing and training configurations and hyperparameters, are elaborated in the Experiments section as well as the section C
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
16

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: The code and dataset will be published on Github with the paper after the anonymous review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https
//wips.cc/publl.. wttpc//ppppppppu..eeeeeeeooopppp guwdee/deeeeee Codeeuumissseeeeeeee ssionooicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see the Experiments section 4 and appendix section ffor all the setting details. Moreover, the code will be published with the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For the comparative study, we have provided the means and standard deviations of the end- and-toint errors for all the models.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the compute resources and training details in section and section
C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://///.popp...cop//aaattttaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
18

---

The proposed SAM2Flow could potentially improve healthcare by enabling the efficient, non-invasive point-of-care microvascular blood flow measurements. There's no direct negative social impact from this work.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
The paper poses no such risks. To prevent PHI leakage of the human dataset, we have conducted de-identification according to the safety protocol before releasing the dataset.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
The Proposed SAM2Flow model is inspired by and built upon the Segment Anything model and SEA-RAFT. Both papers, codes and related licenses have been cited and included in this paper, and the code that will be released.
Justification:
19

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. www.oo/aaaaaooo /dtaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper will release the code of the proposed SAM2Flow model and the microvascular flow dataset. Part of the key details are included in the paper and the appendix. The complete documentation will be released with the code and dataset after the anonymous review session.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: The current manuscript does not include the details of human experiment and compensation due to the anonymous policy. The details of the human experiments, along with the IRB approval, will be disclosed after the double-blind review session.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
20

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The data acquisition experiment in this paper has been IRB approved. The IRB number will be disclosed in the Experiments 4 section after the double- blind review session.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non sta-standard components.
Please refer to our LLM policy (https https:://neuripi cc/Confereeces/ /LLM) for what should or should not be described.
21

---

Technical Appendices and Supplementary Material
A Flow Initialization
ROI Registration. For in vivo microscopic videos, subject movement and camera jitter are inevitable. Large movements pose a significant challenge to optical flow estimation. Therefore, during inference time, we introduce an ROI registration step to initialize the flow map with global offset value. In OBM videos, the background movements are mostly 2D translation, occasionally with slight rotation. To keep ROI registration most efficient, we ignore the rotation and apply the fastest strategy by calculating the center displacement of masks from the ROI decoder. This simple strategy helps remove large background movements without extensive computational overhead. The related ablation study is included at section D.2
Warm Start. As mentioned in section 3.3 the flow patterns within the video are temporally smooth. In addition to the motion memory, we use the flow map prediction from the last frame to initialize the current flow estimation, which has been proven to facilitate flow regression convergence in previous OF models [16. For diagram simplicity, this step is not shown in fig. |
B
Dataset
B.1
Data Collection
The dataset consists of 75 videos from 15 healthy volunteers. The 20x OBM system is applied to superficial capillaries inside the oral cavity. Videos were recorded at 200 FPS ensure less motion blur of fast-flowing blood cells. Each raw video is recorded for at least 90 seconds. Videos are manually reviewed and the most in-focus and stable segments are selected for further processing.
B.2 Dataset Preparation
Video Processing. To generate optical flow map ground truth, OBM videos first go through series of preprocessing steps in ImageJ. 1) Background corrections: a temporally averaged frame over the video is calculated first, then Gaussian blur with a kernel size of 100. The original frames are flattened by dividing by the background estimation. Finally, frames are normalized to [0, 255]. 2) Video stabilization: We use template matching to align slices in the stack with the normalized correlation coefficient.
Flow Estimation. 1) Flow mask: with the stabilized video frames, the flow mask is calculated from the standard deviation over time. The mask is then binarized into flow masks using Otsu thresholding. When there is more than one flow, multiple binary masks are generated. Output masks are further refined by human efforts. 2) Centerline detection: With the refined flow masks, the centerline of each flow is generated by flow mask skeletonization. Small branches and ends are removed so that only the main branches are used for flow estimation, 3) ST diagram: Based on the centerline, the intensity profile along the centerline over time is plotted across the whole frame. Parallel lines around centerlines are also used to generate ST diagrams. 4) Flow velocity estimation: from ST diagrams, the angle to tilted lines is calculated by Hough line transform using sliding time window of 235 frames.
Optical Flow Map Generations. 1) Temporal smoothing: to remove the local noises from the ST diagram, the estimated velocity over time is smoothed by a Butterworth low-pass filter; 2) Velocity profile: velocity profiles are generated along the centerline from the above estimation. We also generate velocity profiles from nearby lines within the flow that are parallel to the centerline. 3) Velocity interpolation: after getting the velocity profile along multiple lines within the flow, the velocity map is generated for each frame by cubic interpolation. 4) Flow direction: The general direction of the flow (upward/downward) is manually decided by scrolling through the video. The local direction is determined by the tangent angle of the centerline curve. 5) Optical flow map: The final optical flow ground truth is generated by multiplying the velocity map with the direction map.
The final dataset made up of 75 long videos paired with 306,800 ground truth flow maps, which significantly longer and larger than most of the existing public datasets, shown in table
22

---

B.3 Public Benchmark: Sintel and Spring
To evaluate SAM2Flow's ability to generalize beyond the microscopic domain, we benchmark it against baseline models on the public Sin- tel [5T] and Spring [52] datasets. As the pro- posed task focuses on ROI optical flow estima- tion, standard whole-frame leaderboard metrics are not informative enough about the model's performance. Therefore, we apply custom splits on the Sintel training set and Spring training set and introduce additional ROI ground truths. The Sintel frames
Table 4: Dataset Size Comparison
Dataset
Videos
Flowmaps
Avg video length
#
(fowmapx/video)
Sintel [1
23
1,041
45
Spring 32]
37
10,000
270
KITTI 2015
400
1,600
HlyingChairs 55
N/A
22.872
N/A
Microvascular
75
306,800
4,091
resized to 436 960 x 3, and the Spring frames are downsampled by 2x to 540 x 960 x 3. The ROI masks are generated via panoptic segmentation with the pretrained SAM2-L model, prompted by 16 x 16 grid points over the whole frame. For evaluation, only retain the segmented objects or regions with motions that are clearly distinct from the backgroumd//cmerra movement.
C
Additional Implementation Details
C.1 SAM2Flow Multi-stage Training
Stage 1: Finetune backbones on whole frame for domain adaptation. For backbones, we select the SEA-RAFT checkpoint pretrained on mix of five optical flow datasets, including Tartan (56) (5ntel[51],, FlyingChairs (55) KITTI[54] and HDIK We select the pretrained SAM2-L checkpoint as the context backbone. Since the backbone SAM2 and SEA-RAFT have no prior information of the OBM domain, we train the two backbones separately on their original tasks for domain adaptation. The SEA-RAFT backbone is initially trained with whole-frame flow ground truth for 50,000 steps, with le 4 learning rate. And the SAM2 adapter is trained with the frozen SAM2 backbone on segmentation masks of flows across videos for 50,000 steps, with a le - learning rate.
Module fusion. The trained SEA-RAFT and SAM2 encoder weights from stage are imported into the SAM2Flow model. We disable the memory module at this stage and train the SAM2Flow on image pairs. We randomly generate 3-6 positive and negative point prompts based on the mask ground truths. The training supervised on the masked optical flow ground truths. We used a smaller learning rate for the SAM2Flow weights at 5e - 5, except for the frozen SAM2 encoder.
Stage 2:
Stage 3: Training with memory. As the last stage of training. we enable context and motion memory modules and input an 8-frame sequence for flow prediction. The point prompts are randomly generated for the first 1-3 frames, with 3-6 points for each frame. During training, start the general learning rate at le - 4, freeze the weights of the SAM2 encoder, and use a learning rate scaling factor of 0.2 for the pre-trained SEA-RAFT feature encoder.
C.2 Comparative Experiments
We elect to use weights from the most comprehensive pre-training checkpoints from each optical flow model. 1) Two-frame baseline models: We take pretrained weights on the mix of five datasets (same as section C for SEA-RAFT, Sintel weights for RAFT, GMA, and FlowFormer++. The pretrained models are fine-tuned according to the training plans suggested by their respective open-source implementations [6, 17
Models were trained on a workstation containing 2 NVIDIA RTX A5500 GPUs Default configurations were used for each model. 2) Multi-frame baseline models: Two multi-frame baseline models are tested, VideoFlow and MemFlow [8 20 pretrained Sintel. VideoFlow was trained using the provided three-frame training methodology due to the GPU memory, The default configuration was used for finetuning on our dataset, with the image size adjusted to fit our dataset. We ensure that there are enough steps so that all the models converge on the validation metrics during training the Microvascular datasct and the Spring dataset.
44 46
23

---

D Additional Results
D.I Public Benchmarking Results
The visual comparison on representative video segment from the Spring test split shown fig. The qualitative error maps verify the quantitative gain. The context- guiitd foreground detection enables SAM22Flww to focus on tracking complex ROI motion with sharp flow edges. In contrast, MemFlow and SEA-RAFT exhibit poor performance and boundary leakage in the highlighted regions.
Impura
GT OptioalFor Flow
MorFloww Ennt map
SEA-RAFT
Ernor
SAM2FIOW Ernor map
18g
Figure Qualitative comparison on the Spring public benchmark. Top: five consecutive animation frames from test video. Second row: foreground GT optical flow fields. Rows 3 - 5: Absolute foreground endpoint-error heatmaps (0 15 px. blue red) overlay on the frames, for MemFlow, SEA-RAFT, and the proposed SAM2Flow. All the models perform well in estimating the optical flow of the object on the right side of the frame with little motion. Yellow arrows and dashed boxes highlight regions where the baselines yield higher errors on fast, complex ROI motions. Meanwhile, SAM2Flow produces robust and accurate flow estimation while preserving motion boundaries across the sequence.
D.2 Ablation Study
Table 5: Ablation study for flow initialization.
Method
Dataset
FEPE,
5pxt
10pxt
15pxt
No
unstable
20.50
0.37
0.52
0.66
unstable
12.32
0.54
0.65
No
stable
0.66
0.86
Figure FEPE vs. number of memories.
Flow initialization for unstable videos. We proposed ROI registration as flow initialization (FI) inference time, table shows the ablation study results on unstable videos without retraining the model. ROI registration as flow initialization effectively improves the estimation accuracy the
24

---

unstable videos, with 40% improvement in FEPE without any additional model adaptation on the background movements.
Number of Memory vs. FEPE. We test the effect of numbers memory frames on the foreground EPE. The results are shown in fig. We only test for the combined context and motion memory; from O to 10 frames. As the plot shown, FEPE first decreases and then converges around 7. SAM2Flow memory module performance exhibits marginal gains by enlarging memory banks beyond frames, with linear compute memory growth. Therefore, we choose 7 frames as memory bank limit for the final SAM2Flow configuration.
25


Reviews:
[{"review_id": "t6XyZ1LrEG", "reviewer": "Reviewer_33dS", "summary": "For microvascular blood flow, this work proposes an optical flow estimation method based on SAM2 to confirm the RoI (Region of Interest) area according to user prompts, and introduces a new microvascular blood flow dataset. The method achieves leading results on both standard optical flow benchmarks and this dataset.", "questions": "Please refer to the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:  \n\n1. The method proposes an effective optical flow estimation approach for the microvascular blood flow task.  \n\n2. It provides a novel microvascular blood flow dataset.  \n\n3. The paper features good visualizations for demonstration.  \n\n\nWeaknesses:  \n1. Lack of comparison with more state-of-the-art optical flow estimation methods, such as the SAM-based method SAMFlow [1] and the multi-frame method StreamFlow [2].\n\n2. Absence of results on more standard optical flow benchmarks (e.g., Sintel). Although the task focuses on microvascular blood flow, since there was no large-scale dataset for this task previously, experiments on more public benchmarks (whether fine-tuned results or 0-shot results) could more effectively validate the method’s effectiveness. After all, the method itself is not specifically designed for microvascular tasks, and the same architecture can be directly applied to optical flow estimation in other scenarios. More comparisons would better demonstrate its validity.\n\n3. The experiments on the Spring dataset should present more metrics such as Fl.\n\n[1] SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model\n\n[2] StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences", "was_revised": true, "final_justification": "The author has addressed most of my concerns in the rebuttal. Overall, I believe the significance of this work is worth accepting.", "rebuttal": "We thank the reviewer for the constructive review. We appreciate the recognition of the effectiveness of our method, the value of our novel dataset, and the strength of our visual demonstrations. We agree with the reviewer that the manuscript would be improved by the suggested additional experiments. In this rebuttal, we provide more details of some choices we made in the original study, and we are happy to share some new results of the suggested additional experiments.\n## 1. Additional Comparative Models: SAMFlow and StreamFlow\n* **SAMFlow**: SAMFlow is a two-frame model that introduces the additional SAM encoder onto the FlowFormer backbone for enhanced estimation integrity per frame, with the challenge of partial occlusion. In our comparative study, FlowFormer++, a variant of FlowFormer, is the slowest model (**134 ms/frame vs/ 48.78 ms/frame of SAM2Flow**) in **Table 1**, while SAMFlow’s speed is similar to FlowFormer according to the original paper [1]. Although both SAMFlow and SAM2Flow leverage the foundation segmentation model SAM, our model design is tailored for efficient and temporally stable flow estimation. In contrast, the design motivation and excessive computational burden make SAMFlow impractical for our task on long, high-framerate microscopy videos (~200FPS, often >1 minute duration).\n* **StreamFlow**: We thank the reviewer for the suggestion to include StreamFlow in our comparative study. StreamFlow is an efficient multi-frame model that avoids recursive flow estimation with the streamlined in-batch multi-frame pipeline. Starting from the checkpoints pre-trained on “C+T+S+K+H” (same as the SEA-RAFT checkpoint, **supplement C.1**), we finetuned StreamFlow on our microvascular dataset, and evaluated it on the test split as follows:\n| Model | EPE_all | 1px | 3px | 5px | EPE_fg | 5px | 10px | 15px | Speed |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| SEA-RAFT | $\\underline{1.28}$ (1.03) | **0.88** | $\\underline{0.92}$ | $\\underline{0.94}$ | $\\underline{6.60}$ ($\\underline{5.47}$) | **0.69** | **0.86** | $\\underline{0.91}$ | **21.14** |\n| MemFlow$^{(MF)}$ | 1.79 (1.40) | **0.88** | 0.91 | 0.93 | 12.47 (10.23) | 0.58 | 0.74 | 0.80 | $\\underline{43.98}$ |\n| StreamFlow$^{(MF)}$ | 1.43 ($\\underline{1.02}$) | **0.88** | 0.90 | 0.93 | 10.13 (8.36) | 0.49 | 0.74 | 0.84 | 79.92 |\n| SAM2Flow$^{(MF)}$ | **1.14 (0.92)** | **0.88** | **0.93** | **0.96** | **5.84 (4.86)** | $\\underline{0.66}$ | **0.86** | **0.93** | 48.78 |\n* $^{(MF)}$ denotes multi-frame models.\n\nThe results indicate that StreamFlow outperforms MemFlow and achieves competitive accuracy in microvascular flow estimation. However, SAM2Flow remains superior in both accuracy (**FEPE=5.84 vs. 10.13**) and efficiency (**48.78 ms/frame vs. 79.92 ms/frame**). These results further highlight SAM2Flow’s unique ability to achieve high accuracy with impressive speed as a multi-frame model, which is critical for long, high-framerate microvascular videos. ***Additional results on StreamFlow on public benchmarks are presented in the following sections.***\n\n    [1] Zhou, Shili, et al. \"Samflow: Eliminating any fragmentation in optical flow with segment anything model.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 7. 2024.\n## 2.  Additional Metrics for Spring Benchmark\nTo provide a more comprehensive view of each model’s performance, we follow the previous works on the Spring dataset and add the following metric, with the standard definition and thresholds from the original Spring dataset, to expand the current **Table 2**:\n* **Fl-fg**: flow outlier (>3pxs and > 5% of GT flow magnitude) rate for the foreground regions;\n* **Fl-epe**: the average EPE of the flow outliers from the foreground regions;\n* **WAUC-fg**: weighted AUC for estimations on foreground pixels.\n| Model | FEPE | 1px (%) | Fl-fg (%) | Fl-epe | WAUC-fg |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| RAFT | 2.25 (6.97) | 74.25 | 9.87 | 7.98 (8.98) | 74.76 |\n| GMA | 2.17 (6.30) | 76.21 | 9.08 | $\\underline{7.67}$ ($\\underline{7.72}$) | 79.67 |\n| SEA-RAFT | $\\underline{1.45}$ ($\\underline{5.61}$) | 86.32 | **5.18** | 8.18 (10.85) | $\\underline{83.85}$ |\n| MemFlow$^{(MF)}$ | 1.56(7.27) | $\\underline{86.49}$ | 7.24 | 8.82 (12.72) | 83.54 |\n| StreamFlow$^{(MF)}$ | 1.54 (7.23) | 85.16 | 5.82 | 7.74 (10.16) | 82.51 |\n| SAM2Flow$^{(MF)}$ | **1.23** (**4.21**) | **87.13** | $\\underline{5.29}$ | **7.30** (**5.29**) | **84.57** |\n\nThese additional metrics provide better insights into model performance regarding estimation outliers. For the estimation regarding the ROIs, SAM2Flow achieves the best fine-grained accuracy (highest 1px, **87.13%**) and most robust estimation, with fewer flow outliers (**Fl-fg=5.29%**), and lowest **Fl-epe (7.30)**. SEA-RAFT performs slightly better in **Fl-fg (5.18%)** but underperforms on **Fl-epe** and **WAUC-fg**, suggesting less stable estimation across the ROIs with some extreme outliers.\n\n## 3. Additional Benchmark: Sintel\nTo study the generalizability of SAM2Flow on non-medical scenarios, we have reported results on the Spring dataset. Both the Sintel and Spring datasets consist of animation sequences. We prioritized Spring because it contains more (**37 vs. 23**) and longer (**270 vs. 45 frames/video**) videos compared to Sintel (as shown in the **Supplement table 4**). Intuitively, such long sequences enable better demonstration of the robust optical flow estimation empowered by the proposed dual-memory module of SAM2Flow. We agree that demonstrating the generality of SAM2Flow on additional benchmarks like Sintel (Clean & Final versions) would strengthen the impact. Similar to Spring, we prepare the dataset with additional annotations of foregrounds. The dataset details will be added to the **Supplements** as **B.4**. We split the total of 23 scenes into training (14), validation (3), and test (6). The training and validation combine both **Clean** and **Final** videos. For evaluation, we report the performance on the Clean and Final videos separately.\n\nFor fair comparison, we start the experiments with the checkpoint pre-trained on the two-frame **Flying-Chair** and **Fly-Things** datasets for all comparative models and SAM2Flow backbones. Next, we train all the models with the prepared **Spring dataset**, which is especially helpful for multi-frame models to incorporate long-term temporal information into learning. Lastly, all the models are finetuned on the Sintel training split. For evaluation, we report the **foreground end-point error (FEPE)** on both clean and final test splits following previous works. The preliminary results are shown in the following table:\n\n| Model | FEPE-Clean | FEPE-Final |\n|:---:|:---:|:---:|\n| RAFT | 5.21 (9.30)  | 5.47 (10.12)  |\n| SEA-RAFT | $\\underline{3.26}$ (7.54) | $\\underline{4.08}$ (8.94) |\n| StreamFlow$^{(MF)}$ | 4.06 (5.37) | 4.43 (5.66) |\n| SAM2Flow$^{(MF)}$ | **3.17** (6.89) | **3.39** (5.97) |\n\nAs for now, we have finished the training and evaluation of the best-performing models, based on the results from the Microvascular and Spring datasets. ***We are currently training more models, including GMA and MemFlow, to complete this table. The complete results will be updated in the final draft.*** For the current results, SAM2Flow consistently demonstrates superior generalization and robustness compared to all baselines on Sintel, particularly on the challenging Final video, where more textures are added to the scene. **These results, combined with the strong performance on the Spring dataset, highlight that SAM2Flow achieves the state-of-the-art ROI-centric optical flow estimation beyond the microscopic domain.**"}, {"review_id": "jytBdZ1vRC", "reviewer": "Reviewer_No5r", "summary": "The paper proposes a noel approach for optic flow estimation. Uniquely to the work, it focuses primarily on in vivo microcirculation analysis. Given a collection of images, the paper allows the user to select which area needs to have its optic flow computed via a prompting mechanism similar to the one in SAM. The rest of the work follows a relatively standard flow estimation pipeline, with the addition of a memory based mechanism for improving results on long video sequences.", "questions": "Overall I do not have significant suggestions for improvement, as I am very much a non expert in the specific application domain. From a pure CV optic flow estimation paper, the paper is not overwhelmingly novel, using concepts from SAM and relatively standard optic flow formulations. The application and dataset however appears to bring sufficient novelty to warrant publication.\n\nIf I had one request, it would be to anchor the design of the approach a bit more into the specific application. The main mention of multivascular flow estimation within the method appears to be on line 227, and this could be expanded.", "limitations": "Yes, these are discussed.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Positive:\n- The paper is fairly well written and easy to understand.\n- I am not an expert in the field, but according to the authors this is one of the first approaches to produce optic flow for this type application.\n- The authors also introduce a novel dataset specific to microcirculation analysis.\n- I very much appreciate that the authors included an application specific result section, commenting on the usefulness of the estimated flow. \n- The paper also appears to produce good results for a more standard (i.e. non-medical) dataset.\n\nNegative:\n- The method section of the paper reads somewhat disconnected from the introduction and experimental section. The latter sections very much link to the target application domain, whereas the former seems quite generic i.e. not specific to microcirculation flow estimation.\n- I am not an expert in the field so do not know if the methods evaluated are adequate for the specific task.", "was_revised": true, "final_justification": "Following the rebuttal and the other reviews, I maintain my overall positive view of the paper and believe it should be accepted.", "rebuttal": "We are grateful for the reviewer's appreciation of our novel application, the contribution of our new dataset, and the model’s generalizability. And we really appreciate your suggestions to enhance the motivations of model design. Below, we provide more details for your comments.\n## 1. Anchoring the Method Section More Firmly in the Application Context\nDuring the design of the SAM2Flow framework, we tailor each component to address the clinical challenges of in vivo microvascular flow analysis. As suggested by the reviewer, ***we will revise the Method section*** to clarify our motivations further and explicitly link each module design to domain-specific challenges of microvascular flow estimation in the following ways:\n* **Prompt-conditioned Context (Sec. 3.2)**: prompt inputs enable user-defined vessel selection in cases where complex overlapping vessel structures or irrelevant background motion (e.g., heartbeat, tongue or jaw movement) can mislead global flow estimation. This is critical in OBM videos, where select vessels are more diagnostically relevant to clinicians.\n* **ROI-guided correlation lookup (Sec 3.2)**: As flows in OBM videos are highly localized, occupying only a small portion of the entire field of view, ROI-guided correlation lookup limits the search within the target vessels to increase computational efficiency and suppress the background noise in flow estimation. This enables practical application of SAM2Flow to match the high framerate (200 Hz) of OBM flow videos.\n* **Dual memory module (Sec. 3.3)**: Motivated by the need for temporal stability across long OBM videos (e.g., 12,000 frames for 60 seconds), SAM2Flow overcomes the limitation of conventional two-frame models by incorporating dual memory modules. The context memory helps to keep track of the identities of target vessels, while the motion memory ensures the long-term estimation smoothness for constantly pulsing flows, which is essential for downstream physiological analyses, such as flow waveform extraction as discussed in **Sec. 5.5**.\n\nAdditionally, ***we will enhance the Introduction by clarifying the motivation of the model designs to tackle a real-world medical imaging challenge***.\n\n## 2. SAM2Flow’s Novelty as a Generic Optical Flow Estimation Application\nWe also note that SAM2Flow has demonstrated its novelty and superior performance for general applications (public datasets: **Spring in Table 2 & Sintel in Q3 of Rebuttal to Reviewer 33dS**). SAM2Flow offers unique features that are not present in previous optical flow models, including **flow stability over longer videos (Fig. 2)** and **User interaction** that enables Object/ROI-specific estimation. This would be a valuable contribution to various downstream tasks, such as motion detection and tracking, as well as object-specific video editing."}, {"review_id": "KB4i7ad1zO", "reviewer": "Reviewer_5zif", "summary": "This paper introduces SAM2Flow, an interactive optical flow estimation model designed for analyzing long in vivo microvascular flow videos captured by Oblique Back-illumination Microscopy (OBM). Inspired by the Segment Anything Model (SAM2), the model enables users to specify regions of interest (ROIs) through prompts, focusing flow estimation on targeted vascular segments. A key innovation is the dual memory attention mechanism, combining motion and context memory, which ensures stable and efficient flow estimation over extended video sequences. SAM2Flow integrates a feature encoder from SEA-RAFT and a context encoder from SAM2, adapted via an adapter to the OBM domain, and employs ROI-guided correlation lookup to optimize computational efficiency. The model is designed to address the limitations of existing methods in handling long sequences and complex vascular structures. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "[Questions]\n\nOverall, the method proposed in this paper is novel, but there are still some confusing minor issues:\n\n1. How reliable is the ROI detection in the proposed method?\n2. When ROI detection is not fully reliable, how does it affect the estimation accuracy of the proposed method?\n3. For the Spring dataset, how are foreground regions defined and generated? Could you provide a more detailed explanation?\n4. In the studied scenarios, vascular overlapping in images seems to exist. How does the proposed method perform in such scenarios? Can the proposed strategies handle such scenarios effectively?\n5. Open discussion: The paper mentions multiple microvascular imaging techniques, but it mainly focuses on OBM. Does the proposed method have generalizability to other microvascular imaging techniques? If its generalizability is limited, what are the reasons?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "[Strengths]\n\n1. This paper proposes a specialized dataset tailored for optical flow estimation in in vivo microvascular studies, comprising 75 long-form videos and 306,800 paired ground-truth flow. The dataset can support downstream research in this area.\n2. This paper Introduces interactive ROI-guided flow estimation via user prompts, enhancing clinical efficiency by focusing on relevant vascular regions and reducing unnecessary calculations.\n3. The proposed methods employs a dual memory module with motion and context memories, improving long-sequence flow estimation stability for sustained microvascular dynamics.\n\n\n[Weaknesses]\n\nThe primary limitation of the proposed method is its dependency on the context branch for ROI (region of interest) detection to guide flow estimation. In complex scenarios with intersecting vascular structures or turbulent blood flow, failed or incomplete ROI detection can directly degrade the accuracy of flow estimation.", "was_revised": true, "final_justification": "Thank you for the detailed responses from the authors. All my questions have been addressed, and I have decided to upgrade my rating to 5. Additionally, there are currently some works on optical flow based on different sensors [1-4]. Citing these works may help make the related work section more comprehensive.\n\nReference  \n[1] Z. Ding, et al., Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation. AAAI 2022.  \n[2] A. Luo, et al., Efficient Meshflow and Optical Flow Estimation from Event Cameras. CVPR 2024.  \n[3] R. Zhao, et al., Learning Optical Flow From Continuous Spike Streams. NeurIPS 2022.  \n[4] L. Xia, et al., Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera. NeurIPS 2023.", "rebuttal": "We sincerely thank the reviewer for the insightful feedback. We are pleased that you find our dataset, interactive ROI-guided flow, and dual-memory mechanism innovative and valuable for clinical and research applications. We appreciate your suggestions to clarify the model’s dependency on ROI detection.\n## 1. How reliable is the ROI detection of SAM2Flow?\nOur ROI detection in the context branch is built upon the SAM2 encoder, which is known for its strong semantic generalization, and further fine-tuned for the microscopic flow domain via a lightweight adapter. Two features of SAM2, user-prompt and context memory, ensure superior ROI segmentation compared to other backbones, especially in terms of temporal stability. As shown in our **ablation study (Table 3)**, the context memory significantly improves estimation accuracy (**FEPE dropped from 9.75 to 6.32**), demonstrating reliable ROI detection and context encoding, which leads to more robust motion estimation over long sequences.\n## 2. How does incomplete ROI detection affect estimation accuracy?\nROI detection from the context branch is used to specify the target motion region and guide the correlation lookup for flow regression. ROI guidance improves model efficiency (as shown in **Table 3**) and suppresses the irrelevant background flow. We acknowledge that failed ROI detection can impact performance (**Sec. 5.6**) in challenging cases, such as blurry or some intersecting vessels in microvascular videos and low contrast or heavy occlusion in Spring videos. However, our model design mitigates the effect on flow estimation from different types of ROI errors:\n* **Over segmentation.** When an over-segmented ROI guides correlation lookup, some background pixels are also indexed and fed to subsequent steps. However, this doesn’t degrade final flow estimations, as the trained motion encoder and RNN further delineate the motion boundary through iterative flow updates. Full-frame flow regression could be considered as an extreme case of over-segmentation.\n* **Transient frames drop or incomplete ROI.** Motion memory module provides redundancy against occasional failures by maintaining estimation consistency with prior memories (**Sec. 3.3**). This feature stabilizes predictions when one frame has incomplete or no ROI detection. This is demonstrated by the ablation study of the motion memory module, where **FEPE dropped from 6.32 to 5.46 (Table 3)**. SAM2Flow incorporates an additional trick, known as warm start (**Supplement A**), to further stabilize the flow estimation in cases of occasional ROI instability.\n* **Missed/incomplete ROI over an extended period.** Failed ROI detection over more consecutive frames would result in failed optical flow estimation, since the model fails to identify any target. However, as an interactive pipeline, SAM2Flow offers the flexibility to add/correct user prompts for these challenging frames during inference. We recommend using multiple positive points spaced over the ROI for larger targets to mitigate prompt ambiguity.\n\n**Table 1 & 2**, along with **Figure 2**, show that SAM2Flow achieves the best foreground flow estimation accuracy and temporal stability, suggesting failure resilience even in challenging scenarios. ***We will clarify this failure-resilience design more explicitly in Sec. 5.6 (Limitations). And for the final version, we plan to include additional visual examples of flow estimation guided by the above three kinds of imperfect ROI detections in the supplement.***\n## 3. How are foreground regions defined in the Spring dataset?\nThank you for requesting more clarity. On the Spring dataset, foreground regions are defined as objects exhibiting significant relative motion to the background/camera movements. In each Spring video, the foreground ROIs are selected manually by reviewing the video and the paired optical flow ground truths.\n\nFor each Spring video, we first manually select key frames for object detection. **Pre-trained SAM2-L** is applied for panoptic segmentation, prompted with a 16×16 grid of points across each selected frame. Among the generated masks, we manually remove tiny objects (e.g., small stones and water drops) and background (e.g., sky, trees). The retained masks are then propagated across the video by SAM2-L to generate a complete segmentation for each object in all frames. Finally, the generated ROIs are reviewed and manually refined to ensure segmentation accuracy and temporal consistency. In the training splits, we utilize all the segmented objects as ROIs to achieve maximum diversity of context and motion. For validation/test sets, we only test the models in 1-2 ROIs with the most prominent motions during evaluation. ***We will add these details of Spring dataset preparation in the revised Supplement B.3.***\n## 4. How does SAM2Flow perform in complex scenarios, such as vascular overlapping or turbulent flow?\n**Structure Complexity.** Our manuscript specifically presents examples that include challenging crossing flow and vessel overlap (**Fig. 2**). These cases are common in clinical microvascular datasets. SAM2Flow is the only model that successfully separates the crossing flows, whereas SEA-RAFT and MemFlow incorrectly merge them. Our SAM2Flow model acheives this performance by utilizing:\n* **User prompts** guide the detection of separate ROIs (**Sec. 3.2**), preventing merging different flows into one.\n* **ROI-specific correlation lookup** allows separate flow regression for each vessel.\n* Propagation of **context and motion memories** maintains the identity of each vessel throughout the video.\n\n**Motion complexity.** Our dataset includes numerous cases of capillary loops that are abundant in the superficial oral mucosa. These vessels present drastic flow direction changes within short vessel segments (as shown in **Fig. 2**), thereby increasing flow complexity compared to long, flat laminar flows. While blood flow in most healthy blood vessels is approximately laminar, flow becomes turbulent at vessel branch points, vessel narrowing,  obstructions (such as in patients with sickle cell disease), or under certain conditions like hyperemia. An efficient flow estimation method like SAM2Flow could enable quantitative in vivo studies of microvascular flows under these clinically relevant conditions.\n## 5. Is the method generalizable to other microvascular imaging techniques beyond OBM?\nWe appreciate this critical discussion. OBM provides rich **phase contrast** at the single-cell level, which enables dense and continuous optical flow estimation. While our current dataset is based on OBM, the design of the SAM2Flow framework is **modality-agnostic**. In principle, it could be applied to other microvascular imaging techniques with high **spatial resolution** to identify blood flows and in-flow features, and sufficient **temporal resolution** to capture fast cell motions. For instance, high-speed fluorescent flow imaging [1] and other phase-contrast flow imaging, such as AOSLO [2], obtain very similar data. Several additional techniques, including Side-stream Dark Field (SDF) and Orthogonal Polarization Spectral (OPS), acquire blood flow data; however, these techniques rely on **absorption contrast** and produce temporally and spatially **sparse landmarks** for flow tracking (white blood cells and plasma gaps). As discussed in **Sec. 2.1**, it is particularly challenging to quantify these flows, particularly for conventional two-frame models, because the landmarks are not visible in every frame. The dual-memory module in SAM2Flow may mitigate such a limitation with enhanced spatial and temporal robustness.  \n\nGenerally, we believe SAM2Flow can generalize to other applicable modalities and various imaging sites, including the eyelid, nailfold, and retina, with available **annotated ground truths** and appropriate domain adaptation. We are actively exploring broader applications of SAM2Flow as future work. Moreover, we hope that by introducing the first optical flow neural network for blood flow analysis and sharing our dataset publicly, other groups will begin sharing their datasets and contributing to generalizable analysis models.\n \n    [1] Meng, Guanghan, et al. \"Ultrafast two-photon fluorescence imaging of cerebral blood circulation in the mouse brain in vivo.\" Proceedings of the National Academy of Sciences 119.23 (2022): e2117346119.\n    [2] Joseph, Aby, et al. \"Label-free imaging of immune cell dynamics in the living retina using adaptive optics.\" Elife 9 (2020): e60547."}, {"review_id": "zNpPwr9RGT", "reviewer": "Reviewer_D1JP", "summary": "This paper proposed a SAM-enhanced model to estimate optical flows of blood in microcirculation analysis. The key contributions are the dataset and dual-memory modules (motion and context memories). On one in-house dataset and another public dataset, the performance is good to estimate the optical flows.", "questions": "1. Please include the complexity comparisons in the tables.\n\n2. How is the performance when using bboxes as prompts? Here, the point prompts are a bit time-consuming.\n\n3. Discussion of failure cases", "limitations": "There is a limitation discussion about the imperfect prompts in the model. No failure cases.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The studied problem is important, and sharing the dataset is a good contribution.\n\n2. Extensive experiments have shown that the model achieves good performance.\n\n\nWeaknesses:\n\n1. The NO-ROI setting can achieve better performance in Table 3. More justifications are needed.\n\n2. [50] can achieve comparable and even better performance. The authors are suggested to clearly discuss its superiority and contributions.\n\n3. More prompts can be explored to validate its generalizability.\n\n4. The authors are suggested to analyze failure cases, especially for some subtle objects which cannot be segmented well by SAM2.", "was_revised": true, "final_justification": "The authors have addressed my concerns. I keep my score as Borderline accept.", "rebuttal": "We would like to thank the reviewer for recognizing the significance of the in-vivo blood flow estimation task and the contribution of our microvascular dataset, as well as the strong performance of the proposed SAM2Flow model. Below, we provide detailed responses to your suggestions and comments.\n \n## 1. Justification of “No-ROI” Performance in Table 3\n**The \"No ROI\" setting in Table 3** refers to a variant where ROI-guided correlation lookup is disabled (**Line 328**), meaning all pixels are considered during flow regression (i.e., full-frame regression). For the complete SAM2Flow framework, we introduce the ROI-guided lookup to guide correlation search within the detected ROI and suppress the computation on the irrelevant background.  As shown in **Table 3**, ROI guidance enhances model efficiency (**inference time improves from 55.25 ms to 48.78 ms**) at the cost of some raw accuracy (**FEPE = 5.46 vs. 5.12**). This trade-off is mainly from imperfect ROI segmentation, particularly false negatives, in some challenging scenarios.\n\nFor the high-frame-rate microvasculation videos (12000 frames for one-minute video), we choose to prioritize the model efficiency, given the comparable estimation accuracy. ***We will clarify this trade-off more explicitly in Sec. 5.3 for the final version.*** And we also include a detailed discussion about the flow estimation under imperfect ROI detection in the next section.\n \n## 2. Failure Case Analysis\nWe agree with the suggestion that an expanded failure case analysis could provide a more comprehensive overview of the model performance, especially under challenging scenarios. As mentioned above and in the **limitations (Sec. 5.6)**, the flow estimation accuracy may be affected by imperfect ROI detections. Here, we provide a detailed analysis of how SAM2Flow copes with different types of ROI errors and the resulting flow estimations:\n* **Over segmentation.** When an over-segmented ROI guides correlation lookup, some background pixels are also indexed and fed to subsequent steps. However, this doesn’t degrade final flow estimations, as the trained motion encoder and RNN further delineate the motion boundary through iterative flow updates. Full-frame flow regression could be considered as an extreme case of over-segmentation.\n* **Transient frames drop or incomplete ROI.** Motion memory provides redundancy against occasional failures by retrieving memories from prior frames (**Sec. 3.3**) that stabilize predictions even if one frame has incomplete or no ROI detection. This is demonstrated by the ablation study of the motion memory module, where **FEPE dropped from 6.32 to 5.46 (Table 3)**. SAM2Flow incorporates an additional trick, known as warm start (**Supplement A**), to further stabilize the flow estimation in cases of occasional ROI instability.\n* **Missed/incomplete ROI over an extended period.** Failed ROI detection over more consecutive frames would result in failed optical flow estimation, since the model fails to identify any target. However, as an interactive pipeline, SAM2Flow offers the flexibility to add/correct user prompts for these challenging frames during inference. We recommend using multiple positive points spaced over the ROI for larger targets to mitigate prompt ambiguity.\n\n***For the final version, we plan to include the above details and explicit visualization of failure cases for more intuitive discussion of limitations (Sec. 5.6).*** These additions will clarify where SAM2Flow still struggles and motivate future improvements in segmentation under ambiguous conditions.\n## 3. Prompt Modality: Bounding Boxes vs Point Prompts\nFor the microvascular videos, we chose point prompts to guide the flow localization in the context branch of SAM2Flow. Based on our observation, positive/negative point prompts can most effectively define any target vessels from single-vessel to complex multi-vessel scenes. Whereas bounding box prompts for different target ROIs would be ambiguous due to significant overlaps when multiple flows are close together and cross each other. For example, see **image $I_t$ in Fig 1**, which has two different vessels close to each other, and **Fig. 2**, which has two different entangled vessels.  Moreover, SAM2Flow identifies ROIs from temporally sparse prompts. Empirically, we only need prompts up to 5 frames for a one-minute-long (12,000-frame) microvascular OBM video. Therefore, the difference in time cost and manual labor is trivial.\n\nHowever, other forms of prompts could be more informative than point prompts when applying SAM2Flow to different videos. Thanks to the flexibility of SAM2's prompt encoder, **SAM2Flow can easily support various prompt inputs, including points, bounding boxes, and masks.**\n\n## 4. Comparison with SEA-RAFT [50]\nSEA-RAFT is a strong two-frame model and serves as our motion encoder backbone. In our experiments, SEA-RAFT achieved competitive performance. However, we want to note that SAM2Flow outperforms SEA-RAFT on both datasets (**FEPE = 5.84 vs. 6.60 on the microvascular dataset, FEPE = 1.23 vs. 1.45 on the Spring dataset**). We have also conducted an additional comparative study on the **Sintel** benchmark (Please find details at **Q3 in Rebuttal to Reviewer 33dS**), where SAM2Flow achieves better foreground estimation accuracy in both “Clean” and “Final” videos (**FEPE-Clean=3.17 vs. 3.26; FEPE-Final=3.39 vs. 4.08**).\n| Model | FEPE in Sintel-Clean | FEPE in Sintel-Final |\n|:---:|:---:|:---:|\n| SEA-RAFT | 3.26 (7.54) | 4.08 (8.94) |\n| SAM2Flow | **3.17 (6.89)** | **3.39 (5.97)** |\n\nAs a two-frame model, SEA-RAFT achieves faster inference (**21.14 ms/frame vs. 48.78 ms/frame** on 512x512 inputs). However, SAM2Flow, inherently a multi-frame model, achieves improvements in practical application and supports unique features: \n* Interactive ROI prompting;\n* Long-term temporal consistency via dual memory;\n* Robustness in multi-object and overlapping flow scenarios.\n\nThese design features are not present in SEA-RAFT and are essential for the physiological analysis of long clinical videos with complex vascular flows, as shown in **Sec. 5.5** and **Fig. 2**.\n## 5. Complexity metrics\nThank you for pointing this out. The current **Table 1** reports **runtime (mspf)** as a measurement of model efficiency. ***We will include additional complexity columns (“number of FLOPs” and “memory use per batch”) in the updated Table 1 for the final draft.***"}]

Author rebuttal:

Meta-review: All reviewers vote for accepting the paper. They state that it provides an "effective optical flow estimation approach for the microvascular blood flow task" with a "novel microvascular blood flow dataset" (33dS), the "paper is fairly well written and easy to understand" (No5r), "The studied problem is important", and "sharing the dataset is a good contribution" (D1JP). Therefore the AC reccomends acceptance of the paper.
Decision: Accept (poster)