TractoTransformer: Diffusion MRI Streamline Tractography using CNN and Transformer Networks
Authors: Itzik Waizman
Yakov Gusakov
Itay Benou
Tammy Riklin Raviv
Venue: NeurIPS 2025 poster ()
Keywords: Neuroimaging
Diffusion MRI
White Matter Tractography
Sequential Modeling
Transformers
Convolutional Neural Networks
Primary area: applications

Abstract: White matter tractography is an advanced neuroimaging technique that reconstructs the 3D white matter pathways of the brain from diffusion MRI data. It can be framed as a pathfinding problem aiming to infer neural fiber trajectories from noisy and ambiguous measurements, facing challenges such as crossing, merging, and fanning white-matter configurations.
In this paper, we propose a novel tractography method that leverages Transformers to model the sequential nature of white matter streamlines, enabling the prediction of fiber directions by integrating both the trajectory context and current diffusion MRI measurements. To incorporate spatial information, we utilize CNNs that extract microstructural features from local neighborhoods around each voxel. By combining these complementary sources of information, our approach improves the precision and completeness of neural pathway mapping compared to traditional tractography models. We evaluate our method with the Tractometer toolkit, achieving competitive performance against state-of-the-art approaches, and present qualitative results on the TractoInferno dataset, demonstrating strong generalization to real-world data. Our code is publicly available at https://github.com/ItzikWaizman/TractoTransformer.

Diffusion MRI Streamline Tractography using CNN and Transformer Networks
Itzik Waizman
Yakov Gusakov!
Itay Benou
Tammy Riklin Raviv
The School of Electrical and Computer Engineering, Ben-Gurion University of the Negev (itzikwei, gusakovy, benoui]@post bgu.bc.......................... ac.u.ac.il, .l, ertammy@bgu.ac. ac. il
Abstract
White matter tractography is an advanced neuroimaging technique that reconstructs the 3D white matter pathways of the brain from diffusion MRI data. It can be framed as a pathfinding problem aiming to infer neural fiber trajectories from noisy and ambiguous measurements, facing challenges such as crossing, merging, and fanning white-matter configurations. In this paper, we propose a novel tractography method that leverages Transformers model the sequential nature of white matter streamlines, enabling the prediction of fiber directions by integrating both the trajectory context and current diffusion MRI measurements. To incorporate spatial information, we utilize CNNs that extract microstructural features from local neighborhoods around each voxel. By combining these complementary sources of information, our approach improves the precision and completeness of neural pathway mapping compared to traditional tractography models. We evaluate our method with the Tractometer toolkit, achieving competitive performance against state-of-the-a art approaches, and present qualitative results on the TractoInferno dataset, demonstrating strong generalization to real-world data. Our code is publicly ormer
available https: //gi giighu.tht... com Itwikwaiiiiaaaa http:///tw.......a.aaaa//aaaaar/
Introduction
Tractography is key technique for analyzing diffusiin--eeighted imaging (DWI) data, aiming to reconstruct the complex 3D trajectories of white matter fibers-a fundamental step in understanding brain connectivity, development and neurological disorders [1] 2). It exploits the principle that water molecules preferentially diffuse along axonal fibers, enabling the indirect estimation fiber orientations from diffusion- tiffuseion-eeightedd measurements acquired via diffusion magnetic resonance imaging (dMRI). Conceptually, tractography can be framed as a pathfinding problem: inferring plausible neural fiber pathways from noisy and ambiguous data while addressing challenges such as crossing, merging, and fanning fiber bundles. Traditional tractography methods rely on mathematical models that fit estimated fiber orientation distribution function (fODF) the measured DWI each voxel, such as diffusion tensor imaging (DTT) 31 multi-tensor models [4), ball-and-sticks (5], Q-ball imaging (QBI) (6), and spherical deconvolution [7]. These orientation functions serve as local directional priors that guide the reconstruction of white matter pathways using deterministic,
probabilistic, combinatorial tracking strategies.
While classical tractography methods have significantly advanced our understanding of white matter architecture, they remain constrained by model- model-based based assumptions-ose such simplified representa- tions of diffusion and voxel-wise independence [8]. These limitations have spurred the development of data-driven alternatives that leam directly from dMRI data (9. Machine learning approaches offer greater flexibility in capturing complex white matter configurations, including fiber crossings and branchings, without imposing explicit assumptions about tissue properties or the dMRI signal.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Although recent rearning-based strategies have shown encouraging results, many still fall short of fully exploiting the underlying structure of the diffusion measurements, as they predict each voxel's orientation in isolation disregarding either spatial dependencies [10]
11 12 13 [4] or the sequential
structure of white matter tracts [15 16 17 18 19 20 Consequently, fiber orientation predictions
tend to degrade in anatomically intricate or ambiguous regions.
In this work, we effectively leverage both the spatial and contextual information inherent in the data by proposing a spatio-sequential formulation of the fODF estimation task. Specifically, local features are first extracted from the dMRI volume using a 3D CNN, then passed to a decoder-only Transformer that predicts the fODF at each point along streamline, conditioned on the preceding trajectory- offering a principled integration of fiber orientation features. Our contributions include:
An algorithmic formulation of tractography as a pathfinding task, inspired by attention- auto-regressive language models and spatially aware encoding.
A tractography model that achieves state-of-the-art performance on widely used benchmark, outperforming existing methods in key metrics.
Open-source code infrastructure for training tractography models on multi-subject datasets, with support for in vivo diffusion MRI scans.
2
Related Work
In recent years, machine learning has emerged as a powerful tool for advancing tractography, moving beyond the limitations of traditional model-based approaches [21). Early work by Neher et al. (2015, 2017) 10 11] introduced a pioneering machine bearning-basedd tractography method that uses a random forest (RF) classifier guide streamline progression based on raw diffusion MRI data. This method demonstrated improved performance, particularly in complex fiber configurations, by taking advantage of data-driven decision-making predict fiber directions and terminations.
Building on the idea of sequential data processing, Poulin et al. (2017) proposed LearnToTrac [12] and Benou et al. (2019) proposed Deep'rract [13]. Both frameworks utilize recurrent neural networks (RNNs) for tractography, but differ in the way they frame the task. The former addressed streamline tractography
regression problem by predicting continuous (deterministic) tracking directions, while the latter takes a classification approach by outputting distribution over discrete directions the unit sphere, thus allowing probabilistic tractography well deterministic. By treating streamlines as sequences of DWI data, RNN models capture the sequential dependencies of the data as context for inferring local fiber orientations. While RNNs enable sequential data processing, they are now often outperformed by Transformers, which offer better parallelization and long-range dependency handling.
Wegmayr et al. (2021) introduced Entrack [4], a probabilistic spherical regression approach that incorporates entropy regularization to manage uncertainty in fiber orientation estimation. Entrack uses the Fisher-von-Min distribution to model the posterior distribution of local streamline directions, enhancing the robustness of the tractography in noisy conditions. This probabilistic approach is particularly well-suited for complex fiber architectures where multiple crossing fibers are present.
The exploration of reinforcement learning for tractography was advanced by Théberge et al. (2021) with the introduction of TrackToLenn [22]. This framework frames tractography as a reinforcement learning problem, where agent learns to navigate white matter pathways by optimizing a reward function based on alignment with principal diffusion directions. This method does not require ground-truth tractograms for training, making it versatile across different datasets.
method combining CNNs and Transformers for fODF estimation. In CTtrack, a CNN projects diffusion MRI data to a nower-dimensiona space, which then processed by a Transformer to estimate fODFs as spherical harmonic coefficients. While both CTtrack and our proposed TractoTransfone combine CNNs and Transformers, their modeling paradigms differ fundamentally. CTtrack processes DWI data a non-sequential manner, while our proposed Tracto Transformer treats tractography as an auto-regressive sequence modeling task, offering a more structured and context-aware framework tailored to the intrinsic sequential nature of tractography.
Hosseini al. (2022) proposed CTtrack (3),

---

3
Methodology
The main goal of the proposed Tracto@Tranfforme method is to extract streamlines sequeecess of (z,, y, 2) coordinates representing fiber pathways- from volumetric DWI data. The core concepts illustrated in Figure I A detailed formulation and description of the data are provided in Section 3.1
The network architecture and its key components presented in Section 3.2 while Section 3.3 outlines the training process for conditional fODFs prediction and the optimization strategies used to enhance model performance. Finally, Section 3,4 describes the inference procedure for streamline tracking on unseen data using the trained model.
3.1 Model Formulation
Streamline tractography aims to reconstruct white matter pathways by inferring plausible fiber trajectories from diffusion MRT data. We frame this problem sequential prediction task, where the likelihood of fiber orientations at a given point along streamline is predicted based on the history of all previous DWI measurements along that path. Our dataset consists of DWI scans and the corresponding tractography data of N subjects. For each subject, we have:
A 4D DWI volume X E H where H.W, and D are spacial dimensions, and G corresponds to the number of magnetic field gradient directions applied during the dMRI scan. Axial views extracted from a volumetric DWI dataset of a single subject, acquired at six (out of 65) different gradient directions are shown in Figure 3
2. A set of reference streamlines S = {s(1) 8 g(M)) representing a whole-brain tractogra-
phy corresponding to X, where each streamline g(m) (&f' 88 ...., &N. is a sequence of 3D points RAS Saatt-ntteroo-Superrinii coordinates, commonly used to standardize anatomical positions.
We feed our model with sequences of DWI values sampled along the coordinate path of stream- line. That is, given streamline s (s1...., sn), the input the model is the sequence (X)))).... X(s.., {s
At each point along streamline, the model is trained to predict conditional fODF, represented as discrete probability distribution over a fixed set of K + 1 classes. Here, K denotes a set of directions uniformly distributed the unit sphere. Specifically, given prefix trajectory (S1,..., s;), the output at point 8; is:
P(f X(s))..... X(:).,
(I)
where = (f1, (fi,...,, fk) is discrete probability distribution over the direction classes defined by the spherical tessellation of K = 724 directions, along with an additional class representing end of fiber (EoF). This conditional formulation reflects the core assumption of our model: the fiber orientation at given point depends not only on the local microstructural context (captured by the
DWI signal), but also the trajectory taken reach that point.
3.2 Model Architecture
The proposed Tracto Transformer leverages the strengths of both Transformers and convolutional neural networks (CNNs) to predict conditional fODFs from DWI data. Its architecture illustrated in Figure T
3D Input Embedding. To embed the input sequence for the Transformer, we first enhance each voxel representation along a streamline by incorporating local spatial context using 3D convolutional neural network (3D-CNN). For each point in the sequence, the 3D-CNN processes a surrounding voxel cube to extract microstructural features from the local diffusion signal. This step improves the voxel-wise representation and expands the effective receptive field, providing the model with spatial context. To reduce computational cost, the 3D-CNN is applied only to the batch of voxels corresponding to the current set of streamlines, rather than the entire brain volume.
The resulting spatially enhanced feature vectors serve as input tokens to the Transformer. To preserve the sequential order of streamline points, we apply standard sinusoidal positional encodings, as introduced by Vaswani et al. (24). This enables the model to account for trajectory history, ensuring
3

---

Pocitional
fODEs estmation
Encoding
Taanformer Decoder Layers
CNN Feature Update
TractoTrrassfun Transformer Model
feature sampled along a streamline
Figure 1: Overview of the Tracto'Transorren model framework. Streamlines represented as sequences of DWI feature vectors, where each vector derived from the raw dMRI data a specific voxel, sampled using spherical harmonies. The TrantoTranffrmmer model consists of 3D-CNN layer, positional encoder, and Transformer layers. The CNN is applied to each streamline voxel (represented by diffusion measurement vector) and its nearby spatial neighbors. The entire encoded streamline is fed into the transformer. The model outputs predicted fODFs at each voxel, which are used guide subsequent tractography.
that each orientation prediction is informed not only by local voxel features but also by the path taken to reach that point-an important consideration for anatomically plausible tractography.
Decoder-Only Transformer. We use a standard decoder-only Transformer architecture to process sequences of streamline data. Each decoder block includes masked multi-head self-atten self-attention and position-wise feed-forward network, both followed by residual connections and layer normalization. causal attention mask enforces autoregressive prediction by preventing access to future positions, while padding mask blocks attention to invalid inputs. This design allows the model to capture long-range dependencies and contextual patterns along the streamline. The final output is mapped to the target space via fully connected layers, followed by softmax function that yields probability distribution over possible directions and an end-of-fiber (EoF) class. Further architectural and implementation details, including specific hyperparameters and training configurations, provided in Section
3.3 Model Optimization and Loss Function
We use the reference streamlines provided in the dataset to construct labels for supervised learning, training the model to predict conditional fODFs during sequence processing. For each reference streamline, direction vectors are computed between consecutive points and normalized to unit vectors. Since the output classes correspond to directions on the unit sphere and possess geometric structure with well-defined angular distances, it is appropriate to weigh classification errors accordingly [13]
To this end, we construct soft label distribution by smoothing each unit direction over the sphere using Gaussian kernel. Formally, given a unit direction 0 (i.e., the direction between two consecutive streamline points) and set of unit directions {a;) defining the discrete class space, we compute the angular distance d, between 0 and each ag and assign weights as:
d3
w;; cxp
202
where a is the standard deviation of the Gaussian kernel. The resulting soft label is a normalized probability distribution over directions:
W!
Ismootth
(3)
K Σx=
Wi
This distribution decays smoothly with increasing angular distance on the unit sphere, as illustrated in Figure 2 and is used supervise the model's predictions.

---

Figure 2: Visualization of the smoothed label distribution on the unit sphere. The gener- ated distribution decays as the distance the unit sphere increases, providing proba- bilistic framework for supervising the model's 1ODF predictions.
To train our TranfoTrrnefrmmer model, employ the Kullback-Leible divergence (KL-Div) loss to measure the divergence between the predicted discrete distribution and the corresponding smooth target label. Given point 8t, target label distribution Usmooth associated with 8;, and model prediction Uprea = IPf X ((1)).... (s;)), the KL-Div is formally defined as:
Yemooth [j]
(3/mmothh Bpeed)
Σ ysmooch [i] log
P (s1),....... X ((:)))
The mean loss is computed at each prediction step along the streamline. The KL-Div loss quantifies the information loss incurred when ypred is used approximate Vuues making it well-suited for evaluating the accuracy of probabilistic predictions against the ground truth distribution of fiber orientations. This loss function particularly appropriate in scenarios where both the predicted outputs and the target labels are probability distributions, as it encourages the model to produce outputs that closely align with the empirical data.
3.4 Streamline Tractography Inference
Once the model is trained, tractography is initiated by sampling random seed points from the provided white matter mask, each defining the starting location of fiber trajectory. Tractography proceeds iteratively: at each step, the model receives the current point along with the accumulated tracking history and predicts conditional fODF, auto-regressively conditioned on previously generated points in the streamline. This design ensures that each orientation prediction incorporates both local features and the full trajectory context, capturing the sequential dependencies inherent white matter pathways.
The tracking direction is selected the with the highest probability in the predicted fODF, resulting in a deterministic propagation scheme. However, unlike classical deterministic methods, predictions are context-aware conditioned the entire streamline history-enabling robust direction selection even in anatomically challenging regions such as fiber crossings or areas of high uncertainty.
After selecting a direction, the streamline is advanced by fixed step RAS space, the new point appended to the trajectory, and the process is repeated until stopping criterion is met:
The class chosen from the prediction of the model EoF class.
2. The next step is outside of the bounds of the MRT image.
3. The next step is outside of the white matter mask.
4. The angle between two consecutive steps exceeds predefined threshold.
5. The fractional anisotropy (FA) values in the next step less than a predefined threshold.
The collection of generated trajectories constitutes a set approximate streamlines which together form the final tractogram. This process is detailed in the pseudocode provided in Algorithm I

---

Algorithm Streamline Tractography Algorithm
Require: Trained model, white matter mask, seed points, stopping criteria
Ensure: Tractogram of streamlines
1: for each seed point do
2: Initialize streamline with seed point
3: while stopping criteria are not met do
4: Feed the current streamline into the model
5: Get conditional fODFs from the model Select direction as argmax of conditional fODF Compute next point by stepping in the selected direction next point satisfies stopping criteria then Terminate streamline
6:
7:
8:
9:
else
11:
Add next point to the streamline
12: end
13: end while
14: Store the completed streamline in tractogram
15:
end for
4 Experiments
4.1
Datasets
For this study, we used two publicly available tractography datasets. The first is the ISMRM 2015 Tractography Challenge phantom dataset [25]. which has been one of the most widely used benchmarks the field over the past decade. It contains a high-quality 4D DWI volume with dimensions 90 108 x 90 100 after resampling, along with comprehensive set of 270,000 ground truth white matter streamlines.
The second dataset is Tractolnferno 26 the largest open-source, multi-site tractography dataset to date, comprising diffusion data from 286 subjects. The dataset partitioned into 198 subjects for training, 60 for validation, and 28 for testing. Each subject includes a 4D diffusion- (DWI) volume with higher spatial resolution than the ISMRM dataset. Although the exact dimensions vary between subjects (for example, the test subject whose tractography is shown in Figure 5 has a spatial volume of 141 X 184 x 120), the number gradient directions also varies, ranging from 22 to 132, and is resampled to 100 directions for consistency. In addition, the dataset provides rich ground--ruu tound tractography, averaging over million streamlines per subject. These streamlines are generally shorter than those in the ISMRM dataset and, after resampling constant step size of 3 mm, can contain up to 100 3D points.
imaging
Figure 3: Axial slices from a volumetric DWI dataset of a single subject where each scan was acquired different diffusion gradient direction. Data source: sub-1024 DWI from TractoInferno dataset (26]
4.2 Preprocessing.
To ensure consistency across subjects and reduce variability arising from acquisition protocols, apply several preprocessing steps. First, represent the DWI signal using spherical harmonic

---

coefficients sampled over fixed set of gradient directions. This step addresses inter-subject variability in gradient schemes and provides standardized input format for the model. Next, we resample the reference streamlines to maintain a consistent step size between consecutive points in the right- anterior-superior (RAS) space, ensuring uniform spatial resolution across all samples and supporting reliable modeling and downstream analysis. Finally, we augment the dataset by reversing streamline orientations, increasing data diversity, and enabling the model to learn more robust features.
4.3 Implementation Details
Architecture. The model configurations are as follows. For the 3D-CNN component, single 3D convolutional layer with a kernel size of 3 x 3 x 3. We evaluated two input variants: (i) the original diffusion diffusiion-wiihhted imaging (DWI) signals, resampled to 100 fixed gradient directions, and (ii) spherical harmonic (SH) coefficients of order 12, representing the diffusion signal in the frequency domain. To mitigate overfitting, dropout with probability of p 0.1 was applied to all layers. The Transformer-bar based network consists of 8 decoder layers, each with 10 attention heads. Every decoder block includes forward network (FFN) with hidden dimension of 512. The final Transformer output is passed through an additional FFN, projecting it to a -is-dimensional vector representing 724 candidate directions on the unit sphere and one end-of-fiber (EoF) class used to signal streamline termination.
Training. All models were trained using the Adam optimizer [27 with an initial learning rate of 0.005. Learning rate decay was applied by multiplying the rate by 0.7 if the accuracy did not improve by least 0.3 over two consecutive epochs. For label smoothing, we employed Gaussian kernel with a standard deviation of a = 0.1. Target labels were represented as discrete probability distributions over 725 classes (K~ 724), corresponding to angular resolution of approximately 3.5". Training was conducted for 30 epochs with a batch size of 20, using up to four NVIDIA V100 GPUs with 32GB of memory each.
Inference. We used an angular threshold of 70 degrees and an FA threshold of 0.05, The step size was set to mm for the ISMRM dataset and mm for Tractolnferno, matching each dataset's native spatial resolution. Streamline generation was performed using approximately 200,000 seed points for ISMRM and about across four GPUs to leverage data-parallel tractography. To further optimize inference, we employ key-value VKV) caching in the Transformer decoder reuse attention states from previous steps, substantially reducing inference time and memory overhead. On average, full-brain tractography for a single subject required approximately 43 minutes on four NVIDIA V100 GPUs.
million for Tractolnferno, processed in batches of 100 and distributed evenly
4.4
Evaluation on the Synthetic ISMRM Dataset
4.4.1 Whole-Brain Tractography Evaluation
To evaluate our model, we trained it on the ISMRM dataset using 80/20 split of reference streamlines for training and validation. Training took 12 hours. Whole-brain tractography was then performed by seeding from random points within the white matter mask. For comprehensive benchmarking and to facilitate future comparisons, we report our results using both the classic Trac- tometer (28) (2015 edition) and the updated Tractometer (29] (2023 edition), the latter incorporating ROI-based segmentation to improve reliability and reproducibility.
We report four key metrics: valid connection (VC) (valid connection rate), overlap (OL) (overlap with ground truth), overreach (OR) (overreach beyond anatomical boundarics), and the F1 score, which balances precision and recall. Results in Table [ demonstrate that forms state-of-the-ar tractography methods. The spherical harmonics input configuration achieves the highest overall performance, with VC of 84%, an overlap of 79%, and the best F1 score (75%). These results indicate accurate and specific reconstruction of white matter connections, with high sensitivity and relatively low overreach (27%).
outper-
4.4.2 Complex Fiber Bundles Reconstruction
To further demonstrate the advantages of our history-aware streamline propagation, we compare the bundle-specific reconstruction of TraccoTrrasffrmee (with DWI-input configuration) with the deterministic tractography algorithm implemented in the MITK Diffusion toolbox. The MITK method
7

---

Table 1: Comparison of the proposed
performance with that of state-of of-the--rtt methods on the Tractolnferno dataset. For each metric, the best results are shown in bold and the second-best are underlined. Our method achieves the top performance in VC, OL, and FI. Entries marked with an asterisk (*) correspond to results obtained using the 2023 Tractometer edition.
Model
VC (%)
OL (%) )
OR (%)
F1 (%)
ISMRM Mean
54
31
23
44
RF 10. 11
67
75
31
Learn To Track 122
42
64
35
64
DeepTraat 13
71
69
23
70
Entrack [14]
65
60
36
58
Track-to-learn 22
68
62
CTtrack [23]
57
50
16
60
Tracto Transformer
82
82
35
71
Tracto Transformer SH input
79
27
75
TractoTransoonn Transformer*
82
84
31
Tracto Transformer SH input
82
81
26
78
reconstructs streamlines by following local diffusion maxima, without accounting for trajectory history or incorporating global contextual information. We focus on complex white matter bundles characterized by extensive fiber crossings and branchings, such the Right Brainstem Pontine tract and the Left Cingulum bundle. As shown in Table 2 Transofransformer achieves markedly higher valid connection counts, overlap, and F1 scores in these challenging bundles. Figure 4 illustrates qualitative differences between the two methods, highlighting the enhanced anatomical plausibility achieved by Tracto Transformer reconstructions.
Tracto Transforer
ISMRM GT
MITK
Figure 4: Visual comparison of tractography results in regions with complex fiber architecture from the ISMRM dataset. Top: Right Brainstem Pontine tract. Bottom: Left Cingulum bundle. Each column shows reconstructions obtained with the proposed TractoTransformel the ISMRM ground truth, and the MITK deterministic approach.
4.5 Ablation Study
Table summarrize ablation results, highlighting the contribution of each component in our frame- work. Whole-brain tractography was evaluated on the ISMRM dataset after removing the 3D-CNN module, reverse streamline augmentation, or label smoothing. All experiments used the TractoTrans. former variant with raw DWT input and were quantitatively assessed using the 2023 Tractometer toolkit 29

---

Table Quantitative comparison between MITK and TrasfoTransformer on complex bundles. VC denotes the number of valid connections.
Bundle
vC
OL (%)1
OR (%)
F1 (%)
Right Brainstem Pontine Tract MITK
11103
39.48
18.37
53.22
Tracto Tranforranf
31996
88.31
27.36
79.71
Left Cingulum Bundle
MITK
7619
46.97
31.71 39.09
55.66
Tracto Transforrnf
16792
89.62
72.52
Table Ablation study of the Tracto framework evaluated for the ISMRM data with the Tractometer toolkit (2023 edition).
Model
VC (%)f
OL (%)f
OR (%)
F1 (%)
Trasforrrrre
81.51
83.72
30.83
74.78
-3D-CNN
69.86(11.65)
80.70(.3.0)
32.84(2200)
72.66(2.12)
-Reverse Streamlines
79.8 (11.00
82.94(
33.08
73.76
-Smooth Labels
79.77(1.74)
82.81 (-0.1)
30.86(+0003)
74.85
The largest performance drop is observed when excluding the 3D-CNN module, which reduces VC by 11.65% and slightly lowers the Fl score, underscoring the importance of local spatial context for accurate trajectory estimation. Removing reverse streamline augmentation leads to a smaller decline, indicating that directional diversity supports regularization but is less critical. Omitting label smoothing has minimal effeet on F1, with only slight decreases in VC and OL. Overall, while all components contribute, the 3D-CNN module remains essential for anatomically plausible reconstructions.
4.6 Evaluation on the In-vivo TractoInferno Dataset
To evaluate our method on in-vivo dMRI data, we used the Tractolnferno dataset 26 Due to computational constraints, training performed ten subjects, validation two, and testing on four. The architecture and training setup matched those used for the ISMRM dataset, except for multi-subject training. Training took approximately seven days. Figure presents whole-brain and bundle-level reconstructions for one test subject (sub-1019), showing generalize across subjects and recover complex fiber pathways. Table reports the average quantitative results across the four test subjects. All basclines were implemented and evaluated by the TractoInferno authors using their official pipeline. Transoerrnsfrrmer achieved the highest Overlap and Dice (FI) scores, demonstrating superior balance between anatomical coverage and precision. The full per-subject results are provided in the Appendix, along with a Pareto visualization illustrating the overlap-overreat that Tranforrrnforrmer lies above the baseline Pareto front, achieving an average overlap gain of 0.027 (~4.8%% at comparable overreach levels.
ability
tradeoff of the compared methods. The Pareto analysis indicates
Table 4: Average performance across four TractoInferoo subjects. Our method achieves the highest Dice and Overlap, indicating a superior balance between coverage and precision.
Det-
Det-SE (Learn2TTacc)
Prob-Sphere (Deep'Tract)
Prob-Gaussian (SOTA)
Prob- Mixture
Tracto Transformrf
Metric
Cosine
(Ours)
Dicef
0.609
0.575
0.596
0.612
0.391
0.628
Overlapt
0.549
0.495
0.540
0.578
0.292
0.589
Overreach.
0.243
0.191
0.219
0.288
0.058
0.263

---

Whole Brain
GT Axial view
Ours Axial view
GT Sagittal view
Ours Sagittal view
Cingulum
GT Axial view
Ours Axial view
GT Sagittal view
Ours Sagittal view
Parieto- Occipital Pontine
GT Axial view
Ours Axial view
GT Coronal view
Ours Coronal view
Corpus Callosum (Frontal)
GT Axial view
Ours Axial view
GT Coronal view
Ours Coronal view
Figure 5: Visual comparison of tractography outputs from TractoInferno (GT) and Tracto Transfomerr (TT) across four regions. Each row shows a different tract, with matched views from both models.
Conclusions, Limitations, and Broader Impact
Conclusions. We presented Tranforransformer, a hybrid CNN WNN-Transformer framework for diffusion MRI tractography that integrates local microstructural context with sequential trajectory modeling. By leveraging the prefix trajectory to guide the tracking process, the model produces anatomically accurate reconstructions and effectively resolves complex fiber configurations such crossing and kissing bundles. Comprehensive evaluations on the ISMRM and IractoInferno datasets, including whole-brain and per-bundle analyses against classical and deep learning baselines, demonstrate the superior performance of the proposed TractoTransfoome Transformer across the tested benchmarks. Limitations. The main bottlenecks
computational, arising from high memory usage and relatively long inference time, which may limit scalability in large or high-resolution datasets. Efficiency could be improved through optimized attention mechanisms (e.g.. PlassAAtentionn or model compression to reduce resource demands and enable broader applicability.
Broader Impact. TructoTransforme provides a high-performing and accessible framework for AI- driven tractography, with potential applications in both neuroscience research and clinical practice. Its modular design, together with publicly available code and pretrained models, promotes reproducibility and ease of adoption, enabling researchers to extend and advance data-driven neuroimaging.
10

---

Acknowledgments and Disclosure of Funding
We thank the authors of the TractoInferno dataset [26 for their courtesy in sharing per-subject evaluation scores of the implemented baselines, which enabled the quantitative analysis presented in the appendix. This work was partially supported by the Israel Ministry of Science and Technology (T.R.R MOST 000731111001111977),
References
[1] Edward Bullmore and Olaf Sporns. Complex brain networks: Graph theoretical analysis of structural and functional systems. Nature reviews. Neuroscience, 10:186-1108 10-986.98, 03 2009.
[2] Olga Ciccarelli, Marco Catani, Heidi Johansen-Berg Clare Clark, and Alan Thompson. Diffusion-based tractography neurological disorders: concepts, applications, and future developments. The Lancet Neurology, 7(8):715-727, August 2008.
[3] Peter Basser, James Matticllo, and Denis LeBihan. Estimation of the effective self-diffusion tensor from the nmr spin echo. Journal of Magnetic Resonance, Series B, 103(3):247-254, 1994.
[4] Matthan WA Caan, H Ganesh Khedoe, Dirk HJ Poot, Arjan den Dekker, Silvia D Olabarriaga, Kees A Grimbergen, Lucas Van Vliet, and Frans M Vos. Estimation of diffusion properties in crossing fiber bundles. IEEE Transactions on Medical Imaging, 29(8):1504-1555 2010.
[5] Timothy EJ Behrens, Mark W Woolrich, Mark Jenkinson, Heidi Johansen-Berg. Rita G Nunes, Stuart Clare, Paul M Matthews, Michael Brady, and Stephen M Smith. Characterization and propagation of uncertainty in diffusion hiffuttio-weighheed mr imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 5065):0077-1088, 2003.
[6] Maxime Descoteaux, Elaine Angelino, Sean Fitzgibbons, and Rachid Deriche. Regularized, fast, and robust analytical q-ball imaging. Magnetic Resonance Medicine, 58(3):497-510, 2007.
[7] Jacques-Donal Tournier, Fernando Calamante, and Alan Connelly. Robust determination of the fibre orientation distribution in diffusion mri: non-negativity constrained super-resolved spherical deconvolution. Neurolmage, 35(4):1459-144 1472, 2007.
[8] H.-W. Chung, M.-. Chou, and C.-Y. Chen. Principles and limitations of computational algorithms in clinical diffusion tensor mr tractography. American Journal of Neuroradiology. 32(1):3-13, 13, 2011.
[9] Philippe Poulin, Daniel Jörgens, Pierre-Man Maree Jodoin, and Maxime Descoteaux. Tractography and machine learning: Current state and open challenges. Magnetic Resonance Imaging. 64:37-48, 2019. Artificial Intelligence in MRI.
[10] Peter F. Neher, Matthias Götz, Tobias Norajitra, Christian Weber, and Klaus H. Maier-Hein machine learning based approach fiber tractography using classifier voting. Nassir Navab, Joachim Hornegger, William Wells, and Alejandro Frangi, editors, Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, volume 9349 of Lecture Notes in Computer Science, pages 54-61. Springer, Cham, 2015.
[11] Peter F. Neher, Marc-Alexandre Côté, Jean- Chan-Chritttphe Houde, Maxime Descoteaux, and Klaus H. Maier-Hein. Fiber tractography using machine learning. Neurolmage, 158:417-429, 2017.
[12] Philippe Poulin, Marc-Alexandre Côté, bean-Christophe Houde, Laurent Petit, Peter F. Neher, Klaus H. Maier-Hein, Hugo Larochelle, and Maxime Descoteaux. Learn to track: Deep learning for tractography. Medical Image Computing and Computer Assisted Intervention MICCAT 2017, pages 540-547, Cham, 2017. Springer International Publishing.

---

[13] Itay Benou and Tammy Riklin Raviv. Deeptract: A probabilistic deep learning framework for white matter fiber tractography. In Medical Image Computing and Computer Assisted Intervention MICCAT 2019, pages 626 635, Cham, 2019. Springer International Publishing.
[14] Valentin Wegmayr and Joachim M. Buhmann. Entrack: Probabilistic spherical regression with entropy regularization for fiber tractography. International Journal of Computer Vision, 129(3):656-688) 2021.
[15] Simon Koppers and Dorit Merhof. Direct estimation of fiber orientations using deep learning in diffusion imaging. In International Workshop on Machine Learning in Medical Imaging, pages 53 60. Springer, 2016.
[16] Jakob Wasserthal, Peter F Neher, and Klaus H Maier-Hein Tract orientation mapping for bundle-specific tractography. In International conference on medical image computing and cemputee--ssssted intervention, pages 66-44. Springer, 2018.
[17] Marco Reisert, Volker A Coenen, Christoph Kaller, Karl Egger, and Henrik Skibbe. Hamlet: hier- archical harmonic filters for learning tracts from diffusion mri. arXiv preprint arXiv:1801 1807.01068 2018.
[18) Vishwesh Nath, Kurt G Schilling, Prasanna Parvathaneni, Colin B Hansen, Allison E Hainline, Yuankai Huo, Justin A Blaber, Ilwoo Lyu, Vaibhav Janve, Yurui Gao, et al. Deep learning reveals untapped information for local white-matter fiber reconstruction in diffusion-weighted mri. Magnetic resonance imaging, 62:220-227, 2019.
[19] Sara Sedlar, Théodore Papadopoulo, Rachid Deriche, and Samuel Reslauriers-Gautiie Diffu- sion mri fiber orientation distribution function estimation using voxel-wise spherical u-net. In Computational Diffusion MRI: International MICCAI Workshop, Lima, Peru, October 2020, pages 95-106. Springer, 2021.
[20] Hongyu Li, Zifei Liang. Chaoyi Zhang, Ruiying Liu, Jing Li, Weihong Zhang, Dong Liang, Bowen Shen, Xiaoliang Zhang, Yulin Ge, et al. Superdti: Ultrafast dti and fiber tractography with deep learning. Magnetic resonance in medicine, 86(6)33334-3377, 2021.
[21] Peter Neher, Philippe Poulin, Daniel Jörgens, Marco Reisert, Itay Benou, and Klaus Maier-Hein. Chapter 17 - machine learning in tractography. In Flavio Dell' Acqua, Maxime Descoteaux, and Alexander Leemans, editors, Handbook of Diffusion MR Tractography, pages 315-345. Academic Press, 2025.
[22] Antoine Théberge, Christian Desrosiers, Maxime Descoteaux, and Pierre-Marc Jodoin. Track- to-learn: A general framework for tractography with deep reinforcement learning. Medical Image Analysis, 72:102093, 2021.
[23] S.M.H. Hosseini, M. Hassanpour, S. Masoudnia, S. Iraji, S. Raminfard, and M. Nazem- Cttrack: A cun-transsfommer-assee framework for fiber orientation estimation & tractography. Neuroscience Informatics, 100099, 2022.
[24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[25] Klaus H Maicr-Hein, Peter Neher, -hrrsshiite Houde, al. The challenge of mapping the human connectome based on diffusion tractography. Nature Communications, 8:1349, 2017.
[26] Philippe Poulin, Guillaume Theaud, Francois Rheault, Etienne St-Onge, Arnaud Bore, Em- manuelle Renauld, Louis de Beaumont, Samuel Guay, Pierre-Marc Jodoin, and Maxime De- scoteaux. TractoInferno- a large-scale, open-source, multi-site database for machine learning dMRI tractography. Sci. Data, 9(1), November 2022.
[27] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
12

---

[28]
Mare-Alexandre Côté, Gabriel Girard, Adrien Boré, Eleftherios Garyfallidis, Houde, and Maxime Descoteaux. Tractometer: towards validation of tractography pipelimes. Medical Image Analysis, 17(7):844-857, 2013.
[29] Emmanuelle Renauld, Antoine Théberge, Laurent Petit, Jean-Christophe Houde, and Maxime Descoteaux. Validate your white matter tractography algorithms with reappraised ismrm 2015 tractography challenge scoring system. Scientific Reports, 13(1):2347, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]-
Justification: The abstract and introduction accurately reflect our contributions and only refer to the results attained in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes].
The technical limitations of our work are detailed in dedicated limitations section. Additional limitations regarding applications are discussed in the "broader impact" section.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer:
[NA].
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
Justification: Our code is attached as supplementary material and will be made publicly available on GitHub upon acceptance. The hyperparameter and detailed architecture with dimensions are also described the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel urchitecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Our code is attached as supplementary material and will be made publicly available on GitHub upon acceptance. The data sets used publicly available and easy to access.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.co wwwpccpppllcee.eeccecceeeeeeeece www.e//gg//eeeeeeeeeeeeeee wttp///odemmsssiiioocoo wommissssoooiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: Data splits, hyperparameters, and choice of optimizer specified in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No].
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: Computer resources, memory, and training time are specified in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httph: https://neurrpst http://wwww.ppppppppp/pppi/c//tt wctppublc/Bhhcccciieee..e.
Answer: [Yes].
Justification: We have reviewed the reviewed the NeurIPS Code of Ethics and confirm that the research conducted the paper conforms with it in every aspect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: The potential positive societal impacts are discussed in the dedicated broader impact section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA].
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
The creators and original owners assets used in the paper, including code, data and models are properly credited and the license and terms of use are properly respected. Links to the data sets we' ve used: ISMRM2015 "Basic dataset" (Creative Commons CCO license) and TractoInferno (Creative Commons Attribution CC BY 4.0 license).
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes].
Justification:
The developed code is thoughtfully documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowd-sourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
The paper does not involve crowd-sourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips: hett::///earrrpsscc//oooerrreeee cc///oofeeeeeecs//22/////.. for what should or should not be described.
20

---

A
Additional Results
Table Performance comparison across four subjects using Dice, Overlap, and Overreach metrics for each method. All baseline methods were implemented and evaluated by the Tractolnferno authors. Our method achieves the highest Dice score on three out of four subjects and the second-best on the remaining one.
Det-Coiine
Det-SK (LearnZrrack)
Prob-Sphere (Deep Tract)
Prob-Caussian (SOTA)
Prob-Mixture
Tranteerannoormeee (Ours)
sub- sub-1006
oondlpp
0.575
0.470
0.525
0.593
0.784
0.590
overeaahh
0,281
0,231
0.248
0.381
0,074
0.262
0.381
sob- 1019
everlapt
0.551
0
0,540
0.566
0.266
0.625
ovenescc)
dice
0.585
0.593
0,600
0.644
0.356
0.602
sub- 1624
overlep
0.511
0,493
0,533
0,579
0.302
0.562
overeach
dicr"
0.688
0,606
0.516
0.60
0.430
0.629
sbb 1961
oveslapt
0.314
0.5M0
ovemosch
0.068
0.34
Tracto Transforrrr vs. Baseline Pareto Front
0.60
Det-Cosine
Det-SE (Leam2Track)
0.55
Prot-Sphere (Deeptract)
Prob-Gaussian (SOTA)
Prob Mixture
0.50
TrsctoTanattrmmer (Ours)
Overlap f
0.45
0.40
0.35
0.30
0.05
0.10
0.15
0.20
0.25
Overreach 4
Figure 6: Pareto analysis of the overlap-overrente trade-off between Tranforrrn and baseline methods. The dashed line represents the Pareto front computed from the baselines, while the red marker denotes TractoIransformer (ours), which lies 0.027 units above the baseline front, equivalent to 4.8% higher overlap at comparable overreach.
21


Reviews:
[{"review_id": "wqWBVuKxfZ", "reviewer": "Reviewer_EBQD", "summary": "This study proposes a CNN-aided transformer-based approach to predict white matter fiber streamlines. The study intuitively poses tractography as a pathfinding algorithm, i.e., it predicts neural fibers from data and deals with challenges such as crossing and merging of fibre bundles. This leads to the formulation of the spatio-sequential formulation of the fibre orientation distribution function (fODF), where local features are captured by convolutional filters, which act as context for the transformer to make a prediction of the fODF in the streamlines. It does it in an autoregressive manner – i.e., it takes the voxel activity corresponding to the previous coordinates and makes the prediction about the next coordinate using a decoder-based transformer. For this, it minimises the KL-divergence between the discretized local streamline vector (soft-binned) and the predicted distribution over directions. This model is then probed like an autoregressive language model to generate the trajectory.", "questions": "* In the inference phase, in spite of taking the argmax over the predicted probabilities, the authors can see if the performance changes if we compute the expected direction. This converts discrete outputs to a continuous prediction space.  \n* What is the train-test split in the ISMRM dataset, as there appears to be only 1 scan? Or is the scan just a test sample.\n* I am open to upgrading my rating if the limitations are addressed satisfactorily", "limitations": "N/a", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The literature review is well structured and contains important details of recent ML-based tractography methods. \n* The approach used to extract local dwi features and use them for decoding using transformer decoders is intuitive and a plausible approach that could extract robust streamlines. \n* The numbers show significant improvement over existing methods. Ablation studies were useful too, to see the significance of the parts of the model. Could become an important neuroscience tool. \n\nMajor Weaknesses \n* The idea of using CNNs and Transformers for tractography prediction is not completely novel, as the authors themselves point out (Hosseini, 2022). Nonetheless, extending it to an auto-regressive model is the novelty (although that too is obtained by using traditional masking as done in LLMs). It would be better if the authors mentioned that more explicitly, just to avoid overselling the method. Although it can have a significant impact on tractography in neuroscience, the novelty in ML models/approaches is limited. \n* The inference time for tractography is 7 hours for V100 GPUs, which is comparable to or longer compared to traditional tractography pipelines. Could the authors confirm that this is correct? This makes it difficult to justify as a useful method.\n* All the results are on a single phantom scan. For human DMRI data, only 10 participants are used, with only 1 test subject. This makes it challenging to evaluate the usefulness of the algorithm at scale, in big datasets with thousands of scans. \n\nMinor Weakness \n* In equation 1, why is X(s_i) being used to make a prediction of the coordinate s_i. We need to predict the next coordinate based on the history of the coordinate. Should the output not be P(f|X(s1)… X(si-1))? \n* Can the authors refer to the experiment section in the methods section for the details of the architecture? Incorporating them into a more detailed figure would be useful for the readers.", "was_revised": true, "final_justification": "I have read the authors' rebuttal responses and discussed these responses, am happy to upgrade my score.", "rebuttal": "**Response to Reviewer EBQD**\n\nWe thank the reviewer for the detailed and constructive feedback. We appreciate your acknowledgment of the strengths of our work and your openness to revising the score if the limitations are properly addressed. We hope that our responses, clarifications, and new analyses below will fully address your concerns.\n\n**Addressing Main Weaknesses**\n\n**1. Novelty**  \nWe thank the reviewer for the feedback. We acknowledge that our method builds on established ML architectures, and we will revise the manuscript to better articulate how our adaptation to the tractography setting constitutes a meaningful and practical contribution.  While CNN and Transformer architectures have been explored in diffusion MRI tractography (e.g., Hosseini et al., 2022), our core novelty lies in adapting and integrating these tools into a sequential, context-aware, auto-regressive framework inspired by language models. Specifically, we reformulate streamline tracking as a sequence modeling task, treating each streamline as a “sentence” of diffusion-derived “tokens,” and predicting the next direction using causal self-attention over the streamline’s history. This enables the model to capture long-range dependencies and better handle complex fiber configurations such as crossings and fanning.\n\nTo underscore the value of our contribution, we conducted an additional experiment comparing our method to the deterministic MITK Diffusion pipeline, which tracks streamlines based solely on local maxima without contextual history. Using the ISMRM 2015 dataset and its evaluation metrics, we focused on the Right Brainstem Pontine and Left Cingulum bundles, chosen for their anatomically complex structure with prominent crossings and branchings. As shown in Table 1, our model outperforms MITK in valid connections, overlap, and F1 score. A whole-brain comparison in Table 2 further demonstrates the benefit of our context-aware formulation, achieving significantly better results. We will include these results in the revised manuscript, along with visualizations of the reconstructed bundles.\n\n**Table 1: MITK vs TractoTransformer on challenging bundles (ISMRM evaluation tool)**  \n*VC = valid connections, OL = overlap, OR = overreach.*\n\n| **Bundle**                    | **Method**        | **VC** | **OL (%)** | **OR (%)** | **F1 (%)** |\n|-------------------------------|-------------------|--------|------------|------------|------------|\n| Right Brainstem Pontine Tract | MITK              | 11103  | 39.48      | **18.37**  | 53.22      |\n|                               | TractoTransformer | **31996** | **88.31** | 27.36      | **79.71**  |\n| Left Cingulum Bundle          | MITK              | 7619   | 46.97      | **31.71**  | 55.66      |\n|                               | TractoTransformer | **16792** | **89.62** | 39.09      | **72.52**  |\n\n**Table 2: Whole-brain tractography comparison**\n\n| **Method**         | **VC (%)** | **OL (%)** | **OR (%)** | **F1 (%)** |\n|--------------------|------------|------------|------------|------------|\n| MITK               | 55.74      | 48.03      | **28.56**  | 54.98      |\n| TractoTransformer  | **81.51**  | **83.72**  | 30.83      | **74.78**  |\n\n**2. Runtime and Practicality**  \nWe acknowledge that the current end-to-end runtime (~7 hours on a single V100 GPU) presents scalability challenges. Our current implementation is research-oriented, prioritizing anatomical fidelity and accuracy over computational efficiency. That said, there is substantial room for optimization. Since streamlines are generated independently, inference time can be significantly reduced through straightforward parallelization across GPUs or CPU threads, enabling near-linear speedup. We are also actively developing strategies to further accelerate inference, including key-value caching for the autoregressive process and optimized batch processing. Alongside open-sourcing our code and model, we plan to release an optimized implementation incorporating these improvements to support more practical runtimes and wider adoption. We will revise the manuscript to better reflect these limitations and our plans to address them.\n\n**3. Evaluation Scale**  \nISMRM has long served as a widely used benchmark for tractography, owing to its high anatomical fidelity and controlled, reproducible setup, despite its limitation to a single phantom scan. To complement this, we initially included a qualitative evaluation on an unseen in-vivo scan from the TractInferno dataset, which provides a valuable alternative perspective. We agree, however, that broader quantitative evaluation across diverse datasets is important for assessing generalizability.\n\nTo address this, we extended our TractInferno experiment to include quantitative evaluation on four test subjects. With the courtesy of the Tractinferno authors, we obtained per-subject evaluation scores (Dice, Overlap, and Overreach) for five baseline methods: Det-Cosine, Det-SE (Learn2Track), Prob-Sphere (DeepTract), Prob-Gaussian, and Prob-Mixture. These scores were not included in the original publication and were shared with us upon request (see the TractInferno paper for method details). Notably, our model was trained on only 10 subjects, whereas the TractInferno baselines were trained on 198. Despite this, as shown in Table 3, our method achieves the highest Dice score on three out of four subjects and second-best on the fourth, demonstrating strong generalization from limited data. Scaling to the full TractInferno test set was not feasible at the time of submission due to hardware constraints. Nonetheless, we will release our full codebase to support future benchmarking and reproducibility at scale.\n\n**Table 3: TractInferno test subjects — Dice, Overlap, Overreach**\n\n| **Subject** | **Metric**   | **Det-Cosine** | **Det-SE** | **Prob-Sphere** | **Prob-Gaussian** | **Prob-Mixture** | **TractoTransformer** |\n|-------------|--------------|----------------|------------|-----------------|-------------------|------------------|------------------------|\n| sub-1006    | dice         | 0.618          | 0.544      | 0.570           | 0.597             | 0.399            | **0.626**              |\n|             | overlap      | 0.575          | 0.470      | 0.525           | **0.593**         | 0.284            | 0.590                  |\n|             | overreach    | 0.281          | 0.231      | 0.248           | 0.381             | **0.074**        | 0.262                  |\n| sub-1019    | dice         | 0.614          | 0.558      | 0.597           | 0.606             | 0.381            | **0.654**              |\n|             | overlap      | 0.551          | 0.493      | 0.540           | 0.566             | 0.266            | **0.625**              |\n|             | overreach    | 0.239          | 0.198      | 0.227           | 0.266             | **0.051**        | 0.279                  |\n| sub-1024    | dice         | 0.585          | 0.593      | 0.600           | **0.624**         | 0.354            | 0.602                  |\n|             | overlap      | 0.511          | 0.493      | 0.533           | **0.579**         | 0.302            | 0.562                  |\n|             | overreach    | 0.196          | 0.153      | 0.192           | 0.245             | **0.039**        | 0.267                  |\n| sub-1061    | dice         | 0.618          | 0.606      | 0.616           | 0.620             | 0.430            | **0.629**              |\n|             | overlap      | 0.560          | 0.523      | 0.562           | 0.575             | 0.314            | **0.580**              |\n|             | overreach    | 0.256          | 0.180      | 0.209           | 0.259             | **0.068**        | 0.244                  |\n\n**4. Equation Clarification**  \nWe thank the reviewer for bringing this to our attention. At each $s_i$, we predict the fODF conditioned on entire preceding path up to $s_i$, and then advance to $s_{i+1}$ in the selected direction. We will correct the indexing and clarify this in the final manuscript.\n\n**5. Manuscript Navigation and Figures**  \nWe agree that clearer section references and a more detailed architecture diagram will improve readability. We will revise the Methods section and figures accordingly.\n\n**Response to Specific Questions**\n\n**ISMRM Split:**  \nReference streamlines were randomly split 80/20 into training and validation. These were used for supervision only, not during inference. While voxel positions may recur, the model conditions on full streamline context, making it highly unlikely to encounter identical input-output pairs. Generalization is further supported by results on unseen subjects (both qualitative and quantitative).\n\n**Expected Direction Inference:**  \nThank you for this suggestion. We evaluated tracking using the expected direction under the predicted fODF, instead of argmax, using identical seeds and evaluation. Results are similar: argmax improves F1 and overlap, while expected direction reduces overreach.\n\n**Table 4: Mode vs Expected Direction on ISMRM**\n\n| **Model**            | **VC (%)** | **OL (%)** | **OR (%)** | **F1 (%)** |\n|----------------------|------------|------------|------------|------------|\n| TractoTransformer    | 81.51      | 83.72      | 30.83      | 74.78      |\n| Expected direction   | 79.37      | 75.43      | **25.52**  | 73.44      |\n\nWe thank the reviewer once again and will integrate this comparison and all clarifications, revisions, and new results into the updated manuscript. We appreciate the reviewer’s constructive comments and suggestions, which have helped us improve both the clarity and scope of our work. Thank you for your thoughtful review."}, {"review_id": "y5CLBW7jt9", "reviewer": "Reviewer_XRRU", "summary": "This paper describes an estimator of white matter tract structure\nthat uses a CNN to characterize diffusion MRI info locally,\nalong with a transformer that marches down the streamlines,\npaying attention to what has come before.\n\nTractography is an important capability that enables research\nin neuroscience.  Empirically we have seen that subtle abnormalities\nin white matter structure are associated with important brain disorders.\nTractography is estimated from specialized MRI scans (MRI is non-invasive).\n\nThe authors summarize the background on what tractography is\nand what characterizes the state of the art in the field.\nTractography, the process of estimating white matter structure\nby constructing streamlines is very challenging, as the voxels\nare quite large compared to the anatomical structure.  While there\nhave been many approaches to this problem, including, e.g., unscented\nKalman filters, the problem is not solved, witness the increasing\nscores in the challenges.\n\nThe present contribution achieves excellent results by harnessing\nthe strengths of CNNs to summarize the diffusion image data locally,\nand by using a transformer to model the sequential nature of\ntractography (the classical method of tractography is sequential:\nfind the direction with the strongest evidence, take a step in\nthat direction, repeat).  This sequential nature suggests use\nof a recurrent mechanism, and currently, transformers are the\nmost successful recurrent mechanisms.  For the local 'image'\nanalysis, CNNs are arguably the most successful mechanism.\n\nThe present paper matches the structure of the problem to two of\nour most powerful mechanisms, transformers and CNNs, and does\nso in an elegant and novel way that yields SOA results.", "questions": "One item I didn't quite get is:   tracking is started from random\nseeds in the WM.   Are these seeds in the 'interior' of the WM\nas a 3d structure?  If so, is tracking carried out in two directions,\none opposite the other?\n\nIn the description of the data, I was wondering how the ground truth\nstreamlines were generated?   Manual tracing by people?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "This is a very interesting and appropriate model for the problem at\nhand.  It produces SOA results, and open source infrastructure will be\nprovided.  The paper is structured very well.  It clearly situates the\nwork in relation to what has come before.  The writing is\nexceptionally clear.", "was_revised": true, "final_justification": "I feel there is substantial novelty in bringing this hybrid method to \na problem of substantial import in neuroscience; that, and the\nresults argue for acceptance, for me.", "rebuttal": "**Response to Reviewer XRRU**\n\nWe sincerely thank the reviewer for the thorough assessment and for the exceptionally positive feedback regarding our methodology, results, and presentation. We appreciate your thoughtful questions and address them below.\n\n**1. Seeding and Tracking Directions:**  \nTracking is initiated from random seed points located within both the interior and, in some cases, the boundaries of the white matter. For each seed, tractography proceeds in a single direction, determined by the probability distribution predicted by our model based on local DWI measurements.\n\n**2. Ground Truth Streamline Generation:**  \n*ISMRM 2015 Phantom Reference:* Ground truth bundles were created by manual segmentation from a Human Connectome Project subject’s tractogram, guided by expert anatomical definitions. Bundle-specific masks enabled deterministic tracking, with post-processing to remove implausible streamlines. Details are provided in the official Tractometer documentation.\n\n*TractInferno Dataset:* Reference bundles were generated using an ensemble of four tractography methods, followed by bundle extraction with RecoBundlesX and extensive manual quality control. About 200 hours of expert QC ensured anatomical plausibility and completeness, as detailed in the TractInferno paper and dataset documentation.\n\nWe hope these clarifications are helpful. We appreciate the reviewer’s constructive comments and suggestions, which have helped us improve the clarity of our work. Thank you for your thoughtful review and support."}, {"review_id": "PBkfPRxa8e", "reviewer": "Reviewer_Nf5j", "summary": "The authors propose a new spatio-sequential tractography algorithm that incorporates spatial information along a predicted streamline. During training, the method leverages a set of ground truth streamlines and, at each point along a streamline, predicts an orientation probability to reach the next point. At inference time, the method can generate novel streamlines in a sequential manner. The approach is validated on two datasets and demonstrates both quantitatively and qualitatively plausible tractograms.", "questions": "* **Q1**: How are the reference streamlines generated and filtered?\n* **Q2**: Why was 3×3×3 chosen, and how does that choice affect performance?\n* **Q3**: Can the CNN be applied once to the full volume to avoid recomputing features at every streamline step?\n* **Q4**: How is interpolation performed when sampling DWI values at off-grid points along a streamline?\n* **Q5**: Does the model predict a full fODF per voxel, or a direction-conditioned distribution tied to the current streamline (closer to an orientation probability)?\n* **Q6**: Which stopping criteria are learned? Please provide statistics on end-of-fiber (EoF) predictions versus other stopping causes.\n* **Q7**: Why not input spherical-harmonic coefficients directly, given that the convolution is blind to spherical geometry?\n* **Q8**: Can you identify regions where classical trackers fail but your model succeeds, and vice versa?\n* **Q9**: Can the quantitative metric from the Tractometer dataset be used on the in-vivo dataset as well?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n\n* **End-to-end learning of tracking decisions**: Predicting the next streamline direction directly from local DWI patches removes the need for intermediate fODF estimation and may capture contextual cues that classical pipelines miss.\n* **Spatial context included**: Even with a simple 3×3×3 voxel cube, the model incorporates local neighbourhood information, improving direction prediction over single-voxel inputs.\n* **Preliminary quantitative gains**: Reported Tractometer scores and qualitative in-vivo bundles suggest performance that is competitive with or better than existing baselines.\n\n**Weaknesses**:\n\n* **Training–test leakage**: The Tractometer experiment uses the same diffusion volume for both training and evaluation; overlap between streamlines likely inflates the reported metrics.\n* **Limited anatomical stress-testing**: The method’s performance in regions with complex fibre configurations (e.g., crossings, kissings, or fans) is not directly compared to existing fODF-based approaches.\n* **Sparse methodological detail**: Important implementation details—such as step size, cube size reasoning, SH decomposition order, and the data split strategy—are not clearly presented.\n* **Tiny convolutional “network”**: A single 3D convolution layer amounts to little more than spatial smoothing; there is no ablation to test whether deeper CNNs or SH coefficients as input would improve results.\n* **Dependence on teacher streamlines**: The model is trained on labels from a single tracking algorithm, meaning any bias or failure in that method is directly inherited.", "was_revised": true, "final_justification": "The rebuttal is strong, providing a deeper analysis of the strengths and weaknesses of the method, as well as useful practical clarifications. I am raising my score accordingly. However, the current draft of the paper requires major revisions to incorporate these new analyses, which would necessitate another round of review.", "rebuttal": "**Response to Reviewer Nf5j**\n\nWe thank the reviewer for the thorough and thoughtful review. Below, we address your main concerns and specific questions.\n\n**1. Training–test leakage**\n\nWe acknowledge that our evaluation on the ISMRM 2015 phantom uses the same diffusion scan for training and testing. While this is a known limitation of the benchmark, it remains standard practice to enable fair, like-for-like comparisons across methods.\n\nImportantly, our model is context-aware: its predictions are based on both the local DWI and the full streamline path up to the current point. Because test streamlines are seeded randomly, even if they traverse seen voxels, the input trajectory differs—preventing simple memorization.\n\nTo directly address the generalization concern, we conducted additional inference on four unseen TractInferno test subjects (sub-1006, 1019, 1024, 1061). We compared our model to five baselines implemented by the TractInferno authors, who kindly shared the per-subject results. We used their official evaluation script to compute Dice, Overlap, and Overreach metrics.\nThese comparisons were not included in the original publication, as TractInferno reports only dataset-wide averages, which we were unable to reproduce due to hardware constraints. Notably, our model was trained on only 10 subjects, whereas the TractInferno baselines were trained on 198. Despite this, as shown in Table 1, our method achieves the highest Dice score on three of the four subjects and the second-best on the remaining one, demonstrating strong generalization from limited training data.\n\n**Table 1: TractInferno test subjects — Dice, Overlap, Overreach**\n| Subject   | Metric    | Det-Cosine | Det-SE | Prob-Sphere | Prob-Gaussian | Prob-Mixture | TractoTransformer |\n|-----------|-----------|------------|--------|--------------|----------------|---------------|-------------------|\n| sub-1006  | Dice      |0.618|0.544|0.570|0.597|0.399| **0.626**|\n|           | Overlap   |0.575|0.470|0.525|**0.593**|0.284| 0.590|\n|           | Overreach |0.281|0.231|0.248|0.381|**0.074**| 0.262|\n| sub-1019  | Dice      |0.614|0.558|0.597|0.606|0.381| **0.654**|\n|           | Overlap   |0.551|0.493|0.540|0.566|0.266| **0.625**|\n|           | Overreach |0.239|0.198|0.227|0.266|**0.051**| 0.279|\n| sub-1024  | Dice      |0.585|0.593|0.600|**0.624**|0.354| 0.602|\n|           | Overlap   |0.511|0.493|0.533|**0.579**|0.302| 0.562|\n|           | Overreach |0.196|0.153|0.192|0.245|**0.039**| 0.267|\n| sub-1061  | Dice      |0.618|0.606|0.616|0.620|0.430| **0.629**|\n|           | Overlap   |0.560|0.523|0.562|0.575|0.314| **0.580**|\n|           | Overreach |0.256|0.180|0.209|0.259|**0.068**| 0.244|\n\n\n**2. Limited anatomical stress-testing**\n\nIn order to address this concern, we conducted a comparative experiment against MITK Diffusion, a deterministic tractography pipeline that follows voxelwise principal directions derived from fODFs, without incorporating streamline history or global context.\n\nWe used the ISMRM phantom and evaluated two challenging bundles—Right Brainstem Pontine and Left Cingulum—which exhibit crossings and branching. Table 2 compares *number* of valid connections (VC) using the official Tractometer tool. Table 3 compares *percentage* metrics for whole-brain tracking.\nOur model clearly outperforms in terms of valid connections, overlap, and F1 score, highlighting its effectiveness in detecting these complex fiber bundles. To further highlight these differences, we will include explicit visual comparisons of bundle reconstructions along with these quantitative results in the revised manuscript.\n\n**Table 2: MITK vs. TractoTransformer on complex bundles (ISMRM, Tractometer)**\n\n| **Bundle**                    | **Method**        | **VC** | **OL (%)** | **OR (%)** | **F1 (%)** |\n|-------------------------------|-------------------|--------|------------|------------|------------|\n| Right Brainstem Pontine Tract |MITK|11103|39.48|**18.37**| 53.22|\n|                               |TractoTransformer|**31996**|**88.31**|27.36|**79.71**|\n| Left Cingulum Bundle          |MITK|7619|46.97|**31.71**|55.66|\n|                               |TractoTransforme|**16792**|**89.62**|39.09|**72.52**|\n\n**Table 3: Whole-brain tractography (ISMRM, Tractometer)**\n\n| **Method**         | **VC (%)** | **OL (%)** | **OR (%)** | **F1 (%)** |\n|--------------------|------------|------------|------------|------------|\n| MITK               |55.74|48.03|**28.56**|54.98|\n| TractoTransformer  |**81.51**|**83.72**|30.83|**74.78**|\n\n**3. Sparse methodological details**\n\nThank you for noting this. We will clarify all implementation details in the final version. For completeness:\n\n*Step size:* 3 mm (TractInferno), 1 mm (ISMRM), matching each dataset’s native resolution.\n*Cube size:* 3×3×3 CNN kernel to balance local context and memory constraints.\n*SH order:* 12, supports a wide range of acquisition schemes (gradient directions vary from 22 to 132 before resampling).\n*Data split:* ISMRM: streamlines are splitted 80% train, 20% val; TractInferno: 10 train subjects, 2 validation subjects, 1 test subject (limited by GPU RAM).\n*Streamlines:* ~200K seeds (ISMRM), ~1M seeds (TractInferno).\n*Preprocessing:* DWI resampled to 100 directions via SH order 12.\n*MRI resolution:* Native resolution used throughout.\n*Augmentation:* Streamline reversal applied during training.\n\n**4. Simple CNN architecture and lack of spherical-harmonic (SH) input ablation**\n\nWe found that a single 3D convolutional layer effectively captures spatial context with minimal compute, as shown in Table 2 of the paper. Deeper CNNs may yield improvements but are beyond the scope of this work given memory and efficiency constraints. This will be discussed in the final manuscript.\nWe agree that using SH coefficients directly is worth studying. We now include an ablation where SH coefficients (order 12) are used as input. See Table 4 in “Additional Experiments” below.\n\n**5. Dependence on teacher streamlines**\n\nThis is an inherent limitation in all supervised methods relying on a single teacher. In our case, TractInferno reference bundles are generated by ensembling results from four different tracking algorithms, followed by RecoBundlesX extraction and ~200 hours of manual QA. This process reduces algorithmic bias and yields more reliable ground truth.\n\n**Response to Specific Questions**\n\n**1. Reference streamlines generation**\n\n- **ISMRM:** Manually segmented from an HCP subject, refined with anatomical masks, deterministic tracking, and post-processing. These are used *only* during training, never at test time. Furthermore, test-time trajectories differ due to random seeds, preventing memorization.\n- **TractInferno:** Ensembled outputs of four classical algorithms, filtered via RecoBundlesX, and curated with ~200 hours of manual QA.\n\n**2. 3×3×3 kernel choice**\n\nThis kernel allows each voxel’s prediction to include 26 neighboring voxels, capturing critical fiber information (e.g., crossings, partial volumes). Larger kernels or deeper CNNs would increase memory costs and are currently infeasible under our hardware limitations.\n\n**3. Can the CNN be applied once to the full volume?**\n\nDuring training, the CNN weights are continuously updated, requiring feature recomputation at each step. During inference, our ablation shows that removing the CNN layer yields only a minor speedup, as the main bottleneck is the transformer's autoregressive nature. Moreover, extracting features for the entire volume would inefficiently allocate resources to non-white matter regions.\n\n**4. How is interpolation handled at off-grid points?**\n\nDuring training, all streamline points are mapped to integer voxel coordinates. During inference, off-grid predictions result in streamline termination. No interpolation is performed.\n\n**5. Fixed fODF or a path-dependent distribution prediction?**\n\nIt predicts a path and context-dependent orientation distribution. The same voxel can yield different predictions depending on the streamline's history. We will clarify this point in the final manuscript.\n\n**6. Learned stopping criteria**\n\nOnly the End-of-Fiber (EoF) token is learned. Others—angular threshold, FA, and anatomical constraints—are rule-based. Percentages for stopping:\n\n- Angular threshold: 70%\n- EoF: 65%\n- FA threshold: 10%\n- Outside image / WM mask / max length: 0%\n\n**7. Using SH-coefficients directly**\n\nWe now evaluated this variant. See Table 4 and discussion below.\n\n**8. Can you identify where classical methods fail vs. your model, and vice versa?**\n\nYes. As shown in our MITK comparison (Table 2 and 3 above), classical methods often fail in regions with complex fiber geometries (e.g., crossings, fanning), where our model excels. In contrast, our model can sometimes overreach in simple regions, reflecting a precision–recall trade-off.\n\n**9. Can the Tractometer metric be used on TractInferno?**\n\nIn theory, yes, but it requires infrastructure adaptation. The ISMRM phantom remains a widely used benchmark for standardized, fair comparison across studies.\n\n---\n\n**Additional Experiments: SH Coefficient Input Ablation**\n\nWe retrained our model with SH coefficients inputs (order 12) instead of raw DWI. All other parameters remained the same.\n\n**Table 4: SH vs. Raw DWI**\n| **Model**                  | **VC (%)** | **OL (%)** | **OR (%)** | **F1 (%)**|\n|----------------------------|------------|------------|------------|------------|\n| Raw DWI (original)         | 81.51      | **83.72**| 30.83| 74.78|\n| SH Coefficients (order 12) | **81.98**  | 80.58| **24.28**  | **77.78**|\n\nSH input reduced overreach and improved F1 by +3%, with minor overlap drop. These results will be included and discussed in the final manuscript. We thank the reviewer for this suggestion.\n\nWe appreciate the reviewer’s constructive comments and suggestions, which have helped us improve both the clarity and scope of our work. Thank you again for your thoughtful review."}, {"review_id": "8DQUlYXHgJ", "reviewer": "Reviewer_LQ4s", "summary": "The manuscript proposed a tractography algorithm based on machine learning, more specifically CNN and Transformer Architecture. The goal is to add more anatomical features (landmarks) and priors (path history) into the tractography process. Framing tractography as an sequence modeling task directly addresses a key limitation of many tractography methods (e.g predicting fiber orientations in a voxel without considering the path taken to get there).", "questions": "For table 1, are the reported results taken for their original publication? Are they reimplementation of the methods that you ran? Or are they standard run (from the original code + container) that you ran? This information would be important for context (default or tuned parameters, weights version, etc.)\n\nThe paper is transparent about the limitations: the 3D-CNN is memory-intensive, and inference takes 7 hours per subject (does resolution impact that? How many streamlines generated? What step-size?). This is a clear contrast with other ML tractography algorithm and possibly a barrier to practical use. While acknowledged the review should explore how this might be improved in future work.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The combination of a 3D CNN for spatial context and a Transformer for sequential context is well-justified. The manuscript demonstrates good performance on the ISMRM 2015 tractography challenge benchmark  and shows interesting results on the more complex, real-world TractoInferno dataset. The ablation study demonstrates the contribution of each model module (CNN, reverse streamlines, smooth labels), supporting the authors design choices.\n\nThe qualitative evaluation on TractoInferno is very promising. However, the model was trained on only ten subjects from this dataset validated on two and tested on one. How well does this represent the full diversity of the 198 training subjects? This might limit the claimed generalization capability. I believe a simpler experiment on data such as the HCP test-retest (44x2 subject) for future work could help with a simpler hypothesis, it is expected that a good algorithm should have have high overlap of bundles (requires a segmentation methods). I recommend this script for comparing bundles (in the same space, co-registered): https://github.com/scilus/scilpy/blob/master/scripts/scil_bundle_pairwise_comparison.py\nAnd this script to compare whole brain tractogram (co-registered): https://github.com/scilus/scilpy/blob/master/scripts/scil_tractogram_pairwise_comparison.py", "was_revised": true, "final_justification": "I believe the author adressed the limitations I observed as well as the other reviewers feedback. Considering my initial review was 'optimistic', I think my initial rating should not change, but the changes to the manuscript and clarifications are in line with what I was expecting.", "rebuttal": "**Response to Reviewer LQ4s**\n\nThank you for your thoughtful and constructive feedback, especially your recognition of our conceptual framing, ablation studies, and qualitative results. Your practical suggestions and questions are highly appreciated and have helped us better communicate the scope and limitations of our work.\n\n**Addressing Main Weaknesses**\n\n**1. Limited training/testing subset from TractInferno; generalization capability:**  \nWe acknowledge the limitation of training and testing on a relatively small subset of the TractInferno dataset, which was due to hardware constraints. Specifically, loading more than ten volumes was not feasible, and expanding further, even with methodological adjustments, would have significantly increased computational demands. While tractography results typically show limited qualitative variation across subjects, we agree that broader evaluation is important for demonstrating generalizability.\n\nTo address this, we have extended our evaluation to include quantitative tractography results on four subjects from the TractInferno test set. For fair and standardized comparison, we contacted the TractInferno authors, who kindly provided us with official evaluation metrics: Dice, Overlap, and Overreach, for five leading baseline methods: Det-Cosine, Det-SE (Learn2Track), Prob-Sphere (DeepTract), Prob-Gaussian, and Prob-Mixture. These per-subject results were not published in the original TractInferno paper (which only reported averaged metrics), so these scores were shared with us upon request. For full implementation details of these baselines, we refer to the original TractInferno paper.\n\nAs shown below, our method achieves the highest Dice score on three out of four subjects (sub-1006, sub-1019, and sub-1061), and the second-highest Dice on sub-1024. Importantly, these results were obtained using a model trained on only 10 subjects, compared to 198 used for the TractInferno baselines—highlighting the strong generalization capability of our approach, even with limited training data.\n\nScaling the evaluation to the full TractInferno test set was not feasible at submission due to hardware constraints. However, to support reproducibility and enable future benchmarking, we are releasing our full codebase as open source. We believe these added quantitative results already provide meaningful evidence of our method’s effectiveness, and we encourage future work, especially from groups with greater computational resources, to build upon our approach and validate it at larger scale.\n\n**Table: Performance comparison (Dice, Overlap, Overreach) for each method on four TractInferno subjects**\n\n| **Subject** | **Metric**   | **Det-Cosine** | **Det-SE (Learn2Track)** | **Prob-Sphere (DeepTract)** | **Prob-Gaussian** | **Prob-Mixture** | **TractoTransformer** |\n|-------------|--------------|----------------|--------------------------|-----------------------------|-------------------|------------------|-----------------------|\n| sub-1006    | Dice         | 0.618          | 0.544                    | 0.570                       | 0.597             | 0.399            | **0.626**             |\n|             | Overlap      | 0.575          | 0.470                    | 0.525                       | **0.593**         | 0.284            | 0.590                 |\n|             | Overreach    | 0.281          | 0.231                    | 0.248                       | 0.381             | **0.074**        | 0.262                 |\n| sub-1019    | Dice         | 0.614          | 0.558                    | 0.597                       | 0.606             | 0.381            | **0.654**             |\n|             | Overlap      | 0.551          | 0.493                    | 0.540                       | 0.566             | 0.266            | **0.625**             |\n|             | Overreach    | 0.239          | 0.198                    | 0.227                       | 0.266             | **0.051**        | 0.279                 |\n| sub-1024    | Dice         | 0.585          | 0.593                    | 0.600                       | **0.624**         | 0.354            | 0.602                 |\n|             | Overlap      | 0.511          | 0.493                    | 0.533                       | **0.579**         | 0.302            | 0.562                 |\n|             | Overreach    | 0.196          | 0.153                    | 0.192                       | 0.245             | **0.039**        | 0.267                 |\n| sub-1061    | Dice         | 0.618          | 0.606                    | 0.616                       | 0.620             | 0.430            | **0.629**             |\n|             | Overlap      | 0.560          | 0.523                    | 0.562                       | 0.575             | 0.314            | **0.580**             |\n|             | Overreach    | 0.256          | 0.180                    | 0.209                       | 0.259             | **0.068**        | 0.244                 |\n\n**2. Test-retest reliability and suggested comparison scripts:**  \nThank you for highlighting the importance of test-retest validation and for recommending specific resources. We agree that evaluation on HCP test-retest data is an excellent way to benchmark robustness and will reference this in our discussion as a priority for future work. By providing our code and weights, we facilitate such analyses by both our team and the broader community.\n\n**Response to Specific Questions**\n\n**1. Table 1 Benchmarking:**  \nAll values in Table 1 are quoted directly from the original publications. We appreciate your request for clarity and will explicitly note this in our revisions to avoid ambiguity for readers.\n\n**2. Practical limitations: runtime, memory, resolution, streamline count, and step size:**  \nInference time and resource consumption are influenced by several factors: (a) Step size: 1mm for ISMRM, 3mm for TractInferno, matching each dataset’s reference sampling. Finer step sizes increase computation; (b) Streamline count: We match reference tractograms ($\\sim$ 200,000 for ISMRM, $\\sim$ 1,000,000 for TractInferno); (c) Resolution: We use native full resolution to preserve anatomical detail, which increases computational cost but benefits qualitative accuracy. (d) 3D-CNN design: The 3×3×3 kernel size was chosen to balance anatomical context and memory consumption; larger/deeper models exceeded our GPU capability. We agree these constraints represent an important direction for practical improvement. Notably, since streamlines are generated independently, inference runtime can be reduced almost linearly simply by parallelizing across computing units. We will explicitly highlight these practical limitations and the opportunities for acceleration in the revised manuscript.\n\nWe appreciate the reviewer’s constructive comments and suggestions, which have helped us improve both the clarity and scope of our work. Thank you again for your thoughtful review and support."}]

Author rebuttal:

Meta-review: This paper presents a method for estimating white matter tract structure by combining a convolutional neural network (CNN) to locally characterize diffusion MRI signals with a transformer that sequentially processes streamlines, attending to prior context along the path.

The approach is evaluated on two datasets and yields both quantitatively and qualitatively plausible tractograms. In particular, the qualitative results on the TractoInferno benchmark are very promising. Nonetheless, there are some concerns regarding the small sample sizes used during training, validation, and testing, as well as the computational cost of the method.

While the use of CNNs and transformers for tractography is not entirely new, the extension to an auto-regressive formulation adds a degree of novelty. The paper is well-structured and clearly written. I recommend it for publication.
Decision: Accept (poster)