Title: TractoTransformer: Diffusion MRI Streamline Tractography using CNN and Transformer Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Neuroimaging

Diffusion MRI

White Matter Tractography

Sequential Modeling

Transformers

Convolutional Neural Networks
Abstract: White matter tractography is an advanced neuroimaging technique that reconstructs the 3D white matter pathways of the brain from diffusion MRI data. It can be framed as a pathfinding problem aiming to infer neural fiber trajectories from noisy and ambiguous measurements, facing challenges such as crossing, merging, and fanning white-matter configurations.
In this paper, we propose a novel tractography method that leverages Transformers to model the sequential nature of white matter streamlines, enabling the prediction of fiber directions by integrating both the trajectory context and current diffusion MRI measurements. To incorporate spatial information, we utilize CNNs that extract microstructural features from local neighborhoods around each voxel. By combining these complementary sources of information, our approach improves the precision and completeness of neural pathway mapping compared to traditional tractography models. We evaluate our method with the Tractometer toolkit, achieving competitive performance against state-of-the-art approaches, and present qualitative results on the TractoInferno dataset, demonstrating strong generalization to real-world data. Our code is publicly available at https://github.com/ItzikWaizman/TractoTransformer.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposed a tractography algorithm based on machine learning, more specifically CNN and Transformer Architecture. The goal is to add more anatomical features (landmarks) and priors (path history) into the tractography process. Framing tractography as an sequence modeling task directly addresses a key limitation of many tractography methods (e.g predicting fiber orientations in a voxel without considering the path taken to get there).\n\nSTRENGTHS AND WEAKNESSES:\nThe combination of a 3D CNN for spatial context and a Transformer for sequential context is well-justified. The manuscript demonstrates good performance on the ISMRM 2015 tractography challenge benchmark  and shows interesting results on the more complex, real-world TractoInferno dataset. The ablation study demonstrates the contribution of each model module (CNN, reverse streamlines, smooth labels), supporting the authors design choices.\n\nThe qualitative evaluation on TractoInferno is very promising. However, the model was trained on only ten subjects from this dataset validated on two and tested on one. How well does this represent the full diversity of the 198 training subjects? This might limit the claimed generalization capability. I believe a simpler experiment on data such as the HCP test-retest (44x2 subject) for future work could help with a simpler hypothesis, it is expected that a good algorithm should have have high overlap of bundles (requires a segmentation methods). I recommend this script for comparing bundles (in the same space, co-registered): https://github.com/scilus/scilpy/blob/master/scripts/scil_bundle_pairwise_comparison.py\nAnd this script to compare whole brain tractogram (co-registered): https://github.com/scilus/scilpy/blob/master/scripts/scil_tractogram_pairwise_comparison.py\n\nQUESTIONS:\nFor table 1, are the reported results taken for their original publication? Are they reimplementation of the methods that you ran? Or are they standard run (from the original code + container) that you ran? This information would be important for context (default or tuned parameters, weights version, etc.)\n\nThe paper is transparent about the limitations: the 3D-CNN is memory-intensive, and inference takes 7 hours per subject (does resolution impact that? How many streamlines generated? What step-size?). This is a clear contrast with other ML tractography algorithm and possibly a barrier to practical use. While acknowledged the review should explore how this might be improved in future work.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new spatio-sequential tractography algorithm that incorporates spatial information along a predicted streamline. During training, the method leverages a set of ground truth streamlines and, at each point along a streamline, predicts an orientation probability to reach the next point. At inference time, the method can generate novel streamlines in a sequential manner. The approach is validated on two datasets and demonstrates both quantitatively and qualitatively plausible tractograms.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n* **End-to-end learning of tracking decisions**: Predicting the next streamline direction directly from local DWI patches removes the need for intermediate fODF estimation and may capture contextual cues that classical pipelines miss.\n* **Spatial context included**: Even with a simple 3×3×3 voxel cube, the model incorporates local neighbourhood information, improving direction prediction over single-voxel inputs.\n* **Preliminary quantitative gains**: Reported Tractometer scores and qualitative in-vivo bundles suggest performance that is competitive with or better than existing baselines.\n\n**Weaknesses**:\n\n* **Training–test leakage**: The Tractometer experiment uses the same diffusion volume for both training and evaluation; overlap between streamlines likely inflates the reported metrics.\n* **Limited anatomical stress-testing**: The method’s performance in regions with complex fibre configurations (e.g., crossings, kissings, or fans) is not directly compared to existing fODF-based approaches.\n* **Sparse methodological detail**: Important implementation details—such as step size, cube size reasoning, SH decomposition order, and the data split strategy—are not clearly presented.\n* **Tiny convolutional “network”**: A single 3D convolution layer amounts to little more than spatial smoothing; there is no ablation to test whether deeper CNNs or SH coefficients as input would improve results.\n* **Dependence on teacher streamlines**: The model is trained on labels from a single tracking algorithm, meaning any bias or failure in that method is directly inherited.\n\nQUESTIONS:\n* **Q1**: How are the reference streamlines generated and filtered?\n* **Q2**: Why was 3×3×3 chosen, and how does that choice affect performance?\n* **Q3**: Can the CNN be applied once to the full volume to avoid recomputing features at every streamline step?\n* **Q4**: How is interpolation performed when sampling DWI values at off-grid points along a streamline?\n* **Q5**: Does the model predict a full fODF per voxel, or a direction-conditioned distribution tied to the current streamline (closer to an orientation probability)?\n* **Q6**: Which stopping criteria are learned? Please provide statistics on end-of-fiber (EoF) predictions versus other stopping causes.\n* **Q7**: Why not input spherical-harmonic coefficients directly, given that the convolution is blind to spherical geometry?\n* **Q8**: Can you identify regions where classical trackers fail but your model succeeds, and vice versa?\n* **Q9**: Can the quantitative metric from the Tractometer dataset be used on the in-vivo dataset as well?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes an estimator of white matter tract structure\nthat uses a CNN to characterize diffusion MRI info locally,\nalong with a transformer that marches down the streamlines,\npaying attention to what has come before.\n\nTractography is an important capability that enables research\nin neuroscience.  Empirically we have seen that subtle abnormalities\nin white matter structure are associated with important brain disorders.\nTractography is estimated from specialized MRI scans (MRI is non-invasive).\n\nThe authors summarize the background on what tractography is\nand what characterizes the state of the art in the field.\nTractography, the process of estimating white matter structure\nby constructing streamlines is very challenging, as the voxels\nare quite large compared to the anatomical structure.  While there\nhave been many approaches to this problem, including, e.g., unscented\nKalman filters, the problem is not solved, witness the increasing\nscores in the challenges.\n\nThe present contribution achieves excellent results by harnessing\nthe strengths of CNNs to summarize the diffusion image data locally,\nand by using a transformer to model the sequential nature of\ntractography (the classical method of tractography is sequential:\nfind the direction with the strongest evidence, take a step in\nthat direction, repeat).  This sequential nature suggests use\nof a recurrent mechanism, and currently, transformers are the\nmost successful recurrent mechanisms.  For the local 'image'\nanalysis, CNNs are arguably the most successful mechanism.\n\nThe present paper matches the structure of the problem to two of\nour most powerful mechanisms, transformers and CNNs, and does\nso in an elegant and novel way that yields SOA results.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a very interesting and appropriate model for the problem at\nhand.  It produces SOA results, and open source infrastructure will be\nprovided.  The paper is structured very well.  It clearly situates the\nwork in relation to what has come before.  The writing is\nexceptionally clear.\n\nQUESTIONS:\nOne item I didn't quite get is:   tracking is started from random\nseeds in the WM.   Are these seeds in the 'interior' of the WM\nas a 3d structure?  If so, is tracking carried out in two directions,\none opposite the other?\n\nIn the description of the data, I was wondering how the ground truth\nstreamlines were generated?   Manual tracing by people?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study proposes a CNN-aided transformer-based approach to predict white matter fiber streamlines. The study intuitively poses tractography as a pathfinding algorithm, i.e., it predicts neural fibers from data and deals with challenges such as crossing and merging of fibre bundles. This leads to the formulation of the spatio-sequential formulation of the fibre orientation distribution function (fODF), where local features are captured by convolutional filters, which act as context for the transformer to make a prediction of the fODF in the streamlines. It does it in an autoregressive manner – i.e., it takes the voxel activity corresponding to the previous coordinates and makes the prediction about the next coordinate using a decoder-based transformer. For this, it minimises the KL-divergence between the discretized local streamline vector (soft-binned) and the predicted distribution over directions. This model is then probed like an autoregressive language model to generate the trajectory.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The literature review is well structured and contains important details of recent ML-based tractography methods. \n* The approach used to extract local dwi features and use them for decoding using transformer decoders is intuitive and a plausible approach that could extract robust streamlines. \n* The numbers show significant improvement over existing methods. Ablation studies were useful too, to see the significance of the parts of the model. Could become an important neuroscience tool. \n\nMajor Weaknesses \n* The idea of using CNNs and Transformers for tractography prediction is not completely novel, as the authors themselves point out (Hosseini, 2022). Nonetheless, extending it to an auto-regressive model is the novelty (although that too is obtained by using traditional masking as done in LLMs). It would be better if the authors mentioned that more explicitly, just to avoid overselling the method. Although it can have a significant impact on tractography in neuroscience, the novelty in ML models/approaches is limited. \n* The inference time for tractography is 7 hours for V100 GPUs, which is comparable to or longer compared to traditional tractography pipelines. Could the authors confirm that this is correct? This makes it difficult to justify as a useful method.\n* All the results are on a single phantom scan. For human DMRI data, only 10 participants are used, with only 1 test subject. This makes it challenging to evaluate the usefulness of the algorithm at scale, in big datasets with thousands of scans. \n\nMinor Weakness \n* In equation 1, why is X(s_i) being used to make a prediction of the coordinate s_i. We need to predict the next coordinate based on the history of the coordinate. Should the output not be P(f|X(s1)… X(si-1))? \n* Can the authors refer to the experiment section in the methods section for the details of the architecture? Incorporating them into a more detailed figure would be useful for the readers.\n\nQUESTIONS:\n* In the inference phase, in spite of taking the argmax over the predicted probabilities, the authors can see if the performance changes if we compute the expected direction. This converts discrete outputs to a continuous prediction space.  \n* What is the train-test split in the ISMRM dataset, as there appears to be only 1 scan? Or is the scan just a test sample.\n* I am open to upgrading my rating if the limitations are addressed satisfactorily\n\nLIMITATIONS:\nN/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}