Title: Spatio-Angular Convolutions for Super-resolution in Diffusion MRI
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion MRI

super-resolution

image synthesis

conditional image synthesis

continuous convolution

parametric continuous convolution
Abstract: Diffusion MRI (dMRI) is a widely used imaging modality, but requires long scanning times to acquire high resolution datasets. By leveraging the unique geometry present within this domain, we present a novel approach to dMRI angular super-resolution that extends upon the parametric continuous convolution (PCConv) framework. We introduce several additions to the operation including a Fourier feature mapping, 'global' co-ordinates, and domain specific context. Using this framework, we build a fully parametric continuous convolution network (PCCNN) and compare against existing models. We demonstrate the PCCNN performs competitively while using significantly fewer parameters. Moreover, we show that this formulation generalises well to clinically relevant downstream analyses such as fixel-based analysis, and neurite orientation dispersion and density imaging.

Spatio-Angular Convolutions for Super-resolution in Diffusion MRI

Abstract

Diffusion MRI (dMRI) is a widely used imaging modality, but requires long scanning times to acquire high resolution datasets. By leveraging the unique geometry present within this domain, we present a novel approach to dMRI angular super-resolution that extends upon the parametric continuous convolution (PCConv) framework. We introduce several additions to the operation including a Fourier feature mapping, global coordinates, and domain specific context. Using this framework, we build a fully parametric continuous convolution network (PCCNN) and compare against existing models. We demonstrate the PCCNN performs competitively while using significantly fewer parameters. Moreover, we show that this formulation generalises well to clinically relevant downstream analyses such as fixel-based analysis, and neurite orientation dispersion and density imaging.

1 Introduction

Diffusion magnetic resonance imaging (dMRI) is a clinical imaging modality that is routinely used to diagnose white matter pathology within the human brain. Advancements in downstream analyses using dMRI data continuously expand the diagnostic capabilities of the modality, but often require high resolution data that is clinically infeasible to acquire [28]. However, recent research demonstrates that deep learning can be used to predict high resolution images using low resolution dMRI data [26, 44, 41], a technique known as super-resolution. These advancements therefore pave the way for clinically accessible advanced dMRI analyses.

dMRI data can be represented using a non-Euclidean geometry. Specifically, a dMRI dataset consists of a series of three-dimensional volumes where each volume measures the water diffusivity in a given direction. Here, the direction is quantified by a 3D vector known as the 'b-vector', and is sparsely sampled from a unit sphere. Each three-dimensional volume, however, is regularly sampled in a discrete grid. Therefore, dMRI data has two distinct resolution types: spatial resolution which denotes the regularly sampled grid size, and angular resolution which denotes the number of diffusion directions. Despite this relatively unique geometry, most dMRI deep learning methods use operations grounded in a Euclidean framework, such as convolutional neural networks (CNNs) [43, 44, 41].

This presents an opportunity as typical CNN architectures do not fully utilise the geometric properties present in dMRI data. For example, implicit within the formulation of the CNN is the assumption that data are densely and regularly sampled in a discrete manner. This is only true when considering the three spatial dimensions of dMRI data, whilst the other dimensions would be more suited using approaches like graph convolutional networks (GCNs) [46], spherical CNNs [7], and point cloud CNNs [22]. Examples of approaches that develop geometrically motivated convolutions in the dMRI

domain include those for tissue segmentation [21], lesion segmentation [29], and fibre orientation distribution (FOD) reconstruction [5, 32].

Another promising candidate for a framework capable of utilising the unique dMRI geometry is the parametric continuous convolution (PCConv) introduced by Wang et al. [39], which extended the discrete convolution into the continuous domain. This was achieved via an inner parameterised hypernetwork [16] responsible for generating convolutional weights. Here, the convolutional weights were not learnt directly but were instead sampled from the hypernetwork given kernel coordinates as input. This operation could therefore be used to model arbitrary geometries.

In this work, we build upon the PCConv operation to convolve across both spatial and angular dimensions of dMRI data via a new, flexible, coordinate embedding. We subsequently create a deep parametric continuous CNN (PCCNN) that is well-suited for inference within this domain. Furthermore, we develop additional modifications to the PCCNN by including supplemental prior information in the coordinate embedding. These modifications include the application of Fourier feature mappings, the inclusion of prior domain information, and the incorporation of global coordinate components.

We demonstrate the effectiveness and flexibility of this approach through the task of angular superresolution of dMRI data in various sampling schemes, including single-shell and multi-shell data [18]. To assess the performance of our method, we compare results across clinically relevant downstream analysis tasks such as fixel-based analysis (FBA) [30], and neurite orientation dispersion and density imaging (NODDI) [45]. We show that the PCCNN performs competitively against a similar recurrent CNN (RCNN) network trained on the same data whilst using approximately a tenth of the number of parameters. Additionally, we demonstrate the generalisability of the network by comparing it against models trained specifically for downstream analysis tasks.

2 Related Works

Whilst there is a body of literature within spatial super-resolution methods in the natural image domain [40], and more specifically the dMRI domain [1, 36, 6, 25], this work focuses on angular superresolution as a means of increasing image quality. Here, the task of inferring high angular resolution data from low angular resolution dMRI acquisitions can broadly be split into two categories: methods that super-resolve raw dMRI data, and those that super-resolve downstream analyses. The former allows for more flexible use of the data and analysis method, but the unconstrained nature of the task makes inference more challenging. The latter constrains the problem by leveraging assumptions made within the analysis method, however the mapping from raw dMRI data to downstream analysis is in general not invertible, therefore the super-resolved data cannot be used to derive other downstream analysis metrics.

Notable works in angular super-resolution of raw dMRI data include Yin et al. [43], whereby a preliminary study constructed a 1D CNN autoencoder that only convolved across the angular dimension, disregarding possible spatial correlations within the data. Lyon et al. [26] developed a 3D RCNN that benefited from parameter sharing in the spatial dimensions, but required angular dimension shuffling to overcome sequence order bias. Both models concatenated the b-vectors as channels within the input, therefore treating the b-vectors as additional features, rather than coordinates of a manifold for which the data lies on. Finally, Ren et al. [31] proposed a conditional generative adversarial network (GAN) that additionally used structural T1w and T2w data, as well as b-vector information to predict unseen dMRI intensity data.

As there exists a myriad of downstream analyses for dMRI, this work focuses on two commonly used methods: FBA [30] and NODDI [45]. FBA relies on segmenting FODs [18] into 'fixels' or fibres within a voxel. Relevant works in FOD super-resolution include Lucena et al. [24] and Zeng et al. [44]. Both of these studies regressed low angular resolution derived FODs with their high resolution counterparts through 3D CNNs. NODDI estimates micro-structural complexity of the underlying brain tissue through modelling axon and dendrite populations within a voxel. Approaches to infer high angular resolution derived NODDI parameters from lower resolution raw dMRI data include Golkov et al. [15], Ye et al. [41], and Ye et al. [42]. These studies use various sub-sampling schemes and network architectures, including a 3D CNN within Ye et al. [41].

3 Methods

This section introduces parametric continuous convolutions, and extends the framework into the dMRI domain. We highlight our key contributions to the PCConv operation, then finally provide implementation details used to perform training and inference within the task of dMRI superresolution.

3.1 Parametric Continuous Convolutions

A discrete convolution within the context of deep learning is defined as follows

where the data function f : N → R and kernel g : G → R are defined over the support domain for the finite integer set N = Z D and G = {-M, -M +1 , . . . , M -1 , M } D respectively. Conversely, a continuous convolution is defined as

where both f and g are continuous functions over the support domain N = G = R D . Typically in deep learning, this integration is analytically intractable but can be approximated as in Wang et al. [39] using the following form:

where a finite number of input points x i are sampled from the support domain for each output point y j . In the parametric continuous convolution framework, we choose to use a multi-layer perceptron (MLP) as the kernel function g ( e ; θ ) = MLP( e ; θ ) , where e is a coordinate embedding that depends on x i and y j , which is described in detail in Section 3.2. The kernel function g ( e ; θ ) : R D → R spans the full continuous domain while being parameterised by a finite number of parameters θ . In contrast to standard convolutional operations, the weights of the PCConv are not learnt directly, and are instead sampled from g ( e ; θ ) . As the kernel function is itself a fully differentiable function, it can be trained through back-propagation.

In typical convolutional layers the set of output points form a subset of the input points set, provided there is no padding. However, within the parametric continuous framework, these two sets can be disjoint. This is key within the task of dMRI angular super-resolution, where the angular component of y j , i.e. the target b-vectors, is not contained within the set of input b-vectors.

Each PCConv layer requires three inputs to produce the output features h k,j ( y j ) : the input features f c,i ( x i ) , the set of input points x , and the set of output points y . Within each layer, following on from the right-hand side of Equation (1), the PCConv operation is defined as

Here, C and K denote the input and output feature dimensions respectively. Within each layer the number of input points N is fixed, therefore the kernel function g will encapsulate the 1 N factor in Equation (1) during training.

3.2 Convolutions in Q-Space

We define q-space Q as the joint spatial-angular domain within dMRI where Q = { ( u, v, w, ρ, ϑ, φ ) ⊤ | u, v, w, ρ ∈ R , 0 ≤ ϑ < 2 π, 0 ≤ φ ≤ π } and x j , y i ∈ Q . Here, ( u, v, w ) ⊤ denote the spatial components, and ( ρ, ϑ, φ ) ⊤ denote the angular components. Specifically, ϑ and φ refer to the azimuthal and polar angles, respectively, which form the set of points on the surface of a sphere of radius ρ . In this context, ρ is the diffusion weighting or b-value. Figure 1 visualises

where

Here p m is the m th component of the coordinate vector p ∈ R E given by p = [ u i -u j , v i -v j , w i -w j , ρ i -ρ j , d r ] . We have opted to enforce rotational invariance across the sphere by omitting ϑ and φ directly, and instead use p 5 = d r ( ϑ j , φ j , ϑ i , φ i ) where d r ( ϑ j , φ j , ϑ i , φ i ) = sin( ϑ j ) sin( ϑ i ) + cos( ϑ j ) cos( ϑ i ) cos( φ j -φ i ) . The function γ : R → R 2 L is a Fourier feature mapping motivated by Tancik et al. [35] and Mildenhall et al. [27] which is used to improve the hypernetwork's ability to learn high-frequency functions. In practice, L is a hyperparameter of the PCConv layer. Figure 2 provides examples of weights sampled from a trained PCConv layer with varying d r . The channels selected within the Figure demonstrate a variety of functions across the sphere, highlighting the network's ability to learn both high and low frequency kernels.

An important aspect of the discrete convolution is its finite support domain M , thus allowing the network to share parameters across the image domain. Within the PCConv framework this is achieved by limiting the number of points to convolve across, given some constraint. When convolving over q-space, the spatial dimensions follow the same constraint as in discrete convolutions. That is, for each dimension given a kernel size n , only the n closest points are selected. As the angular components represent sparsely sampled data across a sphere, a kernel size n selects the n closest points, as determined by d r . We refer to the angular kernel size as k q from this point onward. There is an additional constraint with the angular components that only points within a fixed radius d max , such that d r ≤ d max , are included. To achieve this, a binary mask is applied after the pointwise multiplication of the kernel weight with the input feature. The PCConv is a flexible operation that enables convolutions over non-Euclidean spaces. Additionally, the framework allows for kernel weights to be dependent upon coordinates that are not defined in terms of the relative distance between an input and output point. In this work, we additionally introduce a variant of the PCCNN that uses one such encoding. Denoted as 'PCCNN-Bv', this model uses p = [ u i -u j , v i -v j , w i -w j , ρ i , ρ j , d r ] .

y6 = [0.7,0.8,0.7,1.0, 0.0,0.0]

X6 = [0.7,0.8,0.7,1.0,0.6,1.3]

f(x)

g(e; 0),:

h(y)

X20 = [0.7,0.8, 0.7, 1.0,0.3, 0.9]

convolutions across q-space through Equation (2). To perform convolutions using q-space, we first calculate a coordinate embedding e ∈ R 2 LE via

g

7

,

1

(

e

;

θ

)

g

10

,

1

(

e

;

θ

)

g

17

,

3

(

e

;

θ

)

g

24

,

10

(

e

;

θ

)

g

36

,

10

(

e

;

θ

)

The inclusion of this model is motivated by the non-linear relationship between different shells within multi-shell data, which may not be appropriately captured by the relative difference of b-values.

3.3 Global Information

Discrete convolutional layers are computed over a limited kernel size, and therefore have a limited field of view. This affords the convolutional layer computational efficiency, as the kernel size is typically much smaller than the image. However, this comes at the cost of losing global information. Within dMRI, structures within the brain are highly spatially dependent and follow a similar pattern in each individual. Non-local coordinates, and therefore global context, can be incorporated within the PCConv layer through the coordinate embedding. For example, the relative position of the kernel within an image could be incorporated into the coordinate embedding. This idea is inspired by how Vision transformers (ViTs) [12] actively extract global information through correlations between image patches. The availability of global context allows ViTs to show competitive performance in image recognition tasks, where accurate synthesis of semantic meaning more heavily relies on non-local information [19].

In clinical practice, neuroimaging data are routinely aligned to a reference space, such as Talairach space [3]. As such, coordinates within reference spaces are correlated across different subjects, and therefore can be used to provide context for the location of a voxel within the brain. In this work, after normalising the coordinates with respect to the subject brain mask size, we use the centroid of the training patches, as discussed in Section 3.6, as additional coordinates within p . Denoted as an '-Sp' suffix, both 'PCCNN-Sp' and 'PCCNN-Bv-Sp' models within Section 4 have this modification.

3.4 Factorised Convolutions

dMRI is routinely stored in a dense four-dimensional format, with three spatial dimensions and one angular dimension whose coordinates are defined by b . Performing a convolution in this highdimensional space can be computationally expensive. We mitigate this cost by using factorised convolutions [34] within the PCConv framework. For example, in the two-dimensional case, an ( n × m ) kernel would be factorised into two sequential convolutions of size ( n × 1) and (1 × m ) . As in the non-factorised case, the PCConv layer's weights are sampled from one hypernetwork g ( e ; θ ) . Following the approach used in Wang et al. [39], we further factorise the kernel weight tensor from W ∈ R N × O × C × K into two sequential operations: an input convolution W in = R N × O × C followed by a forward projection matrix W out = R C × K , where O is the number of output points.

3.5 Implementation

Since PCConv layers are capable of convolving over both spatial and angular dimensions, the PCCNN models are constructed exclusively of PCConv layers and non-linear activation functions. In practice, this involves a sequence of spatially pointwise (1 × 1 × 1 × k q ) PCConv layers, followed by blocks of parallel convolutions consisting of spatially pointwise and (3 × 3 × 3 × k q ) factorised convolutions. The hyperparameters for each PCCNN, including the number of PCConv layers, can be found in Figure 5 in the Appendix. Within this work, each PCConv layer has its own independent hypernetwork. This configuration allows for significant weight sharing capabilities, as the number of parameters for each hypernetwork does not depend on the kernel size or dimensionality. Each

PCConv layer or residual PCConv block is followed by a rectified linear unit (ReLU), excluding the final layer. Each hypernetwork is composed of two dense layers, each followed by a leaky ReLU with a negative slope of 0 . 1 , and a final dense layer with output size 1 and no subsequent activation. The code for this work is available at github.com/m-lyon/dmri-pcconv.

3.6 Data and Training

Given the high dimensionality of dMRI data, using complete acquisitions as individual training examples is prohibitively expensive in terms of memory. To mitigate this issue, training sets were derived from input image patches X in ∈ R 10 × 10 × 10 × q in , and target image patches X out ∈ R 10 × 10 × 10 × q out where q in and q out denote the number of input and target b-vectors respectively. During training, to sample the angular dimension of the input sets, an initial b-vector q 0 was randomly selected. The next n , up to q in , b-vectors q were chosen using arg max j (min i ( d r ( q i , q j ))) , ∀ i, j ∈ { 0 , 1 , ..., n -1 } . Similarly, this process was repeated to sample the output set up to size q out .

To reduce inter-subject variability, each dMRI dataset was normalised such that 99% of the data lay between [ -1 , 1] . Initially, a normalisation range of [0 , 1] was used; however, this resulted in training instabilities within the hypernetwork. This was likely due to a greater mismatch between the data distribution and the weight initialisation scheme within the hypernetwork, which followed w ∼ U ( -√ 1 k in , √ 1 k in ) , where k in was the number of input features in a given layer. dMRI data from the WU-Minn Human Connectome Project (HCP) [38] were used for training, validation and testing. This data were initially processed with the standard HCP processing pipeline [14], as well as denoised using the 'patch2self' algorithm [13]. Models were trained on twenty-seven subjects from the HCP dataset, while three subjects were used for validation during development. Hyperparameters for the PCCNN were selected through a random grid search with RayTune [20].

Each acquisition within the HCP dataset had shells of b full = { 1000 , 2000 , 3000 } s / mm 2 , each with 90 directions. Training examples contained randomly selected shells b in , b out ∼ U ( b full ) . This sampling scheme trained each network on single-shell and multi-shell examples simultaneously. The input angular dimension size was determined via q in ∼ U ( q sample ) , q sample = { 6 , 7 , ..., 19 , 20 } . This approach allowed each network to learn the data distribution for a range of angular input sizes. To implement this, training examples had a fixed q in = 20 , with zero-filled entries for input size less than 20. Models were trained using 4 NVIDIA A100's with a batch size of 16, and an ℓ 1 loss function, for 200,000 iterations using AdamW [23]. Each PCCNN model took approximately a week to train. Details of pre-processing and training for other models can be found in the Appendix. The model weights used for test inference in Section 4 were the best performing in the validation dataset.

4 Experiments and Results

We evaluated the performance in dMRI angular super-resolution using eight, previously unseen subjects within the HCP dataset. Our results included four variants of the PCCNN, as listed in Table 1. As dMRI data is most commonly used for downstream analyses, we compared the results in commonly used techniques for both single-shell and multi-shell experiments. Our results included other methods where appropriate, such as spherical harmonic (SH) interpolation for single-shell experiments, FODNet [44] for multi-shell FBA, and SR-q-DL [41] for multi-shell NODDI. FOD-Net is a 3D CNN that maps low resolution FOD data to high resolution FOD data. SR-q-DL is a 3D CNN that maps low resolution dMRI data to high resolution NODDI data. RCNN is a 3D recurrent CNN that maps low resolution dMRI data to high resolution dMRI data. When applicable, we also

compared against low angular resolution data. All error metrics were calculated with respect to the full angular resolution HCP dataset.

The reported error values consist of the mean and standard deviation, with distribution statistics generated from the mean performance per test subject. Only voxels within the brain are included

in the calculations, as determined by a brain mask segmentation. The best mean value in a given category is denoted in bold.

4.1 Single-shell dMRI

Within single-shell inference, we evaluated performance across two analyses. Firstly, on raw dMRI data, of which any downstream analysis task would be derived from. We compared performance across all three shells available within the HCP dataset, as well as in three q-space sub-sampling regimes, as outlined in Table 2. Secondly, we compared performance within FBA, where metrics were derived from the inferred dMRI data. For this, we considered the b = 1000 s / mm 2 shell at three sub-sampling regimes.

4.1.1 dMRI Analysis

Table 2 presents the absolute error (AE) of the inferred dMRI data as compared to the high resolution ground truth. For the b = 2000s / mm 2 results, see Table 8 within the Appendix. The PCCNN-Bv-Sp model had the lowest AE in five out of the nine shell and sub-sampling combinations and had the overall best performance when averaged across all data. Despite having approximately a tenth of the number of parameters of the RCNN [26] model, the PCCNN models had lower mean errors across all data regimes except q in = 6 in shells b = { 2000 , 3000 } s / mm 2 .

4.1.2 Fixel Analysis

FBA is an analysis method used to estimate the intra-voxel fibre populations. To generate fixel data [10], response functions were first estimated in an unsupervised manner [9], and Single-Shell 3-Tissue (SS3T) constrained spherical deconvolution (CSD) [8] was performed to obtain white matter (WM) FODs. WM FODs were then normalised [11], and finally segmented into fixels. This workflow used the MRtrix3 [37] package. FOD data is stored as a finite SH expansion, and as such, was compared using the angular correlation coefficient (ACC) [2], given by Equation (5). To compare FBA data, the AE in the apparent fibre density (AFD) within each voxel was calculated by taking the absolute difference between the magnitudes of each inferred fibre population with respect to the ground truth values.

Table 3 summarises the performance in FBA in the low resolution baseline and model based reconstructions compared to the high resolution ground truth, whilst Figure 3 visualises a subset of reconstructed FODs within an axial slice. Here, all PCCNN models performed better than the low resolution baseline in all three sampling schemes in FOD ACC, and had lower AFD AE in the two lowest sampling schemes. The PCCNN models performed competitively against the RCNN. However, in the smallest sub-sampling scheme q in = 6 , or equivalently 6 . 6% of the original number of voxels, the RCNN performed slightly better. Whilst the low resolution data had lower AFD AE at q in = 20 , the PCCNN models had, in the worse case, a relative increased mean error of approximately 6% , compared to 35% increase in the RCNN model. Additionally, the PCCNN models had greater ACC compared to the low resolution baseline across all sub-sampling schemes, suggesting that their use in probabilistic tractography [33] and connectomics [4] would be beneficial.

(a) ROI

(b) High Resolution

(c) Low Resolution

(d) RCNN

(e) PCCNN-Bv-Sp

4.2 Multi-shell dMRI

To evaluate the performance of multi-shell inference, we used a varying q in number of b = 1000s / mm 2 volumes as input, and inferred the remaining known volumes from all three shells as output. The output volumes set contained the output volumes from single-shell inference, in addition to the other two shells. The q in = 6 , 10 , and 20 sub-sampling schemes correspond to an under-sampling rate of 2 . 2% , 3 . 7% , and 7 . 4% respectively. Outlined in the sections below are the performances of the models evaluated within FBA and NODDI metrics.

4.2.1 Fixel Analysis

Fixel data were generated in the same manner as outlined in Section 4.1.2, except where the MultiShell Multi-Tissue (MSMT) algorithm [18] was used in place of SS3T. Table 4 outlines the performance within multi-shell inference. In contrast to single-shell inference, here FOD ACC and AFD AE for PCCNN models were lower than the baseline in all sub-sampling schemes. The minimum difference in AFD AE, as compared to baseline, was 13 . 8% at q in = 20 , whilst the maximum difference was 43 . 04% at q in = 6 . The FOD-Net model had the best performance overall in this experiment, obtaining the best error rate in three out of the six metrics. Despite having approximately 50-fold less parameters, the PCCNN models still perform competitively, and boasts the highest FOD ACC in q in = { 10 , 20 } .

4.2.2 NODDI Analysis

NODDI metrics were generated from input and inferred output volumes using the cuDIMOT [17] package. The full angular resolution derived NODDI metrics were used as ground truth. Data for q in = { 6 , 20 } are presented in Table 5, whilst q in = 10 can be found in Table 11 within the Appendix.

Input, or low resolution, data were not included in this analysis as multi-shell data is required to reconstruct NODDI parameters. The parameters f intra and f iso refer to the estimated volume fraction for intra-cellular and cerebrospinal fluid respectively, whilst orientation dispersion index (OD) is an estimation of the dispersion of white matter fibre bundles.

In this experiment, the PCCNN models perform competitively, yielding the lowest error in f iso and f intra across all three sub-sampling rates. The relative error in OD across different models mirrors that of the FOD ACC found in Table 4, suggesting that the trained models are able to infer data that generalise across different downstream analyses. The worst performing model across all sub-sampling schemes was the SR-q-DL model, as is demonstrated qualitatively by Figure 4. Whilst this model could benefit from the more constrained task of inferring NODDI data directly, the lack of geometric prior information, such as b-vector coordinates, suggests that these additions within the PCCNN models were important to its relatively high performance in this task.

5 Conclusion

The results of the experiments show that the proposed PCConv operation can be used to build a flexible low parameter network to effectively infer high angular resolution raw dMRI data. It does this through appropriate incorporation of task specific geometric properties, as well as suitable parameter sharing via a hypernetwork. We show that inference of raw dMRI data can be used in downstream analysis with moderately high fidelity, and therefore demonstrates the generalisability of the model.

All three of the PCCNN models with additional modifications (PCCNN-Bv, PCCNN-Sp, PCCNNBv-Sp) performed greater or equal to the standard PCCNN in approximately 47% of the experiments presented in this work. Additionally, at least one of the modified PCCNNs outperformed the standard PCCNN in approximately 87% of the experiments. Overall, this suggests that the inclusion of additional domain specific context through the coordinate embedding was beneficial to the models performance.

(a) High Resolution

(b) SR-q-DL

(c) RCNN

(d) PCCNN-Bv

The application of super-resolution in a medical imaging context presents an inherent risk, as deep learning models can be susceptible to hallucinations, or high error rates, when making predictions on out-of-distribution data. Given that this work only includes data from relatively young healthy adults, future work will be needed to validate these methods in a diverse set of diagnostic settings such as datasets with pathologies, wide age ranges, and varying acquisition parameters.

Modern deep learning frameworks rely on highly optimised subroutines to perform standard discrete convolutions. Subsequently, despite having a lower number of parameters, the PCCNNs inference and train times were longer than the other methods presented in this work. This is because the PCConv uses a bespoke implementation that does not benefit from the aforementioned subroutines. This highlights the need for future research into more efficient implementations of the PCConv operation.

In summary, our work further explores the integration of geometric priors into the distinct geometry of dMRI data, and in doing so provides a flexible framework to incorporate arbitrary geometries in different domains and problem formulations.

References

D. C. Alexander, D. Zikic, A. Ghosh, R. Tanno, V. Wottschel, J. Zhang, E. Kaden, T. B. Dyrby, S. N. Sotiropoulos, H. Zhang, et al. Image quality transfer and applications in diffusion mri. NeuroImage , 152:283-298, 2017.

A. W. Anderson. Measurement of fiber orientation distributions using high angular resolution diffusion imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine , 54(5):1194-1206, 2005.

I. Bankman. Handbook of medical image processing and analysis . Elsevier, 2008.

T. E. Behrens and O. Sporns. Human connectomics. Current opinion in neurobiology , 22(1): 144-153, 2012.

J. J. Bouza, C.-H. Yang, D. Vaillancourt, and B. C. Vemuri. A higher order manifold-valued convolutional neural network with applications to diffusion mri processing. In Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28-June 30, 2021, Proceedings 27 , pages 304-317. Springer, 2021.

S. Chatterjee, A. Sciarra, M. Dünnwald, R. V. Mushunuri, R. Podishetti, R. N. Rao, G. D. Gopinath, S. Oeltze-Jafra, O. Speck, and A. Nürnberger. Shuffleunet: Super resolution of diffusion-weighted mris using deep learning. In 2021 29th European Signal Processing Conference (EUSIPCO) , pages 940-944. IEEE, 2021.

T. S. Cohen, M. Geiger, J. Köhler, and M. Welling. Spherical cnns. arXiv preprint arXiv:1801.10130 , 2018.

T. Dhollander and A. Connelly. A novel iterative approach to reap the benefits of multi-tissue csd from just single-shell (+ b= 0) diffusion mri data. In Proc ISMRM , volume 24, page 3010, 2016.

T. Dhollander, R. Mito, D. Raffelt, and A. Connelly. Improved white matter response function estimation for 3-tissue constrained spherical deconvolution. In Proc. Intl. Soc. Mag. Reson. Med , volume 555, 2019.

T. Dhollander, A. Clemente, M. Singh, F. Boonstra, O. Civier, J. D. Duque, N. Egorova, P. Enticott, I. Fuelscher, S. Gajamange, et al. Fixel-based analysis of diffusion mri: methods, applications, challenges and opportunities. Neuroimage , 241:118417, 2021.

T. Dhollander, R. Tabbara, J. Rosnarho-Tornstrand, J.-D. Tournier, D. Raffelt, and A. Connelly. Multi-tissue log-domain intensity and inhomogeneity normalisation for quantitative apparent fibre density. In Proc. ISMRM , volume 29, page 2472, 2021.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

S. Fadnavis, J. Batson, and E. Garyfallidis. Patch2self: Denoising diffusion MRI with selfsupervised learning. arXiv preprint arXiv:2011.01355 , 2020.

M. F. Glasser, S. N. Sotiropoulos, J. A. Wilson, T. S. Coalson, B. Fischl, J. L. Andersson, J. Xu, S. Jbabdi, M. Webster, J. R. Polimeni, et al. The minimal preprocessing pipelines for the human connectome project. Neuroimage , 80:105-124, 2013.

V. Golkov, A. Dosovitskiy, J. I. Sperl, M. I. Menzel, M. Czisch, P. Sämann, T. Brox, and D. Cremers. Q-space deep learning: twelve-fold shorter and model-free diffusion mri scans. IEEE transactions on medical imaging , 35(5):1344-1351, 2016.

D. Ha, A. Dai, and Q. V. Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016.

M. Hernandez-Fernandez, I. Reguly, S. Jbabdi, M. Giles, S. Smith, and S. N. Sotiropoulos. Using gpus to accelerate computational diffusion mri: From microstructure estimation to tractography and connectomes. Neuroimage , 188:598-615, 2019.

B. Jeurissen, J.-D. Tournier, T. Dhollander, A. Connelly, and J. Sijbers. Multi-tissue constrained spherical deconvolution for improved analysis of multi-shell diffusion mri data. NeuroImage , 103:411-426, 2014.

S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah. Transformers in vision: A survey. ACM computing surveys (CSUR) , 54(10s):1-41, 2022.

R. Liaw, E. Liang, R. Nishihara, P. Moritz, J. E. Gonzalez, and I. Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118 , 2018.

R. Liu, F. Lauze, E. Bekkers, K. Erleben, and S. Darkner. Group convolutional neural networks for dwi segmentation. In Geometric Deep Learning in Medical Image Analysis , pages 96-106. PMLR, 2022.

Z. Liu, H. Tang, Y. Lin, and S. Han. Point-voxel cnn for efficient 3d deep learning. Advances in Neural Information Processing Systems , 32, 2019.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

O. Lucena, S. B. Vos, V. Vakharia, J. Duncan, K. Ashkan, R. Sparks, and S. Ourselin. Enhancing fiber orientation distributions using convolutional neural networks. arXiv preprint arXiv:2008.05409 , 2020.

S. Luo, J. Zhou, Z. Yang, H. Wei, and Y. Fu. Diffusion mri super-resolution reconstruction via sub-pixel convolution generative adversarial network. Magnetic Resonance Imaging , 88: 101-107, 2022.

M. Lyon, P. Armitage, and M. A. Álvarez. Angular super-resolution in diffusion mri with a 3d recurrent convolutional autoencoder. In International Conference on Medical Imaging with Deep Learning , pages 834-846. PMLR, 2022.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

S. Mori and J. Tournier. Moving beyond dti: high angular resolution diffusion imaging (hardi). Introduction to diffusion tensor imaging , pages 65-78, 2014.

P. Müller, V. Golkov, V. Tomassini, and D. Cremers. Rotation-equivariant deep learning for diffusion mri. arXiv preprint arXiv:2102.06942 , 2021.

D. A. Raffelt, J.-D. Tournier, R. E. Smith, D. N. Vaughan, G. Jackson, G. R. Ridgway, and A. Connelly. Investigating white matter fibre density and morphology using fixel-based analysis. Neuroimage , 144:58-73, 2017.

M. Ren, H. Kim, N. Dey, and G. Gerig. Q-space conditioned translation networks for directional synthesis of diffusion weighted images from multi-modal structural mri. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VII 24 , pages 530-540. Springer, 2021.

S. Sedlar, T. Papadopoulo, R. Deriche, and S. Deslauriers-Gauthier. Diffusion mri fiber orientation distribution function estimation using voxel-wise spherical u-net. In Computational Diffusion MRI: International MICCAI Workshop, Lima, Peru, October 2020 , pages 95-106. Springer, 2021.

R. E. Smith, J.-D. Tournier, F. Calamante, and A. Connelly. Anatomically-constrained tractography: improved diffusion mri streamlines tractography through effective use of anatomical information. Neuroimage , 62(3):1924-1938, 2012.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems , 33:7537-7547, 2020.

R. Tanno, D. E. Worrall, E. Kaden, A. Ghosh, F. Grussu, A. Bizzi, S. N. Sotiropoulos, A. Criminisi, and D. C. Alexander. Uncertainty modelling in deep learning for safer neuroimage enhancement: Demonstration in diffusion mri. NeuroImage , 225:117366, 2021.

J.-D. Tournier, R. Smith, D. Raffelt, R. Tabbara, T. Dhollander, M. Pietsch, D. Christiaens, B. Jeurissen, C.-H. Yeh, and A. Connelly. Mrtrix3: A fast, flexible and open software framework for medical image processing and visualisation. Neuroimage , 202:116137, 2019.

D. C. Van Essen, S. M. Smith, D. M. Barch, T. E. Behrens, E. Yacoub, K. Ugurbil, W.-M. H. Consortium, et al. The WU-Minn human connectome project: an overview. Neuroimage , 80: 62-79, 2013.

S. Wang, S. Suo, W.-C. Ma, A. Pokrovsky, and R. Urtasun. Deep parametric continuous convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2589-2597, 2018.

Z. Wang, J. Chen, and S. C. Hoi. Deep learning for image super-resolution: A survey. IEEE transactions on pattern analysis and machine intelligence , 43(10):3365-3387, 2020.

C. Ye, Y. Qin, C. Liu, Y . Li, X. Zeng, and Z. Liu. Super-resolved q-space deep learning. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part III 22 , pages 582-589. Springer, 2019.

C. Ye, Y. Li, and X. Zeng. An improved deep network for tissue microstructure estimation with uncertainty quantification. Medical image analysis , 61:101650, 2020.

S. Yin, Z. Zhang, Q. Peng, and X. You. Fast and accurate reconstruction of HARDI using a 1D encoder-decoder convolutional network. arXiv preprint arXiv:1903.09272 , 2019.

R. Zeng, J. Lv, H. Wang, L. Zhou, M. Barnett, F. Calamante, and C. Wang. Fod-net: A deep learning method for fiber orientation distribution angular super resolution. Medical Image Analysis , 79:102431, 2022.

H. Zhang, T. Schneider, C. A. Wheeler-Kingshott, and D. C. Alexander. Noddi: practical in vivo neurite orientation dispersion and density imaging of the human brain. Neuroimage , 61(4): 1000-1016, 2012.

S. Zhang, H. Tong, J. Xu, and R. Maciejewski. Graph convolutional networks: a comprehensive review. Computational Social Networks , 6(1):1-23, 2019.

A Appendix

A.1 Model Hyperparameters

Key

PCCNN

PCConvBlock

ndu

PCConv

bvec input

bvec input

dMRI input

ReLU

↑

C1 = 256,Wh =(64,64),L = 15

bvec

bvec

bvec

bvec

PCConvBlock

bvec input

bvecinput

(1 × 1 × 1 × 20)

↑

(3× 3 × 3 × 20)

↑

↑

↑

μndino / 1ndu1

C1=128,C3=128,Wh=(128,32),L=5

bvec input

bvec input

↑

↑

Concatenation

C1=128,C3=128,Wh=（128,32),L=5

bvec input

bvecinput

个

↑

C1=128,C3=128,Wh=（128,32),L=5

bvec input

bvecinput

↑

↑

C1 =160,Wh =(32,256),L =15

bvec input

bvec output

↑

↑

C1=128,C3=32,Wh=(16,16),L=5

bvec output

bvec output

↑

↑

C1 =128,C3=32,Wh =(16,16),L = 5

bvec output

bvec output

↑

↑

C1 =128,C3 = 32,Wh =(16,16),L = 5

bvec output

bvec output

↑

↑

C1 = 128,C3 = 32,Wh =(16,16),L = 5

bvecoutput

bvec output

↑

C=128,C3=32,Wh=(16,16),L=5

bvec output

bvec output

↑

↑

C1=64,Wh=(32,64),L=15

bvecoutput

bvecoutput

↑

↑

C1=1,Wh=(256,256),L=10

dMRIoutput

A.2 RCNN Data and Training

Training parameters, the hardware used, and the data sampling scheme used to train the RCNN were the same as outlined within Section 3.6. Training took approximately 3 days to complete.

A.3 FOD-Net Data and Training

In this work we modify the training scheme used within Zeng et al. [44] to more closely match the PCCNN training methodology. Specifically, each training example input was a low angular resolution FOD volume generated from single-shell dMRI data with b-value b in and angular dimension q in determined via

Training parameters such as number of iterations, batch size, learning rate, hardware used, and optimisation method remain consistent with those detailed in Section 3.6. Model training took approximately 6 hours to complete.

A.4 SR-q-DL Data and Training

The data sampling used to train the SR-q-DL for this work differs from that outlined in Ye et al. [41]. As in Section A.3, this was done to match the PCCNN training scheme. Here, each training example input was a low angular resolution single-shell dMRI dataset with b-value b in and angular dimension q in determined via

Training parameters such as number of iterations, batch size, learning rate, hardware used, and optimisation method remain consistent with those detailed in Section 3.6. Model training took approximately 30 minutes to complete.

A.5 Spherical Harmonic Interpolation

SH interpolation of single-shell dMRI data followed the same procedure as outlined in Lyon et al. [26]. Briefly, SH coefficients for each spatial voxel within the low angular resolution dataset were fit using the pseudo-inverse least squares method using a spherical harmonic order of 2. These derived SH coefficients were then used to calculate the interpolated spatial voxels for the target b-vectors.

A.6 Single-shell dMRI PSNR and MSSIM

Within Table 6 we present the peak signal-to-noise ratio (PSNR) of dMRI intensity within the singleshell experiment. Similarly, Table 7 outlines the mean structural similarity index (MSSIM) of dMRI intensity within the same experiment.

A.7 Additional Single-shell dMRI Analysis

This section provides performance metrics of models in single-shell inference using b = 2000s / mm 2 data. Specifically, Table 8 outlines the AE of dMRI intensity, whilst Table 9 and Table 10 outline the PSNR and MSSIM of dMRI intensity respectively.

A.8 Angular Correlation Coefficient

The angular correlation coefficient (ACC) is given by

where α l,m and β l,m denote the SH coefficients for two FODs with degree l and order m .

A.9 Additional NODDI Analysis

Table 11 presents results from the NODDI experiment within the q in = 10 sub-sampling scheme.

A.10 Ablation Studies

This section describes results from various ablation studies performed with the PCCNN model as a baseline. Each of the models were used to infer multi-shell data from single-shell input data across three separate shells. The following ablations were performed:

No Fourier features : The Fourier feature mapping γ ( p m ) , as described in Equation (3.2) were removed from the PCConv operation.

No b-vectors : The b-vector coordinates e b were removed from the PCConv operation, thus reducing the operation to only convolve using spatial based co-ordinates and the b-value difference ρ i -ρ j .

d max = 1 4 π : Only points on the q-space sphere at a spherical distance of d r ≤ 1 4 π away from the centre of the kernel were included within the PCConv operation.

d max = 1 8 π : Only points on the q-space sphere at a spherical distance of d r ≤ 1 8 π away from the centre of the kernel were included within the PCConv operation.

Each modification was applied to all layers within the respective model. The baseline PCCNN model had a d max = 1 . The results of these ablations are shown in Table 12.

Results within Table 12 clearly demonstrate that including b-vector information within the PCConv operation is essential for the model to perform well, as the AE significantly increases within the 'No b-vectors' experiment, as compared to the baseline PCCNN. The two d max ablation results also show that removing information pertaining to the angular co-ordinate system readily reduces performance of the model. It is noteworthy that a d max = 1 8 π has significantly higher AE compared to the 'No b-vectors' variant. This is likely due to the greatly reduced number of points available for each kernel operation within the d max = 1 8 π experiment. This is in contrast to the 'No b-vector' variant which uses all available points within the operation, but with no angular distance information.

The inclusion of Fourier features within the PCConv operation improves performance in two of the shells, but decreases performance in one. This suggests that the inclusion of the Fourier features may improve the ability of the model to generalise to different shells, however this effect is not as apparent as the inclusion of b-vector information. The limited improvement in this case may be twofold.

First, due to the increased complexity of the model, as the co-ordinate embedding dimensionality would increase by a factor of 2 L . Second, whilst Fourier features were shown to vastly improve performance when predicting image intensity from co-ordinates in MLP networks [35], the Fourier feature mapping here is only applied to the co-ordinates within the domain of the kernel, which is a much smaller subset of the entire image domain.

A.11 Noisy Data

This section provides additional results from an experiment using dMRI data that were not denoised prior to training or inference. This analysis includes an additional model, denoted the 'Q-Space CGAN' as detailed within Ren et al. [31]. The weights for this model were trained on noisy HCP data and were obtained from the original authors of the work. The other models presented in this section were trained exactly as outlined previously, with the only difference being that the data were not denoised prior to training.

Figure 6 shows an axial slice of dMRI mean absolute error (MAE) in three models when inferring multi-shell data. Whilst this is only a qualitative example, the Q-Space CGAN has significantly higher MAE across the slice as compared to the other two models. This is likely due to lack of diffusion weighted context provided to the Q-Space CGAN as input.

Table 13 presents results from the multi-shell AFD experiment, as shown in Section 4.2.1 using models trained with non-denoised data. These results follow similar trends as present in Table 4, whereby the FOD-Net generally performs best in AFD AE whilst the PCCNN variants perform best within FOD ACC across sub-sampling schemes.

(a) RCNN

(b) PCCNN-Bv-Sp

(c) Q-space CGAN

1500

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors utilize a parametric continuous convolution network to capitalize on the geometry of diffusion MRI. They enhance prior work (PCConv), integrating domain context and global information. They show that they obtain accurate high resolution dMRI using merely sparsely sampled data and demonstrate performance on two subsequent clinical tasks.\n\nSTRENGTHS:\nTypical CNNs do not utilize the geometry in dMRI, which opens an opportunity. The approach combines existing ideas to seize this opportunity by creating a framework that convolves over both dense grid data and sparsely sampled spherical data concurrently, augmented by context/domain information. I consider analyzing two downstream clinical tasks a considerable strength of the paper. Moreover, the paper is clearly written and provides a proper mathematical description of the method.\n\nWEAKNESSES:\nWhile the paper is clearly written, it is sometimes also hard to understand due to the complexity of the medical context and diffusion MRI terminology/jargon. This makes the motivation and approach of the work somewhat vague. I would suggest elaborating a bit more on the structure of the data (i.e. in paragraph 2 of the introduction) and explaining how the series, b-vector, and multi-shell are connected and what they mean/measure.\nSecondly, it only becomes clear quite late that you opt for super-resolution and inferring the dMRI values from synthetically under-sampled data. I missed why this is a clinically relevant problem. Is data typically under-sampled in clinical practice? Will it speed up the acquisition? Some clinical context would be helpful.\nMoreover, I feel that the results vary quite a lot throughout the experiments. Sometimes they are better than the baseline, other times they are not. While the results are as they are, I miss a proper discussion on these results to discuss why this is the case.\n\nQUESTIONS:\n1.\tProvide more explanation on the structure of the data. Specifically, in the second paragraph of the introduction, clarify the relationship between the series, b-vector, and multi-shell, including what they measure.\n2.\tMake it clear earlier that the goal is super-resolution and deducing the dMRI values from synthetically under-sampled data. Explain why this is relevant in a clinical context. Is under-sampling common in practice? Would this approach speed up acquisition?\n3.\tThink about adding a more detailed explanation of the baseline methods. It's clear that the methods are convolutional, but how do they manage the angular data? Make it obvious what sets your work apart.\n4.\tInclude a thorough discussion of the varying results. While it's understood that results can differ, a deeper analysis could shed light on why the outcomes are sometimes better than the baseline and sometimes not. This would enhance the readers' understanding. E.g. , why do the modifications sometimes work but not always?\n5.\tHow does the FOD model surpasses others in the experiments discussed in section 3.2?\n6.\tTable 5 / Figure 3: The performance differences appear minor. Do these slight enhancements have an impact in a clinical setting? For instance, could these marginal gains influence patient treatments or clinical outcomes?\n\nLIMITATIONS:\nThere is no limitations section present in the manuscript. A suggestion for a limitation is that the method was developed on one data set. There is, however, a very large variation between datasets, institutions, etc, in medical imaging. This also holds for MRI. It would be good to show or at least discuss the performance on other datasets or discuss whether this is a study limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

comments: SUMMARY:
The paper presents a learning-based method for q-space interpolation in diffusion MRI.  The approach constructs particular convolution operators that lend themselves to the structure of the space to be interpolated, then learn convolutions from examples.  Experiments show the learned interpolation substantially outperforms spherical harmonic interpolation and learned methods that use more generic convolutions less appropriate to the problem.

STRENGTHS:
The method is well thought through and seems appropriate to the application domain.

Results are strong for the specific problem at hand.

WEAKNESSES:
Should really use a 3D q-space signal representation as a baseline, e.g. https://pubmed.ncbi.nlm.nih.gov/23587694/.

While the work is nicely formulated, it is a very niche from the point of view of a machine learning conference like NeurIPS.  I struggle to see how the methods developed have broader interest other than to researchers/users of the particular MRI modality under consideration.  The authors make no attempt to explain why this would be relevant to NeurIPS rather than an imaging conference.

QUESTIONS:
How does this relate to other patch-based super-resolution techniques for diffusion MRI?  Well known works on this such as https://pubmed.ncbi.nlm.nih.gov/23791914/ and image-quality transfer (https://pubmed.ncbi.nlm.nih.gov/28263925/, https://pubmed.ncbi.nlm.nih.gov/33039617/) are not mentioned at all, which is a bit worrying.

Do the technical contributions have any more general utility beyond the specific application to diffusion MRI?  Why might they be of broad interest at a machine learning conference?

LIMITATIONS:
Nothing to add
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a parametric continuous convolution (PCConv) framework for Diffusion MRI (dMRI). The PCConv convolves across both spatial and angular dimensions of dMRI data. Meanwhile, the authors introduce a Fourier feature mapping, global coordinates, and domain speciﬁc context into PCConv. Experiments on PCCNN and three variants (PCCNN-Bv, PCCNN-Sp, PCCNNBv-Sp) show that the proposed method is competitive with less parameters.\n\nSTRENGTHS:\n1. Extensive experiments on both single-shell and multi-shell demonstrate that the proposed PCCNN is competitive with previous methods.\n2. This paper is written clearly, and the proposed method is reasonable.\n\nWEAKNESSES:\n1. The PCCNN has been proposed in previous work [31]. The authors mainly apply the PCCNN in MRI tasks. Meanwhile, the proposed Factorised Convolutions (Section .2.5) is also designed in [31]. Therefore, the contribution and novelty of this paper are not enough.\n2. The authors design four models: PCCNN, PCCNN-Bv, PCCNN-Sp, and PCCNNBv-Sp. However, in the comparisons, PCCNN-Bv-Sp combines all additions and does not achieve the best results. The performance of the four models is close to the comparison results. That is, the proposed additional improvements have little effect. The authors should demonstrate the effectiveness of the additional improvements.\n3. The paper lacks an ablation study to indicate the effectiveness of each component. For example, the impacts of the angular kernel size k_q and the ﬁxed radius d_max for PCCNN. I think these are some important hyperparameters.\n\nQUESTIONS:\nIn Line 314, the authors indicate that this method's inference and train times are longer than previous methods. Please give some explanation for this phenomenon, since the model size of PCConv is much smaller than other models.\n\nLIMITATIONS:\nThe authors do not discuss its limitations or potential negative societal in a separate section. But some limitations (e.g., running time) are mentioned in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
The paper with title: Spatio-Angular Convolutions for Super-resolution in Diffusion MRI applies established fully parametric continuous convolution network (PCCNN) to diffusion SR, demonstrating the potentials.

STRENGTHS:
1. This paper proposes a practical method to apply parametric continuous ConvNet to diffusion SR. Q-space in Diffusion MRI is a great example or application of continuous conv. How to leverage the diffusion gradient direction has been an open problem in the community.
2. The authors present analysis on various diffusion coefficients that are clinically useful.

WEAKNESSES:
1. The authors propose a few variations to PCCNN (i.e., BV, SP), however I did not see the clear improvements from the results what are the roles of BV and SP, they are at similar values.
2. The paper needs more comparisons to existing methods, now only RCNN in many experiments.
3. From my viewpoint, this paper misses an important ablation studies, the authors should evaluate the same architecture without the input of diffusion gradient direction and see how it contributes.
4. Needs more visual comparisons, and examples of SR results, very critical.
5. The novelty of this work is quite limited, since the core idea (PCCNN) is adopted from previous work, but application wise, its a fair application paper.

QUESTIONS:
1. Can this work be applied to other diffusion MRI tasks? for example diffusion MRI denoising.

LIMITATIONS:
1. Minor improvements over existing works.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Background for ML audiences**: Multi-shell diffusion MRI (dMRI) is a 6D (3 dims of space + 2 angular dimensions + 1 radial dimension) imaging modality where each 3D voxel contains (potentially concentric) spherical signals which correlate with local white matter properties within the brain. In research settings, each voxel-sphere is sampled densely to recover underlying white matter pathways, but clinical settings often acquire only a few angles for speed in critical settings (e.g. stroke monitoring).\n\n**Summary**: Submission 13443 presents an architecture for processing dMRI images and aims to perform angular super-resolution. To do so, it applies existing parametric continuous convolutional nets (i.e. continuous (x,y,z,r,phi,theta)-location conditioned networks) and adds better positional embeddings and positional conditioning. Experimentally, it evaluates angular super-resolution performance by means of plain predicted image-based performance and downstream analysis such as NODDI parameter fitting.\n\nSTRENGTHS:\n- To my knowledge, fully continuous convolutions for dMRI processing has not been attempted before and the idea is promising due to the extremely nonlinear nature of the 5D spatial and angular images.\n- The submission performs a reasonably thorough ablation of its various moving components.\n- The analysis of downstream applications of dMRI (fixel analysis, NODDI fitting) is well considered as an evaluation strategy of angular super-resolution.\n\nWEAKNESSES:\n### (A) Inadequate evaluation with lower methodological contributions:\n\nThe proposed framework is an application of continuous convolutions to dMRI images alongside combinations with factorized convolutions and Fourier features, the latter of which has been used extensively with coordinate-based hypernetworks and equivalently for [continuous convolutions](https://arxiv.org/abs/2102.02611). In my opinion, this is fine as dMRI processing is a non-trivial and non-traditional application. However, the presented experiments require significant improvements in depth to make this argument. For example,\n\n- Only ~**40 subjects** were used for training/validation/testing with final evaluation only performed on 8 subjects. However, the publicly and freely accessible dataset used in this paper (WU-Minn HCP) has diffusion MRI from 1065 subjects. Why was the entire dataset (or at least a much larger portion) not used? In my opinion, this is a non-trivial limitation as this sample size is small and the evaluation on 8 subjects only reveals no clear results trend amongst all of the various ablations proposed.\n- The paper claims to improve PCCNN networks in performance and computational efficiency with the addition of factorized convolutions and Fourier feature embeddings. However, the gains specific to factorized convolutions and FF embeddings do not appear to be quantified in an **ablation study**, please do so as they are central claims.\n- There is no mention of how the **baselines** were tuned for these experiments. For example, was the degree of the SH interpolation tuned on validation data?\n\n### (B) Potentially suboptimal modeling choices:\n\n- Section 2.4 introduces “**global conditioning**” by means of concatenating coordinates of a reference neuroimaging template. However, this is quite confusing as dMRI are routinely *not* registered to a template space as that requires non-trivial reorientation of each voxel-sphere after registration, so I do not see why this would be helpful and the ablation results are inconsistent and hard to interpret. Further, it is unclear what is technically meant by L159. Please clarify these aspects.\n- It is unclear and unstated why the proposed framework uses **rotationally-invariant kernels**. The main advantage of the proposed work over previous attempts (listed below) is that the hypernetwork can be quite expressive for highly nonlinear data, but rotationally-invariant kernels reduce this to a significant extent. Why was this choice made?\n- The paper performs **zero-filling** of missing angles in its experiments (L208) which in my opinion is not well motivated as 0 diffusivity is a physical value and not a masked missing value to be estimated. As continuous convolutions naturally allow for handling missing values, I do not see why zero-filling is necessary, please clarify.\n- All experiments undergo **denoising** via Patch2Self which would typically significantly reduce the high-frequency content of the sparsely-sampled image which could potentially be useful for nonlinear neural networks. Why is this preprocessing used?\n- As the proposed method is not roto-translation equivariant by design (as in other dMRI processing networks), it is largely data-driven. However, only 27 training images are used and there is no mention of data augmentation which would help with generalization. Was **data augmentation** not used?\n\n### (C) Missing existing work on this topic:\n\nCurrently, most experiments in the paper use 1-2 baseline methods as comparisons, which would be fine if the submission was tackling a recently-formed application. However, there is extensive work on topics relevant to the submission such as dMRI convolutions, angular super-resolution, and regressing microstructural coefficients which are detailed below. It would improve the paper for it to contextualize its contributions w.r.t. existing work and (if possible) add one or more of the most relevant baselines to each experiment.\n\n#### w.r.t. dMRI convolution and/or q-space processing:\nThe current version of this paper presents convolutions for dMRI data as a fundamentally new and unaddressed problem. However, there are several lines of existing work that tackle developing convolutions for dMRI exactly. This includes equivariant and geometrically-motivated work: \n- Using tensor field networks: https://arxiv.org/abs/2102.06942\n- Using separable kernels: https://openreview.net/forum?id=7S1l2zzUZFI\n- Using equivariant graph convolutions: https://arxiv.org/abs/2304.06103\n- Using manifold-valued convolutions: https://link.springer.com/chapter/10.1007/978-3-030-78191-0_24\n- And for completeness, there exist ad-hoc methods which concatenate neighboring voxel-spheres channel-wise such as https://hal.science/hal-02946371/document.\n\nIn my opinion, it would be good to acknowledge and discuss the advantages and disadvantages of the proposed framework w.r.t. one or more of these methods. For example,\n- A potential disadvantage: the proposed method is not equivariant to the underlying symmetries of the data and is purely data-driven.\n- A potential advantage: the above methods cannot directly produce outputs at angular coordinates not in the inputs without interpolation and/or model-fits, whereas the proposed method can as it uses a hypernetwork.\n\n#### w.r.t. dMRI super-resolution\n\nDMRI angular super-resolution has had several previous attempts in the literature that could be discussed and/or compared against when possible. For example, [DeepDTI](https://www.sciencedirect.com/science/article/pii/S1053811920305036) uses 6 angular samples (w/ universally co-acquired structural images) to super-resolve the diffusion tensor. Further, while the paper uses continuous convolutions via hypernetworks both spatially and spherically, the continuous mechanism is not used spatially as all inputs and outputs lie on a regular spatial grid. As a result, it is similar to [this paper](https://arxiv.org/abs/2106.13188) which also uses a q-space-conditioned hypernetwork with spatial gridded convolutions for dMRI super-resolution. Lastly, while SH interpolation is compared against as a non-deep learning baseline, one would typically use SHORE interpolation in practice.\n\n#### w.r.t. regressing dMRI coefficients:\n\nThere are some works that directly regress dMRI scalars (such as NODDI in this paper) which would also be good to discuss and/or compare against if possible such as:\n- https://onlinelibrary.wiley.com/doi/full/10.1002/mrm.27568\n- https://link.springer.com/chapter/10.1007/978-3-031-16431-6_15\n- https://arxiv.org/abs/2207.00572\n- https://link.springer.com/chapter/10.1007/978-3-031-16431-6_11\n\n### Minor comments\n- Why is the related work in the methods section? That choice seems out of place.\n- Please add PSNR and SSIM as evaluation measures on top of AE.\n\nQUESTIONS:\nI have merged my questions and suggestions with the weaknesses (mainly weaknesses A and B), please see above. If the rebuttal adequately addresses weaknesses A and B I would be happy to raise my score.\n\nLIMITATIONS:\nThe paper does not list any limitations beyond long compute times. In my opinion, this is insufficient as angular super-resolution in biomedical applications is a high-risk application as it may introduce hallucinations or incorrectly predict angular samples for unseen populations. Further, the paper only studies 40 “normal” relatively-young adults, whereas angular super-resolution would be most beneficial in time-sensitive clinical settings with lesions and strokes. These limitations should be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-23"}