HUMUS-Net: Hybrid Unrolled Multi-scale Network Architecture for Accelerated MRI Reconstruction
Authors: Zalan Fabian
Berk Tinaz
Mahdi Soltanolkotabi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vision Transformers
MRI reconstruction
inverse problems
deep learning
fastMRI

Abstract: In accelerated MRI reconstruction, the anatomy of a patient is recovered from a set of undersampled and noisy measurements. Deep learning approaches have been proven to be successful in solving this ill-posed inverse problem and are capable of producing very high quality reconstructions. However, current architectures heavily rely on convolutions, that are content-independent and have difficulties modeling long-range dependencies in images. Recently, Transformers, the workhorse of contemporary natural language processing, have emerged as powerful building blocks for a multitude of vision tasks. These models split input images into non-overlapping patches, embed the patches into lower-dimensional tokens and utilize a self-attention mechanism that does not suffer from the aforementioned weaknesses of convolutional architectures. However, Transformers incur extremely high compute and memory cost when 1) the input image resolution is high and 2) when the image needs to be split into a large number of patches to preserve fine detail information, both of which are typical in low-level vision problems such as MRI reconstruction, having a compounding effect. To tackle these challenges, we propose HUMUS-Net, a hybrid architecture that combines the beneficial implicit bias and efficiency of convolutions with the power of Transformer blocks in an unrolled and multi-scale network. HUMUS-Net extracts high-resolution features via convolutional blocks and refines low-resolution features via a novel Transformer-based multi-scale feature extractor. Features from both levels are then synthesized into a high-resolution output reconstruction. Our network establishes new state of the art on the largest publicly available MRI dataset, the fastMRI dataset. We further demonstrate the performance of HUMUS-Net on two other popular MRI datasets and perform fine-grained ablation studies to validate our design.

HUMUS-Net: Hybrid Unrolled Multi-Scale Network Architecture for Accelerated MRI Reconstruction
Zalan Fabian
Berk Tinaz
Dept. of Electrical and Computer Engineering University Southern California Los Angeles, CA afbbanuuucceedu
Dept. of Electrical and Computer Engineering University of Southern California Los Angeles, CA tinazqusc.edu
Mahdi Soltanolkotabi Dept. of Electrical and Computer Engineering University Southern California Los Angeles, CA soltanol@usc. edu
Abstract
In accelerated MRI reconstruction, the anatomy of patient is recovered from set under-sampled and noisy measurements. Deep learning approaches have been proven be successful in solving this ill-posed inverse problem and are capable of producing very high quality reconstructions. However, current architectures heavily rely convolutions, that are content-independent and have difficulties modeling long-range dependencies in images. Recently, Transformers, the workhorse of contemporary natural language processing. have emerged as powerful building blocks for multitude of vision tasks. These models split input images into non- overlapping patches, embed the patches into lower-dimensional tokens and utilize a self-attention mechanism that does not suffer from the aforementioned weak- nesses of convolutional architectures. However, Transformers incur extremely high compute and memory cost when 1) the input image resolution is high and 2) when the image needs to be split into large number of patches to preserve fine detail information, both of which are typical in low-level vision problems such as MRI reconstruction, having a compounding effect. To tackle these challenges, propose HUMUS-Net, hybrid architecture that combines the beneficial im- plicit bias and efficiency of convolutions with the power of Transformer blocks unrolled and multi-scale network. HUMUS-Net extracts high-resolution features via convolutional blocks and refines low-resolution features via a novel Transformer-basee multi-scale feature extractor. Features from both levels then synthesized into high-resolution output reconstruction. Our network establishes new state of the art the largest publicly available MRI dataset, the fastMRI dataset. We further demonstrate the performance of HUMUS-Net on two other popular MRI datasets and perform fine-grained ablation studies to validate design.
1 Introduction
Magnetic resonance imaging (MRI) a medical imaging technique that uses strong magnetic fields to picture the anatomy and physiological processes of the patient. MRI is one of the most popular imaging modalities it is non-invasive and doesn't 't expose the patient harmful ionizing radiation. The MRI scanner obtains measurements of the body in the spatial frequency domain also called
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

k-space. The data acquisition process is MR is often time-consuming Accelerated MRI [Lustig al., 2008] addresses this challenge by undersampling in the k-space domain, thus reducing the time patients need to spend in the scanner. However, recovering the underlying anatomy from undersampled measurements is an ill-posed problem (less measurements than unknowns) and thus incorporating some form of prior knowledge is crucial in obtaining high quality reconstructions. Classical MRI reconstruction algorithms rely on the assumption that the underlying signal is sparse in some transform domain and attempt to recover a signal that best satisfies this assumption in technique known as compressed sensing (CS) [Candes et al., 2006, Donoho, 2006]. These classical CS techniques have slow reconstruction speed and typically enforce limited forms of image priors.
With the emergence of deep learning (DL), data-driven reconstruction algorithms have far surpassed CS techniques (see Ongie et al. [2020] for an overview). DL models utilize large training datasets to extract flexible, nuanced priors directly from data resulting in excellent reconstruction quality. In recent years, there has been flurry of activity aimed at designing DI, architectures tailored to the MRI reconstruction problem. The most popular models are convolutional neural networks (CNNs) that typically incorporate the physics of the MRI reconstruction problem, and utilize tools from mainstream deep learning (residual learning, data augmentation, self-supervised learning). Comparing the performance of such models has been difficult mainly due to two reasons. First, there has been a large variation in evaluation datasets spanning different scanners, anatomies, acquisition models and undersampling patterss rendering direct comparison challenging. Second, medical imaging datasets are often proprictary due to privacy concerns, hindering reproducibility.
More recently, the fastMRI dataset [Zbontar et al., 2019], the largest publicly available MRI dataset, has been gaining ground as a standard benchmark to evaluate MRI reconstruction methods. An annual competition, the fastMRI Challenge [Muckley et al., 2021] I, attracts significant attention from the machine learning community and acts a driver of innovation in MRI reconstruction. However, over the past years the public leaderboard has been dominated by a single architecture, the End-to-End VarNet [Sriram et al., 2020] with most models concentrating very closely around the same performance metrics, hinting at the saturation of current architectural choices.
In this work, we propose HUMUS-Net: Hybrid, Unrolled, MUlti-Scal network architecture for accelerated MRI reconstruction that combines the advantages of well-established architectures in the field with the power of contemporary Transformer-based models. We utilize the strong implicit bias of convolutions, but also address their weaknesses, such as content-independence and inability to model long-range dependencies, by incorporating a novel multi-scale feature extractor that operates over embedded image patches via self-attention. Moreover, we tackle the challenge of high input resolution typical in MRI by performing the computationally most expensive operations on extracted low-resolution features. HUMUS-Ne Nett establishes new state of the art in accelerated MRI reconstruction on the largest available MRI knee dataset. At the time of writing this paper, HUMUS-NG -et is the only Transformer- based architecture on the highly competitive fastMRI Public Leaderboard, Our results are fully reproducible and the source code is available online" 2
2 Background
2.1 Inverse Problem Formulation of Accelerated MRI Reconstruction
An MR scanner obtains measurements of the patient anatomy in the frequency domain, also referred to as k-space. Data acquisition is performed via various receiver coils positioned around the anatomy being imaged, each with different spatial sensitivity. Given a total number of N receiver coils, the measurements obtained by the ith coil can be written as
ky FS,x" + zz, i= 1,
where x= E C is the underlying patient anatomy of interest, S₁ is diagonal matrix that represents the sensitivity map of the ith coil, F is a multi-dimensional Fourier-transform, and z; denotes the measurement noise corrupting the observations obtained from coil i. We use k (k.,..., kx) as
At the time of writing this paper, the number one model the leaderboard is AeRS-Net, however the complete architecture/trining details as well as code this method are not available to the public and we were unable to reproduce their results hased on the limited available information. "Code: https:/ h/tp::///t/ //gi thub. com/ homm/MttFFLDD HHULLDSSS/UU--EEEEEEEESSSSSSSS Net.

---

shorthand for the concatenation of individual coil measurements and æ (#1,.... EN) as the corresponding image domain representation after inverse Fourier transformation.
Since MR data acquisition time is proportional to the portion of k-space being scanned, obtaining fully-sampled data is time-consuning. Therefore, in accelerated MRI scan times are reduced by undersampling in k-space domain. The undersampled k-space measurements from coil i take the form
k; = MK; i == i.,,.,
where M is diagonal matrix representing the binary undersampling mask, that has 0 values for all missing frequency components that have not been sampled during accelerated acquisition.
The forward model that maps the underlying anatomy to coil measurements can be written concisely as h = A (x*), where (-) is the linear forward mapping and & is the stacked vector of all undersampled coil measurements. Our target is to reconstruct the ground truth object x from the noisy, undersampled measurements k. Since have fewer observations than variables to recover, perfect reconstruction in general is not possible. In order to make the problem solvable, prior knowledge the underlying object is typically incorporated in the form of sparsity in some transform domain. This formulation, known as compressed sensing [Candes et al., 2006, Donoho, 2006], provides classical framework for accelerated MRI reconstruction [Lustig al., 2008]. In particular, the above recovery problem can be formulated as a regularized inverse problem
i = arg min A
K
+ R(x)),
(2.1)
where R(-) is regularizer that encapsulates prior knowledge on the object, such sparsity in some wavelet domain.
2.2 Deep Learning-based Accelerated MRI Reconstruction
More recently, data-driven deep learning-basee based algorithms tailored to the accelerated MRI recon- struction problem have surpassed the classical compressed sensing baselines. Convolutional neural networks trained on large datasets have established new state of the art in many medical imaging tasks. The highly popular U-Net [Ronneberger et al., 2015] and other similar encoder-decoder architectures have proven to be successful in a range medical image reconstruction [Hyun et al., 2018, Han and Ye, 2018] and segmentation [Çiçek et al., 2016, Zhou et al., 2018] 18] problems. In the encoder path, the network learns to extract a of deep, -iwddmmeniioall features from images via series of convolutional and downsampling operations. These concise feature representations are then gradually upsampled and filtered in the decoder to the original image dimensions. Thus the network learns hierarchical representation over the input image distribution.
Unrolled networks constitute another line of work that has been inspired by popular optimization algorithms used to solve compressed sensing reconstruction problems. These deep learning models consist of series of sub-networks, also known as cascades, where each sub-network corresponds to an unrolled iteration of popular algorithmss such as gradient descent [Zhang and Ghanem, 2018] or ADMM [Sun et al., 2016]. In the context of MRI reconstruction, one can view network unrolling as solving a sequence of smaller denoising problems, instead of the complete recovery problem in one step. Various convolutional neural networks have been employed in the unrolling framework achieving excellent performance in accelerated MRI reconstruction [Putzky et al., 2019, Hammernik convolutional
et al., 2018, 2019]. Ver-et [Sriram et al., 2020] is the current state-of-the-a model on the fastMRI dataset. V2r-et transforms the optimization problem in (2.1) to the k-space domain and unrolls the gradient descent iterations into T cascades, where the /th cascade represents the computation
- +kk*)
1-+1- E
(2.2)
where &e the estimated reconstruction the k-space domain at cascade 4, jı is a learnable step size parameter and G(-) is learned mapping representing the gradient of the regularization term in 12. The first term also known as data consistency (DC) term as it enforces the consistency of the estimate with the available measurements.
3

---

Fu
Hii-Re Fonhur Extraction
Hi-Res Imsaee
Reconstraction
Noisy image
Denoised Image
Low-Res Festure Eatmetion
P
MUST
Fv
Fyy
Pr: low.res flotures
Fyt deep
RSTB-3
RSTBB
skkn opuartatiion
RSTB-D
RSTB-U
ZERye repromative
sitp conaedia
RSTB-B
Figure I: Overview of the HUMUS-Block architecture. First, we extract high-resolution features FH from the input noisy image through a convolution layer Sи- Then, we apply a convolutional feature extractor f obtain low-resolution features and process them using a maansfoommr-coovolutiin hybrid multi-scale feature extractor. The shallow, high-resolution and deep, low-resolution features then synthesized into the final high-resolution denoised image.
3 Related Work
Transformers in Vision Vision Transformer (ViT) [Dosovitskiy et al., 2020], a fully non- convolutional vision architecture, has demonstrated state-of-the-art performance on image clas- sification problems when pre- pre-trained on large-scale image datasets. The key idea of ViT is to split the input image into non-overlapping patches, embed each patch via a learned linear mapping and process the resulting tokens via stacked self-attention and multi-layer perceptron (MLP) blocks. For more details we refer the reader to the supplementary and [Dosovitskiy al., 2020]. The benefit of Transformers over convolutional architectures in vision lies in their ability to capture long-range dependencies in images via the self-attention mechanism.
Since the introduction of ViT, similar attention-based architectures have been proposed for many other vision tasks such as object detection [Carion al., 2020], image segmentation [Wang al., 2021b] and restoration [Cao et al., 2021b, Chen et al., 2021, Liang et al., 2021, Zamir et al., 2021, Wang et al., 2021c]. A key challenge for Transformers low-level vision problems is the quadratic compute complexity of the global self-attention with respect to the input dimension. In some works, this issue has been mitigated by splitting the input image into fixed size patches and processing the patches independently [Chen et al., 2021]. Others focus on designing hierarchical Transformer architectures [Heo et al., 2021, Wang al., 2021b] similar to popular ResNets [He al., 2015]. Authors Zamir et al. [2021] propose applying self-attention channel-wiss rather than across the spatial dimension thus reducing the compute overhead to linear complexity. Another successful architecture, the Swin Transformer [Liu et al., 2021] I], tackles the quadratic scaling issue by computing self-attention on smaller local windows. To encourage cross- window interaction, windows in subsequent layers are shifted relative to each other.
Transformers in Medical Imaging Transformer architectures have been proposed recently to tackle various medical imaging problems. Authors in Cao et al. [2021a] design U-Net-like architecture for medical image segmentation where the traditional convolutional layers are replaced by Swin Transformer blocks. They report strong results multi-organ and cardiac image segmentation. In Zhou al. [2021] a hybrid convolutional and Transformer- U-Net architecture proposed tailored to volumetric medical image segmentation with excellent results on benchmark datasets. Similar encoder-decoder architectures for various medical segmentation tasks have been investigated in other works [Huang et al., 2021, Wu et al., 2022]. However, these networks are tailored for image

---

segmentation, a task less sensitive to fine details in the imput, and thus larger patch sizes are often used (for instance in Cao et al. [2021a] This allows the network to process larger input images, as the number of token embedding is greatly reduced, but as we demonstrate in Section 5.2 embedding individual pixels as
patches is crucial for MRI reconstruction. Thus, compute and memory barriers stemming from large input resolutions are more severe in the MRI reconstruction task and therefore novel approaches are needed.
Promising results have been reported employing Transformers in medical image denoising problems, such as low-dose CT denoising [Wang et al., 2021a, Luthra al., 2021] and low-count PET/MRI denoising [Zhang et al., 2021]. However, these studies fail to address the challenge of poor scaling to large input resolutions, and only work on small images via either downsampling the original dataset [Luthra et al., 2021] or by slicing the large input images into smaller patches [Wang et al., 2021a]. In contrast, our proposed architecture works directly on the large resolution images that often arise in MRI reconstruction. Even though some work exists on Transformer-basee architectures for supervised accelerated MRI reconstruction [Huang et al., 2022, Lin and Heckel, 2022, Feng et al., 2021], and for unsupervised pre-trained reconstruction [Korkmaz et al., 2022], to the best of our knowledge ours is the first work to demonstrate state-of-the-arr results on large-scale MRI datasets such the fastMRI dataset.
4 Method
HUMUS-NE combines the efficiency and beneficial implicit bias of convolutional networks with the powerful general representations of Transformers and their capability to capture long-range pixel dependencies. The resulting hybrid network processes information both in image representation (via convolutions) and patch-emii michhembedded token representation (via Transformer blocks). Our proposed architecture consists of a sequence of sub-networks also called cascades. Each cascade represents an unrolled iteration of an underlying optimization algorithm in k-space (see (2.2)), with an image- domain denoiser, the HUMUS-Block. First, we describe the architecture of the HUUMUS-Block the core component in the sub-network. Then, we specify the high-level, k-space unrolling architecture of HUMUS-Net in Section 4.3.
4.1 HUMUS-Blon Block Architecture
The HUMUS-BIes Block acts as an image-space denoiser that receives an intermediate reconstruction from the previous cascade and performs a single step of denoising to produce an improved reconstruction for the next cascade. It extracts high-resolution, shallow features and low-resolution, deep features through a novel multi-scale transformer-base block, and synthesizes high- features from those. The high-level overview of the HUMUS -Block depicted in Fig. 1.
High-resolution Feature Extraction- The input to our network is a noisy complex-valued image Pin E TR derived from undersampled k-space data, where the real and imaginary parts of the image are concatenated along the channel dimension. First, we extract high-resolution features FH E hxxwwx from the input noisy image through a convolution layer fH written as FH = {H (.)) This initial 3 x 3 convolution layer provides early visual processing at a relatively low cost and maps the input higher, dæ dimensional feature space. Important to note that the resolution of the extracted features is the same as the spatial resolution of the input image.
Low-resolution Feature Extraction- In case of MRI reconstruction, input resolution is typically significantly higher than commonly used image datasets (32 x 32- 256 x 256), posing a significant challenge contemporary Transformer-baset models. Thus we apply convolutional feature extractor ft. to obtain low-resolution features Fz = fr.(FH), with Fz E Rhi Rebxxwexe xdz where fr. consists of a sequence of convolutional blocks and spatial downsampling operations. The specific architecture is depicted in Figure 4. The purpose of this module is to perform deeper visual processing and provide manageable input size to the subsequent computation and memory heavy hybrid processing module. In this work, we choose h2 = and wT = which strikes balance between preserving spatial information and resource demands. Furthermore, in order to compensate for the reduced resolution we increase the feature dimension to dr. 2 d# :=d. d.
Deep Feature Extraction- The most important part of our model is MUST, a MUlti-scale residual Swin Transformer network. MUST is multi-scale hybrid feature extractor that takes the low-
5

---

resolution image representations FL and performs hierarchical Transformer-convolttionn hybrid processing in an encoder-decoder fashion, producing deep features FD (F(( where the specific a architecture behind SD is detailed in Subsection 4.2.
High-resolution Image Reconstruction- Finally, combine information from shallow, high- resolution features Fи and deep. low-resolution features FD to reconstruct the high-resolution residual image via a convolutional reconstruction module fR. The residual learning paradigm allows us to learn the difference between noisy and clean images and helps information flow within the network [He et al., 2015]. Thus the final denoised image Tout E IR hxwxxs is obtained as Rout Lin + fR(FF.H FD). The specific architecture of the reconstruction network is depicted in Figure 4.
4.2 Multi-scale Hybrid Feature Extraction via MUST
The key component to our architecture is MUST, a multi-scale hybrid encoder-decoder architecture that performs deep feature extraction in both image and token representation (Figure bottom). First, individual pixels of the input representation of shape 1 I x d are flattened and passed through learned linear mapping yield 4 Z tokens of dimension d. Tokens corresponding to different image patches are subsequently merged in the encoder path, resulting in a concise latent representation. This highly descriptive representation passed through a bottleneck block and progressively expanded by combining tokens from the encoder path via skip connections, The final output is rearranged to match the exact shape of the input low- resolution features FL. yielding a deep feature representation FD.
Our design is inspired by the success of Residual Swin Transformer Blocks (RSTB) in image denoising and super-resolution [Liang et al., 2021]. RSTB features stack of Swin Transformer layers (STL) that operate on tokens via a windowed self-attention mechanism [Liu er al., 2021] J, followed by convolution in image representation. However, RSTB blocks operate on single scale, therefore they cannot be readily applied a hierarchical encoder-decoder architecture. Therefore, we design three variations of RSTB to facilitate multi-scale processing as depicted in Figure 2.
RSTB-B is the bottleneck block responsible for processing the encoded latent representation while maintaining feature dimensions. Thus, keep the default RSTB architecture for our bottleneck block, which already operates on single scale.
RSTB-D has a similar function to convolutional downsampling blocks in U-Nets, but it operates on embedded tokens. Given an input with size h1 w; x we pass it through an RSTB-B block and apply PaichMerge operation. PutchMerge linearly combines tokens corresponding to 2 2 non-overlapping image patches, while simultaneously increasing the embedding dimension (see Figure 3, top and the supplementary for more details) resulting in an output of size 2 w, 2 d. Furthermore, RSTB-D outputs the higher dimensional representation before patch merging to be subsequently used in the decoder path via skip connection.
RSTB-U used in the decoder path is analogous to convolutional upsampling blocks. An input with size hú; wy X d is first expanded into larger number of lower dimensional tokens through linear mapping via PatchExpand (see Figure 3, bottom and the supplementary for more details). PatchExpand reverses the effect of PatchMerge on feature size, thus resulting in 2h, 2w; tokens of dimension d Next, mix information from the obtained expanded tokens with skip embeddings from higher scales via TokenMix. This operation linearly combines tokens from both paths and normalizes the resulting vectors. Finally, the mixed tokens are processed by an RSTB-B block.
2.
4.3 Iterative Unrolling
Architectures derived from unrolling the iterations various optimization algorithms have proven to be successful in tackling various inverse problems including MRI reconstruction. These architecture can be interpreted as a cascade of simpler denoisers, each which progressively refines the estimate from the preceding unrolled iteration (see more details in the supplementary).
Following Sriram al. [2020], we unroll the gradient descent iterations of the inverse problem in (2.1) in k-space domain, yielding the iterative update scheme in (2.2). We apply regularization in image domain via our proposed HUMUS-BLLN -lockk, that is we have ((k) F (E (D (R (k)))) where
D denotes the HUMUS- -lock, R(x1, ..., 2N)
Ei -'' S z4 is the reduce operator that combines

---

Paxc Merge
Lew-r Poleer Estnake
24
Laner
NNTRO
1845
Merga
High Insantuctss
Park
2hy 2ml
Figure 2: Depiction of differ- ent RSTB modules used in the HUMUS-Blocd
Figure 3: Patch merge and ex- pand operations used in our multi-scale feature extractor.
Figure 4: Architecture of con- volutional blocks for feature extraction and reconstruction.
HUMUS
200
geha
Figure 5: Adjacent slice reconstruction (depicted in image domain for visual clarity): HUMUS- Net takes a volume of adjacent slices (ā--e, ..... Ø ...., x.+aa) and jointly reconstructs volume (#--",..., x "....,, (..)),++a). The reconstruction loss calculated only on the center slice a
coil images via the corresponding sensitivity maps and E(x) (S SNæ) the expand operator that maps the combined image back to individual coil images. The sensitivity maps can be estimated priori using methods such ESPIRIT [Uecker al., 2014] or learned in an end-to-end fashion via a Sensitivity Map Estimator (SME) network proposed in Sriram et al. [2020]. In this work we aspire design an end-to-end approach and thus we use the latter method and estimate the sensitivity maps from the low-frequency (ACS) region of the undersampled input measurements during training using a standard U-Net network.
4.4 Adjacent Slice Reconstruction (ASR)
We observe improvements in reconstruction quality when instead of processing the undersampled data slice-by-slice, we jointly reconstruct a of adjacent slices via HUMUS-Net (Figure 5). That is, if we have volume of undersampled data kuol (k2 "..,,..,.KK k-K) with K slices, when reconstructing
slice c, we instead reconstruct the volume (k=-a ,..,, ko k. ke+1 .... keta) by concatenating them along the coil channel dimension, where denotes the number of adjacent slices added on each side. However, we only calculate and backpropagate the loss on the center slice the reconstructed volume. The benefit of ASR is that the network can remove artifacts corrupting individual slices it sees larger context of the slice by observing its neighbors. Even though ASR increases compute cost, it is important to note that it does not impact the number of token embeddings (spatial resolution unchanged) and thus can combined favorably with Transformer-bases methods.
Experiments
In this section we provide experimental results our proposed architecture, HUMUS-Net, First, we demonstrate the reconstruction performance of our model on various datasets, including the large-scale fastMRI dataset. Then, we justify our design choices through set of ablation studies.
5.1 Benchmark Experiments
We investigate the performance of HUMUS-NG -et on three different datasets. We use the structural similarity index measure (SSIM) [Wang et al., 2003] basis of our evaluation, which is the most common evaluation metric medical image reconstruction. In all of our experiments, we follow the setup of the fastMRI multi-coil knee track with 8x acceleration in order to provide comparison with state-of-the-an networks of the Public I Leaderboard. That is, we perform retrospective undersampling of the fully-sampled k-space data by randomly subsampling 12.5% of whole k-space lines in the
7

---

fastMRI knee multi-col coil x8 test data
Method
params (approx.)
SSIM(T)
PSNR(1)
E2E- VtrNett [Sriram et al., 2020]
0.8900
36.9
0.0089
30M
E2E-VarNet [Sriram et al., 2020]
(
0.8920
37.1
0.0085
XPDNet [Ramzi et al., 2020]
155M
0.8893
37.2
0.0083
Y-Net [Hammernik ct al., 2019]
676M
0.8877
36.7
0.0091
i-RIM [PPutzky et al., 2019]
300M
0.8875
36.7
0.0091
U-Net [Zbontar et al., 2019]
214M
0.8640
34.7
0,0132
HUMUS-NG (ours)
0.8936
37.0
0.0086
109M
HUMUS- Net (ours)
0.8945
37.3
0.0081
HUMUS Let-L (ours)
0.8944
37.3
0.0083
228M
HUMUS-Net-L. (ours)*
0.8951
37.4
0.0080
Table 1:
Performance of state-of-ther ahrtt accelerated MRI reconstruction techniques on the fastMRI knee test dataset. Most models are trained only on the fastMRI training dataset, available we show results of models trained on the fastMRI combined training and validation dataset denoted by (t).
Validation SSIM
Method
fastMRI knee
Stanford 2D
Stanford 3D
E2E- VarNet [Sriram et al., 2020] HUMUS-Ne (ours)
0.8908
0.8928 0.0168
0.9432 + 0.0063
0.8934
0.8954 0.0136
0.9453 = 0.0065
Table 2: Validation SSIM of HUMUS-Net on various datasets. For datasets with multiple train- validation split runs we show the mean and standard error of the runs.
Ablation studies
Method
Unrolled?
Multi-scale?
Low-re features?
Patch size
Embed, dim.
SSIM
Un-SS
0,9319
Un-MS
X
12
0.9357
0.0038
Un-MS-Patch2
X
2
0.9171 + 0.0075
HUMUS-Ne
SwinlR
0.9336 0.0069
VarNet
0.9432 0.0063
Table 3: Results of ablation studies on HUMUS-Net evaluated on the Stanford 3D MRI dataset.
phase encoding direction, keeping 4% of lowest frequency adjacent k-space lines. Experiments on other acceleration ratios can be found in the supplementary. During training, we generate random masks following the above method, whereas for the validation dataset we keep the masks fixed for each k-space volume. For HUMUS-Net, we center crop and pad inputs to 384 x 384. We compare the reconstruction quality of our proposed model with the current best performing network, E2E- VarNet. For details on HUMÜS-Net hyperparameters and training, we refer the reader to the supplementary. For VarNe, use the hyperparameters specified in Sriram et al. [2020].
The fastMRI dataset [Zbontar et al., 2019] the largest publicly available MRI dataset with competitive baseline models and public leaderboard, and thus provides an opportunity to directly compare different algorithms. Specifically, we run experiments on the multi-coil knee dataset, consisting of close to 35k slices in 973 volumes. We use the default HUMUS-N Net model defined above with 3 adjacent slices as input. Furthermore, we design large variant of our model, HUMUS- Net-L, which has increased embedding dimension compared to the default model (see details in the supplementary). We train models both only on the training split, and also on the training and validation splits combined (additional Rs 20% data) for the leaderboard. Table I demonstrates our results compared to the best published models from the fastMRI Leaderboard evaluated on the test dataset. Our model establishes new state of the art in terms of SSIM this dataset by large margin, and achieves comparable or better performance than other methods in terms PŠNR and NMSE. Moreover, seen in the second column of Table 2, evaluated our model the fastMRI validation dataset as well and compared our results to VarNe,, the best performing model from the leaderboard. We observe similar improvements in terms of the reconstruction SSIM metric to the test dataset. Visual inspection of reconstructions shows that HUMUS- -et recovers very fine details in images that may be missed by other state-of-the-art reconstruction algorithms (see attached
fastMRI
"https://faasee www.eeadeeeoarreeeeeee.ooo
8

---

figures in the supplementary). Comparison based on further image quality metrics can be found in the supplementary. We point out that even though our model has more parameters than E2E- VarNet, our proposed image-domain denoiser is more efficient than the U-Net deployed in E2E- VarNet, even when their number of parameters are matched as discussed in Section 5.3. Overall, larger model size does not necessarily correlate with better performance, as seen in other competitive models in Table Finally, we note that the additional training data (in the form of the validation split) provides a consistent small boost to model performance. We refer the reader to Klug and Heckel [2022] for an overview of scaling properties of reconstruction models.
Stanford 2D Next, we run experiments on the Stanford2D FSE [Cheng] dataset, a publicly available MRI dataset consisting of scans from various anatomies (pelvis, lower extremity and more) in 89 fully-sampled volumes. We randomly sample 80% of volumes as train data and use the rest for validation. We randomly generate different vrainvvaiiattion splits this way to reduce variations the presented metrics. As slices in this dataset have widely varying shapes across volumes, we center crop the target images to keep spatial resolution within 384 384. We use the default HUMUS- defined above with single slices as input. Our results comparing the best performing MRI reconstruction model with HUMUS-Nee is shown in the third column of Table 2. We present the mean SSIM of all runs along with the standard error. We achieve improvements of similar magnitude as on the fastMRI dataset. These results demonstrate the effectiveness of HUMUS-NE on a more diverse dataset featuring multiple anatomies.
Stanford 3D - -inallyy, we evaluate our model on the Stanford Fullysampled 3D FSE Knees dataset [Sawyer et al., 2013], a public MRI dataset including 20 volumes of knee MRI scans. We generate train-validation splits using the method described for Stanford 2D and perform 3 runs. We use the default HUMUS-Net network with single slices as input. The last column of Table 2 compares our results to E2E- VarNet, showing improvements of similar scales as on other datasets we have investigated in this work. This experiment demonstrates that HUMUS-Net performs well not only on large-scale MRI datasets, but also on smaller problems.
5.2 Ablation Studies
In this section, we motivate our design choices through set of ablation studies. We start from SwinIR, a general image reconstruction network, and highlight its weaknesses for MRI. Then, we demonstrate step-by-step how we addressed these shortcomings and arrived the HUMUS-Net architecture. We train the models on the Stanford 3D dataset. More details can be found in the supplementary. The results of our ablation studies are summarized in Table 3.
First, we investigate SwinlR, a saate-of-the-ae image denoiser and super-resolution model that features hybrid Transformer--convlltioro architecture. In order to handle the 10x larger input sizes (320 x 320) in our MRI dataset compared to the input images this network has been designed for (128 x 128), reduce the embedding dimension of SwinIR to fit into GPU memory (16 GB). We find that compared to models designed for MRI reconstruction, such as E2E- VarNet (last row in Table 3) SwinIR performs poorly. This is not only due to the reduced network size, but also due to the fact that SwinIR is not specifically designed to take the MRI forward model into consideration.
Next, we unroll SwinIR and add sensitivity map estimator. We refer to this model as Un-SS. Due to unrolling, we have to further reduce the embedding dimension of the denoiser and also decrease the depth of the network in order to fit into GPU memory. Un-SS, due to its small size, performs slightly worse than vanilla SwinIR and significantly lags behind the 22r-VarNer architectures. We note that SwinIR operates over single, full-resolution scale, whereas state-of-the-ar MRI reconstruction models typically incorporate multi-scale processing in the form of U-Net-like architectures.
Thus, replace SwinIR by MUST, our proposed multi-scale hybrid processing unit, but keep the embedding dimension in the largest-resolution scale fixed. The obtained network, which we call Un-MS, has overall lower computational cost when compared with Un-SS, however Table 3 shows MRI reconstruction performance has significantly improved compared to both Un-SS and vanilla SwinIR, which highlights the efficiency of our proposed multi-scale feature extractor. Reconstruction performance is limited by the low dimension of patch embeddings, which we are unable to increase further due to our compute and memory constraints originating in the high-resolution inputs.
The most straightforward approach to tackle the challenge of high input resolution is to increase the patch size. To test this idea, we take Un-MS and embed 2 x 2 patches of the inputs, thus reducing the

---

number of tokens processed by the network by factor of 4. We refer to this model as 1h-MM--Patcis This reduction in compute and memory load allows us to increase network capacity by increasing the embedding dimension 3-folds (to fill GPU memory again). However, Un-MS-Putch2 performs much worse than previous models using patch size of 1. For classification problems, where the general image context is more important than small details, patches of 16 x 16 or 8 x 8 are considered typical (Dosovitskiy et al., 2020J. Even for more dense prediction tasks such as medical image segmentation, patch size of 4 x 4 has been used successfully [Cao et al., 2021a]. However, our experiments suggest that in low-level vision tasks such as MRI reconstruction using patches larger than x 1 may be detrimental due to loss of crucial high-frequency detail information.
Our approach address the heavy computational load of Transformers for large input resolutions where increasing the patch size is not an option is to process lower resolution features extracted via convolutions. That is we replace MUST in Un-MS by a LEMUUS-Blck, resulting in our proposed HUMUS-Net architecture. We train smaller version of HUMUS- Net with the same embedding dimension as Un-MS. As seen Table 3, our model achieves the best performance across all other proposed solutions, even surpassing E2E- VarNet. This series of incremental studies highlights the importance of each architectural design choice leading to our proposed HUMUS-NG architecture.
5.3 Direct comparison of image-domain denoisers
In order to further demonstrate the advantage of HUMUS-Na Net over E2E- VarNe,, we provide direct comparison between the image-domain denoisers used in the above methods. VarNet unrolls fully convolutional U-Net, whereas we deploy our hybrid HUMUS-Blen (Fig. 1). We scale down the HUMUS-BIos in E2E-Varnet in terms of number of model parameters. To this end, we reduce the embedding dimension from 66 to 30. We train both networks on magnitude images from the Stanford3D dataset. The results are summarized in Table 4. We observe that given fixed parameter budget, our proposed denoiser outperforms the widely used convolutional U-Net architecture in MRI reconstruction, further demonstarting the efficiency of HUMUS-Net. This experiment suggests that our HUMUS- -lock could serve as an excellent denoiser, replacing convolutional U-Nets, in a broad range of image restoration applications outside of MRI, which we leave for future work.
architecture as a denoiser used in HUMUS-NG Net to match the size of the U-Net
Model
# of parameters
SSIM(11
PSNR(1)
NMSE(L)
U-Net
2.5M
0.9348 + 0.0072
39.0 + 0.6
0.0257 + 0.0007
HUMUS-Block
2.4M
0.9378 + 0.0065
39.2 + 0.5
0.0246 + 0.0004
Table 4: Direct comparison of denoisers on the Stanford 3D dataset. Mean and standard error of 3
random training-valiiationn splits is shown.
6 Conclusion
In this paper, we introduce HUMUS-Net, an unrolled, Transformer-convoltiion hybrid network for accelcrated MRI reconstruction. HUMUS-Net achieves statc-of-ther the-art performance on the fastMRI dataset and greatly outperforms all previous published and reproducible methods. We demonstrate the performance of our proposed method on two other MRI datasets and perform fine- grained ablation studies to motivate our design choices and emphasize the compute and memory challenges of Transformer- based architectures on low-level and dense computer vision tasks such as MRI reconstruction. A limitation of our current architecture is that it requires fixed-size inputs, which we intend to address with more flexible design in the future. This work opens the door for the adoption multitude of promising techniques introduced recently in the literature for Transformers, which we leave for future work.
Acknowledgments
M. Soltanolkotabi supported by the Packard Fellowship in Science and Engineering, a Sloan Research Fellowship in Mathematics, an NSF-CAREER under award #1846369, DARPA Learning with Less Labels (LwLL.) and FastNICS programs, and NSF-CIF awards #1813877 and #2008443. This research is also in part supported by AWS credits through an Amazon Faculty research award.
10

---

References
Emmanuel J. Candes, Justin K. Romberg, and Terence Tao. Stable signal recovery from incomplete and inaceurate measurements. Communications Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 59(8): 1007-1233, 2006.
Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. Swin-Unet: Unet-like pure Transformer for medical image segmentation. arXiv:2105.05537, May
2021a. arXiv: 2105.05537.
Jiezhang Cao, Yawei Li, Kai Zhang. and Luc Van Gool. Video super-resolution Transformer. arXiv preprint arXiv:2106.0668 2021b.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with Transformers. In European Conference on Computer Vision, pages 213 213-229. Springer, 2020.
Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12299-12310, 2021.
Joseph Y. Cheng. Stanford 2D FSE. http://mmddaa..ggllsspppgeestttt 2D FSE.
Özgün Çiçek, Ahmed Abdulkadir, Soeren Lienkamp, Thomas Brox, and Olaf Ronneherger. 3D U- Net: Learning dense volumetric segmentation from sparse annotation. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pages 424-432. Springer, 2016.
David L., Donoho. Compressed sensing. IEEE Transactions on Information Theory, 1294):1289-1336 2006.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. arXiv:2010. 1929 [cs], 2020.
Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, and Yong Xu. Task Transformer network for joint MRI reconstruction and super-resolution. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 307-317, 2021.
Kerstin Hammernik, Teresa Klatzer, Erich Kobler, Michael P Recht, Daniel K Sodickson, Thomas Pock, and Florian Knoll. Learning variational network for reconstruction of accelerated MRI data. Magnetic Resonance in Medicine, 3000-3015333 2018.
Kerstin Hammernik, Jo Schlemper, Chen Qin, Jinming Duan, Ronald M Summers, and Daniel Rueckert. -enet: Systematic evaluation of iterative deep neural networks for fast parallel MR image reconstruction. arXiv preprint arXiv: 29/..992229928 2019.
Yoseob Han and Jong Chul Ye. Framing U-Net via deep convolutional framelets: Application to sperse-view CT. IEEE Transactions on Medical Imaging, 3766):418-14299 2018.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv:1512.03385, 2015.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of Vision Transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 11936-11945, 2021.
Jiahao Huang. Yingying Fang, Yinzhe Wu, Huanjun Wu, Zhifan Gao, Yang Li, Javier Del Ser. Jun Xia, and Guang Yang. Swin Transformer for fast MRI. arXiv preprint arXiv:2201.0224 2022.
Xiaohong Huang, Zhifang Deng, Dandan Li, and Xueguang Yuan. Missformer: An effective medical image segmentation Transformer. arXiv preprint 21X 2021.

---

Chang Min Hyun, Hwa Pyung Kim, Sung Min Lee, Sungchul Lee, and Jin Keun Seo. Deep learning for undersampled MRI reconstruction. Physics in Medicine & Biology, 63(13):135007, 2018.
Tobit Klug and Reinhard Heckel. Scaling laws for deep learning based image reconstruction. arXiv preprint arXiv:2209. 13435, 2022.
Yilmaz. Korkmaz, Salman UH Dar, Mahmut Yurt, Muzaffer Özhey, and Tolga Cukur, Unsupervised MRI reconstruction via zero-shot learned adversarial Transformers. IEEE Transactions on Medical Imaging, 2022.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. SwinIR: Image restoration using Swin Transformer. arXiv:2108. 10257, 2021.
Kang Lin and Reinhard Heckel. Vision Transformers enable fast and robust accelerated MRL. In Medical Imaging with Deep Learning, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using shifted windows. 1rX1v:2103.14030 2021.
Michael Lustig, David L. Donoho, Juan M. Santos, and John M. Pauly. Compressed sensing MRI. IEEE Signal Processing Magazine, 25(2):72-88 82, 2008
Achleshwar Luthra, Harsh Sulakhe, Tanish Mittal, Abhishek Iyer, and Santosh Yadav. Eformer: Edge enhancement based Transformer for medical image denoising. arXiv preprint arXiv:2 1e:::209900044 2021.
Matthew J Muckley, Bruno Riemenschneider, Alireza Radmanesh, Sunwoo Kim, Geunu Jeong, Jingyu Ko, Yohan Jun, Hyungseob Shin, Dosik Hwang, Mahmoud Mostapha, al. Results of the 2020 fastMRI challenge for machine learning MR image reconstruction. IEEE transactions on Medical Imaging, 40(9):230--23) 2021.
Gregory Ongie, Ajil Jalal, Christopher Metzler Richard G. Baraniuk, Alexandros G. Dimakis, and Rebecca Willett. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas Information Theory, 2020.
Patrick Putzky, Dimitrios Karkalousos, Jonas Teuwen, Nikita Miriakov, Bart Bakker, Matthan Caan, and Max Welling. I-RIM applied to the fastMRI Challenge. arXiv:1910.08952 2019.
Zaccharie Ramzi, Philippe Ciuciu, and Jean-Luc Starck. XPDNet for MRI reconstruction: an application to the 2020 fastMRI Challenge. arXiv preprint arXiv:2010.07290. 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 234-241, 2015.
Anne Marie Sawyer, Michael Lustig, Marcus Alley, Phdmartin Uecker, Patrick Virtue, Peng Lai, and Shreyas Vasanawala. Creation of fully sampled MR data repository for compressed sensing of the knee. 2013.
Anuroop Sriram, Jure Zbontar, Tullie Murrell, Aaron Defazio, C. Lawrence Zitnick, Nafissa Yakubova, Florian Knoll, and Patricia Johnson. End-to-end variational networks for accelerated MRI re- construction. In Medical Image Computing and Computer Assisted Intervention, pages 64-73, 2020.
Jian Sun, Huibin Li, Zongben Xu, et al. Deep ADMM-Net for compressive sensing MRI. Advances in Neural Information Processing Systems, 29, 2016.
Martin Uecker, Peng Lai, Mark J Murphy, Patrick Virtue, Michael Elad, John M Pauly, Shreyas S Vasanawala, and Michael Lustig. ESPIRiT an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA. Magnetic Resonance in Medicine, 71(3):990-1011, 2014.
Dayang Wang, Zhan WI, and Hengyong Yu. TED-net: Convolution-fres T2T VisionTransfonn Transforme-- based encoder-decoder dilation network for low-dose CT denoising. In International Workshop on Machine Learning in Medical Imaging, pages 416-425, 202
12

---

Wenhai Wang, Enze Xie, Xiang Li. Deng- Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid Vision Transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 568-578, 2021b.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, and Jianzhuang Liu. Uformer: A general U-shaped Transformer for image restoration. arXiv preprint arXiv:2106.031 2021c.
Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1398-1402 leee, 2003.
Yixuan Wu, Kuanlun Liao, Jintai Chen, Danny Chen, Jinhong Wang, Honghao Gao, and Jian Wu. D- D-Former: A U-shaped dilated Transformer for 3D medical image segmentation. arXiv preprint arXiv:2201.00462, 2022.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient Transformer for high-resolution image restoration. arXiv:2111.09881, 2021.
Jure Zbontar, Florian Knoll, Anuroop Sriram, Tullic Murrell, Zhengnan Huang, Matthew J. Muckley,
Aaron Defazio, Ruben Stem, Patricia Johnson, Mary Bruno, Marc Parente, Krzysztof J. Geras, Joe Katsnelson, Hersh Chandarana, Zizhao Zhang. Michal Drozdzal, Adriana Romero, Michael Rabbat, Pascal Vincent, Nafissa Yakubova, James Pinkerton, Duo Wang, Erich Owens, C. Lawrence Zitnick, Michael P. Recht, Daniel K. Sodickson, and Yvonne W. Lui. fastMRI: An open dataset and benchmarks for accelerated MRI. arXiv:1811.08839 2019.
Jian Zhang and Bernard Ghanem. ISTA-Net: Interpretable optimization-inspieed deep network for image compressive sensing. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1828 1837, 2018.
Lipei Zhang, Zizheng Xiao, Chao Zhou, Jianmin Yuan, Qiang He, Yongfeng Yang, Xin Liu, Dong Liang, Hairong Zheng, Wei Fan, et al. Spatial adaptive and Transformer fusion network (STFNet) for low-count PET blind denoising with MRL Medical Physics, 2021.
Hong- Yu Zhou, Jiansen Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang, and Yizhou Yu. nn- Former: Interleaved Transformer for volumetric segmentation. arXiv preprint 2021.
Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested U-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages 3-11. 2018.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See Conclusions.
(c) Did you discuss any potential negative societal impacts of your work? [No] We were unable to identify possible negative effects on society.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Source code is published at https: //ggt/ggibbb com/ Sttp//MaaahLDSHHUUSSSS
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplementary and source code.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] We report the standard error of runs in our tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplementary.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
Source code is published https: //gi thub. Com/MathFLDS, BAS/HUMUS-NNN
(d) Did you discuss whether and how consent obtained from people whose data you're
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "TLowLcbyxH3", "reviewer": "Reviewer_ZMfj", "summary": "This paper proposes a new state-of-the-art for accelerated MRI reconstruction. Different components of the model are properly explained by the authors and most of the design choices are well-justified. There are also several concerns/comments pointed out below.", "questions": "The reviewer understands the point with long-range-dependancy modeling of Transformers. However, it is unclear why that should be beneficial in reconstruction tasks such as MRI reconstruction. Note that locality matters for image reconstruction which is captured by convolutions.\n\nWhile the ablation studies are interesting, two important parts are missing: (1) Adjacent Slice Reconstruction (ASR), and (2) Residual learning. It would be interesting to see how the E2E-VarNet performs if these two elements are added. This is important since if with these modifications, the performance of E2E-VarNet matches HUMUS-Net, then **it sends the message that a complicated HUMUS-Block is not necessary and one can simply regularize the unrolled problem with a simple U-Net (as E2E-VarNet currently does)**.\n\nGiven the marginal improvement of HUMUS-Net over E2E-VarNet, what are some competitive advantages of HUMUS-Net that could convince someone to use it instead of E2E-VarNet? e.g., is it computationally cheaper to train? Does it require less training data? Is it more robust? etc. Any explanations in this direction are highly appreciated.\n\nIt would be nice to add the number of parameters of each model to Table 2 to assure the reader of the fairness of the comparisons.\n\nTypos\n\n- Line 193: a significant challene=\n- Line 199: In this work, ", "limitations": "The reviewer can discuss limitations of the work better if the authors answer the questions/concerns raised above in the Questions/Suggestions section. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++ The paper is very well-written and easy to follow. Having a new state-of-the-art for accelerated MRI reconstruction is also highly appreciated as it has not been pushed since 2020.\n\n++ Most of the architectural design choices are properly justified by the authors.\n\n+- The proposed method is very intuitive in most of its aspects (e.g., high- and low-res processing paradigms). However, some explanations are vague, please see the comment below in the questions/suggestions part.\n\n-- Ablation studies are not conclusive (please see the comment below in the questions/suggestions part).\n\n-- Evaluation experiments can be improved w.r.t efficiency of the model and fairness of the comparisons.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8C0OZA4RGg", "reviewer": "Reviewer_pRk7", "summary": "The paper describes a method for accelerated parallel MRI reconstruction using deep neural networks. This is inspired by recent advancements of vision transformer (ViT) and deep unfolding (also known as algorithm unrolling) network for medical imaging. In specific, a new network architecture (including convolutional and self-attention layers) was proposed and used as a denoising prior in the gradient descent unrolling iterations. The proposed reconstruction network, named as HUMUS-Net, is then trained in an end-to-end supervised fashion, where ground truth images are used during training. The authors compare their proposal against existing SOTAs such as U-Net and the End-to-End Variational network (E2E-VarNet) using the fastMRI and Stanford 2D/3D datasets. Numerical results show that HUMUS-Net achieves comparable or slightly better results to E2E-VarNet and significantly outperforms traditional U-Net.", "questions": "1), The HUMUS-Net models reported in Table 1 do not show a large improvement compared to E2E-VarNet in terms of PSNR and NMSE. This especially true for when models trained on the fastMRI using training dataset only. Since the proposal in this work uses exactly the same unrolling algorithm with jointly coil estimation, it is unclear the rational for using sophisticated transformer as denoising priors, compared to the simple U-Net used in E2E-VarNet. \n\n2), As mentioned by the authors, The MUST module in this work is inspired by the residual Swin transformer blocks. Consider explaining and discussing the relations and differences between the proposal MUST and Swin transformer in more details. A deeper understanding and intuition will strengthen this work and make its contribution more solid.\n\n3), In abstract it is claimed that the HUMUS-Net consume less memory compared to existing transformer-based methods. The memory consumption of proposal should scale with number of iterations. Will unroll more iterations results in larger memory usage? In addition, are the weights of HUMUS-Net shared cross each iteration? If not, this will also affect the disk storage size. \n\n4), The ablation studies in Sec. 5.2, comparisons to SwinIR transformer denoiser seems to be unfair and not very informative. As mentioned by the authors, SwinIR was not originally designed for fMRI reconstruction. Using existing transformers for fMRI will be more precise. In addition, I do not think the simple SwinIR (reduced embedding dimension due to lack of gpu memory) is fine-tuned as much as HUMUS-Net in this work. Maybe the authors can use memory efficient training such as checkpoint or deep equilibrium model (DEQ), while keeping the model’s structure as in its original work. \n\n5), As mentioned earlier, some important numerical results, ablation studies and technical details are missing. \n\n    (a) What is training loss used in this work?\n    (b) PSNR values are missing in Table 2. and Table 3.\n    (c) Visual comparisons of the proposal to baseline methods are missing in the main manuscript. \n    (d) Different acceleration ratios (e.g., X4, X6, etc.) to E2E-VarNet are missing.\n    (e) Number of parameters and running time comparisons to baseline and other transformer-based fMRI methods, such as [R1, R2] are missing.\n\n6), While this may be orthogonal to this proposal, it would be helpful to discuss the robustness of HUMUS-Net to model shifting (e.g., anatomy shift etc.) and how is the HUMUS-Net generalizable to other medical imaging modalities. These will strength the current paper.\n\nRef:\n[R1] Lin. et al., “Vision Transformers Enable Fast and Robust Accelerated MRI”, MIDL, 2022. \n(Github: https://github.com/MLI-lab/transformers_for_imaging)\n\n[R2] Korkmaz. et al., “Unsupervised MRI Reconstruction via Zero-Shot Learned Adversarial Transformers”, IEEE TMI, 2022. \n(Github: https: //github.com/icon-lab/SLATER.)\n", "limitations": "The limitations of this proposal are missing and need to be discussed in the revision.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is overall well written and reports competitive results with both quantitative and qualitative validation for fMRI reconstruction. However, the methodological contributions and rationale for using transformer in unrolling are unclear or at least not well illustrated in its current state. The supportive numerical ablation studies (e.g., running time, memory usage, robustness to model shifting, comparison to existing transformer-based fMRI methods) are missing. Please find my technical comments bellow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "82OXkXaftze", "reviewer": "Reviewer_on3J", "summary": "This paper proposed the HUMUS-Net that used to achieve the goal of reconstructing accelerated  multi-coil MRI images. HUMUS-Net consists several cascades of HUMUL-Blocks, each block complete a step in unrolled iterations of an optimization algorithm to solve inverse problem. Each HUMUL-Block contains three blocks: high-resolution feature extraction, low-resolution feature extraction and high-resolution image reconstruction. Further, the core component of HUMUL-Block was specified. It is a multi-scale hybrid feature extraction component named MUlti-scale residual Swin Transformer network (MUST). It adopted the Residual Swin Transformer Blocks (RSTB) in a hierarchical encoder-decoder architecture to facilitate the multi-scale processing. The MUST is comprised several blocks of RSTB-B, RSTB-D and RSTB-U.  The usage of HUMUL-Block dedicate to solve the regularization term in image domain was indicated. At last, the benefit of reconstruct image using adjacent slices to remove artifacts were explained.\nPerformance of several state-of-the-art reconstruction techniques on the fastMRI knee multi-coil ×8 dataset were compared with respect to metrics of SSIM, PSNR and NMSE. HUMUS-Net and E2E-VarNet were validated as well on three different datasets.  And the ablation studies have done to justify the proposed design.\n", "questions": "Detailed comments:\n1. The model split input images into patches, and embed the patches into lower-dimensional tokens. The way to produce tokens is not clear.\n2. Line 180-182: As shown in Figure 6 in supplementary, HUMUL-Block only dedicates to the regularization term in (Eq 2.2). The high-level unrolled architecture is based on the k-space domain. The expression on line 180-182 seems tell us that each cascade works on the image domain, which is confusing.\n3. Line 192: The used resolution of MRI image is not indicated, makes it hard to compare with (32x32-256x256).\n4. Line 216-218: The explanation of “token representation”, “learned linear mapping” and “positional encoding” are missing.\n5. Line 219: what does the phrase of 'Tokens corresponding to different image patches....in a concise latent representation' means? It is too abstract for me.\n6. Figure 2 and 3: The font of the dimension indicator is too small.\n7. Figure 3: The reshape blocks are not explained.\n8. Line 268: Please specify the dimension of concatenated k-space slices and the dimension of input into the HUMUL-Block\n9. Line 329: How does 10 times was calculated?\n", "limitations": "some ablation study is not provided.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength：\nThe proposed HUMUS-Net claimed to have several strengths:\n1. The utilization of self-attention based Transformer can overcome the weakness of convolutions, like content independence and the inability to model long-range dependencies. \n2. The multi-scale structure in its core block MUST not only could reconstruct fine details but also reduced the computational cost.\n3. The model has achieved the second position in fastMRI leaderboard of ×8 multi-coil knee challenge.\n4. The proposed method performs good at three various dataset: fastMRI knee dataset, Stanford2D and Stanford3D dataset.\n\nWeakness：\n1. The proposed HUMUS-Net achieved better results than the other methods with respect to metrics of SSIM, PSNR and NMSE. But the radiologist's evaluation on specific images is missing.\n2. The HUMUL-Block in HUMUS-Net claimed acts like an intermediate denoiser. But the analysis of this effect is missing. \n3. The ablation experiment of cascade with respect to reconstruction performance is missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "08qaaHnTxtK", "reviewer": "Reviewer_4jKX", "summary": "The paper proposes HUMUS-Net, a NN architecture that combines convolutions and transformer blocks for compressed sensing MRI reconstruction. The approach achieves SOTA on the benchmark fastMRI dataset with 8x acceleration. The code is available with the supplementary materials.", "questions": "1. The experiments are done on 8x acceleration. It seems that this approach might also be fitting well for further acceleration ratios >8x. Did you consider validating the method on higher acceleration ratios and if not, how do you expect the method to perform? How about lower acceleration, 4x?\n2. You mention that the current architecture requires fixed-size inputs. Could you explain why? Both extractor and reconstruction blocks are convolutional, so it shouldn't be the problem.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The novel approach paves way for applied MRI models with hybrid convolutional-transformer architectures.\n2. The paper carefully describes design decisions and includes ablation study of the model.\n\nWeaknesses: -", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept