Noise Reconstruction and Removal Network: A New Way to Denoise FIB-SEM Images
Authors: Katya Giannios
Abhishek Chaurasia
Bambi DeLaRosa
Guillaume THIBAULT
Jessica L. Riesterer
Erin S Stempinski
Terence P Lo
Joe W Gray
Venue: ICLR 2022 Submitted (2022)
Keywords: Neural Network
CNN
LSTM
Unsupervised learning
Denoising
FIB-SEM

Abstract: Recent advances in Focused Ion Beam-Scanning Electron Microscopy (FIB-SEM) allow the imaging and analysis of cellular ultrastructure at nanoscale resolution, but the collection of labels and/or noise-free data sets has several challenges, often immutable. Reasons range from time consuming manual annotations, requiring highly trained specialists, to introducing imaging artifacts from the prolonged scanning during acquisition. We propose a fully unsupervised Noise Reconstruction and Removal Network for denoising scanning electron microscopy images. The architecture, inspired by gated recurrent units, reconstructs and removes the noise by synthesizing the sequential data. At the same time, the fully unsupervised training guides the network in distinguishing true signal from noise and gives comparable/even better results than supervised approaches on 3D electron microscopy data sets. We provide detailed performance analysis using numerical as well as empirical metrics.

Under review as a conference paper at ICLR 2022
NOISE RECONSTRUCTION AND REMOVAL NETWORK: A NEW WAY TO DENOISE FIB-SEM IMAGES
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in Focused Ion Beam- Beam-Scanning Electron Microscopy (FIB-SEM) allow the imaging and analysis of cellular ultrastructure nanoscale resolution, but the collection of labels and/or noise-free free data sets has several challenges, often immutable. Reasons range from time consuming manual annotations, requiring highly trained specialists, to introducing imaging artifacts from the prolonged scan- ning during acquisition. We propose fully unsupervised Noise Reconstruction and Removal Network for denoising scanning electron microscopy images. The architecture, inspired by gated recurrent units, reconstructs and removes the noise by synthesizing the sequential data. At the same time, the fully unsupervised training guides the network in distinguishing true signal from noise and gives com-- parable/even better results than supervised approaches on 3D electron microscopy data sets. We provide detailed performance analysis using numerical as well as empirical metries.
INTRODUCTION
Recent advances in Focused Ion Beam-Scanning Electron Microscopy (FIB-SEM) have led to unprecedented biological tissue visualization and analysis, as well as understanding of cellular ultrastructure and cell-to-cell interactions (Xu et al. (017). High-resolution FIB-SEM data sets often consist volumes sliced into thousands of 6K 4K images with 4nm resolution per voxel, allowing a 3D reconstruction of fraction of tissue volume. Depending
the tissue type, sample preparation, acquisition settings, detector used, etc., the images may contain a significant quantity of noise making any further analysis tedious or even impossible (Kubota et al. 2018 Liu et al. 2018).
By definition, image denoising is the process of taking a noisy image I and separating the noise n from the true signal s:
+ Following the typical assumption for the noise (Foi al. 2008 Wu et al. 2019) and taking the microscope's s characteristics into account, we can assume that the noise is: (i) a zero-mean random noise, that is for any pixel the noise is a discrete random number added to the pixel 'true value'; (ii) each pixel noise is independent, so the noise value at any pixel does not depend the noise at any other pixel, but it is signal dependent. While the noise is random and independent, the signal is not and this is what, typically, denoising methods rely on.
In recent years, deep learning methods, and particularly the Convolutional Neural Networks (CNNs), have established themselves as powerful analytical tools in machine Icarning to name just a few Redmon & Farhadi (2018): Tan al. (2020): Chen et al.] (2017): |Roneberrger et al. (2015): y) Tao et al. (2020), Sun & Chen (2020) In the field of denoising, CNNs have been very useful (Kim et al. [019]] Liu et al. 2019) Yu et al. 2019) especially when the noise characteristics are unknown, making any mathematical modeling difficult. In this paper we apply CNN technique to FIB-SEM acquired images, taking into account the relevant specifics.
FIB-SEM allows 3D imaging of biological fine structure at nanoscale resolution: thin slice of the sample is removed with the ion beam and the newly exposed surface imaged with the electron beam. That results in sequence of images containing isotropic voxels down to 4nm.
Traditionally, training neural networks for denoising demands pairs of noisy and clean images (ground truth images). Theoretically, obtaining denoised images is possible by averaging multiple (up to hundreds) acquisitions of the same sample. As mentioned, this is not feasible with FIB-SEM. The challenges with obtaining ground truth images, as in many biological use cases, is motivation

---

Under review as a conference paper at ICLR 2022
(a) Input Image
(b) U-Net (supervised)
(c) DenoiseNet (supervised)
(d) NRRN 4BU (unsupervised)
The right panel shows the gray original image and left panel shows image after being passed through high-pass filter. A good denoising algorithm should be able to suppress noise while keeping the edges sharp. NRRN's architecture paired with unsupervised training scheme removes the noise and does a better job at preserving the edges of the organelles as compared to U-Net and DenoiseNet which were trained using supervised techniques.
Figure 1:
for developing and utilizing unsupervised techniques, such as a Noise2Noise approach (Wu et al. Lehtinen al. 2018). Our network uses triplet of images input and is trained map one noise realization to the other, using our modified Noise2Noise loss function. We refer to the proposed architecture Noise Reconstruction and Removal Network (NRRN). Fig. shows visual comparison between NRRN and other state-of-th oft-----tthttt art networks trained using supervised techniques. The NRRN is applicable the case of improving the image quality based on two three scans of the same slice, or denoising based on the two adjacent slices the volume stack. Our major three contributions discussed in this paper
novel noise reconstruction module with soft attention and signal boosting, that upon deployment on large images (more than 24M pixels) homogeneously removes the noise, a neural network architecture design using our noise reconstruction module with detailed performance analysis, and
updated Noise2Noise loss function specifically designed for denoising FIB-SEM data.
RELATED WORK
Noise-to-clean (N2C) the traditional supervised learning approach, where the models are trained with pairs of noisy and clean images as inputs and targets respectively. However, when the clean (ground truth) images are not available, the supervised N2C approach is not applicable.
Lehtinen et al. (2018) introduced new idea: Noise2Noise (N2N). Instead of training a network to map noisy inputs to clean images, their N2N trains on pairs of independently degraded versions of the same training sample.
Wu et al. (2019) progressed the N2N idea with a novel loss function applied to medical images with available pairs of noise realizations. The network is trained map one noise realization to the other, with loss function that efficiently combined the outputs from both training subsets. The Noise-to- Void (N2V) training schema (Krull et al.] 2019) and the followed Noise-to-sed self (N2S) (Batson & Royer 2019) go even a step further and train network only on single noisy image. The methods are based on the so-called blind-spot networks where the receptive field of the network does
2

---

Under review as a conference paper at ICLR 2022
Buikding Unit
Building Unit
Building Uni
Conv
ReLU
A2
Conv
RaLU
Cosv
ReLU
NAIL
Feedback
Figure 2: Single Branch Architecture
not include the center pixel. As the blind-spot network has little bit less information than N2N and N2C, it then expectantly cannot quite reach their performances, but still gives very good results.
Independent of the training scheme, lot of work has been done on the neural network architectures design. A very popular choice is the U-Net architecture (Ronneberger al. 2015) and its variations. U-Net was first introduced for segmentation but since has been widely extended in its applications as well as architectures. Alternatively, Remez et al. (2017: 2018) used a CNN that exploits a gradual denoising process. Their DenoiseNet architecture calculates "noise estimate" that is fine-tuned each layer based on the previous layer output, and the results are then added to the input image. They have shown that the shallow layers handle local noise statisties, while deeper layers recover edges and enhance textures. Recurrent neural networks (RNN) have been widely developed and used in many applications, particularly with sequential data such natural language processing and video analysis in order to exploit temporal information. Specifically GRU (Cho et al.) 2014 Yao et al. 2015) and ConvLSTM (Shi et al. 2015 Yang et al. 2017) have influenced the NRRN architecture.
3 NOISE RECONSTRUCTING AND REMOVAL NETWORK
The proposed Noise Reconstruction and Removal Network (NRRN) architecture and the training framework are designed take advantage of the sequential nature of FIB-SEM data.
3.1 DENOISING MODULE
Typically, a FIB-SEM data set consists N images representing N consecutive imaged slices (Zo, ..., ). Our approach is to obtain denoised image 2, from three adjacent noisy images I To do so, we consider the initial tissue sample into a data set of N - independent triplets of images (iJ= We split each triplet into two pairs (ı,) and (z1,, that passed through two parallel identical branches of the proposed network NRRN.
For pair of images (2y, 1) each branch (Fig. 2 consists of:
one convolution layer (3 kernel, 64 channels, and ReLU activation) applied to each of
the two images
+ 1 stacked layers made building unit (BU) coupled with noise attention (NAtt) and feedback blocks
a final convolution layer (3 x kernel, 64 channels, and ReLU activation).
Each BU models noise components (see Fig. 3) by accumulating noise characteristics from previous units and from updated input image after each stage. Within the BU' the information flows through
two major paths highlighted by an upper and lower block. The path through the lower block hy learns noise features from previous BUL-1 and cleaner input '1-1 The upper block N1 N named synthesizer, mostly learns information coming from the adjacent image Ijit- Motivated by LSTM
'Or if available, three of the slice

---

Under review as a conference paper at ICLR 2022
Iell Da
Figure Building Unit
Innge #
Beplace Channe 8
Kg
Syreescar Oxtutt
U
Noñc Compunen
Synthesbre
(a) Noise Attention Block
(b) Feedback Block
Figure 4: NRRN Additional Blocks: a) The Noise Attention Block (NAtt), and b) The Feedback Block after the 7h BU.
and GRU cells, but with reduced computational overhead, BU contains gates: an update gate u' that decides how much of the previous/adjacent information (coming from h f11 and the new one (1 - w') to be kept; a reset gate 7 that regulates how much to be forgotten. This allows the synthesizer at every level to accept useful information coming from the Tj while simultaneously
being aware about the updated image x")
Once the synthesizer has learned the noise components, the new features are passed to the noise attention block (NAtt block). The NAtt block filters the BU output and generates noise component which is removed from the corresponding input image to give new updated image x ' to the next BU. The key equations for every pair of images passed through the BUs are given in the appendix. The NAtt block (see Fig. 4ab receives feature vector with 64 channels ii+1 input. It through all channels to create an attention map that gets element-wise multiplied to the first channel of R
pools
We observe that the output of the NAtt block represents component of the noise present in
previous noisy image x²
Presumably, we obtain a less noisy image nd) after each layer and we want to feed it back to the network. That is achieved in the feedback block (see Fig, 4b) that works buffer by gathering information from both the updated image o²) well the newly learned features 14.+1 from the
4

---

Under review as a conference paper at ICLR 2022
previous BU. Thus the first channel of the synthesizer's output h2=1 is updated by the Feedback block giving an input for next BU. For simplicity we use the same notation hitt1 for both: synthesizer output and input of the next BU.
3.2 LOss FUNCTION
We consider N - triplets of images split into two pairs (z) and (z)+ The NRRN network maps each of these pairs to two outputs =/j = f( and x² [ 0). The two input pairs having independent and zero means noise, generate two denoised versions of the same jth image. The final denoised solution & of the jth image is average of the two. One can view, for every slice Jj, the adjacent images æj-1 and Zj+1 as discrete versions of the signal along the third direction of the volume. In addition, -1-1 and x++1 have their own noise that spatially independent. Due to the FIB-SEM way of imaging (extremely high isotropic resolution along all three dimensions), we consider that the signal between two slices has good enough properties and we can use Taylor expansion along the third-axis. The updated loss derived from the Wu et al. (0019) can be rewritten as (for derivation see appendix)
N-
1
Ln2n
N
2
25
11
2 11a
zj 11-1
(I)
DATA SETS AND METRICS
The NRRN architecture has been specifically designed for FIB-SEM images denoising, and trained and tested images acquired with FEI Helios NanoLab 660 DualBeam using In-Column Detector (ICD) at Oregon Health & Science University (OHSU) Multiscale Microscopy Core (MMC). The OHSU data set is composed of cancer tissue images of dimensions 4K 6K pixels with 4nm resolution. The acquiring technique allowed the creation of "ground truth" images: the tissue sample surface was scanned 10 times before slicing (the number of scans being limited by artifacts produced by extended electron beam scanning), so producing 10 images of the exact same area. The process was repeated 5 times. The "ground truth" images were obtained after applying an affine transformation from an in-house version of the TurboReg algorithm (Thevenaz et al. 1998), removing outliers and finally averaging. Additionally, we experimented with the publicly available data set from EPFL. Here, the ground truth images are not available, and we employed the data set to examine the efficiency the proposed algorithm in removing artificially added Gaussian and/or Poisson noise.
We use two classical measures in the evaluation process (applicable when ground truth is available): the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index Measure (SSIM). We also utilize the interquartile range (IQR) of the signal across straight line in fiat-signal-ares In the FIB-SEM case, the entire workflow from tissue collection to final image acquisition takes roughly two weeks for 1500 images (Riesterer al., 2020). During this process the clinical specimen undertakes several epoxy of resin infiltration steps to fill the space between the cellular structures. In our analysis we look at the noise presence in the resin because is homogeneous material, and as consequence an "efficient" denoiser should produce flat signal.
RESULTS AND COMPARISONS
The OHSU data set was separated into 3 slices for training, for validation and 1 for testing, and the initial large images were cropped into 13650 smaller images of size 256 x 256 pixels. The NRRN architecture was implemented in PyTorch (Paszke et al. 2209), with BUs, the image triplets the training are taken from the same slice, and the training performed with ADAM optimizer for initial learning rate 10 10-4, 0,-4, 31 0.9, B2 = 0.999, e = 10-8 and a batch size of 16.
On the test slice, NRRN achieves PSNR of 31.0197 = 0.1905 dB and SSIM of 0.9705 + 0.0006. Fig. 5 shows an image from the test slice with a PSNR of 31.11dB, significant reduction of the noise the resin (the IQR is reduced to 0.94 F while still keeping sharp edges the cell organelles with reduced noise (see Fig, c.)) Arguably, a model taking triple input a burden, but inference with double even single input is possible as discussed in the appendix.
5

---

Under review as a conference paper at ICLR 2022
(a) NRRN Denoised Image
(b) Input Crop
(c) NRRN Denoised Crop
XQRHSUM
Fa
NRRX-1I 1881
155 170 175
155
200
36
100
182
Villa
(d) TQR across the resin
(c) Signal across the Mitochondria
Figure 5: NRRN denoising results, where the yellow doted line shows where image signal is measured: (a) validation image. (b) zoom in of the noisy input image, (c) zoom in of the denoised image, (d) box plot of the signal across the resin (from subfig. b) showing the significant reduction in the signal/noise variation, and (e) the signal variation across the mitochondria (from subfig. c)) the NRRN denoising keeps sharp edges.
5.1 COMPARISONS
We compare NRRN to the following methods:
Training-free methods: BM3D, non-local means (NLM), median and Gaussian filters. We applied the NLM to a single image and to an average of (here referred as 'NLM sngl' and 'NLM avg' respectively).
Training methods: U-Net and DenoiseNet. We experimented with the classical DenoiseNet (using ground truth images in the training) as presented in Remez et al. (2017) (here referred 'N2C DenoiseNet- angl'), but also a version of DenoiseNet that is applicable to sequence of (three) images Remez al. (2018) (here referred 'DenoiseNet' or 'N2C DenoiseNet').
The U-Net is a classical network with depth of and batch normalization. U-Net is trained on a single image (as input) 'N2C U-Net sngl' and on an average of three consecutive images 'N2C U-Net avg' (or simply U-Net).
The rather large OHSU FIB-SEM input images (4K 6K) do not fit into the GPU memory
typical problem to be faced in this field. A solution is during inference to denoise 256 x 256 cropped patches (345 patches per image), and reconstruct the original image. We noticed that even though U-Net overall performs well, the level of denoising depends significantly on the image (patch). On Fig. 6a we plotted the patches PSNR distribution for each of the compared architectures. NRRN removes the noise homogeneously across all patches, with standard deviation of only 0.52, only marginally better than DenoiseNet and both outperform significantly the U-Net deviation of 1.26.
Here, we argue that the loss function (equation I results in better denoising. The ground truth images have better signal than the input images, but they are not noise free. Using per-pixel loss with ground truth in the training (in the face of L1 or L2 norm) would maximize the PSNR but results in an image that inherits the noise present in the "ground truth". This is avoided by NRRN. On Fig. 6b we compared the noise in the signal flat resin region for U-Net, DenoiseNet and NRRN. The improvements the IQR are significant for NRRN.
Comparisons the final whole image based PSNR and SSIM shown in table I Expectantly, NLM-avg. DenoiseNet and U-Net all outperform NRRN, in terms of PSNR and SSIM. DenoiseNet and U-Net designed minimize the MSE loss (equivalent maximizing the PSNR). In the case

---

Under review as a conference paper at ICLR 2022
100m
11
108-15.72
Cren Track
10R-5.01
60
Crg Vallo
(a) PSNR Histogram
(b) IQR across the resin area
Figure 6: Left Panel: PSNR Histogram for the patches constructing FIB-SEM image denoised with U-Net, NRRN and DenoiseNet. Right Panel: Comparison of the signal across the resin for U-Net, DenoiseNet, NRRN and the Ground Truth image. NRRN with its specific N2N loss function removes more noise than the N2C trained methods. The IQR of NRRN outperform the rest of the networks.
Table 1: PSNR and SSIM for denoising with training-free methods (left) and training methods (right)
Training-free
Training
Method
PSNR (dB)
SSIM
Method
PSNR (dB)
SSIM
Input
22.9680
0.8577
Input
22.9680
0.8577
NLM -avg. input
31.3589
0.9725
N2C DenoiseNet
32,2029
0.9779
BM3D
28.2515
0.8747
N2C U-Net avg
31.4677
0.9760
Median Filter-avg. input
28.0214
0.9495
NRRN
31.1060
0.9708
Gauss Filter-avg. input
27.0649
0.9195
NRRN -2 img
30.4194
0.9666
NLM -sngl
24.0703
0.6195
N2C U-Net sngl
29.1844
0,9571
N2C DenoiseNet -sngl
28.3436
0.9559
of NLM the high PSNR is achieved by aggressively flattening the signal even on biological structures of interest. NRRN does better job in removing noise across the resin while sharpening the biological structures visible on Fig. I and Fig. 6b Additional information is given in the appendix.
Besides the OHSU data, synthesized additional noise the EPFL data set. We fine tuned the OHSU pre-trained models by continuing the training on the small EPFL volume data set. During training, we corrupted the EPFL images with Poisson noise for random peak values ranging in [1, 50] and also added Gaussian noise with random a between 10 and 75. The denoising results are in Table 2 In terms of PSNR, NRRN with its unsupervised training shows very comparable performance to the supervised training marginal improvement in some of the cases and slight mnder-performance in others. A closer examination of Fig. 7 reveals that: U-Net tends to over smooth the details; DenoiseNet adds white speckles for low peak values and high a,, indicating that the network struggles to extract enough information from the significantly damaged images; while NRRN does not display any of these issues because the Building Units able to synthesize the information from the adjacent slices and reconstruct and remove the noise without over smoothing or adding artifacts.
5.2 EXPLORING THE DENOISING PROCESS AND ABLATION STUDY
Our algorithm allows intermediate noise estimates after each Building Unit. Our exploration of the denoising process is carried out for NRRN with 5 BUs on both data sets: OHSU and EPFL. Each BU SSIM shows that the majority of the denoising happens the first 1-3 layers where we observe a more significant and gradual improvement (see Fig, (b). The last two BUs become important for highly corrupted images (Poisson noise with peak=1 and Gaussian noise o = 75). On Fig. 8a we
looked at the signal across the resin after each BU. We scaled the signal with the signal mean value
7

---

Under review as a conference paper at ICLR 2022
POISSON
POISSON 8
POISSON 30
GAUSS 10
GAUSS 75
DeroiseNE Ingot
NRRN
U-NE
Figure 7: Comparison of the denoising performance. Top bottom: Input image, DenoiseNet, NRRN and U-Net. Left to right: added Poisson noise with peak = 1, 8, 30 and Gaussian noise with 10, 75.
Table 2: PSNR in dB for EPFL data set with added Poisson noise with peak 30 and Gaussian
noise with 10, 25, 35, 75
Method
Poisson noise Peak
Gaussian noise o
8
30
10
25
35
75
Input
5.98
10.19
12.52
17.66
28.13
20.17
17.29
11.54
NLM-avg
10.16
14.72
17.03
22.05
27.98
25.78
21.55
16.11
Gauss-avg
16.86
23.89
25.77
26.89
27.13
27.01
26.88
25.57
Median-avg
13.54
18.26
20.14
23.71
27.17
25.09
23.53
19.11
NRRN
17.33
23.77
25.68
27.52
28.57
27.97
27.42
25.38
N2C DenoiseNet
15.02
22.39
25.19
27.53
28.55
27.93
27.43
24.52
U-net
16.93
23.92
25.86
27.49
27.97
27.77
27.47
25.58
and plotted the scaled input image signal (x-axis) versus the scaled BUs signal (y-axis). We that the signal (which on the resin area mostly noise) gradually fattens. Looking at the intermediate denoising results one can tune the modular architecture toward the needs in hand: the higher the level of noise, the higher the number of BUs. We trained a whole family of NRRN networks ranging the on the OHSU data set. Measuring the signal variation on the resin in
number of BUs from terms of IQR, PSNR, and SSIM, we observed that while the SSIM and PSNR have not improved, the variation of the signal (in terms of IQR) on the resin flattened until the 4t BU and leveled there.
5.3 TRANSFER LEARNING AND EFFICIENCY
As we have seen in Table 2 the three architectures achieve similar results the EPFL data set with pre-training. Our final experiment skipping the pre-training step and directly training all the architectures the small EPFL data set. We observed that even with just only I BU the NRRN achieves PSNR of 17.9dB for the case of Poisson noise with peak 1, while the N2C architectures get only PSNR 9.5dB. Similarly, in the
of Gaussian noise with 10 NRRN with I BU reaches a PSNR of 28.5 which is at least 4.7% better than the others.
Furthermore, in the Appendix Fig. II report and measure the giga operations per second (GOPS) and number of parameters of the NRRN family (single branch), (N2C) U-Net and (N2C) DenoiseNet. When looking for trade off between denoising quality and efficiency among the NRRN families,
8

---

Under review as a conference paper at ICLR 2022
+12
L3
.15
Raal
015
30
40
10
31
-10
-30
-1
30
(a)
(b)
Figure 8: Iterative denoising process: a) Input signal scatter plot across the resin vs layers signal, b) SSIM at each layer for OHSU data set and EPFL with added Poisson noise with peak and 30 (P1, P30) and added Gaussian Noise with = 15, 75 (G15,G75)
U-Net and DenoiseNet, the winner is NRRN with 3 Building Units. For more noise corrupted images (as it was for EPFL with Poisson noise with peak 1-8 or Gaussian Noise with a = 75) the 5 BUs with their bigger receptive field would be required.
6 CONCLUSION
The advances in FIB-SEM provide high resolution cell images, unfortunately encompassing a high level of noise. this paper, we propose and further developed the idea of using a sequence images with Noise2Noise approach for removing that noise. That has the benefit of not requiring ground truth images for training, which is typically the case for biomedical deep learning tackled problems. We suggested novel modular architecture (NRRN) that is able to exploit the sequential nature of the images and reconstruct and remove the noise from the original image. We demonstrated that this unsupervised approach leads to consistent noise removal across the entire image without regard to the structures present. Besides, we showed that the architecture allows a glimpse into the denoising process that can be used to adjust the depth of the network to the available computational resources and/or presence of noise.
ACKNOWLEDGMENTS
FIB-SEM data was generated the Multiscale Microscopy Core with technical support from the OHSU Center for Spatial Systems Biomedicine and invaluable specimen acquisition support from SMMART clinical coordination team.The manuscript supported by Prospect Creek Foundation, the Brenden-Coon Colson Center for Pancreatic Care, the NCI Cancer Systems Biology Measuring, Model- ing, and Controlling Heterogeneity Center Grant SSU54CA209982 the NCI Human Tumor Atlas Network (HTAN) Omic and Multidimensional Spatial (OMS) Atlas Center Grant (5U2CCA233280), the OHSU KCI and NCI Cancer Center Support Grant (P30CA069533), and the OCSSB. This study was approved by OHSU IRB#16113.
ETHICS STATEMENT
Here with we would like to disclaim potential conflicts of interest: one of the co-authors has licensed technologies to Abbott Diagnostics; has ownership positions in Convergent Genomics, Health Technology Innovations, Zorro Bio and PDX Pharmaceuticals: serves as paid consultant to New Leaf Ventures; has received research support from Thermo Fisher Scientific (formerly FEI), Zeiss, Miltenyi Biotech, Quantitative Imaging, Health Technology Innovations and Micron Technologies; and owns stock Abbott Diagnostics, Abb Abb, Alphabet, Amazon, Amgen, Apple, General Electric, Gilead, Intel, Microsoft, Nvidia, and Zimmer Biomet.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We have undertaken the following steps ensure the reproducibility of our results: the source code is attached as supplemental material, the description of the model, including figures, are in section 3.1 syper-parametess are in section [|] additional formulas and information are the Appendix A.1 and A.2
REFERENCES
Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. In International Conference on Machine Learning, pp. 524-533. PMIR, 2019.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation, 2017.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation, 2014.
EPFL. EPFL-Electron Microscopy Dataset. http::///ww..ggllccc///////ooob data-em, 2013.
Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian. Practical poissonian- gaussian noise modeling and fitting for angle-image raw-data. IEEE Transactions on Image Processing, 17(10):1337-175 2008
Dong-Wood Kim, Jae Ryun Chung, and Seung-Wor Won Jung. Grongrouped residual dense network for real image denoising and gan-based real-world noise modeling. 2019.
Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void-leaminn denoising from single noisy images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1129-2137, 2019.
Yoshiyuki Kubota, Jaerin Sohn, and Yasuo Kawaguchi. Large volume electron microscopy and neural microcireuit analysis. Frontiers in neural circuits, 12:98, 2018.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2Noise: Learning Image Restoration without Clean Data. arXiv e-prints, art. arXiv:1803.041 March 2018
Tengfei Liang, Yi Jin, Yidong Li, and Tao Wang. Edenn: Edge enhancement- densely con- nected network with compound loss for low-dose et denoising. 2020 15th IEEE International Conference Signal Processing (ICSP), volume 1, pp. 193-198. IEEE, 2020.
Jiaming Liu, Chi-Hao Wu, Yuzhi Wang, Qin Xu, Yuqian Zhou, Haibin Huang, Chuan Wang, Shaofan Cai, Yifan Ding, Haoqiang Fan, and Jue Wang. Learning raw image denoising with bayer pattern unification and bayer preserving augmentation, 2019.
S Liu, L Sun, Gao, and K Li. A fast curtain-removal method for 3d fib-sem images of heterogeneous minerals. Journal of microscopy, 272(1)33-11, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luea Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. Curran Associates, Inc., 2019. URL httpp///ntttpppppp...rrrrrrsss.s wttp:/ttnncaaa-aeeaaaiiiiiieeeea imperative streiiieessyyeeeeee hi oegeapappeeeeiiaaaprrrppoooeeeee
9015 1e155--ytrrc---e an
pdf
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement, 2018.
10

---

Under review as a conference paper at ICLR 2022
Tal Remez, Or Litany, Raja Giryes, and Alex M. Bronstein. Deep Class Aware Denoising. arXiv e-prints, art, 1rX1v:1701.0168,, January 2017.
Tal Remez, Or Litany, Raja Giryes, and Alex M. Bronstein. Class-Aware Fully Convolutional Gaussian and Poisson Denoising. IEEE Transactions on Image Processing, 27(11):5777-5722. November 2018. doi: 10. (000//TP.201.228590000
Jessica L Riesterer, Claudia S López, Erin S Stempinski, Melissa Williams, Kevin Loftis, Kevin Stoltz, Guillaume Thibault, Christian Lanicault, Todd Williams, and Joe W Gray. A workflow for visualizing human cancer biopsies using large-format electron microscopy. In Methods in Cell Biology, volume 158, pp. 163-181. Elsevier, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv e-prints, art. M.Kiv:1505.04999 May 2015.
Xingjian Shi, Zhourong Chen, Hao Wang, Din Yeung. Wai-kin Wong, and Wang-chun Woo, Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. arXiv e-prints, art. 1ee.i110....2111 June 2015.
Wanjie Sun and Zhenzhong Chen. Learned image downscaling for upscaling using content adaptive resampler. IEEE Transactions Image Processing, 29:4027-40400, 2020. ISSN 1941-0042. doi: 20.200094 URL http: dx doi httppp///.......ooog//0001111112 20w.0000...2222222
Mingxing Tan, Ruoming Pang, and Quoe V. Le. Efficientdet: Scalable and efficient object detection, 2020.
Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scal attention for semantic segmentation. arXiv preprint arXiv:2005. 2020.
Philippe Thevenaz, Urs E Ruttimann, and Michael Unser. A pyramid approach to subpixel registration based on intensity. IEEE transactions on image processing, 7(1):27-41, 1998.
Dufan Wu, Kuang Gong, Kyungsang Kim, and Quanzheng Li. Consensus Neural Network for Medical Imaging Denoising with Only Noisy Training Samples. arXiv e-prints, art. arxiv: 1906.03639, June 2019.
C Shan Xu, Kenneth Hayworth, Zhiyuan Lu, Patricia Grob, Ahmed M Hassan, Jose G Garcia- Cerdan, Krishna K Niyogi, Eva Nogales, Richard Weinberg, and Harald Hess. Enhanced fib-sem systems for large-volume 3d imaging. Elife, 6:e25916, 2017.
Dong Yang, Tao Xiong, Daguang Xu, S. Kevin Zhou, Zhoubing Xu, Mingging Chen, JinHyeong Park, Sasa Grbic, Trac Tran, Sang Chin, Dimitris Metaxas, and Dorin Comaniciu. Deep imagc-to-image recurrent network with shape basis learning for automatic vertebra labeling in large-scale 3d et volumes. pp. 398-506, 09 2017. ISBN 778-3-319-661788 doi: 10. 10.7/09777---- 1---3--611999-77-7771
Kaisheng Yao, Trevor Cohn, Katerina Vylomova, Kevin Duh, and Chris Dyer. Depth-gated Istm, 2015.
Songhyun Yu, Bumjun Park, and Jechang Jeong. Deep iterative down-up enn image denoising. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 BUILDING UNIT EQUATIONS
For every pair of images (15)1, the key equations are given by:
M+1 j+1 = tanh(W1 11
u' o oW h/j + Vu * {1z11)
,' oWW, h2 + V, ((t1)
(2)
d' = Lanh(W, hi + r' o h * Vo)
1 (1 u²) i / + ut o
for 1 E {0, [O,....J). J.. {1, -
- 2} and h 1 (e zイ=1) and h ReLU(We * zz)
and W1 is with shared parameters across all BUs. Here a denotes the Hadamard product and denotes the convolution operator.
A.2 Loss FUNCTION
Wu et al. 2019 showed that for clean image sy and two noisy realizations of it T(k.1) = 8k + ((k.1)
(k.2) Sk ((k22) the function
((1( 0) ( 0) If (I((.2))
8k
2
0-20"
approaches the real signal 8y for, 0" = arg min Lnnn (0) where
I,2n
2 [f( 211.2)l12
N
(z 11111)11
- (1 (= 0)1)), (3)
The only requirements for such loss function are that the noise nik.1) and (1.22) are with zero means and spatially independent.
In our case, we consider - 2 triplets of images split into two pairs (z-- xz) and (1)+1, The NRRN network maps each of these pairs to two outputs z/J f 6) and x (1.++, f). The two input pairs with independent and zero mean noise, generate two denoised versions of the same 7th image. The final denoised solution ft, of the jth image is given by:
f = (o)) + x.)).
One view, for every slice wj, the adjacent images - and zz- as discrete versions of the signal along the third direction of the volume. In addition, z5-1 and rs+1 have their own noise that is spatially independent. Due the FIB-SEM way of imaging (extremely high isotropic resolution along all three dimensions), we consider that the signal between two slices has good enough properties that we can use Taylor expansion along the third-axis. Thus, for every image r, one can view Jj-1 = 8j + noise Co, where 8j is the real signal, eo = 0(s; 8j). and similarly for x5+1- In such cases the eq. can rewritten as
Σ L N j=1
1
1 + 11 -H-1
1 11os ="'13²
2
12

---

Under review as a conference paper at ICLR 2022
Input
Tmage
2 Tmagus
Tmages
City Villa
103
0,858
SSIM
179
0.967
0.971
Distanne (pixelss)
(b) Signal acTOSS the
22
169
20.133
Gray Valae
PSNR
31.106
0.000
10.000
20,000
30.000
40.000
Drstance (picelss) (c) Signal actoss Mitochondna
(a) Inference with 1, and 3 images
Figure 9: Inference with 2, and 3 input images. a) PSNR and SSIM, b) The signal across the resin for inference with one, two and three images, c) The signal across an organelle.
A.3 RESULTS
A.3.1 INFERENCE WITH 1, 2 AND 3 IMAGES
NRRN requires triplet of images in the denoising process, but one can use NRRN for denoising with double or even single inputs during inference. However, as shown in Figure 9 the higher the number input images, the better the PSNR and SSIM. Even so, an inference with double input architecture gives very satisfactory quality results in comparison to the triple input by degrading the overall results (measured PSNR and SSIM) by only 2%. The signal variation in the flat resin area across the organelle for two images is almost indistinguishable from the three images. Using single image or two images would, in addition, reduce the computational burden since we would pass the input through the single branch of the architecture.
A.3.2 EDGE PRESERVING
A good denoising algorithm should be able to suppress noise and at the same time keep the edges sharp. NRRN's architecture removes the noise and does a better job at preserving the edges of the organelles as compared to Non-local mean, U-Net and DenoiseNet (the last two were trained using supervised techniques). In Figure I0 we visually inspect the denoised images (the right panels), the image being passed through high- tass filter (left panel) and the signal variation across straight line (the bottom graphs).
Additionally, have experimented with small modification in architecture (Fig. 2}. exchanging the very first convolution layer for a concatenation of four edge enhancement modules (Liang et al.) (2020)) and a convolution with 32 channels. The edge enhancement module contains four trainable Sobel convolutions (detecting vertical, horizontal, and the two diagonal directions edge information). That resulted in a slightly boosted PSNR of 31.12 dB and SSIM of 0.9708 (around 0.3% better than the current architecture). A visual inspection showed no significant enhancement on the edges. Due the incremental improvement we did not proceed further with that architecture change.
A.3.3
IQR vs OPFRATIONS/PAAAMEERRR
In Fig. II report and measure the giga operations per second (GOPS) and number of parameters of the NRRN family (single branch), (N2C) U-Net and (N2C) DenoiseNet. If we are to find the best trade off denoising quality versus efficiency, then the NRRN with 3 Building Units gives, in
13

---

Under review as a conference paper at ICLR 2022
Figure 10: Comparison of the denoising performance (from left to right): Input image, NLM, U-Net, DenoiseNet and NRRN
3.5-
U.Net
5.5
U-Ne
5.0
4.5
4.0
4,0
3.0
DenoiseNet
3.0
DemotseNet
2.5-
2.0
1.5
NRRN(()
NKRNN()
2.0
NRRN(
0.5
NRRNN()
NBK(N144 NRRNN()
1.0
NRRN(3) NRRN(S)
NRRN(4)
0
20
40
60
100
120
140
160
180
0.02
0.1
0.5
Ops
Pammetccologpriimii
(a) IQR vs GOPS across the resin
(b) IQR vs number of parameters across the resin
Figure 11: Comparison (a) IQR vs GOPS (b) IQR vs number of parameters (in million) for NRRN family, DenoieNet, and U-Net. For the NRRN family the GOPS are for only one (of the two) branches, the Noise2Noise training requires two branches which will double the GOPS but avoids the need of ground truth images.
our opinion, the best trade off from the whole NRRN family, DenoiseNet, and U-Net for the OHSU data set. For more noise corrupted images (as for EPFL with Poisson noise with peak 1-8 or Gaussian Noise with a 75) the 5 BUs with their bigger receptive field would be required.
14


Reviews:
[{"review_id": "ukizC8DyNwV", "reviewer": "Reviewer_Uq3h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The text does a good job of introducing the problem and providing a short review of prior work in this area. The paper is overall well written and easy to read. \n\nThe authors propose to use triplets of consecutive sections in a noise2noise setting with a modified loss, and a custom architecture composed of stacked Build Units and Noise Attention Blocks, reducing noise at every level in the stack. The idea of using multiple consecutive sections in an image stack has been proposed before (https://arxiv.org/abs/2011.05105), and this should be mentioned (and ideally compared to) in the present work. Evidence that the BUs really do reduce noise with subsequent application is shown in Fig. 8, though it would be helpful to also see this visually (perhaps in the supplementary materials if space is a concern). It would also be interesting to see a visualization of the noise that is removed at every step.\n\nThe denoising results are compared to training-free methods, as well as supervised baselines. The proposed model performs comparably to the supervised methods in terms of PSNR and SSIM, and where the other models perform better, the authors argue that this is because they learn to reproduce the noise still present in the ground truth data. The improvements offered by the proposed NRRN model are somewhat subtle -- e.g. fewer artifacts and flatter profiles in empty (resin) areas. This is shown in Fig. 6. I found the attached movie very helpful to intuitively understand the impact of this, so it could be helpful to include some high-res figures in the text as well. I also appreciated the computational complexity comparison to DenoiseNet.\n\nThe paper would be strengthened by including a noise2noise baseline (this should address the concern of the baseline models learning to reproduce ground truth noise). It would also be helpful to have some ablations of the proposed BU architecture. The architecture is quite complex, and there is no experimental data showing that e.g. the LSTM-inspired internal gates are actually necessary. \n\n\nQuestions:\n- Has it been tried to modify the building unit to take 2 sections of context directly instead of feeding a section pair at a time and averaging the results?\n- What is the total number of images and Mpx contained in the dataset?\n- Please comment on how section alignment quality impacts the denoising results.\n- Please comment on image degradation due to electron irradiation in case of multiple acquisitions. Is this a problem at all within the 10 scans you did?\n- A.3.1 mentions inference with a single image. How does this work, given that 2 inputs are shows in Fig. 2?\n\nMinor nits:\n- Fig. 5 does not show the yellow dotted line mentioned in the caption.\n- The paper mentions that supplementary materials contain source code, but only a single AVI \nfile is present.\n\n", "summary_of_the_paper": "The paper proposes a neural network architecture (NRRN) implementing a variant of the noise2noise scheme, and applies this to section triplets of a FIB-SEM stack. The results are evaluated with PSNR, SSIM, and IQR on a custom dataset with low-noise ground truth, as well as on a public dataset with no ground truth (using artificially introduced noise). NRRN is shown to improve the images in ways that are hard to measure with traditional metrics.\n", "summary_of_the_review": "The proposed neural network architecture and training scheme improve denoising over baselines in relatively subtle ways that are hard to measure with traditional metrics. To me, in the current version it slightly misses the bar for acceptance due to some missing baselines (noise2noise, noise2stack) and insufficient experimental data motivating the architecture choices.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rkydarJv673", "reviewer": "Reviewer_4BoC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ Noise in the FIBSEM, and more generally in the EM images, is a real problem and many approaches in different applications would benefit from a noise removal technique.\n+ An unsupervised technique for denoising is also highly desirable.\n+ I must also appreciate the effort to design an architecture and loss functions for this task.\n\nCons:\n- **Generality:**  However,  I am concerned the utility of this method on different types of medical images. In fact, the assumption of structural continuity in the z dimension or having multiple images taken from the same region of tissue may be too restrictive for the EM images itself. The SEM and TEM imaging approaches, that are widely popular because their capability to record larger tissue volume, has a z-resolution of approx ~30nm. The contents of two slices in these types of images vary significantly and violates the working assumption of this method. \n\nAs FIBSEM imaging destroys the tissue samples, recording multiple images from the same region may not possible in practice, not only because of scarcity of the same tissue but also because the extensive, elaborate sample preparation necessary to produce images of the same quality. \n\nNot sure if the requirement of having multiple slices of the same region or consecutive slices with very similar content can be satisfied in other medical images or not.\n\n- **Experimental validation:** Since there exist other methods (Lethinen'2018, Wu'2019) already in the literature that removes noise in an unsupervised fashion, the proposed method must be compared with both or at least one of these algorithms. I could find this comparison in the manuscript.\n\n- **Technical/conceptual/theoretical justification:** Although the paper describes the network architecture in Sec 3.1, it does not explain why and how the architecture is removing noise. Why and how do \"we obtain a less noisy image x'_j^l after each layer\" conceptually/theoretically or ate least intuitively? What is the conceptual/theoretical reason \"the output of the NAtt block represents component of the noise present in previous noisy image x'_j^{l-1}\"? How does the synthesizer learn the noise components? Why is the attention map multiplied by the *first* layer of \\hat{h}_j_{l+1}? Without proposer justification, it sounds like a trial-and-error strategy for the network design.\n\n- **Presentation:** The description of Sec 3.1 was not easy for me to understand. It would be better to have the equations shown here rather than the appendix. That way, perhaps all the components would be accounted for -- now the description misses c^l, c^{l+1}, it does not what they are and how are they computed. There is also some inconsistency in the statements and figures. For example, Fig 4(a) shows connection between synthesizer and NAtt, but the text mentions the upper block is the synthesizer which should not have any connection with NAtt according to Fig 3. ", "summary_of_the_paper": "This paper proposes a method for noise removal from the FIBSEM images. The method is unsupervised, it does not require a corresponding clean image to learn denoising. Instead the method utilizes consecutive slice from a FIBSEM volume or multiple images from the same region to remove noise. The core assumption is, while the noise pattern is random and varies across different z-slices, the actual content, i.e., the cellular structure it is capturing remains largely similar.\n\nThe paper proposes a network architecture and loss functions for this purpose. The architecture is is inspired by the LSTM/GRU designs, perhaps in the hope of modeling the common structure between a pair of images. \n\nThe approach has been tested on two real datasets.", "summary_of_the_review": "My current understanding is, the paper lacks the general applicability and sufficient experimental validation that are essential for acceptance in ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ys5wECW6lde", "reviewer": "Reviewer_xdev", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-----------------Strengths--------------------------\nThe paper is specifically aimed at FIB-SEM denoising which is an important application area in the biomedical domain. Although image denoising is a well-studied problem, the impact of image denoising for biomedical images has only been investigated recently. I appreciate the authors looking into this direction.\n\n-----------------Weaknesses---------------------------\n\n1. Novelty: The method proposed in this paper is a variation of Noise2Noise. But I am wondering why would Noise2Noise not work in their case. The authors claim that they need triplets of images (which are consecutive image slices or three images of the same slice) having the same content but different noise realizations. This seems like the perfect case for application of Noise2Noise and that too needing only two images. For instance, Cryo-CARE (Buchholz et al.) uses Noise2Noise in a similar manner fore noising of cryo-electron microscopy images. Why should the same not be applicable to FIB-SEM images?\n\n2. Motivation for architecture and concerns with ablations: As mentioned above, I do not understand the motivation for the proposed architecture. Although the paper claims the novelty of the proposed architecture, I am unsure why is it needed in the first place given we have simpler and theoretically well-founded methods such as Noise2Noise. Additionally, the proposed NRRN has many modules and it is hard to grasp which module is responsible for NRRN's performance. For example, how useful is the Noise Attention block, Feedback block and the Synethesizer block? How do gating blocks affect the performance in the BU module? I think there are no reported ablations for these. Overall, I do not understand the motivation for the complexity of the proposed architecture.\n\n3. Baselines: The proposed method has not been compared to many important baselines. Many of these baselines such as  Self2Self (Quan et. al) , Noise2Same (Xie et al.), Cryo-CARE (Buchholz et al.), PN2V (Krull et al.), PPN2V (Prakash et al.), etc. will be directly applicable to the datasets considered in this paper with no modification. Additionally, these baselines will not need triplet of images as proposed in this paper and most of them can be applied on single noisy images as well.\n\n4. Related Work: Image denoising is a well-studied problem and the paper does not acknowledge many important prior works in general as well as in the biomedical image denoising domain. For instance, the related work section does not mention popular non-DL based methods such as BM3D, NLM, etc (although they have been used as baselines). Additionally, many recent DL based methods such as Self2Self (Quan et. al) , Noise2Same (Xie et al.), Cryo-CARE (Buchholz et al.), PN2V (Krull et al.), PPN2V (Prakash et al.), etc. have also not been mentioned or compared against even though all these methods are unsupervised denoisers targeting same noise types (Gaussian and Poisson) as studied in this paper.\n\n5. Limited Applicability: As the paper admits, the proposed method is specifically designed for FIB-SEM images which although being important is a limited class of biomedical image domain. There are many works (see my comments on ```Related Work```) which target the same noise types but are applicable to different biomedical image domains as well as natural images. ", "summary_of_the_paper": "The paper proposes a denoising algorithm for FIB-SEM images using triplets of images. These triplets can either be consecutive imaged slices or three scans of the same slice. The proposed NRRN network takes two sets of image pairs (constructed from triplets) and generates two denoised images corresponding to the two sets with the final denoised solution being the average of the two denoised images. The method is tested on two datasets and compares against few existing supervised denoising methods.", "summary_of_the_review": "In summary, I vote for rejecting this paper. \n\nThe paper misses comparisons against many important and recent unsupervised denoising works which are directly applicable for the presented application domain without needing any modifications (see my comment in weaknesses). A motivation for choosing the proposed architecture is missing (see my comment in weaknesses) when in my opinion, simpler architectures can arguably perform similar or better. Moreover, the paper is rather limited in its application (is specifically designed for FIB-SEM images) and is unclear if it can be extended for any other domains. Hence, my rating of reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "WS6-ihPHwBF", "reviewer": "Reviewer_34aJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the current work include addressing the challenging biomedical application of denoising *FIB-SEM* images. A novel recurrent neural network architecture employing soft attention is proposed and the *N2N*- loss formulation is modified in order to suit the current biomedical application.\n\nI have gone through sections 1, 2, 3.2, 4, 5 and Appendix A.2 and in my opinion, the weaknesses of the paper (without being able to look under the hood at any code since it was not made available by the authors as of now) is as follows:\n1. It would be interesting to see how well this loss formulation performs with a U-Net like network architecture just in order to distinguish the contribution of the loss formulation and the architecture towards the performance of the method. Additionally, it would be interesting to investigate  the same  recurrent neural architecture employed in this work but training the model parameters with a N2V (*Noise2Void, Krull et al*) blind spot  strategy, for example.\n\n2. In A.2, the authors state that Taylor’s expansion is used\n>“Due to the FIB-SEM way of imaging (extremely high isotropic resolution along all three dimensions), we consider that the signal between two slices has good enough properties that we can use a Taylor expansion along the third-axis.”\n\n* Here, the third-axis should be clarified to be the axis of milling.  \n* Furthermore, the authors should clarify that they use a zeroth-order Taylor Approximation and ignore all higher order terms.\n* This is a more strong weakness in my opinion - *NRRN* rests on the assumption that the signal in any slice is the same as the signal in the slice before and the signal in the slice after, since the authors ignore all higher order terms in the Taylor’s expansion. Hence, the authors should highlight that this method starts to break down if the voxel size along the  z or third dimension is large. Probably an ablation study highlighting this drawback by considering every second slice in the *EPFL* dataset to form a triplet would highlight this.\n\n3. It would be useful to compare against baseline methods designed especially for electron microscopy image data such as [Buchholz et al](https://pubmed.ncbi.nlm.nih.gov/31326025/) , if possible.\n\n4. Rather few slices during evaluation in the OHSU dataset- the OHSU dataset comprises 5 slices, 3 of which are used for training, 1 for validation and 1 for testing. Since their proposed approach relies on feeding in a triplet of images to the network, hence feeding in only one test image during evaluation seems somewhat ill-suited to show the merits of the proposed method. \n\n5. Minor point: The EPFL dataset is commonly referred to as the `Lucchi` dataset in some publications (for example, [this](https://arxiv.org/pdf/2104.03577.pdf) or [this](https://ieeexplore.ieee.org/document/6619103) ) after the surname of the first author. The correct reference for this dataset should be *Lucchi, A., Smith, K., Achanta, R., Knott, G., Fua, P.: Supervoxel-based segmentation of mitochondria in em image stacks with learned shape features. IEEE Transactions on Medical Imaging 31(2), 474–486 (2011)*\n\n6. Minor point, regarding  the **Introduction** section:\n>“Theoretically, obtaining denoised images is possible by averaging multiple (up to hundreds) acquisitions of the same sample. As mentioned, this is not feasible with FIB-SEM.” \n\nI think that the reason why several acquisitions at the same depth are not acquired in the *FIBSEM* context can be better motivated in the introduction section perhaps by stating explicitly that it leads to sample degradation/damage, the idea of a finite electron budget can be introduced perhaps etc. \n\n7. Minor point, regarding the **Results and Comparison** section: \n>“Initial large images were cropped into 13650 smaller images”\n\nHow is the cropping done? Would be nice to mention that the crops were generated randomly with or without any overlap, etc \n\n8. Minor point, regarding the **Dataset and Metrics** section:\n>“The process was repeated 5 times”. \n\nThis statement was not clear enough to me- does this mean that the ground truth was generated from 50 images of the same area? An explanation to this effect would be welcome.\n\n9. Minor point, regarding the **Denoising Module** subsection:\n>“L+1 stacked layers made of a building unit (BU) coupled with noise attention (Natt) and feedback blocks”. \n\nWhat does the variable `L` stand for? It would be good  to introduce it before in the text, prior to using it in the aforementioned sentence.  \n\n10. Minor point: The authors state that the “the source code is attached as supplemental material”, but that doesn’t seem to be the case yet. \n\n11. Minor point: Some of the figures (Fig. 6, 8, 9, 10) are hard to read: small font and smudgy bitmaps. \n\n\n\n\n\n", "summary_of_the_paper": "The authors propose an unsupervised learning framework called `NRRN` for denoising focused ion beam scanning electron microscopy  (*FIBSEM*) images - which are typically acquired with a high resolution along all dimensions - without requirement of any clean ground truth data during training. They do so by considering a triplet of images obtained from neighboring consecutive slices, employing a mean-squared error like loss formulation inspired from *Lehtinen et al, 2018 (N2N)* and *Wu et al, 2019* and proposing a novel recurrent neural network architecture. The authors show comparable results to supervised denoising approaches (which have access to clean ground truth data during training) on the *OHSU* dataset and on the simulated-noisy dataset from *EPFL*. ", "summary_of_the_review": "`NRRN` is a novel approach to address the challenging application of denoising *FIB-SEM* images. The authors back this up with state of the art results on a real and a noisy-simulated dataset. I rate this work slightly below acceptance threshold currently because even though the solution to this difficult biomedical problem  is very welcome, it is not evident to me how much of the performance of the method owes itself to a new architecture and how much benefit is gained by the employed loss formulation.  Furthermore, because of the zeroth-order approximation to the Taylor series along the z (third) axis implies to me that as soon as the voxel size along this dimension increases, the results would become sub-par. While this is okay and not a problem perhaps in the *FIB-SEM* context (?) I think it would be important to mention this explicitly or comment on how this can be fixed/handled if one were to extend this method and the proposed architecture to other domains. The writing in general is good but at times lacks some motivating sentences leading to the argument. Since the work concerns this specific biomedical application, perhaps the authors can consider submitting their work to a more domain-relevant conference or journal, in order to accrue a higher impact and relevant target audience.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ar4b0atKIJT", "reviewer": "Reviewer_aPxW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: biomedical imaging is one of the most important application areas of computer vision and machine learning. Designing methods that take into account the specific properties of a certain type of biomedical imaging is surely impactful and thus welcomed. The authors clearly demonstrate their expertise in FIB-SEM imaging. This method explores unsupervised models on very large high-resolution images, which is challenging.\n\nWeaknesses: Although this biomedical application is meaningful, the novelty behind this method is limited especially for a major ML conference. Other than the loss function, which is only a minor novel design, the proposed method is mainly an adaptation of an established approach. The authors did the experiments on only two datasets. I am not sure how generalizable is the proposed method to other FIB-SEM datasets or other similar imaging domains. \n\nMost importantly, the experiments showing structural details are preserved whereas noises are removed, are missing. The two metrics, PSNR and SSIM are essentially distance functions which are dominated by low-frequency features. However, the main advantage of learning-based methods over traditional methods, such as the Gaussian filter, should be to distinguish high-frequency features from noises. Since the ground truth is available for one dataset, I would suggest the authors to only keep the high-frequency region of the ground truth and compare with different denoising results.\n\nMinor concerns: \n1. Texts too small in Figure 1-4, 6, and 8. \n2. Related works should be better organized by including subsections.\n3. It is unclear how section 5.3 is related to transfer learning.\n", "summary_of_the_paper": "This paper proposes an unsupervised denoising convolutional neural network model for Focused Ion Beam-Scanning Electron Microscopy (FIB-SEM) images. The whole framework is adapted from the Noise2Noise approach. The proposed architecture Noise Reconstruction and Removal Network (NRRN) contains stacked layers of building units and noise attention modules to gradually denoise the input image. Additionally, a loss function, modified from the original loss function of Noise2Noise is specifically designed for FIB-SEM image taking into account its imaging properties. The results are demonstrated on two datasets, one public and one private, which outperformed both traditional and learning-based baselines. An ablation study is performed by varying the noise type and noise level.", "summary_of_the_review": "It is a good work overall with a clear presentation. However, the novelty is limited and more experiments should be performed to show the utility and advantage of the proposed method.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the challenging application of denoising FIB-SEM images. State-of-the-art results are reported on a real and a noisy-simulated dataset. Unfortunately, this paper failed to convince the reviewers and received 4 negative ratings. The paper misses critical comparisons against baselines and appears rather limited in scope. The authors failed to provide adequate answers to some of the reviewers' points.
Decision: Reject