Frequency Regularized Deep Convolutional Dictionary Learning and Application to Blind Denoising
Authors: Nikola Pavle Janjusevic
Amirhossein Khalilian-Gourtani
Yao Wang
Venue:  (2021)
Keywords: Dictionary learning
unrolled algorithm
convolutional sparse coding
interpretable deep learning
inverse problems
blind denoising
LISTA

Abstract: Sparse representation via a learned dictionary is a powerful prior for natural images. In recent years, unrolled sparse coding algorithms (e.g. LISTA) have proven to be useful for constructing interpretable deep-learning networks that perform on par with state-of-the-art models on image-restoration tasks. In this study we are concerned with extending the work of such convolutional dictionary learning (CDL) models. We propose to construct strided convolutional dictionaries with a single analytic low-pass filter and a set of learned filters regularized to occupy the complementary frequency space. By doing so, we address the necessary modeling assumptions of natural images with respect to convolutional sparse coding and reduce the mutual coherence and redundancy of the learned filters. We show improved denoising performance at reduced computational complexity when compared to other CDL methods, and competitive results when compared to popular deep-learning models. We further propose to parameterize the thresholds in the soft-thresholding operator of LISTA to be proportional to the estimated noise-variance from an input image. We demonstrate that this parameterization enhances robustness to noise-level mismatch between training and inference.

Under review as a conference paper at ICLR 2021
FREQUENCY REGULARIZED DEEP CONVOLUTIONAL DICTIONARY LEARNING AND APPLICATION TO BLIND DENOISING
Anonymous authors Paper under double-blind review
ABSTRACT
Sparse representation via learned dictionary is powerful prior for natural im- ages. In recent years, unrolled sparse coding algorithms (e.g. LISTA) have proven to be useful for constructing interpretable deep-learning networks that perform on par with mtateoof-the-an models on image-restoration tasks. In this study we are concerned with extending the work of such convolutional dictionary learn- ing (CDL) models. We propose to construct strided convolutional dictionaries with single analytic low-pass filter and set of learned filters regularized to occupy the complementary frequency space. By doing so, we address the neces- sary modeling assumptions of natural images with respect convolutional sparse coding and reduce the mutual coherence and redundancy of the learned filters. We show improved denoising performance at reduced computational complexity when compared to other CDL methods, and competitive results when compared popular deep-learning models. We further propose parameterize the thresh- olds in the soft-thresoolding operator of LISTA to be proportional to the estimated noise-varians from an input image. We demonstrate that this parameterization enhances robustness to noise-level mismatch between training and inference.
INTRODUCTION
Sparsity in a transform domain an important and widely applicable property of natural images. This property can be exploited in variety of tasks such signal representation, feature extraction, and image processing. For instance, consider restoring an image from a degraded version (noisy, blurry, missing pixels). These inverse problems are generally ill-posed and require utilizing ade- quate prior knowledge, for which sparsity has proven extremely effective (Mairal et al. 2014).
In recent years, such problems have been tackled with deep neural network architectures that achieve superior performance but are not well- sell-understood in terms of their building blocks. In this study, we interested in utilizing the knowledge from classical signal processing and spare coding literature to introduce a learned framework which interpretable and that can perform on-par with state-of- the-art deep-learning methods. We choose explore this method under the task of natural image denoising, in line with much of the recent literature (Sreter & Giryes) 2018 Simon & Elad 2019) Lecouat et al. 2020). As a benefit of this interpretability, we are able extend the framework for blind-denoising setting using ideas from signal processing.
In sparse representation we seek to approximate signal as a linear combination of few vectors from a set of vectors (usually called dictionary atoms). Olshausen & Field (1996). following a neuroscientific perspective, proposed adapt the dictionary to set of training data. Later, dictio- nary learning combined with sparse coding was investigated in numerous applications (Mairal et al.) 2009a Protter Hlad 2008). More specifically, for a of N image patches (reshaped into column vectors) X [.1,... ØNN
RRxN we seek to find the dictionary D' E m and the sparse xxy E - such that
representation Z
[zt, 1
N
D' Z' arg min Σ
z subject to: Dz, = Di, -i=1, -,,..,, ...
(1)
D,Z

---

Under review as a conference paper at ICLR 2021
This formulation is not tractable for large signals since minimizing the fu-pseudo-non posmu involves combinatorial optimization (Naaarajan 1995). To address this complication, a popular technique is to relax the problem by using the & noom with inverse problems such as denoising, learning the dictionary from the degraded signal has shown effective (Toič & Frossard, (011). Let y a; ni Ikm represent the noisy signal where n;
surrogate (Sreter & Giryes 2018). When dealing
follows an additive white Gaussian distribution, N I.). Then, the relaxed formulation can be written as
N 1
min
E1 s.t. 2 Dz, y: 1 S De D.Z
or min D.Z
Dz. v:122 A||zi||1
(2)
where À is a regularization parameter and is nontrivialy related to the representation error e. We will refer to this as the basis-pursuit denoising (BPDN) formulation of dictionary learning. Many iterative algorithms have been proposed in the literature to solve this problem (Mairal et al., 2014). A majority of these algorithms split the problem into step updating the dictionary followed by step solving for the sparse codes.
Note that learning a dictionary over independent image patches neglects the dependencies be- tween these patches. As a result, the models involving patch processing are inherently sub-optimal (Batenkov et al. 2017; [Simon & Elad, 2019). Although enforeing local priors on merged images Sulam & Elad] 2015) and utilizing self self-similarrty between patches (Mairal et al., 2009b) have been proposed as ideas to mitigate this flaw, ideally a global shift-invariant model is more appropriate. By constraining the dictionary to have a Toeplitz structure, the Convolutional Sparse Coding (CSC) model has been introduced which replaces the local patch processing with a global convolution (Grosse et al. 2007 Papyan et al. 2017).
Algorithms for solving the CSC model are also discussed in (Moreau & Gramfort 2019 Wohlberg, 2017). In this study, we are interested in interpretable CSC-based deep-learning models. A metric known as the mutual-coherence is well known to be related to the representation capability of the dictionary and is of special concern in using the CSC model with natural images (Simon & Elad,
2019). We take an alternative route to SSimnn& & Elad (209)) in addressing the mutual-coherence of CSC-based deep-learning models, which is both less computationally expensive and improves the denoising performance. We continue the discussion about CSC-based deep-learning models in Sec.
Another important aspect of the sparse representation is the sparse coding algorithm. For a given signal y Th and dictionary D, iterative soft-thresholding algorithm (ISTA) (Beck & Teboulle, 2009) finds the solution the BPDN functional, z" arg ming 1/2 || Dz 31 + A||=||1, by repeating the following iteration until a convergence criterion is reached:
=(++1) SS,,(1) (a" (7 (x) DT (D=(*)
(D)(") - -)) where So(w)) - sgn(x)) ([zr 0)+, 0 020. (3)
Here, 17( is the step-size of the descent algorithm at iteration k. Note that performing sparse coding with an iterative method like ISTA for all patches is computationally exhausting and slow. To resolve this issue, Gregor & LeCun (2010) proposed to approximate the sparse coding via learned differentiable encoder, dubbed LISTA. Further extensions of LISTA both in terms of practice and theory have been studied in the literature (Wu al. 2019 Chen al., 2018). More recently, using LISTA combined with dictionary learning has been research highlight (Sreter & Giryes| 2018 Simon & Flad, 2019, Lecouat et al. 20200. We refer to this type of models that leverages L.STA for convolutional dictionary learning as CDL models.
issue./rregor
1.1 RELATED WORKS
In this study, we are interested in the CDL model that concatenates LISTA network with linear convolutional synthesis dictionary. Let D be convolutional dictionary with M filters (and their integer shifts). We denote the filters in D by d where j E (1,.... M). Let Z, denote the sparse code for the data sample y, corresponding subband signal to d in Z; can be denoted as x/² Then the convolutional dictionary learning problem is written as
+ where i E 21,2.-. ... NJ and n ~ (N(0, T). The
M Σ
minimize di Z.
Σ d'
Il.
(4)
1=
2

---

Under review as a conference paper at ICLR 2021
Sreter & Giryes (2018) introduce the approximate convolutional sparse coding (ACSC) framework for "task--driver convolutional sparse coding". combining a convolutional extension of LISTA with a linear convolutional decoder. The proposed framework offers strategy for training an approximate convolutional sparse coding network and a corresponding convolutional dictionary in an end-to-end fashion. They demonstrate competitive performance against classical patch-hased methods such as K-SVD (Aharon et al., (006), on image denoising and image inpainting. Our proposed baseline model (CDLNet) differs from the ACSC model by use of mean-subtraction preprocessing, employ- ing small-strided convolutions, and imposing a norm-come corm-constraint on the synthesis dictionary.
Simon & Elad (2019) extend the framework of Sreterr & Giryes (2018) by considering the role of stride in the stable recovery of signals and proposed the "CSCNet" framework. They argue that the CSC model for image representation in a sparse domain is limited by the inclusion of "smooth filters", which are required to represent the piecewise smooth characteristics of natural images. This limitation manifests itself in the maximum cross-correlation between atoms of the dictionary, known as the mutual-coherence. They empirically show that using relatively large stride, while processing shifted-duplicates of the input, improves denoising performance of the model. Although using large stride reduces the mutual coherence of the learned filters, all possible shifts of the image need to be processed and averaged, yielding a model very similar to patch- processn..sssing.. We propose a frequency regularization strategy to mitigate the problem of smooth-varying filters which does not require shift-averaging.
Note that the parameter À in equation 4 depends on the desired sparsity, relative the noise-level, and is directly related to the threshold values in ISTA. /Sreter & Giryes) (2018) propose to learn different thresholds for each channel, effectively changing the regularizer term in equation to Σ5=1 2 ||. Inspired by the benefit of minimax-concave (MC) penalty (Selesnick) 2017) over l norm, PPokala et al. (2020) propose "ConFirmNet" where trn-thresholding function is used in the network. Kim & Park (2020) propose a signal adaptive threshold scheme for LISTA where the threshold is decreased if the previous estimate of an element is large.
Mohan et al. (020) explore the role of bias-vectors in popular deep-learning network's convolution operators. They advocate for eliminating the biases completely to improve generalization blind- denoising where there is mismatch between training and inference noise level. [sogawa et al. (2017) propose altering the biases of deep neural-networds by scaling them with the input noise standard- deviation. Their method is ultimately non-blind denoising scheme as they use the ground-truth noise statistics during training and inference. In contrast, we propose a blind-denoising scheme that is motivated by the interpretation of the biases in LISTA as thresholds and employ a scaling by the noise variance (in the last layer of LISTA), estimated from the input signal during training and inference. Performance of different denoising techniques on other noise distributions have also been studied in the literature, which is not the focus of this study (Abdelhamed et al.) 20118 Plotz & Roth, 2017).
1.2 CONTRIBUTION OF THIS STUDY
The unrolled convolutional sparse coding and dictionary learning frameworks have led to the field dubbed "interpretable deep-learning". The networks constructed in such way have the benefit of interpretability and decreased parameter count while performing quite closely to other state-of-the- art deep-learning models. In this study we further extend such frameworks. We propose utilizing a strided convolutional dictionary with a fixed low-pass channel and set of frequency-regullaizze learnt filters (Section 2.2). Our experimental results demonstrate that such frequency regularization and small stride leads to more interpretable dictionary filters than the prior work. Consequently, by limiting the number of low-pass atoms the dictionary and using stall-strided convolutions, we address the modeling assumptions associated with the convolutional sparse coding model (Section 2. Additionally, leveraging interpretability of our network, we propose to parameterize the soft- thresholding operator in LISTA such that the thresholds are proportional to the estimated input noise- level for a given image (Section 2.3) Experimentally, we show improved denoising performance at reduced computational complexity compared to other frameworks (Section 3.2 Furthermore, our parameterization of the learned thresholds greatly improves robustness to noise-level mismatch between training and inference and increases the generalizability of the network (Section 3.3
3

---

Under review as a conference paper at ICLR 2021
2 PROPOSED FRAMEWORK
2.1 CONVOLUTIONAL DICTIONARY LEARNING NETWORK (CDLNET)
We seek to solve the natural image denoising problem via the convolutional dictionary learning model on the BPDN functional,
minimize d.Zz
Σ j=1
d 2
|| Σ j=1
subject to: ||d 1 S siyjee (1. M). (5)
norm constraint imposed on the dictionary atoms to remove the arbitrary scaling of coefficients, in Mairal et al.
(2014). We propose the following learned CDL model. dubbed CDLNet, which
involves a LISTA module followed by a learned convolutional synthesis dictionary, D,
④ (2( (k+1) Sgie) (z( A (B())(() v)) b ..... K 1, (0) 0 (6)
where ISTA has been unrolled for steps. Here, A and ((k) are small-strided convolution analysis and synthesis operators respectively. We untie the parameters at each iteration of LISTA following the theoretical analysis of Chen et al. (2018). A threshold vector 0 < ((²) E R is
learned corresponding to the M subbands the convolutional sparse code iteration k.
The reconstructed signal is given by L. The total learnable parameters are given by Q {{A(k), (k), (*)) K-1 T== {d' [d'),,1} Note that traditional LISTA network requires supervised training on sparse codes computed from ISTA. On the other hand, the CDLNet can learn to ap- proximate sparse coding and the dictionary in an unsupervised fashion by minimizing suitable loss function designed for the image reconstruction task (Sreter Giryes 2018) (i.e. unsupervised the code-domain, but supervised in the dictionary learning strategy of alternating between computing sparse codes and updating the dictio- nary, however, the sparse coding is done via a learned algorithm with fast inference. An alternative unsupervised LISTA training strategy, which minimizes the BPDN functional (equation 2 was pre- sented in Ablin et al. (2019). As in (Sreter & Giryes 2018 Simon & Elad, 2019), we employ & loss between the restored image and its ground-truth clean image throughout this study.
In this sense the network mimics the common
=
8 D
Synthesis Conv ((4
Analysis Cony A
Conv Dictionary
Soft-tureholding Spo
Figure 1: Block diagram of CDLNet.
2.1.1 A DISCUSSION ON MUTUAL COHERENCE OF THE LEARNED DICTIONARY
The approximately piecewise smooth nature of natural images will require a synthesis dictionary to contain "smoothly- varootthy-varriigg low-pass atoms. Simon & Elad (2019) discuss, such low-pass atoms pose problem for BPDN, A sufficient condition for the faithful recovery of the tn sparse code from an ε basis pursuit can be given in terms of the dictionary's 's mutual coherence, (DD). Note that for matrix A with normalized columns a,, we have {{AA) max |a. a,y- For the convolutional Ar dictionary, the atoms of D are composed of the shifts of its filters, {d' jj=1- This poses a problem will
in that the inner product between any of such low-pass filters and their own greatly increase the mutual coherence and potentially harm the reconstruction performance of the system.
Sreter & Giryes (0018) do not address this issue in the ACSC framework. Simon & Elad (2019) propose to use large strides on the order of the filter size, along with averaging reconstructions from shifted input signals effectively returning patch- approach. In CDLNet we use small
4

---

Under review as a conference paper at ICLR 2021
strided convolutions (stride=2, in both horizontal and vertical directions) without averaging construction scheme. Furthermore, we use a preset low-pass filter, and parameterize other filters to be in the complimentary frequency space of the low-pass. We empirically show that the combi- nation of the proposed regularization scheme and small stride reduces the mutual coherence of the dictionary, improves denoising performance of the model, and reduces the computational cost.
2.2 FREQUENCY REGULARIZATION OF A CONVOLUTIONAL DICTIONARY
In this section we propose a method for regularizing the synthesis dictionary to contain only a single low-pass filter. Note that in the BPDN formulation, the hyperparameter determines trade-off between data-fidelity to the observation, y, and sparsity of the transform domain coefficients, x. Following Sreter Giryes (2018). we extend this a vector, on the expected levels of sparsity in different subbands of the decomposition. The learned thresholds, ((k) ultimately reflect these weights, representing sparsity priors on each subband. In the case of natural image reconstruction, their piecewise smooth nature necessitates a subband decomposition which contains an approximation signal, for which sparsity prior is ill-suited.
R-M to reflect prior knowledge
To address these assumptions, we designate the first channel of the sparse code as the approximation signal and fix its corresponding synthesis filter to an analytic low-pass filter. Note that total variation regularization of the low-pass signal has been previously studied (Elad et al.) 2005 Lalanne et al,, 2020), however, we're concerned with regularizing the dictionary elements for reasons concerning mutual coherence. Knowing which subband the approximation signal lives allows us to remove it from the soft-thresholding operation (0) = 0), thereby removing any misplaced assumption of sparsity. Further, we wish to ensure no additional now-pass filters are learned during training so that we are not inadvertently violating the sparsity assumptions of the model (i.e. thresholding other low- frequency subbands) and reduce the mutual coherence of dictionary. This restriction on the number of low-pass filters has the added benefit of improving stable recovery bounds of the dictionary as discussed in Section 2.1.I
The issue of learning figh-pass/bannie pass filters is both non-trivial and ill-posed. If we naively assert that such a set of filters must simply be "non-low-pass" we may consider projecting filters onto the set of zero-mean filters there by removing their DC-component. However, this allows for the learning of filters whose frequency response is arbitrarily close to DC. Alternatively, preprocessing the signal by removing its low-pass component is not effective it produces noisy -reeuequency signal and does not properly regularize the learned filters. As demonstrated in Appendix [5]] even when the input images are preprocessed in this way, the learned dictionary can still contain low-pass filters leading to high mutual coherence.
A more apt characterization is consider the learning of filters occupying the frequency-space complement to that of the low-pass filter. Let h denote a fixed low-pass filter and ö h be its high-pass complement, where is the discrete Dirac delta function. We formalize the regularization by considering the following effective dictionary elements,
d' hh d' = d', |d'|l221 11 j-
We refer to {d' (d and {d' 1 as the effective and learned filters respectively. Signal recon- struction is ultimately performed with the effective filters which compose D. Note that the norm constraint is necessary to avoid large responses in the transition band of the low-pass filter. By explicitly denoting which subbands of our decomposition are expected to be sparse, this regu- larization technique forms sufficiently expressive model for the reconstruction of natural im- ages. This has the added benefit of nearly eliminating the correlation between the atoms corre- sponding the lowpass filter, d' and the atoms corresponding to high-freuuency filters, d², as dt d' h (6 h) d (h- h h) d' R for j # 1.
2.3 BLIND DENOISING: NOISE- LEARNED THRESHOLDS
As presented, the CDLNet model and any similar network utilizing LISTA is not amenable to gen- eralizing denoising performance across a set of noise levels. Note that the threshold values in soft- thresholding operator are directly proportional to the expected sparsity and the noise level in each subband Bayram& & Selesnick (0010). As a result, the sparsity hyperparameter, X, and consequently the threshold values should be functions of the noise variance, i.e. ((k) (kk) (a
5

---

Under review as a conference paper at ICLR 2021
We thus propose to parameterize the thresholds in the last layer in CDLNet as ((K) ok00 where o, is the estimated noise variance which can be estimated from the input noisy image, and (K)) is a vector containing the learned scaling factors for different subbands. We employ a com- monly used estimator, On Mcdian (lc])/0.6744 where denotes the diagonal-detail Wavelet subband of an input image (Chang et al.) 2000; Mallat [008] [Dnohoo & Johnstone| 1994, 1995). The proposed parmeterization of thresholds is inspired by the MAP estimate of orthogonal Wavelet denoising under a Laplace distribution prior of the high-frequency coefficients and the Gaussian distribution prior on the noise(Bayram & Selesnick] 2010). This parameterization enables the pro- posed CDLNet to handle varying ipput noise-levels while maintaining the integrity of CDLNet as an unfolded dictionary learning model.
3 EXPERIMENTAL SETUPS AND RESULTS
Models: are trained via stochastic gradient descent on the Ey-loss with parameter constraints,
minimize 9= (=(( B00 000) - (2))
- E(y; 1 subject to: k²k) 0 Ak, || 1
Vj..
where the parameter constraints are enforced by projection onto the constraint set after each gradient step. Models of different capacity are trained by varying the number of unrollings and number of subbands M. Filters are of size 7 x 7. CDLNet is used to refer to our proposed base-model, differing from other mentioned CDL methods (ACSC (Sreter & Giryes) 2018 and CSCNet (Simon & Elad 2019)) by its use of stride-2 convolutions, mean-subtraction of ipput signals, and the above projection operations during training. A 3 3 isotropic Gaussian filter (o=0.6) is used as the analytic low-pass filter for frequency-regulariieed models, denoted FCDLNet. We use (F)CDL.Ne++ to refer to networks with noise-adaptive thresholds as in section 2.3 In blind denoising cases, the noise level is estimated using the estimator in section 2.3 both during training and inference. Implementation and trained models are provided herd
Dataset: All CDLNet models and variants are trained on the BSD432 dataset (Martin et al. 2001 Random crops of size 128 x 128 are flipped, rotated, and batched online during training. Independent identically distributed Gaussian noise is drawn from on E trtain uniformly within each batch and added to the ground-truth signal. As preprocessing, all images are normalized by 255 to have range of [0, 1] and mean of each image is subtracted. Testing is performed on the associated BSD68 test-set (Martin et al. 2001
Training: performed with the Adam optimizer (Kingma & Ba 2015), using its default settings in PyTorch. Mini-batches consist of 10 samples. A learning rate of 1e-3 is set at the start of training and reduced by a factor of 0.95 every 50 epochs. Training is run until convergence. As advised by Lecouat et al. (2020), backtracking is used to correct for model divergence by reloading the most recent checkpoint within the last 10 epochs and reducing the learning rate by a factor of 0.8.
Initialization: A single set of M filters are initialized by drawing from standard normal distri- bution and subsequently normalized w.rt each filter. This corresponds to our expectation that most filters will learn to be approximately zero-mean and spatially localized. We found that this initial- ization greatly improves convergence speed over drawing from a standard uniform distribution. All convolution operators are initialized with this same weight. Following Simon & Elad (2019). we then normalize A (k) by the spectral norm L (A)
(*) B (x))) [|2, which corresponds initializing the step-sizes of ISTA to (*() L/L. Thresholds are initialized to ((x) 1e-1/L.
3.1 EFFECT OF FREQUENCY REGULARIZATION AND STRIDE ON LEARNED DICTIONARIES
To validate the effectiveness of small-stride and the proposed frequency regularization on the learned synthesis dictionary, we train three CDLNet models containing convolutions with (a) no stride, (b) stride 2, and (c) stride 2 with frequency regularization. For all models M=32, K=20, and oraain=25 Figure 2 shows the learned filters in the spatial and frequency domain. Without stride, the learned dictionary consists of some "noise-like" filters with non-localized frequency responses and few directional filters. The stride 2 model (b) learns more directional filters and overall dictionary with

---

Under review as a conference paper at ICLR 2021
lower mutual-coherenee compared to (a). However, both (a) and (b) produce multiple low- frequeeyycc filters in unpredictable channels. With frequency regularization added in (c), we are able to control the subband in which our ww-frequency information is located. The learned filters in (c) are all directional or texture high-pass, and the mutual-coherence is decreased as predicted.
(n) Stride-1, 1, Freq.reg. reg. n(D) 0.87
(b) Stride=2. No Freq, reg.
(e) Stride=2, with Freq. ((D)) 0.59
p(D) - 0.77
Figure 2: Learned Filters (top row) and their corresponding frequency responses (bottom row).
Boxes highlight low-pass filters as seen in the frequency domain. Observe that non-frequency regu- larized dictionaries (a.b) have multiple of such filters in non-determined locations, in contrast to (c). The mutual coherence between dictionary elements (D) is reduced by incorporating stride and frequency regularization. Filters in (c)
interpretable than those reported in ACSC (Sreter
& Giryes 2018 and CSCNet (Simon & Elad 2019)
3.2 DENOISING PERFORMANCE AGAINST OTHER FRAMEWORKS
In this section we demonstrate the efficacy of the proposed methods on single noise-level grayscale image denoising. We train two FCDLNet models of varying capacity (FCDLNet with M/64, K=10 and Big FCDLNet with M= and K=30 f We compare these to the classic collaborative filtering method BM3D (Dabov al. 2007), popular convolutional neural network based methods FFDNet (Zhang et al. 2018) and DnCNN (Zhang et al. 2017), and CDL method proposed by Simon & Elad (2019), CSCNet. All learned methods have been trained on the same dataset, BSD432. Average peak signal-to-noise ratio (PSNR) on BSD68 testset is shown in Table Visual comparison between the above mentioned models and FCDLNet presented in Figure
The FCDLNet with trainable parameters on the order of CSCNet shows improved performance across neise- Interestingly. Big FCDLNet observed to compete very well with state-of- the-art deep-learning denoising networks. This is done without the use of common deep-learning tricks such as Datch-normaiization or residual learning (both of which are employed DnCNN). The ability to train larger CDLNet models of competitive performance without such methods may suggest an appeal to more interpretable networks.
The average run-time at inference of different models averaged over Set-12 (Sreter & Giryes 2018) images of size 512 x 512 also given in Table The timing experiments were conducted with an Intel Xeon E5 at 2.6GHz CPU, Nvidia P40 GPU, and 4GB of RAM, running Linux version 3.10... We observe that by leveraging small-strided convolutions and forgoing the "shift-duplicate processing" of CSCNet, FCDLNet has significantly reduced (10x 20x) computation time both GPU and CPU compared to CSCNet, while having better denoising quality.
Corresponding filters available here.

---

Under review as a conference paper at ICLR 2021
Table I: Denoising performance (PSNR) on BSD68 testset (o
0
tain
0
BM3D
FFDNet
DrCNN
CSCNet
FCDLNet
Big FCDLNet
15
31.07
31.63
31.72
31.40
31.45
31.66
25
28.57
29.19
29.22
28.93
28.99
29.22
50
25.62
26.29
26.23
26.04
26.11
26.30
Params
486k
556k
64k
66k
510k
CPU time (sec)
17.06
14.76
0.76
9.93
GPU time (sec)
0.34
0.03
0.14
(s) Original
(b) Noisy (20.13 dB)
(e) BM3D (28.08 dB) (d) CSCNet (29.34 dB) (e) FCDLNet (29.38 dB)
Figure 3: Visual comparison of different models for noise level on = 25. PSNR value for each
image is given in parentheses. Details are better visible by zooming images.
3.3 ROBUSTNESS TO NOISE LEVEL MISMATCII IN TRAINING AND INFERENCE
In this section we provide experimental results regarding the generalization of the networks across noise-levels. The main focus to investigate the effect of the proposed blind denoising framework (section 2.3 especially for cases with mismatch between noise-range during training (a frainn)) and testing (a
In Figure we show the average PSNR values for three different training noise ranges: (a) [0, 20), (b) [15, 35,. and (c) 50). Networks are trained by uniformly sampling the noise-level within the training range at each iteration. All networks have close to 120k learnable parameters with M=64 and K=20. The trained networks are then tested on different noise levels test [0, 50), and average PSNR is calculated over the BSD68 dataset.
As shown in Figure 4 all networks perform closely over the training noise-range. On the other hand, when tested noise-levels outside the training range, the networks with adaptive thresholds (as in Section 2.3p perform superior compared to others. In spite of increasing input signal- signal-to-noise to-noise ratio for noise-levels below the training range, observe that models without noise-adaptive thresholds have diminishing performance returns (note the plateau of CDLNet/FCDLNet in af in (b) and !est [0, 30) in (c)). On the other hand, denoising behavior of models with noise- adaptive thresholds (CDLNet+Blind and FCDLNet+Blind) extends to the lower noise-range. Simi- larly, we observe that models without noise-adaptive thresholds have more significant performance drop compared to noise-adaptive models when generalizing above the training noise level. Another notable observation is that FCDLNet models perform better than their non-frequency regularized counterparts in low noise-levels due to the proper treatment of the low-pass signal.
test [0, 15]
We also compare the generalization of the proposed networks against other CDL methods, Pokala et al, (2020) propose ConFirmNet model where they use firm-thresholding in LISTA and show bet- ter performance compared to ACSC (Sreter & Giryes) 2018) when training and testing noise levels different. Results from Pokala al. (2020) are summarized and compared to our framework in Table 2 FCDLNet performs on par with ConFirmNet when o train 20. To allow the proposed
scaling parameters (K)) to properly fit to the noise-variance we train over o [18, 22). As in Table and from discussion above, simply training over noise range gives marginal improvement. However, when combined with noise-adaptive thresholds (FCDLNet+Blind), we ob- serve significant improvement in generalization over other methods.
8

---

Under review as a conference paper at ICLR 2021
34-
Training
Range
CDLNNet
40-
FCDLNe:
PSNR (d6))
35
CDLNet-Blied
JCDL Ne-BBlad
SAN (003
30-
FCDD Net Bllnd
30
25
Traiting Noise Rango
CDLNet
20
TCDL
20
CUUNet Ulind
PCDL iind
10
20
30
40
50
10
20
30
00
50
20
30
40
Tesling Notie Lovel
Toking Noise Level
Toding Netoo Level
(a) o.tain = [0, 20 o
[0, 50)
(b) mmin
35], of = (0,50)
(c) o.tain = [30, 50), o. [0, 50
Figure 4: Generalization of the network for different training noise range. Average PSNR is calcu- lated over BSD68 testset.
Table 2: Generalization of the network: mean (sd) PSNR on Set-9 (Pokala et al.
2020
testing set.
Comparisons with ACSC (Sreter & Giryes 2018) and ConFirmNet (Pokala et al.
2020
20
18, 22
ACSC
ConFirmNet
CDLNet
FCDLNet
CDLNet
FCDLNet
FCDLNet Blind
3
32.02 (0.02)
32.23 (0.01)
32.04 (0.01)
32.17 (0.01)
32.76 (0.01)
32.81 (0.01)
34,25 (0.02)
15
31.88 (0.03)
32.04 (0.03)
32.00 (0.03)
32,06 (0.03)
32.24 (0.03)
32.30 (0.04)
32.45 (0.03)
30
22.89 (0.03)
23.13 (0.04)
23.68 (0.04)
23.70 (0.05)
24.34 (0.06)
24.51 (0.05)
25.31 (0.06)
4
DISCUSSION AND CONCLUSION
In this study we investigated unrolled convolutional sparse coding and dictionary learning frame- works. These frameworks have the benefit of interpretability while maintaining similar performance compared to other state-of- of-theeart deep learning models. We proposed employing strided convo- lutional dictionary constructed with a fixed lowpass filter and a set of learned frequency regularized filters. As illustrated, small-strided and frequency regularized convolutions give the benefit of re- duced mutual coherence of the dictionary and properly address the modeling assumptions regarding convolutional sparse coding. We showed that learned high-pass filters are more structured cover- ing different orientations and textures. In comparison to other CDL models of similar parameter count, proposed framework showed improved denoising performance whilst reducing the com- putational cost. The learned dictionary filters are more interpretable with lower mutual coherence. Additionally, experimental results with FCDLNet models of similar size to the state-of-the-ar de- noising models showed competitive denoising performance.
We further investigated the generalizability of CDL networks in scenarios where noise-level mis- match exists between training and inference. Leveraging the interpretability of CDLNet, we pro- posed to parameterize the thresholds in LISTA such that they are scaled based on the estimated input noise variance. Experimental results demonstrated that this reparamcterization greatly improves the robustness to noise-level mismatch between training and testing and increases the generalizability of the network.
In future work we aim explore the possible extensions of the proposed models and further leverage the interpretability of this framework. The proposed frequency regularization scheme provides the required grounds for multiresolution representation learning. Note that by further processing of the fixed lowpass channel one can achieve multiresolution representation while in other frameworks the lowpass information is represented in multiple, non-predetermined channels, making this exten- sion challenging (see discussion in Section 3.1). Further augmenting the thresholds of the CDLNet model to be employed at each layer of LISTA, with both signal and noise adaptivity, is a promis- ing direction for improved generalization of the network. Additionally, investigating other noise distribution models is an exciting avenue of research.
REFERENCES
Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEF Conference on Computer Vision and Pattern

---

Under review as a conference paper at ICLR 2021
Recognition, pp. 1692-1700, 2018.
Pierre Ablin, Thomas Moreau, Mathurin Massias, and Alexandre Gramfort. Learning step sizes for unfolded sparse coding. In Advances in Neural Information Processing Systems, pp. 13100- 13110, 2019.
Michal Aharon, Michael Elad, and Alfred Bruckstein. K-SVD: An algorithm for designing over- complete dictionaries for sparse representation. IEEE Transactions on Signal Processing, (1): 2006.
Dmitry Batenkov, Yaniv Romano, and Michael Elad. On the global-local dichotomy in sparsity modeling. In Compressed Sensing and its Applications, pp. 1-53. Springer, 2017.
llker Bayram and Ivan W. Selesnick. A subband adaptive iterative bhrinkagetthresholdinng algorithm. IEEE Transactions on Signal Processing, 58(3):1131-114: March 2010.
Amir Beck and Mare Teboulle. A fast iterative shrinkage-thrresholdnng algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2()):183-202, 2009.
S Grace Chang, Bin Yu, and Martin Vetterli. Adaptive wavelet thresholding for image denoising and compression. IEEE' Transactions on Image Processing, 99): 2000.
Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Theoretical linear convergence of unfolded ISTA and its practical weights and thresholds. In Advances in Neural Information Pro- cessing Systems, pp. 9061-9071, 2018.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16 (8):2080-2095, 2007.
D. Donoho and Johnstone. Ideal spatial adaptation by wavelet shrinkage. Biometrika, 81:425-455, 1994.
David Donoho and lain Johnstone. Adapting to unknown smoothness via wavelet shrinkage. Journal of the American Statistical Association, 90: 90:1200-1224, 1995.
Michael Elad, L. Starck, P. Querre, and D. L. Donoho. Simultancous cartoon and texture image in- painting using morphological component analysis (MCA). Applied and Computational Harmonic Analysis, 19(3):340-355 35800 2005.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th International Conference on Machine Learning, ICML'10 pp. 399-406, 2010.
Roger Grosse, Rajat Raina, Helen Kwong, and Andrew Y. Ng. Shift-Invariant Sparse Coding for Audio Classification. Cortex, 8:9, 2007.
Kenzo Isogawa, Takashi Ida, Taichiro Shiodera, and Tomoyuki Takeguchi. Deep shrinkage con- volutional neural network for adaptive noise reduction. IEEE Signal Processing Letters, 25(2): 224-228, 2017.
Dohyun Kim and Daeyoung Park. Flement- adaptive thresholds for learned iterative shrinkage thresholding algorithms. IEEE Access, 8:45874 45886, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Clement Lalanne, Maxence Rateaux, Laurent Oudre, Matthieu P. Robert, and Thomas Moreau. Extraction of Nystagmus Patterns from Eye-Tracker Data with Convolutional Sparse Coding. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 928-931, Montreal, QC, Canada, July 2020. IEEE.
Bruno Lecouat, Jean Ponce, and Julien Mairal. Fully trainable and interpretable non-local sparse models for image restoration, 2020.
10

---

Under review as a conference paper at ICLR 2021
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. In Proceedings of the 26th International Conference on Machine Learning, pp- 689-696, 2009a.
Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, and Andrew Zisserman. Non-local sparse models for image restoration. In Proceedings of 12th IEEE International Conference Computer Vision, pp. 2272-2279, 2009b.
Julien Mairal, Francis Bach, and Jean Ponce. Sparse Modeling for Image and Vision Processing. Foundations and Trends@ in Computer Graphics and Vision, 8(2-3):85-288 2833,2233222 2014.
Stephane Mallat. A Wavelet Tour of Signal Processing: The Sparse Way. Elsevier Science, 2008.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics, In Proceedings of Eighth IEEE International Conference Computer Vision, volume 2, pp. 2001.
Sreyas Mohan, Zahra Kadkhodaie, Eero P. Simoncelli, and Carlos Gemandez-Granda Robust and interpretable blind image denoising via fies-free convolutional neural networks. In International Conference Learning Representations, 2020.
Thomas Moreau and Alexandre Gramfort. Distributed convolutional dictionary learning (dicodile): Pattern discovery in large images and signals. arXiv preprint arXiv: 09235,, 2019.
Balas Kausik Natarajan. Sparse approximate solutions to linear systems. SIAM Journal on Comput- ing. 24(2):227 2344, 1995.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, (8666)))) 1996.
Vardan Papyan, Jeremias Sulam, and Michael Elad. Working locally thinking globally: Theoretical guarantees for convolutional sparse coding. IEEE Transactions on Signal Processing, 65(21): 5687-5701, 2017.
Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 1586-1595, 2017.
Praveen Kumar Pokala, Prakash Kumar Uttam, and Chandra Sekhar Seelamantula. ConFirmNet: Convolutional FirmNet and application to image denoising and inpainting. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8663- 8667, 2020.
Matan Protter and Michael Elad. Image sequence denoising via sparse and redundant representa- tions. IEEE Transactions on Image Processing. 18(1):27-35, 2008
Ivan Selesnick. Sparse regularization via convex analysis. IEEE Transactions on Signal Processing, 65(17):448144494 2017.
Dror Simon and Michael Elad. Rethinking the CSC model for natural images. Advances in Neural Information Processing Systems, pp. 2274-2284, 2019.
Hillel Sreter and Raja Giryes. Learned convolutional sparse coding. In Proceedings of IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2191-2195, 2018.
Jeremias Sulam and Michael Elad. Expected patch log likelihood with sparse prior. In Interna- tional Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 111 Springer, 2015.
Ivana Toić and Pascal Frossard. Dictionary learning: what is the right representation for my signal. IEEE Signal Processing Magazine, 28(2):27-38, 2011.

---

Under review as a conference paper at ICLR 2021
Brendt Wohlberg. Sporco: A python package for standard and convolutional sparse representations. In Proceedings of the 15th Python in Science Conference, Austin, 7X, USA, pp. 1-8, 2017.
Kailun Wu, Yiwen Guo, Ziang Li, and Changshui Zhang. Sparse coding with gated learned ISTA. In International Conference on Learning Representations, 2019.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng. and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 26(7):3142-3155 2017.
Kai Zhang, Wangmeng Zuo, and Lei Zhang. FFDNet: Toward a fast and flexible solution for CNN- based image denoising. IEEE Transactions Image Processing, 27(9):4608 4622, 2018.
5 APPENDIX
5.1 REMOVING THE LOW-PASS INFORMATION FROM THE INPUT IMAGES IS NOT EFFECTIVE FOR REDUCING MUTUAL COHERENCE
Instead of the proposed frequency regularization approach, it may be tempting to simply remove the low-pass information of the signal preprocessing step. More specifically, y be the noisy image, one can process the high-pass signal y h+ y with the proposed network without adding any frequency regularization. Note that this approach is not equivalent to proposed frequency regu- larization scheme as the removed low-past channel (h- * y) is still noisy. Although the thresholds for the low-pass channel in FCDLNet are set to zero, this does not mean that the low-pass information removed from the denoising framework. As result, in FCDLNet, the pass filtering each LISTA stage is ultimately a filtering of an incrementally cleaner image, producing a full subband de- composition of cleaned up low and high frequency bands the final stage of LISTA. Additionally, note that removing the low-frequency component of the signal does not stop the learned dictionary from learning multiple (redundant) low-pass filters. Even though the input signal does not contain low-frequency information, the filters are not necessarily regularized to take advantage of this prop- erty. As shown figure we observed that the filters learned in this scheme have multiple low-pass channels.
(a) Stride=2,, No Frq. rog. Low-pass subtraction p(D) 0.66
Stride=2, with Frq. reg.
((D)) 0.59
Figure 5: Learned Filters (top row) and their corresponding frequency responses (bottom row). Boxes highlight low-pass filters as seen in the frequency domain. Observe that non-frequency reg- ularized dictionary when low-pass information removed (a) has multiple of such filters in non- determined locations, in contrast to (b). The mutual coherence between dictionary elements (μ(D) reduced by incorporating the frequency regularization.
12


Reviews:
[{"review_id": "y8ty_4C6Md", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper proposes a denoising method with a neural network inspired from convolutional dictionary learning. In the proposed method, one atom of the dictionary is constrained to be a low frequency filters and all other atoms are to be high frequency atoms. The authors also propose to make the threshold depends on the noise level to better adapt to different noise level and to use strided convolution to reduce the computational cost of the method. The method is then evaluated on images from BSD68.\n\n----\n\n*For extra citations, see bibtex at the end. Sorry if I ended up putting a lot of them but I feel the bibliography is a bit lacking.*\n\n## Overall assessment\n\n- A major weakness of this work is that it seems to be equivalent to doing CDL on a signal filtered with high pass filter $g^{-1} * y_i$ because $\\theta_0 = 0$. Indeed, using parseval on the data fitting term and the equivalence between convolution and point wise multiplication in the frequency space, one can easily show that $z^1_i$ will correspond to the low frequency part of $y_i$ and as the filters $d^i$ all only have high frequency, they will try to reconstruct the high frequency part of $y_i$. This means that only keeping high frequency of $y_i$, one can use the classical CDL approach algorithm to recover the same method as the one proposed in this paper. To me, the method boils down to stating an equivalent model where the preprocessing is integrated in the model. If the authors think I am wrong, they could try to perform experiment 3.1 and show that there is significant difference between filtering $y$ and using the proposed method. At least, this point should be mentionned and discussed in the manuscript. Note that such approach of integrating multiple component can be related to the work on morphological component analysis (see `[Elad2005]`) and the integration of preprocessing step in CDL have recently been proposed with detrending in `[Lalanne2020]`.\n- Apart from this point, the novelty of this work is not very significant. The adaptation of the thresholds with the input noise level in the context of denoising has been proposed in `[Isogawa2017]` and `[Ramzi2020]` and  the use of strided convolution are proposed in `Simon & Elad (2019)`.\n- The effect of the stride on the denoising is not evaluated. This would be interesting to look at how the denoising performance change when changing the stride. In particular, how does FCDLNet compare with FCDLNet with `stride` $\\in\\\\{1, 3, 4\\\\}$. Does it impact the performances a lot?\n- It would also be interesting to study the impact of the noise level estimator. If the estimator is biased, how does it impact the performances? This is also related to Question 3, as I suppose if the estimator is biased but used for training, the thresholds $\\nu$ can also be adapted to cope for this bias. This would be interesting to add such experiments.\n- For the computational complexity, I would be interested to see comparison with modern convolutional sparse coding algorithm such as the one in `sporco` (`[Wohlberg2017]`) or the LGCD algorithm from `[Moreau2019]`, which scales to much larger images.\n- The writing is not very clear and not always correct and there are many typos (see bellow).\n\n\n## Some question\n\n1. Could the authors comment on why the learned network is more interpretable than a classical network?\n2. Why  does the authors change the constraint to the one in (7) ? This makes the model somewhat incomparable with other approaches as $d^j$ won't have the unit norm property. Moreover, I don't really see the point, as simply stating that $\\\\|d^j\\\\| \\le 1$ also constrains appropriatly $\\\\|\\tilde d^j\\\\|$ to be smaller than roughly $\\frac{1}{\\\\|g\\\\|}$, which is a similar constraint but comparable to the original one.\n3. When training the network, is the true noise level given as an input of the model or is it also estimated using the wavelet based estimator? This is unclear from the text and should be clarified.\n4. What is the training time for the proposed model? This is not discussed, but I guess this is similar to the training time of other models.\n\n\n## Minor comments, nitpicks and typos\n\n- For the citation, when they are between parenthesis, could the authors use `\\citep` to have proper formating.\n- p.1: `Mairal et al. (2014)`: This is not an arXiv paper, the proper citation is `[Mairal2014a]`.\n- p.1: The first paragraph could be improved a lot. It is unclear why this would be an inverse problem (there is no sensing matrix here, the dictionary correspond to the prior knowldge in the inverse problem literature). This is mainly denoising so the paragraph should be fixed to reflect this.\n- p.1: `a linear combination of a collection of vectors`: this is simply linear representation. The sparse linear representation also promotes the usage of only a few atoms.\n- p.1: `where $n_i \\sim \\mathcal N(0, \\sigma^2 \\bm I)$.` The authors should add in plain word that this `is an additive Gaussian white noise.`\n- p.2: `is nontrivialy` -> `non trivialy`? What does it mean to be trivialy related? I would remove this as it is unclear.\n- p.2: `the Convolutional Sparse Coding (CSC) model has been introduced`: the original work introducing such model is `[Grosse2007]`.\n- p.3: `interpretabile` -> `interpretable`\n- p.4: Note that there is a third option for training LISTA networks which is to use loss in Eq.(5) as a training loss, as it is done for instance in `[Ablin2019]`.\n- p.5: `the Guassian distribution prior` -> `Guassian`\n- Table.1: Could the authors highlight the leading method in the table?\n- p.7: It is unclear whether the timing for FCDLNet is performed for full image denoising or on 128x128 patches.\n- Figure.4: Could the authors add the error bars in this plot? As the gain is small between CDLNet and FCDLNet, it would be interesting to see the confidence here.\n\n### References\n\n```bibtex\n@inproceedings{Ablin2019,\n  title = {Learning Step Sizes for Unfolded Sparse Coding},\n  booktitle = {Advances in {{Neural Information Processing Systems}} ({{NeurIPS}})},\n  author = {Ablin, Pierre and Moreau, Thomas and Massias, Mathurin and Gramfort, Alexandre},\n  year = {2019},\n  pages = {13100--13110},\n  address = {{Vancouver, BC, Canada}},\n  archivePrefix = {arXiv},\n  copyright = {All rights reserved},\n  eprint = {1905.11071},\n  eprinttype = {arxiv}\n}\n\n@article{Elad2005,\n  title = {Simultaneous Cartoon and Texture Image Inpainting Using Morphological Component Analysis ({{MCA}})},\n  author = {Elad, Michael and Starck, J. L. and Querre, P. and Donoho, D. L.},\n  year = {2005},\n  volume = {19},\n  pages = {340--358},\n  journal = {Applied and Computational Harmonic Analysis},\n  number = {3},\n  pmid = {16370462}\n}\n\n@article{Grosse2007,\n  title = {Shift-{{Invariant Sparse Coding}} for {{Audio Classification}}},\n  author = {Grosse, Roger and Raina, Rajat and Kwong, Helen and Ng, Andrew Y.},\n  year = {2007},\n  volume = {8},\n  pages = {9},\n  journal = {Cortex}\n}\n\n@article{Isogawa2017,\n  title={Deep shrinkage convolutional neural network for adaptive noise reduction},\n  author={Isogawa, Kenzo and Ida, Takashi and Shiodera, Taichiro and Takeguchi, Tomoyuki},\n  journal={IEEE Signal Processing Letters},\n  volume={25},\n  number={2},\n  pages={224--228},\n  year={2017},\n  publisher={IEEE}\n}\n\n@inproceedings{Lalanne2020,\n  title = {Extraction of {{Nystagmus Patterns}} from {{Eye}}-{{Tracker Data}} with {{Convolutional Sparse Coding}}},\n  booktitle = {2020 42nd {{Annual International Conference}} of the {{IEEE Engineering}} in {{Medicine}} \\& {{Biology Society}} ({{EMBC}})},\n  author = {Lalanne, Clement and Rateaux, Maxence and Oudre, Laurent and Robert, Matthieu P. and Moreau, Thomas},\n  year = {2020},\n  month = jul,\n  pages = {928--931},\n  publisher = {{IEEE}},\n  address = {{Montreal, QC, Canada}},\n}\n\n@article{Mairal2014a,\n  title = {Sparse {{Modeling}} for {{Image}} and {{Vision Processing}}},\n  author = {Mairal, Julien and Bach, Francis and Ponce, Jean},\n  year = {2014},\n  volume = {8},\n  pages = {85--283},\n  journal = {Foundations and Trends\\textregistered{} in Computer Graphics and Vision},\n  number = {2-3}\n}\n\n@article{Moreau2019,\n  title={Distributed Convolutional Dictionary Learning (DiCoDiLe): Pattern Discovery in Large Images and Signals},\n  author={Moreau, Thomas and Gramfort, Alexandre},\n  journal={arXiv preprint arXiv:1901.09235},\n  year={2019}\n}\n\n@inproceedings{Ramzi2020,\n  title = {Wavelets in the {{Deep Learning Era}}},\n  booktitle = {European {{Signal Processing Conference}} ({{EUSIPCO}})},\n  author = {Ramzi, Zaccharie and Starck, Jean-Luc and Moreau, Thomas and Ciuciu, Philippe},\n  year = {2020},\n  month = jul,\n  pages = {1417--1421},\n}\n\n@inproceedings{Tolooshams2018,\n  title = {Scalable Convolutional Dictionary Learning with Constrained Recurrent Sparse Auto-Encoders},\n  booktitle = {{{IEEE International Workshop}} on {{Machine Learning}} for {{Signal Processing}} ({{MLSP}})},\n  author = {Tolooshams, Bahareh and Dey, Sourav and Ba, Demba},\n  year = {2018},\n  archivePrefix = {arXiv},\n  eprint = {1807.04734v1},\n  eprinttype = {arxiv}\n}\n\n@inproceedings{Wohlberg2017,\n  title={SPORCO: A Python package for standard and convolutional sparse representations},\n  author={Wohlberg, Brendt},\n  booktitle={Proceedings of the 15th Python in Science Conference, Austin, TX, USA},\n  pages={1--8},\n  year={2017}\n}\n\n```", "extra_scores": {}, "extra_text": {}}, {"review_id": "YOiJgsusmrf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new regularization for the dictionary in the learned convolutional sparse coding model of Sreter & Giryes '18. The main contribution is that the dictionary is regularized to be composed of 1) a fixed low-pass filter and 2) a set of learned filters to occupy the complementary high-frequency space. A second contribution is that the thresholding in the network is adjustable according to the estimated noise level in the image. \n\nComments:\n\n- Presentation-wise, I didn't get the motivation of the work after reading the introduction. Everything before Sec. 1.2 is a narrative of the existing methods, and all of a sudden Sec. 1.2 states the proposed method, but what is the problem that the paper wants to address?\n\n- The major contribution is the introduction of a fixed low-pass filter in the dictionary, but I don't see a clear justification as to why it is needed. If you want to model the DC component, why not simply use a bias term?\n\n- The paper tells a story that the dictionary need to be incoherent, that some prior work enforces this by using large stride convolution, and that the proposed method does not need to use large stride. But ultimately there is no incoherence regularization for the high-frequency part of the dictionary in the proposed method. So how do you enforce those high-frequency filters to be incoherent?\n\n- It is stated as a second main contribution of the paper that the thresholding in the network can be made to be dependent on the noise level in the image, so that the network is capable of performing blind denoising. However, the explanation for how it works (Sec. 2.3) is very sketchy. Why is the adaptive threshold only used in the last layer? Also, the threshold is set to be a multiple of noise variance, but how do you estimate the noise variance in practice? Where does the number 0.6745 come from?\n\nOverall, the presentation of the paper needs major improvement to make the motivation as well as technical details clear. In addition, the technical contributions appear quite minor and are not fully justified: it is unclear how fixed low-pass filter compares with a bias term, and it is unclear how effective the adaptive threshold is in practice. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "-VtbbWc50C", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a convolutional dictionary learning (CDL) work for image denoising. Compared with existing CDLnet, the proposed method used stride convolutions with pre-defined low-pass channels  to improve the performance and reduce complexity. Results on image denoising showed comparable performance to the state-of-the-arts.\n\n\nStrength:\n\n1. The idea of constraining low-pass atoms in learned dictionary is interesting, which limits the space of learned atoms to have the desired behavior.\n\n2. The proposed method parameterize the soft-thresholding operator in LISTA such that the thresholding is directly linked with the estimated image noise level.\n\nWeaknesses:\n\n1. The contribution of this paper is incremental. The proposed method is heavily based on existing CDLNet with some modifications. Although such modifications improved results a little bit, the motivation is not very clear and the implementation is empirical, e.g., Eq. (7).\n\n2. The proposed method did not substantially improve the results. It improves the baseline CSCNet a little bit, but still not good as the state-of-the-art methods such as DnCNN. In addition, all the experiments are using synthetic data with ideal Gaussian noise. Actually image noise is more complex and is a mixture of Poisson and Gaussian. Under controlled environment, the proposed method still cannot outperform the state-of-the-art. The results on real image noise might be even worse.\n\nAbdelhamed, Abdelrahman, Stephen Lin, and Michael S. Brown. \"A high-quality denoising dataset for smartphone cameras.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n\nPlotz, Tobias, and Stefan Roth. \"Benchmarking denoising algorithms with real photographs.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n\n3. More analysis of the proposed components are missing. For example, there is no ablation study for the proposed components.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a deep learning approach to blind image denoising based on deep unrolling. In particular, the proposed network is derived from convolutional sparse coding algorithms, which are unrolled, untied across layers and learned from data. The paper proposes a frequency domain regularization scheme in which the filters consist of a single analytically defined low-pass filter and a large collection of filters which are constrained to reside in the mid-to-high frequency ranges. It also proposes to tie the thresholds in the soft-thresholding stages of the learned network to estimates of the noise variance, making the proposed scheme more robust to variations in the noise level. 

Pros and Cons:

[+] Having a single low-pass dictionary atom reduces redundancy (and potentially coherence) in the learned dictionary. This type of regularization may also reduce the time/data required to learn. 

[+/-] Using noise estimators and a noise adaptive threshold renders the model more robust to variations in the noise level. This is important, since in most denoising applications the noise level is not known a-priori. As the reviewers note, the idea of tuning thresholds in an unrolled sparse coding method based on the noise level is not a novelty of the paper; the novelty here is coupling this with a wavelet-based estimate of the noise level. 

[-] All three reviewers raise concerns regarding the novelty of the work compared to existing convolutional sparse coding-based neural networks. The structure of the network is similar; the main difference is the frequency restriction for learned atoms, which is enforced by prefiltering the learned atoms with a high-pass filter. 

[-] The paper is not entirely clear in its motivation and argumentation. Reducing the coherence of the learned dictionary makes sense from the perspective of certain worst case results from sparse approximation. However, the coherence is a worst case quantity; moreover, certain approaches to coherence control (e.g., using large stride) control coherence at the expense of the expressiveness of the dictionary, and hence may not actually improve its ability to provide sparse reconstructions of natural signals. The proposed frequency domain regularization is a sensible approach to controlling coherence, since low-frequency atoms will tend to be highly coherent, but would benefit from a crisper analytical motivation. 

[-] Reviewers found the experiments lacking in some regards. In particular, the paper only evaluates its proposals on synthetic experiments with Gaussian noise. While this is in line with some previous work on deep learning based denoising, more extensive and realistic experiments would have bolstered the paper's argument. 

Overall, the paper makes a sensible proposal regarding the adaptivity to unknown noise levels, and introduces a potentially useful frequency-domain restriction on the learned filters in a CSC network. However, the reviewers did not find that the paper made a clear argument for the significance of these proposals, and raised other concerns regarding the clarity and experiments. The consensus of the reviewers is to recommend rejection. 
Decision: Reject