Sparse Coding with Multi-layer Decoders using Variance Regularization
Authors: Katrina Evtimova
Yann LeCun
Venue: Accepted by TMLR ()

Abstract: Sparse representations of images are useful in many computer vision applications.
Sparse coding with an $l_1$ penalty and a learned linear dictionary requires regularization of the dictionary to prevent a collapse in the $l_1$ norms of the codes. Typically, this regularization entails bounding the Euclidean norms of the dictionary's elements.
In this work, we propose a novel sparse coding protocol which prevents a collapse in the codes without the need to regularize the decoder. Our method regularizes the codes directly so that each latent code component has variance greater than a fixed threshold over a set of sparse representations for a given set of inputs. Furthermore, we explore ways to effectively train sparse coding systems with multi-layer decoders since they can model more complex relationships than linear dictionaries.
In our experiments with MNIST and natural image patches, we show that decoders learned with our approach have interpretable features both in the linear and multi-layer case. Moreover, we show that sparse autoencoders with multi-layer decoders trained using our variance regularization method produce higher quality reconstructions with sparser representations when compared to autoencoders with linear dictionaries. Additionally, sparse representations obtained with our variance regularization approach are useful in the downstream tasks of denoising and classification in the low-data regime. 

Published in Transactions on Machine Learning Research (08/2022)
Sparse Coding with Multi-layer Decoders using Variance Regularization
Katrina Evtimova
k
Center for Data Science New York University
Yann LeCun
Courant Institute and Center for Data Science
New York University
Meta AI FAIR
Reviewed on OpenReview: https: // net/ forum? id=44uls11j7
Abstract
Sparse representations of images are useful in many computer vision applications. Sparse coding with penalty and learned linear dictionary requires regularization of the dic- tionary to prevent collapse in the 4 norms of the codes. Typically, this regularization entails bounding the Euclidean norms of the dictionary's elements. In this work, we propose a novel sparse coding protocol which prevents a collapse in the codes without the need to regularize the decoder. Our method regularizes the codes directly so that each latent code component has variance greater than a fixed threshold over a set of sparse representations for a given set of inputs. Furthermore, we explore ways effectively train sparse coding systems with multi-layer decoders since they model more complex relationships than linear dictionaries. In our experiments with MNIST and natural image patches, show that decoders learned with our approach have interpretable features both in the linear and multi-layer case. Moreover, we show that sparse autoencoders with multi-layer decoders trained using our variance regularization method produce higher quality roconstructions with sparser representations when compared to auloencoders with linear dictionaries. Ad- ditionally, sparse representations obtained with our variance regularization approach are useful in the downstream tasks of denoising and classification in the low-data regime.
1 Introduction
Finding representations of data which are useful for downstream applications is at the core of machine learning and deep learning research (Bengio et al.) 2013). Self-supervised learning (SSL. is branch of representation learning which is entirely data driven and in which the representations are learned without the need for external supervision such as semantic annotations. In SSL, learning can happen through various pre-text tasks such as reconstructing the input with autoencoders (Vincent et al. 2008), predicting the correct order of frames in a vidco [Misra et al.) 2016 Fernando al.] (017), enforcing equivariant representations across transformations of the same image (He et al.) 2019; Grill et al. 2020 Zbontar al.) 2021 Bardes
et al..
2021 enforcing other invariance properties (Földiák 1991 Wiskott & Sejnowski 2002 Hyvärinen
et al.
2003 Chen et al. 2018). The pre-text tasks should be designed to extract data representations usefuu
in other downstream tasks.
We focus on learning representations of images through sparse coding, a self-supervised learning paradigm basod on input reconstruction (Osshausen & Field 1996). reprosentation (or code) in the form of feature vector e R! is considered sparse if only small number of its components are active for given data sample. Sparsity introduces the inductive bias that only small number of factors relevant for single data point

---

Published in Transactions on Machine Learning Research (08/2022)
and it is desirable property for data representations as it can improve efficiency and interpretability of the representations (Bengio 2009; Bengio et al. 2013). While it is an open debate, sparse representations of images have been shown to improve classification performance (Ranzato et al. 2007 Kavukcuoglu et al.
2010
Makhzani & Frey 2013 and are popular for denoising and inpainting applications (Elad & Aharon
2006 Mairal et al. 2007]. They are biologically motivated as the primary visual cortex uses sparse activations
of neurons to represent natural scenes (Olshausen & Field 1996 Yoshida & Ohki, 2020).
R which can reconstruct imput y E R" using learned decoder D. Typically, the decoder is linear, namely D E gdxi, , and the sparse code z selects a linear combination of few of its columns to reconstruct the input. In practice, a sparse code z is computed using an inference algorithm which minimizes z's 4 norm and simultaneously optimizes for a good reconstruction of the imput y using D's columns. Learning of the decoder D typically happens in an iterative EM-like fashion. Inference provides set of sparse codes corresponding to some inputs (expectation step). Given these codes, the decoder's paramcters are updated using gradient descent to minimize the error between the original inputs and their reconstructions (maximization step). Sparse coding systems in which the decoder is non-linear are hard to learn and often require layer-wise training (Zeiler et al. 2010 He et al., 2014). In this work, we explore ways to effectively train sparse coding systems with non- lincar decoders using end-to-end learning.
In sparse coding, the goal is to find a latent sparse representation
In order to successfully train a sparse coding system which learns a linear dictionary D, it is common to bound the norms of the dictionary's 's weights in order to avoid collapse in the I norms of the codes. Indeed, consider a reconstruction ÿ obtained from a dictionary D and code 2, namely ÿ Dz. The same reconstruction can be obtained from a re-scaled dictionary D' = cD and code z' = z/c for any non-zero multiple c. When 1, the representation z' has smaller ( norm than but produces the same reconstruction. In practice, if D's weights are unbounded during training, they become arbitrarily large, while the 4 norms of the latent representations become arbitrarily small which leads to a collapse in their I norm. Typically, in order to restrict a lincar dictionary D, its columns are re-scaled to have a constant l2 norm (Lee et al., 20066 or weight decay is applied to its weights (Sun et al., 2018). However, in the case when D a multi-layer neural network trained end-to-end, it is not clear what the best strategy to bound the decoder's weights is in order to avoid collapse in the codes and at the same time learn useful features.
Instead of restricting D's weights, we propose to prevent collapse in the 4 norm of the codes directly by applying variance regularization to each latent code component. In particular, we add a regularization term to the energy minimized during inference for computing set of codes. This term encourages the variance of each latent code component to be greater than a fixed lower bound. This strategy is similar to the variance principle presented in Bardes et al. (2021) and related to methods that directly regularize the latent codes (Földiak 1990 OIshausen & Field, 1996; Barello et al.] 2018). Our strategy ensures that the norms of the sparse codes do not collapse and we find that it removes the need to explicitly restrict D's parameters. Since this variance regularization term is independent from D's architecture, we can use it when the decoder is deep neural network. In this work, we experiment with fully connected sparse autoencoders with a one hidden layer decoder and successfully train such models using our variance regularization strategy.
The main contributions of our work are:
We introduce an effective way to train sparse autoencoders and prevent collapse in the I norms of the codes that they produce without the need to regularize the decoder's weights but instead by encouraging the latent code components to have variance above a fixed threshold.
We show empirically that linear decoders trained with this variance regularization strategy are comparable to the ones trained using the standard 4-- g-normalization approach in terms of the features that they learn and the quality of the reconstructions they produce. Moreover, variance regularization successfully extends sparse coding to the case of a non-linear fully connected decoder with one hidden layer.
Additionally, our experiments show that sparse representations obtained from autoencoders with a non- linear decoder trained using our proposed variance regularization strategy: 1) produce higher quality reconstructions than ones obtained from autoencoders with linear decoder, 2) are helpful in the down- stream task of classification in the low-data regime.
2

---

Published in Transactions on Machine Learning Research (08/2022)
y
y
E
((Z,ZE)
Z
D
R(z)
C yy y y
minimize w.r.t. 2
(a) Inference of a sparse code 2.
E C(z", _228)
y
y
E
*
D
C( y ÿ
minimize w.r.t. & and D
(b) Learning of the encoder & and decoder D.
Our sparse autoencoder setup. C' denotes mean squared error. The inference step in Figure la finds a code x" which minimizes the reconstruction error between the input and its reconstruction ğ. The code is subject to sparsity and variance regularization, denoted by R(z). Additionally, the code is regularized stay close the encoder's prodictions zg. During the learning step in Figure [b the encoder & is trained to predict the codes z" computed during inference and the decoder D learns to reconstruct inputs y from the sparse codes z².
Figure 1:
2 Method
A schematic view of our sparse autoencoder setup is presented in Figure [ At a high level, given an input y and a fixed decoder D, we perform inference using modified version the FISTA algorithm (Beck & Teboulle, 2009) find sparse code x' which can best reconstruct y using D's elements. The decoder's weights trained by minimizing the mean squared error between the input y and the reconstruction ý computed from z". The encoder & is trained to predict the codes 2" computed during inference. More details on each of the components in Figure I are presented in the following sections.
2.1 Background: Sparse Coding and Dictionary Learning
Inference Sparse coding algorithms with an I sparsity penalty and a linear decoder E Rdxl perform inference to find a latent sparse representation z e IR' of a given data sample y E IR which minimizes the following energy function:
E(z,y,D) - - 1ily-D2133 +||2||1-
(1)
The first. lerm in T is the reconstruction error for the sample using code and decoder D. The second term is a regularization term which penalizes the I norm z. In essence, each code z selects a linear combination of the decoder's columns. The decoder can thus be thought as dictionary of building blocks. Inference algorithms find solution for the minimization problem in (1 keeping D fixed, namely
arg min E(z, y, D).
3

---

Published in Transactions on Machine Learning Research (08/2022)
Learning the Decoder The decoder's parameters can be learned through gradient-based optimization by minimizing the mean squared reconstruction error wetween elements in the training set. y {y),.... 9y C
Rd and their corresponding reconstructions from the sparse representations Z = {zi..... 2 J C R' obtained during inference, namely:
arg ('
,)) arg min
y
D2j
i=1
where D's columns are typically projected onto the unit sphere avoid collapse in the latent codes, namely ||D., for D., Rd.
2.2 Background: Inference with ISTA and FISTA
We provide an overview of the Iterative FISTA which we adopt in our setup. ISTA is a proximal gradient method which performs inference to find a sparse code z" which minimizes the energy in [1 for a given input and a fixed dictionary D. We can re-write the inference minimization problem as:
Algorithm (ISTA) and its faster version
arg min E(z,y, D) = arg min f(=) + g(=),
(3)
2
1
where f(z) == 211y fly- D=113 is the reconstruction term and g(z) == A||=||a the I penalty term. In ISTA,
2
the code z can be given any initial value and is lypically set to the zero vector: z [0) O R². An ISTA iteration consists of two steps: gradient step and a shrinkage step.
Gradient Step To compute code (k) from its previous itcration (k-11), where E (1...., K}, ISTA first performs the gradient update by taking gradient step size Ti with respect the squared error between the input and the reconstruction from (k=1);
ISTA gradient step: {()) = (R-1) - 1/. V ((((1-1)),
(4)
The size of the gradient step т at iteration & can be determined with backtracking BBeck & Teboulle 2009
Shrinkage Step The shrinkage step is applied to the intermediate output from the gradient step in ISTA. It uses the shrinkage function which sets components of its input to 0 if their absolute value is less than a or otherwise contracts them by a, namely [ra(=)], = sign(z;))(zg) - a)+ for some threshold a > 0 where j traverses the components its input vector æ. The shrinkage step for ISTA is given by:
ISTA shrinkage step: 2(h) Tym (z(+))
where the hyperparameter which determincs the sparsity level. In the case when the codes are restricted be non-negative, a property that we adopt in our setup, the shrinkage function can be modified to set all negative components in its output to 0: [=)(;)) (a)))))
FISTA In our setup, we use FISTA (Beck & Teboulle 2009 by adding momentum term in each of its iterations. Namely, 2 (k-1) in 140 is replaced by:
which accelerates the convergence of ISTA
(k) = (k-11) + tk-1 (z (k-1) (k-2))
(6)
tx
1+ 11 and 4 = 1. Thus, the FISTA gradient step becomes:
for k 2 2 where la
(k)) (k) mh V (x(
(7)
FISTA employs the same shrinkage step as ISTA.
4

---

Published in Transactions on Machine Learning Research (08/2022)
2.3 Modified Inference with Variance Regularization the Latent Code Components
In order to prevent collapse in the 4 norm of the latent codes, propose to ensure that the variance of each latent code component stays above pre-set threshold. To achieve this, we add a regularization term to the energy in (3) which encourages the variance of all latent component across mini-batch of codes to
1 be greater than pre-sel threshold. In particular, we replace the function [(z) 2 ly D=ll2 in (3) with a new function f(Z) defined over a set of codes Z Rix" corresponding to a mini-batch of data samples Y (V....., yn as:
f(Z) Σ ly. -Z.,11 + Σ 3 (T - (( /ar (a.) \ar(Zz)),_]²' 1
(8)
The first sum in 8
is over the reconstruction terms from each code Z E R². The second sum in is over squared hinge terms involving the variance of each latent component Zj.; E R" across the batch
(
where Var 2j.))
E m3)² and Py is the mean across the j-th latent component, namely The hinge terms non-zero for any latent dimension whose variance is below the fixed
My = 1 i== Zz, threshold of VT.
Modified Energy: Variance Regularization Using f(Z) from (8) and sctting g(Z)
Σ g (Z.),, our
modified objective during inference is to minimize the energy:
= ((z)
2 Σ
2 Di - -DZ,,111 i=1
Σ 3 (T- V (Va((z;))) j=1
2 ]
+ Σ All Z.,ll
(9)
with respect to the codes Z for mini-batch of samples Y and fixed dictionary D. Note that the hinge terms counteract with the l penalty terms in ( The hinge terms act regularization terms which encourage the variance of each of the latent code components to remain above the threshold of vT. This should prevent collapse in the 4 norm of the latent codes directly and thus remove the need to normalize the weights of the decoder.
Gradient Step The first step our modified version of FISTA is to take a gradient step for each code Z, e R':
ž (1)) (k-1)) Th V
Z (()))
(10)
In practice, since we use FISTA, the code - (10) replaced by a momentum term as defined in (6)
but here we omit this notation for simplicity. The gradient of the sum of reconstruction terms in ( with respect Zg,t, one of the latent components of code Ze,, for a linear decoder D is:
yi - DZ. 12 = D. - y:),
(11)
07,
1=1
where D. denotes the s-th column of the dictionary D respect to Za is:
The gradient of the sum of hinge terms in with
29 T- Var Za.)) Var(Z Zs.))
0
Σ 3 [( - /Vr(
(Za,s Us), if /Var (Z..) T
(12)
OZ.t
otherwise.
The gradient for any neural network decoder be computed through automatic differentiation.
5

---

Published in Transactions on Machine Learning Research (08/2022)
Even though the hinge terms in (8) are not smooth convex functions] the fact that the gradient in is line implies that the hinge term behaves locally as convex quadratic function. A visualization of h(()) (1-V [1 for IR² and the full derivation of (12) available in Appendix A.2 and A.3 respectively.
12
Shrinkage Step Similarly to the regular FISTA protocol, the shrinkage step in our modified FISTA is applied to the intermediate results from the gradient step in 10
2.4 Encoder for Amortized Inference
One limitation of the proposed variance regularization is that the modified inference procedure depends on batch statistics (see equation 12) and thus the codes are not deterministic. To address this limitation, we propose to train encoder simultancously with the decoder DD to predict the sparse codes computed from inference with modified FISTA. After training of the encoder and decoder is done, the encoder be used to compule sparse codes for different inputs independently from each other in deterministic way. Additionally, using an encoder reduces the inference time: instead of performing inference with FISTA, the encoder computes sparse representations directly from the inputs and thus provides amortized inference.
Learning the Encoder The encoder trained using mini-batch gradient descent minimize the following mean-squared error between its predictions and the sparse codes Z Rix computed from inference with FISTA for inputs Y {yi,... ..,,....,..
argmin LE(E, Z", () arg min
[8(ys) 1:.
(13)
Note that the codes Z" are treated as constants. Training the encoder this way similar to target propa- gation [Le et al. 2015).
Modified Energy: Variance and Encoder Regularization To encourage FISTA to find codes which can be learned by the encoder, we further modify the inference protocol by adding term to f(Z) which penalizes the distance between FISTA's outputs and the encoder's s outputs:
j(Z)
Σ
8 3 DZ.
+3 Σ [T (T- Var(Zz)))
1² + Z E(1))1
(14)
This regularization term ensures that the codes computed during inference with FISTA do not deviate too much from the encoder's predictions. In our setup, the encoder's predictions are treated as constants and are used the initial value for codes in FISTA, namely _ Eyy.. This reduce the inference time by lowering the number of FISTA itcrations needed for convergence if the encoder provides good initial values. With the new f(Z) from equation 14 the energy minimized during inference becomes:
B(Z.Y.D)
n Σ
2 11 DZ.
DZ
Σ (T V Var Z,.)) ) +7 Σ 1=1
72
Z
8(1)) + Σ All Z..lh.
(15)
2 requirement the FISTA algorithm.

---

Published in Transactions on Machine Learning Research (08/2022)
3 Experimental Setup
3.1 Encoder and Decoder Architectures
LISTA Encoder The encoder's architecture is inspired by the Learned ISTA (LISTA) architecture Gregor & LeCunj 2010). LISTA is designed to mimic outputs from inference with ISTA and sembles recurrent neural network. Our encoder consists of two fully connected layers U E Rdxi and S E Rixi, bias term b E R², and ReLU activation functions. Algorithm T describes how the encoder's outputs are computed. Our version LISTA pro- duces non-negalive codes by design. We refer to our encoder as LISTA in the rest of the paper.
Algorithm LISTA encoder E: forward pass
Input: Image y E Rd number of iterations L
Parameters: U E Rdxi, S E R b E R!
Output: sparse code Zg E R! u Uy +
20 = ReLU(u) for 1 do
2; ReLU(u + S2,-1)
end for
_g 21.
Linear Decoder A linear decoder is paramcterized simply as a linear transformation W which maps codes in R' to reconstructions in the input data dimension R L.e W Rdxi No bias term used in this linear transformation.
Non-Linear Decoder In the case of non-lincar decoder D, we use a fully connected network with one hidden layer of size m and ReLU as an activation function. We refer to the layer which maps the input codes to hidden representations as W1 E Rm and the one which maps the hidden representations to the output (reconstructions) as W2 E Rdxm There is a bias term bj E RM following Wi and no bias term after W2. Given an input 2 E R², the decoder's output is computed as: y W2(W1 + b1)+.
3.2 Evaluation
We compare sparse autoencoders trained using our variance regularization method to sparse autoencoders trained using other regularization methods. We refer to our approach as uuriance-repllarzeed dictionary learning (VDL) in the case when the decoder is a linear dictionary and as VDL-NL when the decoder is non-linear. We refer to the traditional approach in which the norm of the decoder's columns is fixed standard dictionary learning (SDL) in the case the decoder is linear and SDL-NL in the case it is non-lincar natural extension of SDL to the case of non-linear decoders is to fix the l2 norms of the columns of both layers W1 and W2. Additionally, autoencoders in which only weight decay is used to avoid collapse which we refer to as WDL and WDL-NL in the case of a lincar and a non-linear decoder, respectively. Finally, we consider a decoder-only sctup in which the standard approach of fixing the l2 norm of the decoder's columns is applied and the codes are computed using inference with FISTA. We refer to this setup as DO when the decoder is linear and as DO-NL when the decoder is non-linear.
compare our variance regularization method to sparse
In order to compare the different models, train them with different levels of sparsity regularization X, evaluate the quality of their reconstructions in terms of peak signal-to-noise ratio (PSNR), and visualize the features that they learn. We evaluate models with a linear decoder on the downstream task of denoising, Ad- ditionally, we measure the pre-trained models' performance on the downstream task MNIST classification in the low data regime.
3.3 Data Processing
MNIST In the first set of our experiments, we use the MNIST dataset (LeCun & Cortes 2010 consisting of 28 x 28 hand-written digits and do standard pre-processing by subtracting the global mean and dividing by the global standard deviation. We split. the training data randomly into 55000 training samples and 5000 validation samples. The test set consists of 10000 images.
Natural Image Patches For experiments with natural images, we use patches from ImageNet ILSVRC-
2012 DDng et al. 2009). We first. convert the ImageNet. images to grayscale and then do standard pre-
7

---

Published in Transactions on Machine Learning Research (08/2022)
processing by subtracting the global mean and dividing by the global standard deviation. We then apply local contrast normalization to the images with 13x 13 Gaussian filter with standard deviation of 5 following Jarrett et al. (2009) and "Zeiler et al. (2011). We use 200000 randomly selected patches of size 28 x 28 for training. 20000 patches for validation, and 20000 patches for testing.
3.4 Inference and Training
We determine all hyperparameter values through grid search. Full training details can be found in Appendix
B.1
The codes Z are restricted to be non-negative as described in scction 2.2 The dimension of the latent codes in our MNIST experiments is | - 128 and in experiments with ImageNett patches is / = 256. The batch size is set to 250 which we find sufficiently large for the regularization term on the variance of each latent component in g for VDL and VDI-NI models. We set the maximum number of FISTA itcrations K to 200 which we find sufficient for good reconstructions.
Inference
Autoencoders Training We train models for a maximum of 200 epochs in MNIST experiments and for 100 epochs in experiments with natural image patches. We apply carly stopping and save the antoencoder which outputs codes with the lowest average energy measured on the validation set f In SDL and SDL-NL experiments, we fix the la norms of the columns in the decoders' fully connected layers W, Wi, and W2 to be equal to 1. We add weight decay to the bias term b in models with non-linear decoder well as to the bias term b in LISTA encoders to prevent their norms from inflating arbitrarily.
Implementation and Hardware
Our PyTorch implementation is available on GitHub at We train our models on one NVIDIA RTX 8000 GPU card
com wtt.g/tuwb.oom///ettooraaaa and all our experiments take less than 24 hours to run.
4 Results and Analysis
4.1 Visualizing Learned Features
Linear Decoders Figures 2 shows the dictionary elements for two SDL and two VDL decoders trained with different levels of sparsity regularization X. As evident from Figures 2a and 2b we confirm that, in the case of standard dictionary learning on MNIST, the dictionary atoms look like orientations, strokes, and parts of digits for lower values of the sparsity regularization À and they become looking like full digit prototypes as X increases (Yu 2012). A similar transition from strokes to full digit prototypes observed in figures [c] and [2d displaying the dictionary clements of models trained using variance regularization in the latent code components. The same trend is observed in dictionary elements from WDL and DO models (Figure 12 in the Appendix). Figure shows dictionary elements for two SDL and two VDL. models trained ImageNet patches. All of these models contain gratings and Gabor-like filters. We observe that models which are trained with a lower sparsity regularization parameter X (Figures 3a and 3c contain more high frequeney atoms than ones trained with higher values of À (Figures 3b and
Non-linear Decoders Each column W. E I2 in a linear dictionary D (parameterized by W) reflects what a single latent component encodes since W., = ( where el)) R' is a one-hot vector with 1 in position and Os elsewhere. We use a similar strategy to visualize the features that non-linear decoders learn. Namely, we provide codes (f)) with only one non-zero component as inputs to the non-linear decoders and display the resulting "atoms" in image space While the linear case each code component selects a single dictionary column, in the case of non-linear decoders with one hidden layer, each code component selects linear combination of columns in W2. Two SDL-NL and two VDL-NL models trained on MNIST with different levels of sparsity regularization À yield the atoms displayed in Figure 4 They contain strokes and orientations for smaller values of X and fuller digit prototypes for larger values of X and are very similar to the dictionary elements from linear decoders in Figure 2 We use the same strategy to visualize the features that non-linear decoders trained on ImageNet patches learn. These features are displayed in Figure 5 for
3 As reminder, codes computed using amortized inference with the encoder during validation. 4n practice, we remove the effect of the hias term by after W1 and display D(e(0)- D(0).
8

---

Published in Transactions on Machine Learning Research (08/2022)
6
6
7
7
6
6
6
3
8
9
8
5
9
5
9
6
0
2
3
?
9
7
0
S
0
8
6
7 7
8
6
9
2
7
(a) SDL, À 0.001
(b) SDL, 0.005
(c) VDL., À 0.005
(d) VDL. 0.02
Figure 2: Dictionary elements for linear dictionaries from SDL and VDI. models with latent dimension = 128 and different levels of sparsity regularization x. The SDL models in (2a] and (2b) produce reconstructions with PSNR of 21.1 and 18.6 for codes with average sparsity level of 63% and 83% on the lest set, respectively,
The VDL models in (2ch and (2d) produce reconstructions with PSNR of 20.7 and 17.7 for codes with average sparsity level of 69% and 91.8% on the test set, respectively.
SDL, = 0.002
(b) SDL, 0.005
(c) VDL, 0.005
(d) VDL, X 0.01
Figure Dictionary elements which resemble gratings and Gabor filters for SDL and VDL models with latent dimension 1 256 trained on ImageNet patches with different levels of sparsity regularization A. The SDL models in 3a and 3b produce reconstructions with average PSNR of 26.9 and 24.0 and average sparsity level in the codes of 74.6% and 90.5% on the lest set, respectively. The VDL models in Figures 3c and 3d produce reconstructions with average PSNR of 27.3 and 25.3 and average sparsity level in the codes of 77.3% and 89.2% on the test set, respectively.
SDL-NL and VDL-NL models trained with different levels of sparsity regularization A. As in the case of lincar decoders, we observe that both SDL-NL and VDL-NL models learn gratings and Gabor-like filters.
Encoders Figure 14 in the Appendix shows visualizations of the U Rdxi layer of the LISTA encoder SDL, VDL, and WDL models with code dimension 128 trained on MNIST images and different levels of sparsity regularization X. Similarly to the decoders in 11 the features learned by the encoders resemble orientations and parts of digits but in the case of WDL there is larger proportion of noisy features compared to the other models as À increases.
4.2 Reconstruction Quality
Figure plots regret curves for the trade-off between average sparsity in the codes and reconstruction quality from these codes for models with linear decoders (dashed lines) and non-linear decoders (solid lines) trained MNIST images (Figure (aa) and ImageNet patches (Figure 6bj. The sparsity level is measured by the average percentage of non-active (zero) code components and the reconstruction quality is mcasured by the average PSNR. Both are evaluated the test set over 5 random seeds. As expected, higher sparsity levels result in worse reconstructions. 5 Moreover, models trained with our variance regularization approach and a non-linear decoder (VDL-NL) on both MNIST and ImageNet patches produce better reconstructions than
5 As reference reconstruction quality reconstructions of the images in the top row with PSNR of around 17.3 and 17.7, respectively.
PSNR, the second row of images Figure contains

---

Published in Transactions on Machine Learning Research (08/2022)
2
3
S
5
1
3
8
5
6
9
4
6
6
5
8
0
(a) SDL.-N,, À 0.003
(b) SDL.NI, À 0.01
(c) VDI.NI, À 0.003
(d) VDL-NI, À 0.02
Figure 4: Visualizing reconstructions from SDL-NL and VDL-NI decoders trained MNIST and imput codes in which only one component is active and the rest zero. The models are trained with different levels of sparsity regularization X. Each code component encodes stokes, orientations, parts of digits, and full digit prototypes very similar to those in Figure 2 The SDL-NL models in 4a and 4b produce reconstructions with average PSNR of 20.9 and 18.4 and average sparsity level in the codes of 69.2% and 85.6% on the test set, respectively. The VDL-NL models in 4c and 4d produce reconstructions with average PSNR of 22.7 and 18.3 and average sparsity level in the codes of 58.8% and 92.2% on the test set, respectively.
(a) SDL-NL, À = 0.003
(b) SDL-NL, À = 0.01
(c) VDL-NL, = 0.003
(d) VDL-NL, À = 0.01
Figure 5: Visualizing reconstructions from SDL-NL and VDL-NL decoders trained on ImageNet patches and input codes which only one code component is active and the rest are zero. The models are trained with different levels of sparsity regularization x. Code components encode gratings and Gabor-like filters. The SDL-NL models in (5ah and (5b) produce reconstructions with PSNR of 27.1 and 25.0 for codes with average sparsity level of 69.8% and 83.4%, respectively. The VDL-NL models in 55 and (5d] produce reconstructions with PSNR of 28.4 and 24.9 for codes with average sparsity level of 67.6% and 90.8%, respectively.
other models for higher levels of sparsity. This implies that VDL-NL models can better utilize the additional reprosentational capacity from the larger number of layers and parameters the non-linear decoder compared to the other models when the sparsity level is high.
4.3 Denoising
We evaluate the performance of SDL and VDL autoencoders on the downstream task of denoising. In this setup, we corrupt the input images by adding Gaussian noise and proceed with computing their corresponding sparse codes through amortized inference using the encoder. We then feod these codes the docoder obtain reconstructions of the noisy inputs.
Figure 7 shows the reconstructions of MNIST test set images and of the same images corrupted with Gaussian noise with zero mcan and standard deviations a = 1 and = 1.5 using the SDL and VDL models in Figures 25 and 2d We observe that both SDL and VDL models are able to produce reconst which are closer to samples from the manifold of training data than to the noisy inputs. This implies that the sparse coding systems have learned detect features usefuu for the data they are trained on and cannot reconstruct random noise, as desired. SDL and VDL models trained on ImageNet patches are also robust the addition of Gaussian noise to their inputs. This is evident, in Figure g which contains recononsuctions ÿ and ú. of
10

---

Published in Transactions on Machine Learning Research (08/2022)
ImageNet
MNIST VDL
ImageNet VIL
MNIST SDL-NN
ImageNet SDL-NL
26
MNIST SDL
ImageNet SDL
MNIST DO NL
MNIST
WDL-NL
30
MNIST WDL
PSNR
PSNR
20
26
16
0.2
1.0
Average fraction of zero elements in codes
Average fraction of zero elements in codes
(a) MNIST
(b) ImageNet patches
Figure Trade-off between sparsity measured by average percentage inactive (zero) code components and reconstruction quality measured by PSNR for different models. Figure (6a) plots regret curves for SDL, SDL-NL, VDL, VDI-NI, WDL, WDL-NI, DO and DO-NL. models trained on MNIST with code dimension = 128 and different levels of sparsity regularization X. Figure (6b) plots regret curves for SDL, SDL-NL, VDL, and VDL-NL models trained ImageNet patches with code dimension 256 and different levels of sparsity regularization A. Higher PSNR reflects better reconstruction quality.
y
0
/
2 3
4
5
6
7
8 9
y
0
/
2
3
4
5
6
7
8
y
0
/
2
3
4
5
6
7
8
9
0
/
2
3
4
5
6
7
8
9
Y==1
0
/
2
3
/
5
6
8
9
y==1
0
2
3
7
5
6
7
8
y /==1
0
/
2
3
4
3
6
8
9
80=1
0
/
2
3
4
3
6
7
8
9
1-55.5 1.5
8
y==
0
1-5
0
/
2
3
4
3
6
7
8
9
1==1.5
0
/
.
3
4
3
6
8
(a) SDL, 0.005
(b) VDL, 0.02
Figure 7: Examples of denoising abilitics of encoders from SDL and VDL models trained on MNIST which produce codes with average sparsity of 89.7% and 91.8%, respectively. Top two the test set (top) and their reconstructions (bottom); middle two rows: the inputs corrupted with Gaussian noise with standard deviation corrupted with Gaussian noise with standard deviation = 1.5 (top) and their reconstructions (bottom).
original inputs from
(top) and their reconstructions (bottom), bottom two the inputs
images y and their noisy version yo for the SDL and VDL models from Figures codes with similar sparsity on average.
3a and
3c
which produce
Table I shows summary of the reconstruction quality of images with varying levels of Gaussian noise corruption when they are passed through SDL and VDL autoencoders. The results are evaluated on the test set over random seeds. Both the SDL and VDL encoders are able to produce codes from corrupted inputs yo which, when passed through their corresponding decoders, produce outputs ú, that are less noisy than the corrupted inputs. This observation is significant since the sparse autoencoders in our experiments are not explicitly trained to perform denoising.
11

---

Published in Transactions on Machine Learning Research (08/2022)
y
y
)=11
Y==1 ) )==1
9 o=1
(a) SDL, = 0.002
(b) VDL, À = 0.005
Figure 8: Examples of denoising abilities of encoders from SDL and VDL models trained on ImageNet patches which produce codes with average sparsity of 74.6% and 77.3%, respectively. Top two rows: original inputs from the test set (top) and their reconstructions (bottom); bottom two rows: the same inputs corrupted with Gaussian noise with standard deviation 1 (top) and their reconstructions (bottom).
The performance of SDL and VDL models trained on MNIST and ImageNet patches with different levels of sparsity regularization À on the the task of denoising inputs corrupted with Gaussian noise with zero mean and standard deviation = 1.0 = 1.5. We report the average sparsity of codes computed with these models and the PSNR of: the reconstruction ÿ of the original inpul y, the corrupted input yo and the reconstruction y, of the corrupted imput yo Evaluated on the test set over 5 random seeds.
Table 1:
MNIST
MODEL
Lo(zg)
PSNR
PSNR Y
Lo(zg.)
PSNR 7o
SDL
0.005
89.7% + 0.2%
17.3 + 0.028
1.0
10.2
87.3% + 0.2%
16.9 + 0.053
VDL
0.02
91.8% + 0.1%
17.7 + 0.023
1.0
10.2
88.1% + 0.2%
15.4 + 0.175
SDL
0.005
89.7% + 0.2%
17.3 + 0.028
1.5
6.7
84.6% + 0.5%
15.7 + 0.204
VDL
0.02
91.8% + 0.1%
17.7 + 0.023
1.5
6.7
84.5% + 0.3%
12.2 + 0.275
IMAGENET PATCHES
SDL
0.002
74.6% 1 0.1%
+ 0.007
20.0
72.7% 1 0.1% 73.5% + 0.2%
25.6 + 0.011
VDL
0.005
77.3% 1 0.2%
27.3 + 0.058
1.0
20.0
25.2 + 0.035
4.4 Ablation: No Variance Regularization
VDL and VDL-NL experiments from Figure G show that our proposed variance regularization strategy can successfully be used train sparse coding models without collapse the norms of the sparse codes. experiments without our variance regularization strategy or any regularization restricting the dictionary's weights, do observe collapse, both for models with lincar and non-linear decoder. This shows that our proposed variance regularization protocol is an effective alternative to the standard l2 normalization weight decay regularization for preventing collapse, both in the case of models with linear decoder and with a fully connected decoder with one hidden layer.
4.5 Classification the Low Data Regime
We investigate whether self-supervised pre-training with sparse coding improves classification performance over training from scratch in the low data regime. For this purpose, we compare the classífication error of classifiers trained on the raw MNIST data the classification error of lincar classifiers trained on features from pre-trained frozen LISTA encoders in SDL, SDL-NL, VDL, and VDL-NL autoencoder:[ and pre-trained frozen DO and DO-NL decoders when few training samples are available for supervised learning.
In particular, we train lincar classifier on top of fcatures from these pre-trained models when 1, 2, 5, 10, 20, 50, 100 training samples per class are available as well as when the full dataset is available. For each type of ppe-trained model and each number of training samples per class, we select the model among the
oTe autoencoders trained the full MNIST dataset in unsupervised way as described section
12

---

Published in Transactions on Machine Learning Research (08/2022)
40
Linear on VDL-NL
Linear on VDL-NL
Linear on VDL
Linear on VDL
Linear SDL-NL
Linear SDL-NL
Linear SDL
Linear SDL
50
Linear
30
Linear DO-NL
Top-1 Error (%)
on
Top-3 Error (%)
25
Linear
LISTA + Linear on raw
LISTA + Linear on raw
Linear on raw
Linear on raw
30
15
20
10
10a
10a
10
10a
10*
10a
10
10
10a
10
Number of training samples per class (log scale)
Number of training samples per class (log scale)
(a) Top-1 error in %.
(b) Top-3 error in %.
Figure 9: Lincar separability (measured by classification error) of the raw MNIST data and of features from pre-trained SDL, SDL-NL, VDL, VDL-NL encoders and decoder-only DO and DO-NL models with different numbers of training samples per class. Additionally, classification error of classifier consisting of LISTA enoeder followed by linear layer (LISTA + Linear) trained from scratch the raw MNIST data. Models trained with variance regularization in the codes and non-linear decoder (VDL-NL) outperform other models in linear classification with up to 10 training samples per class. Test set performance is reported over 5 random seeds.
ones presented in Figure 6a| which gives the best validation accuracy. As baselines, we train two classifiers from scratch on the raw MNSIT data. The first one is linear classifier referred to "Linear raw". The second one consists of linear layer on top of randomly initialized LISTA encoder and is referred to "LSITA + Lincar on raw".
Figure g summarizes the top-1 and top-3 classification errors of these classifiers on the test sel evaluated over random seeds. With small exceptions all linear classifiers trained on top of the pre-trained LISTA features give better top-1 and top-3 classification performance than classifiers trained on the raw MNIST data with up to 100 samples per class. This observation supports a claim made by Raina et al. (2007) that SSL is expected to be useful especially when labeled data is scarce. Notably, classifiers trained top of VDL-NL features have the best performance among the rest of the models with up to 10 training samples per class. In other words, sparse autoencoders with non-linear decoder pre-trained using our proposed variance regularization on the latent codes can boost the classification performance when few labeled training samples are available.
5 Related Work
Our work is related to the extensive literature on sparse coding and dictionary learning, Mairal et al. (2014) provide a comprehensive overview of sparse modeling in computer vision, discuss its relation the fields of statistics, information theory, and signal processing, well as its applications in computer vision including image denoising, inpainting, and super-resolution.
Ways to Avoid Collapse There are different ways to avoid the collapse issue in a sparse coding model. One approach to constrain the weights of the decoder, for example by bounding the norms of its columns
Loe ot al. 2006 Raina et al. 2007 Mairal et al.) 2009 Hu & Tan 2018 or by applying weight decay regularization (Sun et al. 2018): Another approach to regularize the latent codes directly a strategy we
Thii randomly initialized LISTA encoder trainable and with the architecture the Algorithm Namely, classifiers trained DO, DO-NT. and SDI. features with training samples per
13

---

Published in Transactions on Machine Learning Research (08/2022)
incorporate in model. In the spiking model of Földiak [1900], an anti-Hebbian adaptive thresholding mechanism encourages each latent, component to be active with a pre-set level of probability. Our variance regularization strategy is also related to Olshausen & Field 1996 1997] where the norm of the basis elements is adjusted adaptively over time so that each of the latent components has a desired level of variance. Their method works well in the linear dictionary setup but is not directly exlendable to the case of non- linear decoder. In variational inference models such as Kingma & Welling (2013) and its sparse coding extensions such as SVAE Barello et al. (2018) and [Tonolini et al. (2020), the latent codes have pre-set level of variance determined by the prior distribution. In SVAE, there is also variance regularization term which determines the trade-off' between the input reconstruction and the sparsity level imposed by the prior. Our variance regularization term is most closely related to the variance principle presented in Bardes et al. b in which each latent component is encouraged to have variance above a pre-set threshold.
Sparse Coding and Classification Performance There an active area of research which studies whether sparse coding is uscful in classification tasks. Some existing literature supports this claim Raina
et al. 2007; ; Ranzato et al. (2007) Mairal et. al. (2008a b); Yang et. al. (2009); Kavukcuoglu et al. (2010)
Makhzani & Frey| (0013); He et al. (0014) while other research refutes it Coates & Ng (2011); Rigamonti
ct al. )0 Luther & Seung (2022).
p. Tang et al. (2020) show that adding layers to a hicrarchy of sparse
dictionaries can improve classification accuracy. In our work, we provide empirical evidence that sparse coding models trained with a reconstruction objective, especially models with a non-linear decoder trained using our variance regularization method, can give better classification performance than classifiers trained on the raw data when very few training samples are available, supporting a claim made by Raina et al. (007 that SSL is expected to be useful when labeled data is scarce.
Sparsity Constraints Sparse representations can be obtained using many different objectives. The 4 penalty term in equation I can be replaced by other sparsity-inducing constraints including lo, 4p (0 <0<pp< < 1). 12, or 12,1 norms. In particular, the 4 term is convex relaxation of the non- non-differentiable to constraint of the form ||01lo S k requiring& z to have at most is active components. Zhang ct al. (2015) provide a survey of existing algorithins for sparse coding such as K-SVD which generalizes K-means clustering (Aharon el al., 2006). Makhzani & Frey (2013) propose an approach to train a sparse autoencoder with a linear encoder and decoder in which only the top l activations in the hidden layer are kept and no other sparsifying penalty is used. In our model, we modify the commonly used sparse coding setup with 4 penalty term in by including term which encourages pre-set level of variance across each latent component in 9
Inference There are many variations of the ISTA inference algorithm which address the fact that it is computationally expensive and are designed to speed it up, such as FISTA (Beck & Teboulle 2009 which is used in our setup (for details please refer to section 2.2 Gated LISTA (Wu et al. 2019) introduces a novel gated mechanism for LISTA (Gregor & LeCun 2010) which enhanses its performance. In Zhou et al.) (2018), it is shown that deep learning with LSTMs can be used to improve learning sparse codes in ISTA by modeling the history of the iterative algorithm and acting like a momentum. Inspired by LISTA, we train an encoder to predict the non-negative representations computed during inference using the original and modified FISTA protocols as explained in section 2.4
Multi-layer Sparse Coding There exist methods which greedily construct hierarchies of sparse rep- resentations. Zeiler et al. (2010) use ISTA in a top-down convolutional multi-layer architccture to learn sparse latent representations at each level in a greedy layer-wise fashion. Other convolutional models which produce a hierarchy of sparse representations are convolutional DBNs (Lee et al. 2009) and hierarchical convolutional factor analysis (Chen et al. 2013) which also use layer-wise training. Other greedy approaches are proposed by He et al. (2014) and Tariyal et al. (2016) who train multiple levels of dictionaries. Chen et al. (2088) combine sparse coding and manifold learning in a multi-layer network which is also trained wise. In contrast, our work is about finding sparse representations by utilizing multi-layer decoders which are trained end-to-end. Hu & Tan (2018) propose a non-linear sparse coding method in which a linear dictionary is trained in the encoding space of non- mon-linear autoencoder but it differs from our method as the sparse representations of this encoding layer are stand-alone, i.c. they are not fed as inputs to the multi-layer decoder. Other multi-layer sparse coding models include Sun et al. (0018), Mahdizadehaghdam et al. (2019) and Tang et al.) (2020) in which linear dictionaries are stacked at different scales. Rather than using linear dictionaries, our method learns non-linear decoder to decode sparse codes.
14

---

Published in Transactions on Machine Learning Research (08/2022)
6 Conclusion
In this work, revisit the traditional sctup of sparse coding with an 1 sparsity penalty. We propose to apply variance regularization to the sparse codes during inference with the goal of training non-linear decoders without collapse in the 1 norm of the codes. We show that using our proposed method we can successfully train sparse autoencoders with fully connected multi-layer decoders which have interpretable features, outperform models with linear decoders in terms reconstruction quality for a given average level of sparsity the codes, and improve MNTST classification performance the low data regime. Future research directions include scaling up our method to natural images using deep convolutional decoders.
Broader Impact Statement
This work proposes general-purpose machine learning algorithm for representation learning which is trained using large amounts of unlabeled data. The representations learned by this algorithm reflect any biases present in the training data. Therefore, practitioners should apply techniques for identifying and mitigating the biases in the data when applicable. This notice is doubly important since the learned representations can be incorporated in numerous real-world applications such as medical diagnosis, surveillance systems, autonomous driving, and so on. These applications can have both positive and negative effect on society, and warrant extensive testing before being deployed in order to prevent harm or malicious use.
Acknowledgments
We would like to thank our colleagues at the NYU CILVR Lab and NYU Center for Data Science for insightful discussions, and to the TMLR anonymous reviewers and action editor for their feedback. Part of this research was conducted while Katrina was interming at FAIR. This work is supported in part by AFOSR award FA9550-19-1-0344 and by the National Science Foundation under NSF Award 1922658.
References
Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse reprosentation. IEEE Transactions signal processing, 54(11)::311-4222 2006.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varfance-ivvrianccccovaaaaa regularization for self-supervised learning. arviv preprint arXiv:2105.0 04900, 2021.
Gabriel Barello, Adam S Charles, and Jonathan W Pillow. Sparso-coding variational anto-encoders. BioRrin, pp. 399246, 2018.
Amir Beck and Mare Teboulle. A fast iteralive shrinkagg- algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183-22) 2020 2009.
Yoshua Bengio. Learning deep architectures for ai. Foundations and trends in Machine Learning, 2(1):1-127, 2009.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEFE Iransactions on pattern analysis and machine intelligence, 37(8):1798-1888 2013.
Bo Chen, Gungor Polatkan, Guillermo Sapiro, David Blei, David Dunson, and Lawrence Carin. Deep learning with hierarchical convolutional factor analysis. IEEE transactions on pattern analysis and machine intelligence, 358)):1887-1900 2013.
Yubei Chen, Dylan Paiton, and Bruno Olshausen. The sparse manifold transform. Advances neural information processing systems, 31, 2018.
Adam Coates and Andrew Y Ng. The importance of encoding versus training with sparse coding and vector quantization. In TCML, 2011.
15

---

Published in Transactions on Machine Learning Research (08/2022)
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image processing, 15(12):3736-3745, 2006.
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervieed video representation learning with odd-one-out networks. In Proceedings of the TEFE conference on computer vision and pattern recognition, pp. 3636-3645, 2017.
Peter Földiak. Forming sparse representations by local anti-hebbian learning. Biological cybernetics, 64(2): 165 170, 1990.
Peter Földiák. Learning invariance from transformation sequences. Neural computation, 3(2):194 200, 1991.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 399 406, 2010.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, ko- ray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent proach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Aduances in Neural Information Processing Systems, volume 33, pp. 21271- 21284. Curran Associates, Inc., 2020. URL https://proceeediggs htwpoeeeddinnnnnuuiiss..o.mo www.nips...ppppee//eee.... (itaa0ddccee7/111222211888122222
new ap-
ppf
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast. for unsupervised visual representation learning. arxiv preprint arXiv.1911.05799 2019.
Yunlong Hc, Koray Kavukeuoglu, Yun Wang, Arthur Szlam, and Yanjun Qi. Unsupervised feature learning by deep sparse coding. In Proceedings of the 2014 SIAM international conference on data mining pp- 902 910. SIAM, 2014.
Junlin Hu and Yap-Peng Tan. Nonlinear dictionary learning with application image classification. Pattern Recognition, 75:282 291, 2018.
Aapo Hyvärinen, Jarmo Hurri, and Jaakko Väyrynen. Bubbles: a unifying framework for low-level statistical properties of natural image sequences. JOSA A, 2077):1277 2003.
Kevin Jarrett, Koray Kavukeuoglu, Mare' Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In 2009 IEEE 12th international conference computer vision, pp. IEFF, 2009.
Koray Kavukeuoglu, Pierre Sermanet, Y lan Boureau, Karol Gregor, Michael Mathieu, and Yann L. Cun. Learning convolutional feature hicrarchies for visual recognition. In J. D. Lafferty, C. K. I. Williams, J. Shawe- Taylor, R. S. Zemel, and A. Culotta (eds.), Advances in Newral Information Pro- cessing Systems 23, pp. 1090 1098. Curran Associates, Inc., 2010. URL httt:://pppppee aaraaoottttttaaaaaauueeeeerreree
ctppscer///p wre.oniiiio...otti.ooo.oo..o
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ar preprinl arXiv:1412. 6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes arXiv preprint or arxi::1922.611 2013.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. URL http: www..aann..eccccccomm exdb/mnist/ 2010.
16

---

Published in Transactions on Machine Learning Research (08/2022)
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint european conference machine learning and knowledge discovery in dalabases, pp- 498-59--8 118-51.. Springer, 2015.
Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Ng. Efficient sparse coding algorithms. Advances in neural information processing systems, 19, 2006.
Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y Ng. Convolutional deep belief networks for scalable unsupervised learning of hicrarchical representations. In Proceedings of the 26th annual interna- lional conference machine learning, pp. 609 616, 2009.
Kyle Luther and H Sebastian Seung. Sensitivity of sparse codes image distortions. Neural Compulation, pp. 1-20, 2022.
parametric network approach. IEEE Transactions on Image Processing, 28(10):4790 00044990--80,,, 2019. Shahin Maldizadelaghhaam, Ashkan Panahi, Hamid Krim, and Liyi Dai. Deep dictionary learning: A
Julien Mairal, Michael Elad, and Guillermo Sapiro. Sparse representation for color image restoration. IEEE Transactions on image processing, 17(1):53-60) 2007.
Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, and Andrew Zisserman. Discriminative learned dictionaries for local image analysis. In 2008 TEFE conference on computer vision and pattern recognition, pp. 8, IEEE, 2008a.
Julien Mairal, Jean Ponce, Guillermo Sapiro, Andrew Zisserman, and Francis Bach. Supervised dictionary learning. Advances in neural information processing systems, 21, 2008b.
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding, In Proceedings of the 26th annual international conference on machine learning, pp. 689-696, 2009.
Julien Mairal, Francis R. Bach, and Jean Ponce. Sparse modeling for image and vision processing. CoRR, abs/1411.3230, 2014. URL htll: wwwp:/wwwww.//////11..22222 3230
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXio:1312.566: 2013.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. European Conference on Computer Vision, pp. 527 544. Springer, 2016.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature,
1996.
Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis sct: A strategy employed by v1? Vision research, 37(23):3311 3325, 1997.
Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y Ng. Self-taught learning: transfer learning from unlabeled data. In Proceedings of the 24th international conference on Machine learning, pp. 759-766, 2007.
Mare' Aurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann L Cun. Efficient learning of sparse representations with an energy-based model. In Advances in neural information processing systems, pp. 1137-1144, 2007.
Roberto Rigamonti, Matthew A Brown, and Vincent Lepetit. Are sparse representations really relevant for image classification? In CVPR 2011, pp. 1545 1552. IEEE, 2011.
Xiaoxia Sun, Nasser M Nasrabadi, and Trac D Tran. Supervised deep sparse coding networks. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 346-350. IEEE, 2018.
Hao Tang, Hong Liu, Wei Xiao, and Nicu Sebc. When dictionary learning moets deep learning: Deep dictionary learning and coding network for image recognition with limited data. IEEE transactions on neural networks and learning systems, 22(5):2129 2114-111,, 2020.
17

---

Published in Transactions on Machine Learning Research (08/2022)
Snigdha Tariyal, Angshul Majumdar, Richa Singh, and Mayank Vatsa. Deep dictionary learning. Access, 4:10096-1000 10109, 2016.
1EEE
Francesco Tonolini, Bjorn Sand Jensen, and Roderick Murray-Smith. Variational sparse coding. In Uncer- tainty in Artificial Intelligence, pp. 690-700. PMT.R, 2020.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoin Manzagol. Extracting and compos- ing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Newral compulation, 14(4):715 770, 2002.
Kailun Wu, Yiwen Guo, Ziang Li, and Changshui Zhang. Sparse coding with gated learned ista. In Inter- national Conference on Learning Represenlalions, 2019.
Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. Linear spatial pyramid matching using sparse coding for image classification. In 2009 IEEE Conference on computer vision and pattern recognition, pp. 1794-1801. IEEE, 2009.
Takashi Yoshida and Kenichi Ohki. Natural images are reliably represented by sparse and variable popula- tions of neurons in visual cortex. Nature communications, 11(1):1-19, 2020.
Kai Yu. Tutorial on deep learning: Sparse coding. CVPR, 2012.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103. 0rX0:2200.00220, 2021.
Matthew Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In 2010 IEEE Computer Sociely Conference on compuler vision and pollern recognition, pp. 2528 2535. IEEE, 2010.
Matthew D Zeiler, Graham W Taylor, and Rob Fergus. Adaptive deconvolutional networks for mid and high level feature learning. In 2011 International Conference on Computer Vision, pp. 2018 2025. IEEE, 2011.
Zheng Zhang, Y. Xu, Jian Yang, Xuelong Li, and David Zhang. A survey of sparse representation: Algorithms and applications. IEEE Access, 3:490-530, 2015.
Joey Tianyi Zhou, K. Di, J. Du, Xi Peng, Н. Yang, Sinno Jialin Pan, 1. Tsang, Yong Liu, Z. Qin, and R. Goh, Sc2net: Sparse Istms for sparse coding. In AAAI, 2018.
A
Appendix
A.1
Notation
Table 2
contains descriptions of the notation and symbols used in this work.
A.2
Variance Regularization Term
Figure
10
visualizes the variance regularization term h(xx) = [(1 - /Va(())) 1 for ar E R² in 3D.
A.3
Gradient Derivation
We compute the gradient of the hinge term in with respect to one latent code's component Zn
9 (I /Va((2,)))
(16)
0Z, 1=1
18

---

Published in Transactions on Machine Learning Research (08/2022)
Table Descriptions of symbols and notation used the paper.
Notation
Description
LISTA-inspired encoder.
D
Decoder: linear or fully connected with one hidden layer.
m
Dimension of the decoder's hidden layer.
X
Hyperparameter for the sparsity regularization.
B
Hyperparameter for the variance regularization of the latent codes.
Hyperparameter encouraging the FISTA codes to be close to the encoder's predictions. Dimension of the input data. Input sample in R
N
Total number training samples. Mini-batch size.
n
T
Number ISTA ISTA/FISTA iterations. Step size for ISTA/FISTA gradient step. Learning rate for the decoder.
nD
ng
Learning rate for the encoder.
Shrinkage function [T.(m)], sign (1=5) aa).
Ta
Non-negative shrinkage function [fa (2)]; ([Ta (Taa())))++
Dimension of the latent code. Latent, code in R!.
z
Z
Mini-batch of latent codes in R
W
Parametrization in Rdxi of linear dictionary D.
W1
Parametrization in Rmxl of the bottom layer of fully connected decoder D with one hidden layer.
W2
Parametrization of a the top layer fully connected decoder D with one hidden layer in Rdxm
bi
Parametrization of the bias term Rm following the bottom layer of a fully connected decoder D with one hidden layer.
b
Parametrization of the bias term in the LISTA encoder (sce 3.1
SDL
Standard Dictionary Learning with LISTA encoder and linear decoder whose columns have fixed l2 norm.
SDL-NL
Standard Dictionary learning with LISTA encoder and non-linear fully connected decoder. Each layer in the decoder has columns with a fixed ly norm.
VDL
Variance-reggulaiizel Dictionary Learning with a LISTA encoder and linear decoder which regularization applied to the sparse codes encouraging the variance across the latent components to be above a fixed threshold.
VDL-NL
Variance-regularrzzed Dictionary Learning (as above) with non-linear decoder. Dictionary Learning with LISTA encoder and linear decoder trained with weight decay regularization.
WDL
WDL-NL
Dictionary Learning with a LISTA encoder and a non-linear decoder trained with weight decay regularization.
DO
Standard Dictionary Learning without an encoder and with a lincar dccoder whose columns have a fixed la norm.
DO-NT.
Standard Dictionary Learning without encoder and with non-linear decoder whose columns have fixed le norm.
As reminder of our notation, / the latent dimension, n is the batch size, Z R stores the codes for all elements in batch, and Zj.; E R stores code values for the j-th latent component.
19

---

Published in Transactions on Machine Learning Research (08/2022)
Variance Regularizatien Term in 3D
1.0
0.8
0.6
0.4
0.0
1
Figure 10: Visualizing the variance regularization function h(x) = [(1 v Var(x)) _] for æ R² 3D.
Note that:
1
Var Zj.))
Σ
(zj pz;)²
(17)
i=1
Zz..-
(18)
n i=1
The gradient in 16
non-zero only for and v Var Z.,)) T. Thus,
Σ 3 (T
V Var (g.))
)
(T V /Vr((zs))
(19)
OZ.,1
OZ,,1
23 T Var( z..))
T /Var(z.))
(20)
02,,1
-22 / Varr Z..:)
/Va( (Z.:)
(21)
1Z,,1
T Var Zo.))
θ Var(Zs.)
-B
(22)
Var(Za.)
Za.)
0Z,t
Now, continuing with:
θ Σ ZZs 12)
Var (Za:) = aZ,,t In -
(23)
aZ.t
i=1
20

---

Published in Transactions on Machine Learning Research (08/2022)
(Zat maa)2 2 Σ
(Zoi - -ma
(24)
11 sZ
8zat
171
1
2(Zz, - 1s)
(Ze - Us)+ +
Σ
2(Zs - -4s) 0
(Zoss - Hai)
(25)
n
1Z
oZ.,1
2
(40, 1a))
Zz,1 Σ
Zamm)
+
(Ko,, - 1o)) 8Z
(Zo,i
1 Σ
Z m
)]
8Zatt
m=1
m=1
(26)
2
(Zo,1 [o0
(1) +
11 n)
((z,,
(Ze, - (-a)
(27)
n 1
2
(Za, - 14))
1 n
ZZ. [o)]
(28)
n
11
2
(Zo,1 Mo))
1
Σ
Ze
+
1
(29)
n1 1
11 i=1
n
2
[2a, - Ma)) - 1/s + Msg
(30)
1
2
n 1 (Zait - -1,),
(31)
and using the result in 22 conclude that:
T (rrvaa(Zs.))
3 [ V Var (2.))) ]
1 =
29
(Z Mb) if Var(Z <
n-1
Var(2 Zn.))
(32)
8Z..b j=1
0
otherwise.
B
Additional Visualizations
Figures 11 and 13 display 128 of the dictionary elements for the same SDL and VDL models as in Figures 2
and 3 Figure 12 displays 128 of the dictionary elements for the WDL and DO models trained with different levels of spansity regularization.
B.1 Training Details
All hyperparameter values are selected through grid search. For the constant step size ÿe in FISTA, we consider values in the range of 0.01 to 100. In VDL experiments, consider values for the hinge regular- ization coefficient 33 between 1e-1 and 100. In all our experiments, we use Adam (Kingma & Ba 2014) as an optimizer for both the encoder & and decoder D with batch size of 250. We use L = 3 iterations in the LISTA encoder (sce Algorithm 1 We run the FISTA algorithm for a maximum of 200 iterations. The
(1 < 1e-3. FISTA's output is x= =x*)) where k* is the (2
convergence criterion we set is:
index of the first itcration for which the convergence criterion holds. Table 3 contains the hyperparamcter values we use in all our experiments expect for the ones with WDL, WDL-NI, DO and DO-NL models. In the case of DO and DO-NL models, we use the same number of epochs, tp,, wd((), wd(bi), and 7 in SDL and SDL-NL experiments, respectively. In the case of WDL and WDI-NI models, we use grid scarch to find the optimal weight decay value of 5e-4 (in terms of reconstruction quality regret curve as displayed in Figure D this is also the value used in Sun et al. (2018) and use the same number of epochs, fp, ", ne: and 7]s as in SDL and SDL-NL experiments, respectively.
21

---

Published in Transactions on Machine Learning Research (08/202)
O
M
8
9
2
G
S
2
3
5 4
7
G
6
2
O
Z
7
S
O
G
6
2
7
3
5
9
6
6
O
O
5
2
5
O
4
5
8
8
9
4
O
7
4
3
4
3
6
0
G
9
5
2
O
5 2
(a) SDL, A = 0.001
(b) SDL, 0.005
(c) VDL, X = 0.005
(d) VDL, = 0.02
Figure 11: Dictionary elements for linear dictionaries trained on MNIST with latent dimension / 128 and different levels sparsity regularization x. SDL stands for dictionary learning in which the l/2 norm of the decoder's columns is fixed 1. VDL stands for dictionary learning in which norms of the decoder's columns not explicitly bounded but our proposed variance regularization on the latent codes is used. The SDL models in [a and 2b produce reconstructions with average PSNR of 21.1 and 18.6 for codes with average sparsity level of 63% and 83% on the test set, respectively. The VDL models in 2c and 2d produce reconstructions with average PSNR of 20.7 and 17.7 for codes with average sparsity level of 69% and 91.8% on the test set, respectively.
22

---

Published in Transactions on Machine Learning Research (08/2022)
C
U
DO
S
9
S
Q
3
9
7
Z
Z
5
6
S
6
O
3
8
4
2
/
6
4
7
(a) WDL, x = 0.001
(b) WDL, À = 0.005
(c) DO, À = 0.005
(d) DO, À 0.005
Figure 12: Dictionary elements for linear dictionaries trained on MNIST with latent dimension = 128 and different levels of sparsity regularization X. WDL stands for dictionary learning in which weight docay is applied to the decoder's weights and there is LISTA encoder. DO stands for dictionary learning in which norms of the decoder's columns are fixed to 1 and there is LISTA encoder. The WDL models in 12a and 12b produce reconstructions with average PSNR of 20.8 and 17.6 for codes with average sparsity level of 67.7% and 86.7% on the test set, respectively. The DO models in 12c and 12d produce reconstructions with average PSNR of 21.1 and 17.6 for codes with average sparsity level of 64.9% and 89.5% on the test set, respectively.
23

---

Published in Transactions on Machine Learning Research (08/2022)
(a) SDL, 0.002
(b) SDL, 0.005
(c) VDL, 0.005
(d) VDL, À 0.01
Figure 13: Dictionary elements which resemble Gabor filters for SDL and VDI. models with latent dimension
256 trained on ImageNet patches with different levels of sparsity regularization x. The SDL models in
3a and 3b produce reconstructions with average PSNR of 26.9 and 24.0 and average sparsity level in the
codes of 74.6% and 90.5% the test set, respectively. The VDL models in Figures 3c and 3d produce
reconstructions with average PSNR of 27.3 and 25.3 and average sparsity level in the codes of 77.3% and 89.2% on the test set, respectively.
24

---

Published in Transactions on Machine Learning Research (08/2022)
(a) SDL, À 0.001
(b) SDL, À 0.005
(c) VDL À = 0.005
(d) VDL, 0.02
(e) WDL, X = 0.001
(1) WDL, = 0.005
Figure 14: Visualization of the U E Rdxl layer of the LISTA encoder for SDL, VDL, and WDL models with code dimension 128 trained on MNIST images and different levels of sparsity regularization Similarly the decoders in [1 the features learned by the encoders resemble orientations and parts of digits but the case of WDL models there are more noisy features compared to the other models À increases.
25

---

Published in Transactions on Machine Learning Research (08/2022)
dataset
model
ep
X
B
1/D
wd(bi)
wd((b)
7/1
MNIST
SDL
200
0
0
1e-3
3e
MNIST
SDL
200
1e-4
1
0
1e-3
30
MNIST
SDL
200
5e-4
0
le-3
3e
0
1
MNIST
SDL
200
1e-3
0
le-3
3e
0
1
MNIST
SDL
200
3
0
le-3 3
3e-4
0
1
MNIST
SDL
200
5c 3
0
1c
3c
MNIST
VDL
200
0
5
10
0.5
30
le
0.5
MNIST
VDL
200
1e-3
5
10
0.5
30
le-4
0.5
MNIST
VDL
200
3c 3
10
0.5
30
1c
0.5
MNIST
VDL
200
5
5
10
0.5
3e
le-4
0.5
MNIST
VDL
200
1e 2
5
10
0.5
3e 4
le-4
0
0.5
MNIST
VDL
200
20 2
5
10
0.5
30 4
le-4
0
0.5
MNIST
SDL-NL
200
0
0
le-3 3
le-4
le-3
0
MNIST
SDL-NL
200
1e-3
0
le-3
le-4
le-3
0
1
MNIST
SDL-NL
200
3
0
le-3
le-4
le_3
MNIST
SDL-NL
200
5
0
le-3
le-4
le-3
MNIST
SDL.NI,
200
1e-2
0
le-3
le-4
le-3
MNIST
SDL-NL
200
2e 2
0
le 3
le
le-3
MNIST
VDL-NL
200
0
100
10
0.5
3e
le
le-3
0.5
MNIST
VDL-NL
200
1e-3
100
10
0.5
3--4"
le
le-3
0.5
MNIST
VDL-NI.
200
3
100
10
0.5
3--4"
1e
le-3
0.5
MNIST
VDL-NL
200
3
100
10
0.5
3--4"
1c
le-3
0.5
MNIST
VDL-NI
200
1e-2
100
10
0.5
3e-4"
le
le-3
0.5
MNIST
VDL-NL
200
2
100
10
0.5
3e-4"
le
le-3
0.5
ImageNet
SDL
100
0
0
le-3
le
le-2
0.5
TeageNet
SDL.
100
5e-4
0
le-3
le-4
le 2 0.5
ImageNet
SDL
100
1e-3
0
le-3
le
le-2 0.5
ImageNet
SDL
100
2
0
1e 3
le
le-2 0.5
ImageNet
SDL 100
3e 3
0
le 3
le-4
le- 0.5
ImageNet
SDL
100
-
0
le 3
le-4
le-2 0.5
ImageNet
VDL
100
0
5
10
0.5
3c 4
1e-4
le-2 0.5
ImageNet
VDL
100
1e-3 3
5
10
0.5
3e
le-4
le-2 0.5
ImageNet
VDL
100
3e-3
10
0.5
3e
le-4
le-2 0.5
ImageNet
VDL
100
50
10
0.5
3c 4
1c
le-2 0.5
ImageNet
VDL
100
1e 2
10
0.5
3e
le-4
le-2 0.5
ImageNet
VDL.
100
1.5e- 2
10
0.5
3e
le-4
le-2 0.5
ImageNet
SDL-NL
100
0
0
le-3
le
lc-2
le
0.5
ImageNet
SDL-NL
100
1e-3
0
le-3
le-4
le-2
le-2
0.5
ImageNet
SDL-NI.
100
3e-3
0
le-3
le-4
le-2
le-2
0.5
ImageNet
SDL-NL
100
5e-3
0
le-3
le-4
le-2
le-2
0.5
ImageNet
SDL-NL
100
8c 3
0
le-3
le-4
le-2
le-2
0.5
ImageNet
SDL-NL.
100
1e
0
1e-3
le
le-2
le-2 0.5
ImageNet
VDI-NI.
100
0
20
10
5c
le
le
le-2 2 0.5
ImageNet
VDL-NL
100
1e-3
20
10
0.5
5e-5"
le
le-1
le-2
0.5
ImageNet
VDI.NI.
100
2e 3
20
10
0.5
5e-5"
le
le-1
le 2 0.5
ImageNet
VDL-NL
100
3e-3
20
10
0.5
5e-5"
le-4
le-1
le-2 0.5
ImageNet
VDL-NL
100
5c-3
20
10
0.5
5e-5"
le-4
le-1
le-2 0.5
ImageNet
VDL-NL
100
le-2
40
10
0.5
5e-5"
le-4
le-1
le-2
0.5
ImageNet
VDL-NI.
100
2e-2
40
10
0.5
5e-5"
1c-4
1e-1
1e-2
0.5
Table 3: Training hyperparameters. SDL indicates models in which columns of the decoder's layers have fixed norm of VDL indicates models in which variance regularization is applicd the codes during inference. (*) means that the learning rate for the decoder up is annealed by half every 30 epochs. wd stands for weight decay.
26


Reviews:
[{"review_id": "pr1AJ74N83", "reviewer": "Reviewer_Lkv3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros.\nA simple idea, the experiments show some minor improvements.\n\nCons:\nAs long as the dictionary atoms in D are regularized well (such as using unit norm), I do not see how the l1 norm on the sparse codes will collapse. What is the issue with normalizing the columns of the dictionary that we need a new method to regularize the sparse codes? This is the key contribution of this paper and will need a clear motivation. Regularizing the sparse codes using a statistical measure will lead to a stochastic sparse code, which may not be what one would want for an application.\n\nWhat is the point of predicting the sparse codes produced by FISTA using a neural network? Perhaps this contribution is besides the main topic of the paper, and could be omitted.\n\nOverall, this paper does not seem to have a mathematically convincing argument for its contributions. Experiments do not bring out the advantages clearly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents the use of a variance regularization term within the classic sparse coding/dictionary learning formulation. The paper also proposes to integrate a neural encoder to predict the sparse codes produced by the FISTA algorithm directly from the input vectors. The experiments show some promise.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "There are minor typos: \na. In (2), there must be a term ||d_i|| = 1 for d_i being the i-th column of D, and \\forall i in \\set{1,2,...|D|}. The optimization on D is constrained to be in this product space of unit spheres.\n\nb. The FISTA iterations in (6) should be z^(k) instead of x^(k).\n\nc. For the Figure 1, y and D must have arrows going into FISTA algorithm.\n\nAlso, please address the above concerns regarding a clear motivation for the need for a new method.", "broader_impact_concerns": "I do not see any ethical concerns with this paper. Nevertheless, the paper has provided a paragraph on such concerns already.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Eal6Ti1qCG", "reviewer": "Reviewer_G1LD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well written in general. The descriptions of the sparse coding, ISTA, and FISTA algorithms are succinct and complete. I’m also glad to see that they extended their experiments beyond the typical reconstruction/denoising regime into classification. The provided gradient calculations and weight visualizations in the appendix are appreciated – they give a more complete understanding of what the model learned and reinforce their argument of interpretability.\n\n**Weaknesses**\n\nI have two primary concerns with the submission: first is a lack of research into related work in this area and second is the depth of experimentation. I will discuss each of these in turn below. Together they trace one underlying request: for the authors to take more care in providing appropriate context for how their results fit into the broader field. They should include a more thorough background investigation, provide additional comparisons to previous attempts, and conduct additional experiments investigating the effects of their own modifications.\n\n**Related work**\n\nThe idea of using latent variance to regularize sparse codes enforcing a fixed dictionary norm is not new. In fact, one of the first sparse coding methods introduced by Foldiak has this term explicitly [1], which was extended to a learned model in [2]. This was extended further by [3] to produce a modern variant that does not use backpropagation for learning. However, it was recently shown to learn similar image features to more traditional sparse coding models by [4]. In early work from Olshausen & Field [5], the normalization of the weights was actually set to match the variance of the sparse codes, not scaled to a constant norm as the authors suggest (they note “The vector length (gain) of each basis function was adapted over time so as to maintain equal variance on each coefficient.”, although I admit that the practice of normalizing the dictionary columns to constant norm is much more common today). While somewhat orthogonal, the more modern SVAE model proposed by [6] also includes a variance-based adaptation scheme in the weight update rule and the inference method has constrained latent variance by definition due to the VAE framework. Some works focus on normalizing the weights to control the variance, while others focus on normalizing the variance itself. Both approaches are valid and provide core insights that give necessary context for assessing the novelty of the submitted work.\n\nOne of the core research areas that the authors are addressing can be summed up by their phrasing: “Sparse representations of images have also been shown to improve classification performance.” However, they lack references to a large volume of works in this space, leading me to assume that they are not aware of the many previous attempts to test this hypothesis. What’s more, there is active research in the area that both supports and refutes this claim. Examples of the former can be found from a number of papers [7-12]. Studies claiming to refute the benefit of sparse coding models also provide compelling counter-points that should be considered in this work, for example [13-15].\n\nMinor note - The authors note that SSL has been achieved by “enforcing equivariant representations across transformations of the same image”. It might be of interest to readers if they also included examples more closely related to the sparse coding literature, such as [16-19], the first of which is also a hierarchical sparse coding model with a non-linear decoder.\n\n**Summary**\nI am not suggesting that it is necessary for the authors to provide an extensive background of “unsupervised learning” or “sparse coding”. However, their choice of breadth of citations will matter for context and assessing novelty. The citation list I’ve provided is far from exhaustive, and some may be less relevant than others. My suggestion is that the authors take some more time to consider how their work fits into the greater context of this area of research, and perform their own literature review in the context of the works I’ve named.\n\n**2) Depth of experimentation**\n\nThe authors propose several modifications to the classic sparse coding setup: a non-linear decoder, a variance regularization term, a learned inference network (LISTA), and a modification to iterative inference (eq 13).\n\nTo ensure that I understand correctly, I will outline each of the models tested & the primary experiments:\n\nSDL is a standard sparse coding setup. It uses the unmodified FISTA algorithm for inference (i.e. not including the extra term from eq 13), a linear decoder, and the decoder weights are projected to have an l2 norm of 1 after each weight update. SDL-NL is the same as SDL, but with a two-layer nonlinear decoder, where the columns of both layer weights are fixed to have an l2 norm of 1.\n\nVDL is the proposed variance-regularized model. During training, a modified FISTA is used for inference, which includes the variance regularization term and a new LISTA regularization term (eq 13). During testing, LISTA is used for inference. The decoder weight norms are unconstrained. VDL-NL is the same as VDL, but with a two-layer nonlinear decoder with weights that have no normalization constraints.\n\n\nFor the denoising experiments, they provide corrupted images as inputs to the networks and measure the PSNR of the reconstructions.\n\nFor the few-shot classification experiments, all four networks are trained without supervision on the training set, then the weights are frozen, then a linear classifier is trained with some limited number of labeled samples.\n\nAssuming these descriptions are correct, I have the following questions:\n\nThe authors note on pg 5 footnote 1 that the modified FISTA algorithm lacks the momentum term. Why, then, are they calling it FISTA? As they explain, FISTA “accelerates the convergence of ISTA by adding a momentum term in each of its iterations.” So this is really a modified ISTA, correct? (I’ll continue to follow their convention of calling it FISTA below, assuming I missed something.)\n\nHow does the LISTA regularization term influence performance? The authors are adding two new terms to FISTA inference: first is variance regularization, the second is l2 distance to the predicted LISTA code. This second one seems circular – LISTA is training to emulate the FISTA outputs, but the FISTA outputs are constrained to look like the LISTA outputs. I would like to see more detail on why this was chosen and what happens if they don’t include it (what is the difference in performance on their chosen metrics and the weights learned).\n\nFor the reconstruction & denoising experiments, are the images used from the test set? I may have missed this. If not, they should be. I assume they also used the test set for the classification results (fig 9; table 1). Can the authors point me to where they say this in the text?\n\nIs the SDL decoder weight matrix constrained to be the transpose of the encoder weight matrix used for FISTA? From the equations in 2.1 it seems as though the encoding & decoding weights are tied. However in LISTA they are not tied. If they are not tied, it would be useful to investigate how they compare, similar to what was done in [20].\n\nIn their words, LISTA was argued to only be designed for “input vectors drawn from the same distribution as our training samples.” Although they do not clarify, it is assumed that they mean the training and test sets are each independently drawn from the same distribution, as is typical in the classification literature. The submitted work has extended LISTA to explicitly test generalization performance in the classification regime with a test set of images. It would be helpful if the authors provided a direct comparison of generalization classification performance between LISTA and iterative inference methods like ISTA or FISTA. That is, without including their variance regularization energy function.\n\nRegarding figure 9, if I am understanding correctly, the top 1 accuracy for 1-shot (1 example per class) is between 35% and 50% for all models. I believe a zero-shot (the network itself is “trained” without labels) K-NN can achieve 95% accuracy [21]. It is my understanding that one-shot classification accuracy for nonlinear encoders is readily also in the high 90s. So am I correct in assuming that the objective here is not to perform well in general in this setting, but simply to compare SDL against VDL? If so, it would be helpful if the authors made this more explicit. It would also be helpful if they extended the curve in fig 9 until the number of training samples is maximal – that is using the full training set. As I mentioned earlier, there is much disagreement in the field as to whether sparse coding is useful for this classification task at all. Extending their results to the domains that are tested by other works will allow the reader to easily compare their experimental setup and model performance to similar alternatives.\n\n**Summary**\nIt is necessary to investigate each of all of the proposed modifications individually and in an intersectional manner in order to properly assess how they compare. To that end, I would suggest that the authors perform a larger set of ablation experiments.\n\n**References**\n\nVARIANCE NORMALIZATION\n\n[1] P Foldiak (1989) - Adaptive Network for Optimal Linear Feature Extraction\n\n[2] P Foldiak (1990) - Forming Sparse Representations by Local Anti-Hebbian Learning\n\n[3] J Zylberberg, J Murphy, MR DeWeese (2011) - A Sparse Coding Model with Synaptically Local Plasticity and Spiking Neurons Can Account for the Diverse Shapes of V1 Simple Cell Receptive Fields\n\n[4] EVM Dodds, MR DeWeese (2019) - On the Sparse Structure of Natural Sounds and Natural Images- Similarities, Differences, and Implications for Neural Coding\n\n[5] BA Olshausen, DJ Field (1996) - Emergence of Simple-Cell Receptive Field Properties by Learning A Sparse Code For Natural Images\n\n[6] G Barello, AS Charles, JW Pillow (2018) - Sparse-Coding Variational Auto-Encoders\n\nSPARSE CODING IMPROVES CLASSIFICATION\n\n[7] J Mairal, F Bach, J Ponce, G Sapiro, A Zisserman (2008) - Discriminative Learned Dictionaries For Local Image Analysis\n\n[8] J Mairal, F Bach, J Ponce, G Sapiro (2009) - Online Dictionary Learning for Sparse Coding\n\n[9] R Raina, A Battle, H Lee, B Packer, AY Ng (2007) - Self-Taught Learning: Transfer Learning from Unlabeled Data\n\n[10] J Yang, K Yu, Y Gong, T Huang (2009) - Linear Spatial Pyramid Matching Using Sparse Coding for Image Classification\n\n[11] A Coates, H Lee, AY Ng (2011) - An Analysis of Single-Layer Networks in Unsupervised Feature Learning\n\n[12] DM Paiton, CG Frye, SY Lundquist, JD Bowen, R Zarcone, BA Olshausen (2020) - Selectivity and Robustness of Sparse Coding Networks\n\nSPARSE CODING DOES NOT IMPROVE CLASSIFICATION\n\n[13] A Coates, AY Ng (2011) - The Importance of Encoding Versus Training with Sparse Coding and Vector Quantization\n\n[14] R Rigamonti, MA Brown, V Lepetit (2011) - Are Sparse Representations Really Relevant for Image Classification\n\n[15] K Luther, HS Seung (2022) - Sensitivity of Sparse Codes to Image Distortions\n\nLEARNING FROM EQUIVARIANCE\n\n[16] Y Chen, DM Paiton, BA Olshausen (2018) - The Sparse Manifold Transform\n\n[17] P Foldiak (1990) - Learning Invariance from Transformation Sequences\n\n[18] A Hyvarinen, J Hurri, J Vayrynen (2003) - Bubbles- A Unifying Framework for Low-Level Statistical Properties of Natural Image Sequences\n\n[19] L Wiskott, T Sejnowski (2002) - Slow feature analysis-Unsupervised learning of invariances\n\nOTHER REFERENCES\n\n[20] JT Rolfe, Y Lecun (2013) - Discriminative Recurrent Sparse Auto-Encoders\n\n[21] Y LeCun, L Bottou, Y Bengio, P Haffner (1998) - Gradient-Based Learning Applied to Document Recognition", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a modification to the original sparse coding model that replaces the dependency of normalized dictionary weights with a latent code variance regularization term. They argue that this allows them to avoid mode collapse resulting from the existing l1 regularization. They additionally extend both the classic and their modified sparse coding model with a non-linear decoding network. They claim that these modifications improve reconstruction for a given sparsity level, denoising performance, and few-shot classification scores over a standard sparse coding-based classifier, while maintaining interpretability.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I have asked that the authors provide additional context for how their work fits into the broader field. I have also asked that they perform additional ablation experiments to understand how each of their contributions impact performance.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FZDg2lLdkw", "reviewer": "Reviewer_unXH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I enjoyed reading the paper! The idea of regularizing the sparse codes is new and has shown to lead to comparable reconstructions with standard normalization.\n\nIt will be great if the authors could clarify the following points:\n\n1. I understand that the specific use the variance regularization is new, but the question that I am asking myself is this: why variance regularization is preferred over the use of weight decay (L2 regularization over the weights) or standard normalization? All those methods will avoid the code collapse issue, why one is better than the other?\n\n2. In other words, is there any theoretical motivation that flashes out why variance regularization is preferred over the standard L2 norm normalization?\n\n3. Moreover, in the new approach, there are two additional hyper-parameters to tune (beta and T) compared to SDL, and one additional hyper-parameter compared to weight decay. Why should one prefer to increase the number of hyperparameters and use the variance regularizer if the performance is similar to standard methods?\n\n4. Weight decay is widely used in end-to-end multi-layer dictionary learning. I believe it will valuable to include a comparison with weight decay, as the latter also avoids the sparse codes to collapse.\n\nOther, minor issues:\n\n5. What is the effect of T and beta: how sensitive is the learning to the choice of this pair?\n\n6. How should one tune the hyper-parameters?\n\n7. I do not understand the “Non-Linear Decoder” architecture. Please include a mathematical/algorithmic description of this part.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper suggests a regularization that prevents the estimated sparse codes to collapse. The idea is to ensure that the sparse codes will have variance above a pre-defined threshold. This regularization can be used for learning data-adaptive representations via dictionary learning / auto-encoder. In contrast with standard dictionary learning schemes (e.g., KSVD), there is no need to normalize the dictionary atoms in an L2 sense, but there are additional hyperparameters to tune.\n\nThe authors train the entire dictionary learning pipeline using end-to-end learning, by unfolding FISTA iterations. The sparse code variance regularization is shown to yield comparable reconstructions to that of the standard dictionary atom normalization.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please address points (3)-(7) above. Also, I believe that it will be great if the authors could discuss and clarify points (1)-(2).\n\n\n", "broader_impact_concerns": "The authors explain the ethical implications clearly. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission introduces a new sparse coding approach which avoids collapse in the codes without having to regularize the decoder. Instead, variance regularization is applied to each latent code component in the form of a squared hinge term encouraging the code component's variance over a given mini-batch to reach some pre-determined threshold value.  Since inference now introduces a dependency across latent codes, the submission also proposes to train an encoder which maps inputs to their corresponding latent codes.

The proposed approach (VDL) is compared against the usual sparse coding approach (SDL) which enforces a constant L2 norm on the decoder weight columns. Various sparsity regularization coefficients are used during training, and the approaches are evaluated in terms of peak signal-to-noise ratio (PSNR) for MNIST and ImageNet patches, and VDL is found to retain better PSNR at larger code sparsity values. The models are also evaluated on two downstream tasks, namely image denoising and MNIST classification in the low data regime. For the latter, performance is measured in terms of the linear separability of the latent codes, and VDL with a non-linear decoder is shown to perform better than the alternatives in the very low data regime.

Reviewers enjoy the writing quality (unXH, G1LD) and clarity (G1LD), and note that the proposed approach is straightforward (Lkv3) and the evaluation ventures beyond reconstruction and denoising to also consider classification in the low-data regime (G1LD). The main concerns they raise are:

- The submission lacks references to multiple related works (G1LD). Reviewer G1LD provides a comprehensive list of citations which the authors acknowledge and promise to incorporate in the submission. Reviewer G1LD concedes that they haven't seen the specific variance regularization implementation proposed in the submission before. This point has been addressed to their satisfaction, as their official recommendation leans towards acceptance (provided the promised changes are materialized in a submission update).
- It's unclear why one would prefer the proposed approach over alternatives such as weight decay or weight normalization (unXH, Lkv3). The authors clarified that the proposed approach is a direct way of preventing latent code collapse (as opposed to the indirect alternatives), and that extending L2-normalization to the non-linear decoder case might be harder to optimize. The authors also ran additional experiments comparing VDL with weight decay which shows that VDL outperforms it in terms of PSNR on MNIST, and reported observing more dictionary atoms dying off at higher sparsity levels when using weight decay. Reviewer unXH remains unconvinced of the advantages of VDL over simple alternatives. Reviewer G1LD points out that although the performance improvements observed are modest, the submission's value to the research community resides in its extensive testing across the different model variations with consistent methods for parameter selection, architecture, etc. I agree with Reviewer G1LD, given that TMLR "emphasizes technical correctness over subjective significance"; to me the proposed approach is an interesting alternative for avoiding latent code collapse, and the empirical evaluation demonstrates that it performs as advertised.

Given the above, I recommend accepting the submission with minor revision. The revision should incorporate the expanded discussion on related work requested by Reviewer G1LD and the additional experiments currently contained in the supplementary material.
Decision: Accept (poster)