Title: LEARNING LESS-OVERLAPPING REPRESENTATIONS

Abstract: In representation learning (RL), how to make the learned representations easy to interpret and less overfitted to training data are two important but challenging issues. To address these problems, we study a new type of regularization approach that encourages the supports of weight vectors in RL models to have small overlap, by simultaneously promoting near-orthogonality among vectors and sparsity of each vector. We apply the proposed regularizer to two models: neural networks (NNs) and sparse coding (SC), and develop an efficient ADMM-based algorithm for regularized SC. Experiments on various datasets demonstrate that weight vectors learned under our regularizer are more interpretable and have better generalization performance.

1. INTRODUCTION:

In representation learning (RL), two critical issues need to be considered. First, how to make the learned representations more interpretable? Interpretability is a must in many applications. For instance, in a clinical setting, when applying deep learning (DL) and machine learning (ML) models to learn representations for patients and use the representations to assist clinical decision-making, we need to explain the representations to physicians such that the decision-making process is transparent, rather than being black-box. Second, how to avoid overfitting? It is often the case that the learned representations yield good performance on the training data, but perform less well on the testing data. How to improve the generalization performance on previously unseen data is important.

In this paper, we make an attempt towards addressing these two issues, via a unified approach. DL/ML models designed for representation learning are typically parameterized with a collection of weight vectors, each aiming at capturing a certain latent feature. For example, neural networks are equipped with multiple layers of hidden units where each unit is parameterized by a weight vector. In another representation learning model -sparse coding (Olshausen & Field, 1997) , a dictionary of basis vectors are utilized to reconstruct the data. In the interpretation of RL models, a major part is to interpret the learned weight vectors. Typically, elements of a weight vector have one-to-one correspondence with observed features and a weight vector is oftentimes interpreted by examining the top observed-features that correspond to the largest weights in this vector. For instance, when applying SC to reconstruct documents that are represented with bag-of-words feature vectors, each dimension of a basis vector corresponds to one word in the vocabulary. To visualize/interpret a basis vector, one can inspect the words corresponding to the large values in this vector. To achieve better interpretability, various constraints have been imposed on the weight vectors. Some notable ones are: (1) Sparsity (Tibshirani, 1996) -which encourages most weights to be zero. Observed features that have zeros weights are considered to be irrelevant and one can focus on interpreting a few non-zero weights. (2) Diversity (Wang et al., 2015) -which encourages different weight vectors to be mutually "different" (e.g., having larger angles (Xie et al., 2015) ). By doing this, the redundancy among weight vectors is reduced and cognitively one can map each weight vector to a physical concept in a more unambiguous way. (3) Non-negativeness (Lee & Seung, 1999) -which encourages the weights to be nonnegative since in certain scenarios (e.g., bag of words representation of documents), it is difficult to make sense of negative weights. In this paper, we propose a new perspective of interpretability: less-overlapness, which encourages the weight vectors to have small overlap in supports1 . By doing this, each weight vector is anchored on a unique subset of observed features without being redundant with other vectors, which greatly facilitates interpretation. For example, if topic models (Blei et al., 2003) are learned in such a way, each topic To encourage less-overlapness, we propose a regularizer that simultaneously encourages different weight vectors to be close to being orthogonal and each vector to be sparse, which jointly encourage vectors' supports to have small overlap. The major contributions of this work include:

• We propose a new type of regularization approach which encourages less-overlapness, for the sake of improving interpretability and reducing overfitting. • We apply the proposed regularizer to two models: neural networks and sparse coding (SC), and derive an efficient ADMM-based algorithm for the regularized SC problem.

• In experiments, we demonstrate the empirical effectiveness of this regularizer.

2. METHODS:

In this section, we propose a nonoverlapness-promoting regularizer and apply it to two models.

2.1. NONOVERLAPNESS-PROMOTING REGULARIZATION:

We assume the model is parameterized by m vectors W = {w i } m i=1 . For a vector w, its support s(w) is defined as {i|w i = 0} -the indices of nonzero entries in w. We first define a score õ(w i , w j ) to measure the overlap between two vectors:

EQUATION

)

which is the Jaccard index of their supports. The smaller õ(w i , w j ) is, the less overlapped the two vectors are. For m vector, the overlap score is defined as the sum of pairwise scores

o(W) = 1 m(m -1) m i =j õ(w i , w j ).

(2)

This score function is not smooth, which will result in great difficulty for optimization if used as a regularizer. Instead, we propose a smooth function that is motivated from õ(w i , w j ) and can achieve a similar effect as o(W). The basic idea is: to encourage small overlap, we can encourage (1) each vector has a small number of non-zero entries and (2) the intersection of supports among vectors is small. To realize (1), we use an L1 regularizer to encourage the vectors to be sparse. To realize (2), we encourage the vectors to be close to being orthogonal. For two sparse vectors, if they are close to orthogonal, then their supports are landed on different positions. As a result, the intersection of supports is small.

We follow the method proposed by Xie et al. (2017b) to promote orthogonality. To encourage two vectors w i and w j to be close to being orthogonal, one can make their 2 norm w i 2 , w j 2 close to one and their inner product w i w j close to zero. Based on this, one can promote orthogonality among a set of vectors by encouraging the Gram matrix G (G ij = w i w j ) of these vectors to be close to an identity matrix I. Since w i w j and zero are off the diagonal of G and I respectively, and w i 2 2 and one are on the diagonal of G and I respectively, encouraging G close to I essentially makes w i w j close to zero and w i 2 close to one. As a result, w i and w j are encouraged to be close to being orthogonal. In (Xie et al., 2017b) , one way proposed to measure the "closeness" between two matrices is to use the log-determinant divergence (LDD) (Kulis et al., 2009) . The LDD between two m × m positive definite matrices X and Y is defined as D(X, Y) = tr(XY -1 )log det(XY -1 ) -m where tr(•) denotes matrix trace. The closeness between G and I can be achieved by encouraging their LDD D(G, I) = tr(G) -log det(G) -m to be small.

Combining the orthogonality-promoting LDD regularizer with the sparsity-promoting L1 regularizer together, we obtain the following LDD-L1 regularizer

EQUATION

)

where γ is a tradeoff parameter between these two regularizers. As verified in experiments, this regularizer can effectively promote non-overlapness. The formal analysis of the relationship between Eq.( 3) and Eq.( 2) will be left for future study. It is worth noting that either L1 or LDD alone is not sufficient to reduce overlap. As illustrated in Figure 1 (a) where only L1 is applied, though the two vectors are sparse, their supports are completely overlapped. In Figure 1 (b) where the LDD regularizer is applied, though the two vectors are very close to orthogonal, their supports are completely overlapped since they are dense. In Figure 1 (c) where the LDD-L1 regularizer is used, the two vectors are sparse and are close to being orthogonal. As a result, their supports are not overlapped.

2.2. CASE STUDIES:

In this section, we apply the LDD-L1 regularizer to two models.

Neural Networks In a neural network (NN) with L hidden layers, each hidden layer l is equipped with m (l) units and each unit i is connected with all units in layer l -1. Hidden unit i at layer l is parameterized by a weight vector w (l)

i . These hidden units aim at capturing latent features underlying data. For m (l) weight vectors

W (l) = {w (l) i } m (l)

i=1 in each layer l, we apply the LDD-L1 regularizer to encourage them to have small overlap. An LDD-L1 regularized NN problem (LDD-L1-NN) can be defined in the following way: min

{W (l) } L l=1 L({W (l) } L l=1 ) + λ L l=1 Ω(W (l) )

where L({W (l) } L l=1 ) is the objective function of this NN.

Sparse Coding Given n data samples X ∈ R d×n where d is the feature dimension, we aim to use a dictionary of basis vectors W ∈ R d×m to reconstruct X, where m is the number of basis vectors. Each data sample x is reconstructed by taking a sparse linear combination of the basis vectors x ≈ m j=1 α j w j , where {α j } m j=1 are the linear coefficients and most of them are zero. The reconstruction error is measured using the squared L2 norm x-m j=1 α j w j 2 2 . To achieve sparsity among the codes, L1 regularization is utilized:

m j=1 |α j | 1 .

To avoid the degenerated case where most coefficients are zero and the basis vectors are of large magnitude, L2 regularization is applied to the basis vectors: w j 2 2 . We apply the LDD-L1 regularizer to encourage the supports of basis vectors to have small overlap. Putting these pieces together, we obtain the LDD-L1 regularized SC (LDD-L1-SC) problem

min W,A 1 2 X -WA 2 F + λ 1 |A| 1 + λ2 2 W 2 F + λ3 2 (tr(W W) -log det(W W)) + λ 4 |W| 1 (4)

where A ∈ R m×n denotes all the linear coefficients.

Algorithm 1 Algorithm for solving the LDD-L1-SC problem:



Initialize W and A repeat:

Update A with W being fixed, by solving n Lasso problems defined in Eq.( 6). repeat Update W by solving the Lasso problem defined in Eq.( 10)

U ← U + (W -W) repeat for i ← 1 to n do

Update the ith column vector w i of W using Eq.( 23) end for until convergence of the problem defined in Eq.( 12) until convergence of the problem defined in Eq.( 9) until convergence of the problem defined in Eq.( 4)

3. ALGORITHM:

For LDD-L1-NNs, a simple subgradient descent algorithm is applied to learn the weight parameters. For LDD-L1-SC, we solve it by alternating between A and W: (1) updating A with W fixed; (2) updating W with A fixed. These two steps alternate until convergence. With W fixed, the subproblem defined over A is

EQUATION

which can be decomposed into n Lasso problems:

for i = 1, • • • , n min ai 1 2 x i -Wa i 2 2 + λ 1 |a i | 1 (6)

where a i is the coefficient vector of the i-th sample. Lasso can be solved by many algorithms, such as proximal gradient descent (PGD) (Parikh & Boyd, 2014) . Fixing A, the sub-problem defined over W is:

EQUATION

We solve this problem using an ADMM-based algorithm. First, we write the problem into an equivalent form

EQUATION

Then we write down the augmented Lagrangian function

EQUATION

We minimize this Lagrangian function by alternating among W, U and W.

Update W The subproblem defined on W is

EQUATION

)

which is a Lasso problem and can be solved using PGD (Parikh & Boyd, 2014) .

Update U The update equation of U is simple.

EQUATION

The subproblem defined on W is

EQUATION

)

which can be solved using a coordinate descent algorithm. The derivation is given in the next subsection.

3.1. COORDINATE DESCENT ALGORITHM FOR LEARNING W:

In each iteration of the CD algorithm, one basis vector is chosen for update while the others are fixed. Without loss of generality, we assume it is w 1 . The sub-problem defined over w 1 is min w1

1 2 n i=1 x i - m l=2 a il w l -a i1 w 1 2 2 + λ2+λ3 2 w 1 2 2 -λ3 2 logdet(W W) + u w 1 + ρ 2 w 1 -w 1 2 2 (13)

To obtain the optimal solution, we take the derivative of the objective function and set it to zero. First, we discuss how to compute the derivative of logdet(W W) w.r.t w 1 . According to the chain rule, we have ∂logdet(W W)

EQUATION

where

EQUATION

According to the inverse of block matrix

EQUATION

where

EQUATION

Then

EQUATION

)

where

EQUATION

To this end, we obtain the full gradient of the objective function in Eq.( 13):

n i=1 a i1 (a i1 w 1 + m l=2 a il w l -x i ) + (λ 2 + λ 3 )w 1 -λ 3 Mw1 w 1 Mw1 + ρ(w 1 -w 1 ) + u. (21)

Setting the gradient to zero, we get

(( n i=1 a 2 i1 + λ 2 + λ 3 + ρ)I -λ 3 M/(w 1 Mw 1 ))w 1 = n i=1 a i1 (x i - m l=2 a il w l ) -u + ρ w 1 . (22) Let γ = w 1 Mw 1 , c = n i=1 a 2 i1 + λ 2 + λ 3 + ρ, b = n i=1 a i1 (x i - m l=2 a il w l ) -u + ρ w j , then (cI -λ3 γ M)w 1 = b and w 1 = (cI -λ3 γ M) -1 b.

Let UΣU be the eigen decomposition of M, we have

EQUATION

The matrix A = W ¬1 (W ¬1 W ¬1 ) -1 W ¬1 is idempotent, i.e., AA = A, and its rank is m -1.

According to the property of idempotent matrix, the first m -1 eigenvalues of A equal to one and the rest equal to zero. Thereafter, the first m -1 eigenvalues of M = I -A equal to zero and the rest equal to one. Based on this property, Eq.( 24) can be simplified as

γ d s=m (U b) 2 s (rc -λ 3 ) 2 = 1 (25)

After simplification, it is a quadratic function where γ has a closed form solution. Then we plug the solution of γ into Eq.( 23) to get the solution of w 1 .

4. EXPERIMENTS:

In these section, we present experimental results. The studies were performed on three models: sparse coding (SC) for document modeling, long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) network for language modeling and convolutional neural network (CNN) (Krizhevsky et al., 2012) for image classification.

4.1. DATASETS:

We used four datasets. The SC experiments were conducted on two text datasets: 20-Newsgroups2 (20-News) and Reuters Corpus3 Volume 1 (RCV1). The 20-News dataset contains newsgroup documents belonging to 20 categories, where 11314, 3766 and 3766 documents were used for training, validation and testing respectively. The original RCV1 dataset contains documents belonging to 103 categories. Following (Cai & He, 2012) , we chose the largest 4 categories which contain 9625 documents, to carry out the study. The number of training, validation and testing documents are 5775, 1925, 1925 respectively. For both datasets, stopwords were removed and all words were changed into lower-case. Top 1000 words with the highest document frequency were selected to form the vocabulary. We used tf-idf to represent documents and the feature vector of each document is normalized to have unit L2 norm.

The LSTM experiments were conducted on the Penn Treebank (PTB) dataset (Marcus et al., 1993) , which consists of 923K training, 73K validation, and 82K test words. Following (Mikolov et al.) , top 10K words with highest frequency were selected to form the vocabulary. All other words are replaced with a special token UNK.

The CNN experiments were performed on the CIFAR-10 dataset 4 . It consists of 32x32 color images belonging to 10 categories, where 50,000 images were used for training and 10,000 for testing. 5000 training images were used as the validation set for hyperparameter tuning. We augmented the dataset by first zero-padding the images with 4 pixels on each side, then randomly cropping the padded images to reproduce 32x32 images.

4.2. LDD-L1 AND NON-OVERLAPNESS:

First of all, we verify whether the LDD-L1 regularizer is able to promote non-overlapness. The study is performed on the SC model and the 20-News dataset. The number of basis vectors was set to 50. For 5 choices of the regularization parameter of LDD-L1: {10 -4 , 10 -3 , • • • , 1}, we ran the LDD-L1-SC model until convergence and measured the overlap score (defined in Eq.2) of the basis vectors. The tradeoff parameter γ inside LDD-L1 is set to 1. Figure 2 shows that the overlap score consistently decreases as the regularization parameter of LDD-L1 increases, which implies that LDD-L1 can effectively encourage non-overlapness. As a comparison, we replaced LDD-L1 with LDD-only and L1-only, and measured the overlap scores. As can be seen, for LDD-only, the overlap score remains to be 1 when the regularization parameter increases, which indicates that LDD alone is not able to reduce overlap. This is because under LDD-only, the vectors remain dense, which renders their supports to be completely overlapped. Under the same regularization parameter, LDD-L1 achieves lower overlap score than L1, which suggests that LDD-L1 is more effective in promoting non-overlapness. Given that γ -the tradeoff parameter associated with the L1 norm in Table 1 : Representative words of 9 exemplar basis vectors LDD-L1 -is set to 1, the same regularization parameter λ imposes the same level of sparsity for both LDD-L1 and L1-only. Since LDD-L1 encourages the vectors to be mutually orthogonal, the intersection between vectors' supports is small, which consequently results in small overlap. This is not the case for L1-only, which hence is less effective in reducing overlap.

4.3. INTERPRETABILITY:

In this section, we examine whether the weight vectors learned under LDD-L1 regularization are more interpretable, using SC as a study case. For each basis vector w learned by LDD-L1-SC on the 20-News dataset, we use the words (referred to as representative words) that correspond to the supports of w to interpret w. Table 1 shows the representative words of 9 exemplar vectors. By analyzing the representative words, we can see vector 1-9 represent the following semantics respectively: crime, faith, job, war, university, research, service, religion and Jews. The representative words of these vectors have no overlap. As a result, it is easy to associate each vector with a unique concept, in other words, easy to interpret. Figure 3 visualizes the learned vectors where the black dots denote vectors' supports. As can be seen, the supports of different basis vectors are landed over different words and their overlap is very small.

4.4. REDUCING OVERFITTING:

In this section, we verify whether LDD-L1 is able to reduce overfitting. The studies were performed on SC, LSTM and CNN. In each experiment, the hyperparameters were tuned on the validation set.

Sparse Coding For 20-News, the number of basis vectors in LDD-L1-SC is set to 50. λ 1 , λ 2 , λ 3 and λ 4 are set to 1, 1, 0.1 and 0.001 respectively. For RCV1, the number of basis vectors is set to 200. λ 1 , λ 2 , λ 3 and λ 4 are set to 0.01, 1, 1 and 1 respectively. We compared LDD-L1 with LDD-only and L1-only. (2) orthogonality-promoting regularizers based on cosine similarity (CS) (Yu et al., 2011) , incoherence (IC) (Bao et al., 2013) , mutual angle (MA) (Xie et al., 2015) , decorrelation (DC) (Cogswell et al., 2015) , angular constraint (AC) (Xie et al., 2017a) and LDD (Xie et al., 2017b) .

Table 3 shows the perplexity on the PTB test set. Without regularization, PytorchLM achieves a perplexity of 72.3. With LDD-L1 regularization, the perplexity is significantly reduced to 71.1. This shows that LDD-L1 can effectively improve generalization performance. Compared with the sparsity-promoting L1 regularizer and orthogonality-promoting regularizers, LDD-L1 -which promotes non-overlapness by simultaneously promoting sparsity and orthogonality -achieves lower perplexity. For the convenience of readers, we also list the perplexity achieved by other state of the art deep learning models. The LDD-L1 regularizer can be applied to these models as well to potentially boost their performance. (Zagoruyko & Komodakis, 2016) . The depth and width are set to 28 and 10 respectively. The networks are trained using SGD, where the epoch number is 200, the learning rate is set to 0.1 initially and is dropped by 0.2 at 60, 120 and 160 epochs, the minibatch size is 128 and the Nesterov momentum is 0.9. The dropout probability is 0.3 and the L2 weight decay is 0.0005. Model performance is measured using error rate, which is the median of 5 runs. We compared with (1) L1 regularizer; (2) orthogonality-promoting regularizers including CS, IC, MA, DC, AC, LDD and one based on locally constrained decorrelation (LCD) (Rodríguez et al., 2016) .

Table 4 shows classification errors on CIFAR-10 test set. Compared with the unregularized WideResNet which achieves an error rate of 3.89%, the proposed LDD-L1 regularizer greatly reduces the error to 3.60%. LDD-L1 outperforms the L1 regularizer and orthogonality-promoting regularizers, demonstrating that encouraging non-overlapness is more effective than encouraging sparsity alone or orthogonality alone in reducing overfitting. The error rates achieved by other state of the art methods are also listed.

5. RELATED WORKS:

The interpretation of representation learning models has been widely studied. Choi et al. (2016) develop a two-level neural attention model that detects influential variables in a reverse time order and use these variables to interpret predictions. Lipton (2016) discuss a taxonomy of both the desiderata and methods in interpretability research. Koh & Liang (2017) propose to use influence functions to trace a model's prediction back to its training data and identify training examples that are most relevant to a prediction. Dong et al. (2017) integrate topics extracted from human descriptions into neural networks via an interpretive loss and then use a prediction-difference maximization algo-Network Error Maxout (Goodfellow et al., 2013) 9.38 NiN (Lin et al., 2013) 8.81 DSN (Lee et al., 2015) 7.97 Highway Network (Srivastava et al., 2015 ) 7.60 All-CNN (Springenberg et al., 2014) 7.25 ResNet (He et al., 2016) 6.61 ELU-Network (Clevert et al., 2015) 6.55 LSUV (Mishkin & Matas, 2015) 5.84 Fract. Max-Pooling (Graham, 2014) 4.50 WideResNet (Huang et al., 2016) 3.89 CS-WideResNet (Yu et al., 2011) 3.81 IC-WideResNet (Bao et al., 2013) 3.85 MA-WideResNet (Xie et al., 2015) 3.68 DC-WideResNet (Cogswell et al., 2015) 3.77 LCD-WideResNet (Rodríguez et al., 2016) 3.69 AC-WideResNet (Xie et al., 2017a) 3.63 LDD-WideResNet (Xie et al., 2017b) 3.65 L1-WideResNet 3.81 LDD-L1-WideResNet 3.60 ResNeXt (Xie et al., 2016) 3.58 PyramidNet (Huang et al., 2016) 3.48 DenseNet (Huang et al., 2016) 3.46 PyramidSepDrop (Yamada et al., 2016) 3.31 Table 4 : Classification error (%) on CIFAR-10 test set rithm to interpret the learned features of each neuron. Our method is orthogonal to these existing approaches and can be potentially used with them together to further improve interpretability.

6. CONCLUSIONS:

In this paper, we propose a new type of regularization approach that encourages the weight vectors to have less-overlapped supports. The proposed LDD-L1 regularizer simultaneously encourages the weight vectors to be sparse and close to being orthogonal, which jointly produces the effects of less overlap. We apply this regularizer to two models: neural networks and sparse coding (SC), and derive an efficient ADMM-based algorithm for solving the regularized SC problem. Experiments on various datasets demonstrate the effectiveness of this regularizer in alleviating overfitting and improving interpretability.

:

The support of a vector is the set of indices of nonzero entries in this vector.

http://qwone.com/ ˜jason/20Newsgroups/

http://www.daviddlewis.com/resources/testcollections/rcv1/

https://www.cs.toronto.edu/ ˜kriz/cifar.html

Training accuracy minus test accuracy.

https://github.com/pytorch/examples/tree/master/word_language_model

Figures and Tables:

Figure 1: (a) Under L1 regularization, the vectors are sparse, but their supports are overlapped; (b) Under LDD regularization, the vectors are orthogonal, but their supports are overlapped; (c) Under LDD-L1 regularization, the vectors are sparse and mutually orthogonal and their supports are not overlapped.

Figure 2: Overlap score versus the regularization parameter

Figure 3: Visualization of basis vectors

Classification accuracy on the test sets of 20-News and RCV1, and the gap between training and test accuracy.To evaluate the model performance quantitatively, we applied the dictionary learned on the training data to infer the linear coefficients (A in Eq.4) of test documents, then performed k-nearest neighbors (KNN) classification on A. Table2shows the classification accuracy on test sets of 20-News and RCV1 and the gap 5 between the accuracy on training and test sets. Without regularization, SC achieves a test accuracy of 0.592 on 20-News, which is lower than the training accuracy by 0.119. This suggests that an overfitting to training data occurs. With LDD-L1 regularization, the test accuracy is improved to 0.612 and the gap between training and test accuracy is reduced to 0.099, demonstrating the ability of LDD-L1 in alleviating overfitting. Though LDD alone and L1 alone improve test accuracy and reduce train/test gap, they perform less well than LDD-L1, which indicates that for overfitting reduction, encouraging non-overlapness is more effective than solely promoting orthogonality or solely promoting sparsity. Similar observations are made on the RCV1 dataset. Interestingly, the test accuracy achieved by LDD-L1-SC on RCV1 is even better than the training accuracy.LSTM for Language ModelingThe LSTM network architecture follows the word language model (PytorchTM) provided in Pytorch 6 . The number of hidden layers is set to 2. The embedding size is 1500. The size of hidden state is 1500. The word embedding and softmax weights are tied. The number of training epochs is 40. Dropout with 0.65 is used. The initial learning rate is 20. Gradient clipping threshold is 0.25. The size of mini-batch is 20. In LSTM training, the network is unrolled for 35 iterations. Perplexity is used for evaluating language modeling performance (lower is better). The weight parameters are initialized uniformly between [-0.1, 0.1]. The bias parameters are initialized as 0. We compare with the following regularizers: (1) L1 regularizer;

Table content: <table><tr><td>Method</td><td/><td>20-News</td><td/><td>RCV1</td></tr><tr><td>-</td><td colspan="4">Test Gap between train and test Test Gap between train and test</td></tr><tr><td>SC</td><td>0.592</td><td>0.119</td><td>0.872</td><td>0.009</td></tr><tr><td>LDD-SC</td><td>0.605</td><td>0.108</td><td>0.886</td><td>0.005</td></tr><tr><td>L1-SC</td><td>0.606</td><td>0.105</td><td>0.897</td><td>0.005</td></tr><tr><td colspan="2">LDD-L1-SC 0.612</td><td>0.099</td><td>0.909</td><td>-0.015</td></tr></table>

Word-level perplexities on PTB test set CNN for Image Classification The CNN architecture follows that of the wide residual network (WideResNet)

Table content: <table/>

Peer review (iclr2018):
 The paper proposed a new regularization approach that simultaneously encourages the weight vectors (W) to be sparse and orthogonal to each other. The argument is that the sparsity helps to eliminate the irrelevant feature vectors by making the corresponding weights zero. Nearly orthogonal sparse vectors will have zeros at different indexes and hence, encourages the weight vectors to have small overlap in terms of indices of nonzero entries (called support). Small overlap in support of weight vectors, aids interpretability as each weight vector is associated with a unique subset of feature vectors. For example, in the topic model, small overlap encourages, each topic to have unique set of representation words. The proposed approach used L1 regularizer for enforcing sparsity in W. For enforcing orthogonality between different weight vectors (wi, wj), the log-determinant divergence (LDD) regularization term encourages the Gram Matrix G (Gij = wiTwj) to be close to an identity matrix I. The authors applied and tested the performance of proposed approach on Neural Network and Sparse Coding (SC) machine learning models. The authors validated the need for their proposed regularizer through experiments on 4 datasets (3 text and 1 images). Major * The novelty of the paper is not clear. Neither L1 no logdet() are novel regularizers (see the literature of Determinatal Point Process). With the presence of the auto-differentiator, one cannot claim the making derivative a novelty. * L1 is also encourages diversity although as explicit as logdet. This is also obvious from Fig 2. Perhaps the advantage of diversity is in interpretability but that is hard to quantify and the authors did not put enough effort to do that; we only have small anecdotal results in section 4.3. * The Table 1 is not convincing because one can argue, for example, gun (vec 1) and weapon (vec 4) are colinear. * In section 4.2, the authors experimented with SC on text dataset. The overlap score decreases as the strength of regularization increases. The authors didn’t show the effect of increasing the regularization strength on the model accuracy and convergence time. This analysis is important to make sure, the decrease in overlap score is not coming at the expense of model accuracy and performance. * In section 4.4, increase in test set accuracy and difference between test and train set accuracy is used to validate the claim, that the proposed regularizer helps reducing over fitting. In Table-2, , the test accuracy increases between SC and LDD-L1 SC while the train accuracy remains almost the same. Also, the authors didn’t do any cross validation to support their claim. The difference is numbers is too small to support the claim. * In section on LSTM for Language Modeling, the perplexity score of LDD-L1 regularization on PytorchLM received perplexity score of 1.2 lower than without regularization. Although, the author mentions it as a significant reduction, the lowest perplexity score in Table 3 is significantly lower than this result. It’s not clear how 1.2 reduction in perplexity is significant and why the method should be preferred while much better models already exists. * Results of the best perplexity model, Neural Architecture Search + WT V2, with proposed regularization would also help, validating the generalizability claims of the new approach. * In CNN for Image Classification section, details of increase interpretability of the model, in terms of classification decision, is missing. * In Table-4, the proposed LDD-L1 WideResNet is not the best results. Results of adding the proposed regularization, to the best know method (Pyramid Sep Drop) would be interesting. * The proposed regularization claims to provide more interpretable representation and less overfit model. The given experiments are inadequate to validate the claims. * A more extensive experimentation is required to validate the applicability of the method. * In SC, aj are the linear coefficients or the coefficient vector of the j-th sample. If A ∈ Rm×n then aj ∈ Rm×1 and j ranges between [1,n] as in equation 6. The notation in section 2.2, Sparse Coding section is misleading as j ranges between [1,m]. * In Related works, the authors mention previous work done on interpreting the results of the machine learning models. Related works on enhancing interpretability and reducing overfitting by using regularization is missing.