Title: ON THE STATISTICAL AND INFORMATION THEORETI-CAL CHARACTERISTICS OF DNN REPRESENTATIONS

Abstract: It has been common to argue or imply that a regularizer can be used to alter a statistical property of a hidden layer's representation and thus improve generalization or performance of deep networks. For instance, dropout has been known to improve performance by reducing co-adaptation, and representational sparsity has been argued as a good characteristic because many data-generation processes have only a small number of factors that are independent. In this work, we analytically and empirically investigate the popular characteristics of learned representations, including correlation, sparsity, dead unit, rank, and mutual information, and disprove many of the conventional wisdom. We first show that infinitely many Identical Output Networks (IONs) can be constructed for any deep network with a linear layer, where any invertible affine transformation can be applied to alter the layer's representation characteristics. The existence of ION proves that the correlation characteristics of representation can be either low or high for a well-performing network. Extensions to ReLU layers are provided, too. Then, we consider sparsity, dead unit, and rank to show that only loose relationships exist among the three characteristics. It is shown that a higher sparsity or additional dead units do not imply a better or worse performance when the rank of representation is fixed. We also develop a rank regularizer and show that neither representation sparsity nor lower rank is helpful for improving performance even when the data-generation process has only a small number of independent factors. Mutual information I(z l ; x) and I(z l ; y) are investigated as well, and we show that regularizers can affect I(z l ; x) and thus indirectly influence the performance. Finally, we explain how a rich set of regularizers can be used as a powerful tool for performance tuning.
In this section, we briefly address the most popular statistical characteristics and information theoretic characteristics of representations. Consider a neural network N A whose architecture A is fixed and the weights for the l th layer are given by {W l } and {b l } after training. Notation-wise, we simply write N A = (W, b) to define a network and y or N A (x) to refer to its deterministic output for a given input x. The layer index l is omitted when the meaning is obvious. The l th layer's activation vector for the given input x is noted as z l (x) or simply z l , and the i th element of z l is noted as z l,i . The mean, variance, and standard deviation of z l,i are defined as µ z l,i , v z l,i , and σ z l,i , respectively. The covariance of z l is defined as C l . Then, the basic representation characteristics can be summarized as in Table 1 . Six of them are statistical characteristics, and the last two are information theoretic characteristics.

1. INTRODUCTION:

A learned representation can significantly affect the performance of deep networks, and the representation's distributed and deep natures are the essential elements for the success of deep learning (Bengio et al., 2013) . As a consequence, deep networks have a greater expressiveness compared to the other machine learning algorithms (Hinton et al., 1986) or shallow networks (Montufar et al., 2014; Telgarsky, 2015; Eldan & Shamir, 2016; Raghu et al., 2016) . Besides the distributed and deep natures that have been intensively studied, a hidden layer's representation characteristics are considered to be important as well. Nonetheless, a relatively smaller number of studies have been completed on the topic, and the goal of this work is to understand the representation characteristics better. Therefore, the meaning of representation in this work is restricted to the activation vector of a single hidden layer and a unit refers to a neuron of the hidden layer.

A few previous studies considered manipulating statistical characteristics of representations such as reducing covariance among hidden units (Cogswell et al., 2015; Xiong et al., 2016) , encouraging representational sparsity (Glorot et al., 2011) , or forcing parsimonious representations via clustering (Liao et al., 2016) . In some of the similar works, a popular argument has been that the representation regularization reduces the generalization error via altering a representation characteristic. This argument, however, has not been rigorously studied. Another popular argument has been the reduction of effective capacity via regularization. This argument has been recently disproved by Zhang et al. (2016) where they empirically show that explicit regularization methods like L2 weight decay and dropout cannot sufficiently limit the effective capacity of deep networks. 

| z| E i [| z l,i |] COVARIANCE c E i =j [c i,j ]

, where c i,j {C l } i,j = E[(z l,i -µ zl,i )(z l,j -µ zl,i )] CORRELATION ρ E i =j [ρ i,j ], where ρ i,j {C l } i,j /σ zl,i σ zl,j = E[(z l,i -µ zl,i )(z l,j -µ zl,i )]/σ zl,i σ zl,j SPARSITY P s E i,n [1(z n l,i )], where 1 is an indicator function whose output is 1 only when z n l,i = 0

DEAD UNIT P d E i [1(z l,i )]

, where 1 is an indicator function whose output is 1 only when z n l,i = 0 for all n = 1, .., N RANK r rank(C l ); numerical evaluations are approximated as the stable rank C l Since a novel information-theoretic analysis method was proposed for deep learning (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017) , information theoretic characteristics of representation have become an important research topic. In their work, mutual information I(z l ; x) and I(z l ; y) are used to address the learning dynamics and generalization of deep learning, where z l is the hidden layer l's representation, x is the input, and y is the output. It is further discussed that a good representation is the one that contains a minimal amount of information from the input while containing a sufficient amount of information from the output. In Achille & Soatto (2017) , the Information Bottleneck Lagrangian (Tishby et al., 2000) is decomposed into the sum of a crossentropy term and a regularization term. The regularization term turns out to be I(z l ; x) that needs to be minimized. Some of the recent works will be additionally addressed in Section 5.

Previous studies on statistical characteristics are often based on regularizers. Srivastava et al. (2014) addresses dropout for preventing co-adaptation among hidden units by randomly putting zeros in a layer's activation vector. Ioffe & Szegedy (2015) explains batch normalization (BN) that reduces internal covariate shift via normalizing activations of each unit to speed up network training. Cogswell et al. (2015) suggest DeCov that utilizes a penalty loss function to reduce activation covariance among hidden units. Choi & Rhee (2018) considers extension to class-wise regularization and provides four representation regularizers. Among them, CR (Covariance Regularizer) is equivalent to DeCov, and we adopt the notation of CR. Glorot et al. (2011) Because the true distribution of data is not accessible, the numerical results in the following sections are evaluated using the empirical distribution of the test dataset. Then, the expectations in Table 1 are with respect to the empirical distribution. For instance, C l is calculated as the covariance matrix of N activation vectors {z 1 l , ..., z N l } where z n l corresponds to the activation vector for the n'th test data sample, x n . Rank can be calculated by examining C l , but often there are tiny eigenvalues that hinder a proper assessment of the rank. Therefore, we evaluate stable rank instead, and it will be explained further in Section 4. Two information-theoretical characteristics, I(z l ; x) and I(z l ; y), are estimated using upper and lower bounds explained in Kolchinsky & Tracey (2017) ; Kolchinsky et al. (2017) . Further details are provided in Section 5. ReLU is the only activation function that is considered in this work. When ReLU is used, ACTIVATION AMPLITUDE, COVARIANCE, and CORRELATION are calculated using only the positive activation values such that the numerical evaluations can provide meaningful insights on what is happening to the non-zero representation values.

3. SCALING, PERMUTATION, COVARIANCE, AND CORRELATION:

After training is completed for a deep network N A , the output of the network becomes a deterministic function of the input x. Without an activation function, i.e. a linear layer, z l = W T l z l-1 + b l . When ReLU is applied to layer l, the activation vector becomes z l = ReLU (W T l z l-1 + b l ) = max(W T l z l-1 + b l , 0). In this section, we investigate the most flexible affine transformation that can be applied to a layer's representation z l without influencing the output N A (x) for any arbitrary input vector x. While complicated transformations over multiple layers can be explored, we limit our focus to manipulating only the weights of layer l and layer l + 1 for the analytical tractability. Because scaling and permutation are well known results, covariance and correlation are the main focus of this section.

3.1. IDENTICAL OUTPUT NETWORK (ION):

We first consider a linear layer l. For a linear layer, it turns out that any affine transformation can be applied as long as the transformation does not cause an information loss. Theorem 1. (ION for a linear layer) For a deep network N A = (W, b) whose layer l is linear, there exists N A = ( W, b) that satisfy the following conditions:

EQUATION

)

where Q is any nonsingular square matrix of a proper size and m is any vector of a proper size.

The first condition says that the two networks generate identical outputs for any input x. The second condition says that z l can be affinely transformed using any nonsingular matrix Q. The proof is straightforward and can be found in Appendix A.

While simple, Theorem 1 has significant implications on the representation characteristics of z l . Let's inspect covariance and correlation (normalized version of covariance) first. If N A is a network that is globally optimal for a task and has at least one linear layer l, then N A 's covariance C l can be whitened to have C l = I by choosing m as the expected mean and Q as a whitening matrix. The resulting network N A will have zero correlation between any pair of units in layer l, but will be globally optimal, too. In fact, there are infinitely many globally optimal networks with different covariance characteristics, and one can easily construct an ION with an arbitrary covariance matrix C l as long as its rank is the same as C l 's rank. With this result, it becomes unclear why one should pursue a lower correlation when training a deep network. Unless regularization for a low correlation somehow helps optimization to reach a better performing network, there seems to be no reason to pursue low (or high) correlation.

For dead neurons, a similar claim can be made. If globally optimal N A has no dead neurons in layer l and C l is not full rank, one can make an affine transformation to align the null spaces of C l to some of the neurons. Then, the resulting network N A will be still globally optimal, but with some dead neurons in layer l. For higher layers of classification tasks, typically the rank of C l is close to the number of classes. For classification tasks with only 2∼10 classes, it is possible to construct an ION that has as many dead neurons as the size of C l 's null space. This can be done without negatively affecting the performance, and the wisdom of 'reduce the number of dead neurons' becomes dubious.

For scaling and permutation, their influences are rather insignificant. As for the scaling that can affect activation amplitude, it often has no effect on the network's performance. For instance, scaled activation amplitude can affect the probability of classification tasks when softmax is in the last layer, but the class with the highest probability remains the same anyway. When representation regularizers are used, often activation amplitude is squashed to reduce the cost of representation penalty function but the network can still perform well. As reported in Choi & Rhee (2018) , such an activation squashing can make covariance much smaller, but the effect is removed when correlation is calculated. As for the permutation, it is considered to be meaningless because the index number itself is not important.

Before discussing further, a similar result is developed for ReLU layers. The resulting Q, however, is much more limited. The proof can be found in Appendix A as well.

Theorem 2. (ION for a ReLU layer) For a deep network N A = (W, b) whose activation function of layer l is ReLU, there exists N A = ( W, b) that satisfy the following conditions:

EQUATION

)

where Q is any permuted positive diagonal matrix of a proper size. Furthermore, it can be shown that any Q that satisfy the above two conditions must be a permuted positive diagonal matrix.

Using a permuted positive diagonal matrix Q, covariance can be affected by independently scaling activation amplitudes of layer l's units. As explained above, such a scaling is canceled out when calculating correlation and therefore a linear transformation cannot affect correlation while keeping the output identical. There are a few possibilities for overcoming the limitations of Theorem 2, and they are discussed in the following subsection.

For rank and mutual information, the invertible affine transformation has no effect. They are discussed in the following sections.

3.2. POSSIBLE EXTENSIONS OF ION:

We discuss three possible extensions of ReLU's ION.

Insertion of a linear layer One way to overcome the limitations of Q is to insert an extra linear layer near the target ReLU layer and to consider its implications. When representation characteristics are analyzed or interpreted, researchers do not care much about the layer's activation function, regularization, etc. The activation vector's representation characteristics are the essential components for understanding and assessing the representations. Therefore, one can apply the insights from Theorem 1 when the extra linear layer shows similar statistical properties as the ReLU layer. In Table 2 , statistical properties of immediately neighboring ReLU and linear layers are compared. Compared to the representation characteristics of the first ReLU layer, the 5th ReLU layer and the inserted 6th linear layer show very similar characteristics. Then the correlation and dead unit characteristics are not so important as the consequence of Theorem 1, and the same might be conjectured for the 5th ReLU layer.

Non-affine transformations over multiple layers In the ION derivations, we have considered only an affine transformation applied to the layers l and l + 1 only. If we remove the constraints and borrow the results from expressivity of DNN and universal approximation theorem, it might be possible to derive more powerful and general results. In the extreme case, one can divide a deep network N A into two parts: N A lower and N Aupper . Then,

N A lower (x) = z l and N A (x) = N Aupper (N A lower (x)).

In theory, there exist N A lower and N Aupper that can result in N A (x) = N Aupper ( N A lower (x)) while allowing z l to have a completely different characteristics compared to z l . Such N A lower and N Aupper , however, might be infeasibly large or fail to learn in the way we desire. Therefore, it might be more practical to consider a reasonable extension of Theorem 1 and Theorem 2.

Comparable Performance Network (CPN) According to Theorem 2, only permuted positive diagonal matrices can form IONs. If we ignore the result and apply an affine transformation in the same way as in the ION of a linear layer, the resulting network N A will not form an ION but instead we might be able to find a Comparable Performance Network (CPN) that achieves a comparable performance while showing different representation characteristics. We tried this brute-force method, and two sample results along with the baseline and a positive diagonal matrix case are shown in Table 3 . In the first row where Q is identity, the values are for the original network N A . In the next row, 'Random positive diagonal', uniformly random values between 0 and 1 (U (0, 1)) were used as the diagonal values. Note that this choice of Q satisfies Theorem 2, and therefore the error performance remains the same while affecting activation amplitude and covariance only. In the 'Ones with random positive diagonal', Q was chosen as a matrix of ones with its diagonals replaced with random values chosen from U (0, 1). We randomly generated 100 of such random matrices and selected the one that resulted in a higher correlation while showing a comparable performance. Despite of the very high correlation of 0.80, the selected network N A can perform comparably well. In the last row, we applied a whitening filter where Q and m were calculated while ignoring ReLU. The resulting network does not end up with zero correlation because the whitening is not perfect in the presence of ReLU. But the correlation is considerably reduced to 0.09 from 0.28 while achieving a slightly worse error rate of 5.48%.

To find the examples in Table 3 , all we had to do was to construct a meaningful matrix Q or to try 100 random matrices and choose one. The fact that it is almost painless job to find a CPN also implies that the relevant representation characteristics might not be essential for achieving high performance. 

P s = E i,n [1(z n l,i )].

This can be interpreted as the probability of z n l,i (unit i's activation for n'th test sample x n ) being zero, because 1(z n l,i ) = 1 when z n l,i = 0 and 1(z n l,i ) = 0 when z n l,i = 0. Similarly, dead unit is defined as

P d = E i [1(z l,i )]

and it can be interpreted as the probability of z l,i (unit i's activation) being always zero or at least for all M test samples.

Because 1(z l,i ) ≤ 1(z n l,i ) for any pair of (i, n), P d ≤ P s can be shown by taking expectations on both sides. The rank r in Table 1 is defined as the rank of C l . For layer l with M units, this means that r out of M linearly independent dimensions are used by the codewords {z 1 l , ..., z N l } and that the other M -r dimensions form a null space of C l . When dead units are considered, M P d units need to be constant zero by the definition of the dead unit and it implies that at least M P d dimensions need to be included in the null space. Therefore, M P d ≤ M -r. These results can be summarized as below.

EQUATION

Between sparsity P s and rank r, there is no clear relationship. The codeword z n l for a test sample x n can be very sparse, and yet the set of codewords {z 1 l , ..., z N l } collectively might use all of the M dimensions. Conversely, rank r can be very small and yet P s can also be very small when {z n l,1 , ..., z n l,M } are strongly correlated and the basis vectors are not sparse over the M units. From the viewpoint of signal processing or information theory, sparsity is a property that is related to individual signals or individual codewords while rank is a property that is related to the total number of dimensions used by the set of signals or the entire codebook. Therefore, sparsity is not directly responsible for the efficiency of the signals or codebook while rank is directly responsible for the efficiency. From the viewpoint of deep learning, rank can be associated with the maximum number of latent factors that are independent. As for the dead unit, we know from equation 6 that it is upper bounded as a function of rank. When the bound is met, the value of P d is merely an artifact of how the representation vectors {z l } are aligned with the eigenvectors of C l . If each neuron is aligned to an eigenmode of C l , then P d = 1 -r/M will be achieved. From our experience, however, such a perfect alignment never happens when using the backpropagation based learning process. This has been true even when L1R or other advanced representation regularizers were applied. ION, however, can easily meet the requirement for a linear layer. Motivated by the above discussion, we have designed a rank regularizer and examined common wisdom that says 'most of the data generation processes have a small number of independent factors and therefore increasing sparsity of representation can be helpful.' For instance, see Bengio et al. (2013) . We first explain the design of rank regularizer.

4.2. RANK REGULARIZER:

In deep learning, a low-rank approximation of convolutional filters (Jaderberg et al., 2014; Lebedev et al., 2014; Tai et al., 2015) and weight matrices (Xue et al., 2013; 2014; Nakkiran et al., 2015; Masana et al., 2017) has been widely used for network compression and fast network training. Some of the works applied a singular value decomposition to weight matrices after network training ends but not to representations. In this work, as L1 representation regularizer was designed to encourage a higher sparsity by adding a penalty loss term ] T as a surrogate. Note that N M B instead of N activation vectors are used for each mini-batch. Stable rank of Z is defined as

Ω L1R = n i |z n l,i |, Rank

EQUATION

where Z F is the Forbenius norm, Z 2 is the spectral norm, and {s i } are the singular values of Z.

From i s 2 i maxi s 2 i

, it can be clearly seen that stable rank is upper bounded by the usual rank that counts strictly positive singular values. Because the spectral norm is based on a singular value decomposition, calculating stable rank's derivative for every mini-batch is a computationally heavy operation. To reduce the computational burden, we introduce an approximation using a special case of Holder's inequality.

EQUATION

The inequality Z 2 ≤ Z 1 Z ∞ was used where Z 1 is the maximum absolute column sum of the matrix Z (sum of all activation values of unit i) and Z ∞ is the maximum absolute row sum of the matrix Z (sum of all activation values of sample n). Then the gradient of approximation Ω RR can be as below.

EQUATION

where

∂ Z 2 F ∂z n i = 2z n i , ∂ Z 1 ∂z n i = 1 (i=i * ) • sign(z n i ), ∂ Z ∞ ∂z n i = 1 (n=n * ) • sign(z n i ), i * = arg max 1≤i≤M N M B n=1 |z n i |, and n * = arg max 1≤n≤N M B M i=1 |z n i |.

4.3. A CONTROLLED EXPERIMENT ON DATA GENERATION PROCESS:

We have designed two datasets where the number of independent factors is fully controlled. The first dataset is a synthetic 10-class classification dataset that was created using Python scikit-learn library (Pedregosa et al., 2011) . The number of independent Gaussian factors, d, was controlled to be 10, 50, 100, 250, and 500, and the independent factors were mixed using a randomly generated 1000 × d rotation matrix. The second dataset is a PCA-controlled MNIST data that was created by including only the top 10, 50, 100, 250, and 500 dimensions of MNIST's PCA dimensions.

For the two datasets, we have chosen N A to be the same 5-layer MLP as before and repeatedly performed training while applying either L1R or RR with different loss weights. The results are shown in Figure 2 and Figure 3 . The sparsity and rank plots show that indeed sparsity is increased and rank is reduced by increasing the loss weight. The accuracy performance, however, does not show any meaningful dependency on d. For instance, even when d = 10 and there were only ten independent factors in the data generation process, strongly applying L1R or RR did not result in improved performance. On the contrary, the accuracy often suffered when loss weight was increased.

According to the discussion in subsection 4.1, it is not surprising that the level of learned representation's sparsity does not affect the accuracy performance. Perhaps it is more surprising that even the level of learned representation's rank does not affect the accuracy performance. We move on to the analysis of mutual information for a further discussion on this issue. 

5. MUTUAL INFORMATION:

So far, we have investigated popular statistical characteristics of representation z l where none of the statistical characteristics showed a strong and clear relationship to deep network's performance. In this section, we examine two information-theoretical characteristics: I(z l ; x) and I(z l ; y). In the original and pioneering work of Shwartz-Ziv & Tishby (2017) , the two characteristics of z l were used to explain the concept of information bottleneck on deep networks. Basically, the work shows that the task-relevant information should be maximized via I(z l ; y) while the task-irrelevant information should be minimized via I(z l ; x). A further development was made in Achille & Soatto (2017) , where the Information Bottleneck Lagrangian L(p(z l | x)) = H(y | z l ) + βI(z l ; x) was explainedthe first term is the usual cross entropy cost function, the second term is a penalty term on I(z l ; x), and β is parameter for controlling a trade-off between sufficiency (the first term) and minimality (the second term). In Achille & Soatto (2018) , they develop 'information dropout' method that implicitly minimize I(z l ; x). In their limited performance experiments, they showed that information dropout can improve MNIST classification performance by about 0.25% for the best case. In another work by Kolchinsky et al. (2017) , an upper bound derived using a non-parametric estimator of mutual information and a variational approximation is used to develop a gradient-based optimization method. They showed I(z l ; x) and I(z l ; y) are indeed reduced by the method, but did not report anything about performance.

In this work, we neither tried the aforementioned techniques nor explicitly implemented an I(z l ; x) regularizer. Instead, we simply applied the twelve regularizers (including baseline) and calculated the upper and lower bounds of I(z l ; x) and I(z l ; y). The bounds can be calculated using the results of Kolchinsky & Tracey (2017) where a pairwise distance function between mixture components is used. They prove that the Chernoff α-divergence and the Kullback-Leibler divergence provide lower and upper bounds when they are chosen as the distance function, respectively. Figure 4 shows the results for the last hidden layer together with the generalization error where the same network and dataset as in Figure 1 were used, and regularizers were also applied to the last hidden layer. One x) can be seen to be strongly dependent on which regularizer is used, and the upper and lower bounds show a similar pattern as the generalization error's pattern. In fact, the correlation between the lower bound and generalization error can be calculated to be 0.84, and the correlation between the upper bound and generalization error can be calculated to be 0.78. Therefore, it can be surmised that the regularizers might be indirectly affecting the performance by influencing I(z l ; x).

When a mutual information regularizer is excluded, the rest of the representation regularizers fail to provide general reasoning on why any of the statistical characteristics should be pursued. In fact, one can argue that even a single-neuron in layer l (activation becomes a scalar) can be a sufficient condition for encoding to have a chance to achieve the maximum possible I(z l ; y), i.e., lossless in terms of relevant information. Such an encoding on a scalar activation might be very inefficient, and a practical learning method might never reach such an encoding. Nonetheless, there is no reason why such an encoding should be impossible. Obviously, many of the statistical characteristics become meaningless for such a scalar representation, and it is high time to reconsider the so-called conventional wisdom on representation characteristics.

6. IMPROVING PERFORMANCE WITH REPRESENTATION REGULARIZERS:

In the previous sections, we have investigated representation characteristics and their relationship to the performance. All the results, except for mutual information that is shown in Figure 4 , indicate that there might be no firm ground to believe that z l 's representation characteristics are strongly related to the performance. But there have been numerous reports that performance was improved by utilizing newly designed regularizers. In this section, we investigate if (representation) regularizers can indeed consistently improve the performance for a given task condition. Here, a task condition means a learning task with small data size, a small layer width, a specific dataset, a large number of classes, or a specific optimizer. We perform experiments on MNIST, CIFAR-10, and CIFAR-100 datasets using twelve regularizers, and the representation regularizers are explained in Appendix B. The details of experimental settings and architecture parameters can be found in Appendix D.

6.1. CONSISTENTLY WELL PERFORMING REGULARIZER:

We analyze if there is a consistent dependency between a regularizer and its effect on the performance when a particular regularizer is applied to a particular task condition. Our results, as shown in Table 4 , Table 8 , and Table 9 indicate that there is no consistently well-performing regularizer for a specific task condition. As an example, consider the entire CIFAR-10 dataset results in Table 4 . While task conditions change over different columns, the data remains common for all the tasks. If there is a representation characteristic that fits the data-generation process well and one of the regularizers can match the representation well, it might have outperformed across all the columns. In the table, the best performing regularizer for each task (column) is marked in bold, and any other regularizer whose performance overlaps with the best one is highlighted in green. Looking at the bold and green-highlight patterns, one can easily conclude that there is no single regularizer that works well for all the tasks of CIFAR-10. A similar observation can be made for other task conditions. For instance, one can examine the data size of 1k. For the 1k columns of the three tables, there is no single regularizer that always performs distinctively well. In fact, we have experimented many more settings than what are shown in this paper. The hope was to find a strong match between a task condition and a representation regularizer, but we have failed to find anything that looks consistent. Many of the previous works on regularizers have compared their regularizer with only a small number of known regularizers. When many regularizers are compared over many different tasks as in our work, one can easily conclude that there is no obvious relation to declare where a certain representation characteristic is advantageous.

6.2. PERFORMANCE IMPROVEMENT USING REGULARIZERS AS A SET:

Even though no single representation characteristics consistently outperforms, it can be seen that one can improve performance by using the twelve regularizers as a set and by choosing the best performing regularizer for the given task. This is in line with the usual theme of tuning in many areas of deep learning. Looking into Table 4 more carefully, we can see that cw-VR and L1R often had the best performance for CIFAR-10 test cases. In our experiments, we have observed that one of representation regularizers often outperform weight regularizers (L1W, L2W), dropout, and BN. Even though representation regularizers do not seem to have a direct impact on the performance, they might have indirect effects on mutual information as we have seen in Section 5 or on the optimization process. When many representation regularizers are tried as a set, perhaps there is a larger chance of one of such indirect effects improving the performance.

7. DISCUSSION AND CONCLUSION:

We have studied the most popular statistical characteristics and information theoretic characteristics of DNN representations. All the statistical characteristics that were studied failed to show any general or causal pattern for improving performance. Some of the conventional wisdom were analytically dismissed. Empirical results consistently showed that none of the studied statistical characteristics is a requirement for achieving good performance. While we could not identify any systematic pattern, the popular regularizers have been frequently observed to provide a healthy performance improvement.

To understand this phenomenon, we have tried applying twelve different regularizers over many classification tasks with different task conditions. The results show that still no systematic pattern can be found, but the set of regularizers can be used as a very compelling tool for tuning the performance.

In contrast to the statistical characteristics, information theoretic characteristic I(z l ; x) showed a strong correlation with the performance of a classification task. Regularizers were able to affect the mutual information, and possibly they ended up affecting the performance as well. In this work, we have directly addressed and dismissed several conventional wisdom. However, perhaps the most important contribution of this work is to provide an early work on developing rigorous and general theories and methodologies that can be used to better understand the learned representations.

APPENDIX A PROOFS:

Theorem 1 For a deep network N A = (W, b) whose layer l is linear, there exists N A = ( W, b) that satisfy the following conditions:

∀ x, N A (x) = N A (x); ∀ x, z l = Q(z l -m),

where Q is any nonsingular square matrix of a proper size and m is any vector of a proper size.

Proof. Proof based on a simple construction. Choose the weights of N A as below.

W T l = Q W T l b l = Q(b l -m) W T l+1 = W T l+1 Q -1 b l+1 = b l+1 + W T

l+1 m For all the other layers, choose the same as N A 's weights. Then, clearly z l-1 = z l-1 and therefore

z l = W T l z l-1 + b l = Q(W T l z l-1 + b l -m) = Q(z l -m). Also, z l+1 = W T l+1 z l + b l+1 = W T l+1 Q -1 Q(z l -m) + b l+1 + W T l+1 m = z l+1 .

Because the activation vector of layer l + 1 is the same for N A and N A , the resulting outputs N A (x) and N A (x) are exactly the same as well.

Theorem 2 For a deep network N A = (W, b) whose activation function of layer l is ReLU, there exists N A = ( W, b) that satisfy the following conditions:

∀ x, N A (x) = N A (x); ∀ x, z l = Q z l ,

where Q is any permuted positive diagonal matrix of a proper size. Furthermore, it can be shown that any Q that satisfy the above two conditions must be a permuted positive diagonal matrix.

Proof. For simplicity, we denote a + = ReLU(a) and a -= ReLU(-a). Then a = a + + a -. We denote h l = W T l z l + b l , which is the representation before applying the activation function, such that

z l = h + l . If we choose W T l = Q W T l and W T l+1 = W T l+1 Q -1 , our focus is to find an invertible matrix Q that satisfy W T l+1 Q -1 (Q h l ) + + b l+1 = W T l+1 h + l + b l+1 for all x. This reduces down to finding Q that satisfies (Q h l ) + = Q h + l .

We denote i'th row of Q as q T i , and the statement mentioned above can be written as: (q T i h l ) + = q T i h l . For q T i that satisfy (q T i h l ) ≥ 0, obviously (q T i h l ) + = q T i h i l . If we substitute h l with h + l -h - l , then q T i h - l = 0. For q T i that satisfy q T i h l < 0, we can derive the condition q T i h + l = 0. Now, we will show that Q should be a permuted positive diagonal matrix using the statements proved above. For a permuted positive diagonal matrix, {q T i } are linearly independent and each q T i has only one positive element. Because Q needs to be invertible (otherwise information loss occurs and N A (x) = N A (x) cannot be achieved), it is trivial that each q T i is linearly independent. To show each q T i has only one positive element, let's assume q T i has more than one non-zero elements. If we denote q T ik as the q T i 's k-th element, and h lk as the h l 's k-th element, we can divide the element indexes as follow.

A = {k| q T ik = 0 and h lk > 0} B = {k| q T ik = 0 and h lk < 0}

Under review as a conference paper at ICLR 2019 Then, we can consider h l such that A = ∅ and B = ∅. If q T i h l >= 0, then

j∈A q T ij h lj > j∈B q T ij h lj .

In this case, the right side should be zero because q T i h - l = 0. However, q T ij should be zero to satisfy q T i h - l = 0 because h l be chosen arbitrary in the range that A = ∅ and B = ∅ and q T i h l >= 0. This is a contradiction due to the definition of B. We can prove the case of q T i h l < 0 similarly, which shows that each q T i has only one non-zero element. To show the q T i 's one element is positive, we denotes the q T i 's only one element as q T ij , in which j is the index of the non-zero element. When q T ij < 0, we can consider h l such that h lj < 0. In this case, (q T i h l ) > 0, but q T i h - l > 0, which contradicts the condition q T i h - l = 0. If we consider h l such that h lj > 0, (q T i h l ) < 0, but q T i h + l > 0, which contradicts the condition q T i h + l > 0. Therefore, q T ij is positive.

B REPRESENTATION REGULARIZERS:

In this section, we briefly introduce the representation regularizers that are used in our experiments.

Based on the notations provided in Section 2, we define class-wise statistics that are calculated using only class k's samples out of a total of K labels in the mini-batch. Class-wise mean, covariance, and variance are defined as below.

EQUATION

EQUATION

Here, S k is the set that contains indexes of the samples with class label k. Note that superscripts with and without parenthesis indicate class label and sample index, respectively. Penalty loss functions of the representation regularizers are summarized in Table 5 . 

Ω CR = i =j (c i,j ) 2

Covariance of representations calculated from all-class samples.

EQUATION

i,j ) 2 Covariance of representations calculated from the same class samples.

Ω V R = i v z l ,i

Variance of representations calculated from all-class samples.

Ω cw-V R = k i v (k) z l ,i

Variance of representations calculated from the same class samples.

Ω L1R = n i | z n l,i |

Absolute amplitude of representations calculated from all-class samples.

Ω RR = Z l 2 F Z l 2 2

Stable rank of representations calculated from all-class samples.

Ω cw-RR = k Z (k) l 2 F Z (k) l 2 2

Stable rank of representations calculated from the same class samples.

C REPRESENTATION CHARACTERISTICS:

In this section, we investigate the statistical characteristics of the learned representations when different regularizers are applied. We used the same network and dataset as the ones used for generating Figure 1 . All the regularizers were applied only to the fifth layer, and the representation characteristics were calculated using the fifth layer as well. In Table 6 , typical evaluation results of statistical characteristics are shown. We can confirm that the statistical characteristics targeted by each representation regularizer are indeed manipulated as expected (Bold). In particular, RR and cw-RR designed in this work to regularize the stable rank work as expected. The two weight regularizers (L1W: L1 Weight Regularizer, L2W: L2 Weight Regularizer) have similar characteristic values as the baseline's, and this can be taken for granted because the regularizers do not directly regularize representations. A few conventional beliefs mentioned in the paper are quantitatively confirmed as well. A large number of dead units is known to be harmful because they do not contribute toward improving the performance of deep networks. Our result shows even 39% of DEAD UNIT caused by L1R does not hurt the performance, which is in line with our analysis in Section 4.

For dropout, COVARIANCE is reduced as in Cogswell et al. (2015) , but CORRELATION is actually increased compared to the baseline. In fact, COVARIANCE is reduced simply because ACTIVATION AMPLITUDE is reduced as mentioned in Section 3, and the correlation between two active units is actually made larger by applying dropout. Therefore, it cannot be said that the relationship between a pair of neurons becomes weaker by applying dropout. This is in contrary to the 'reduction of coadaptation' idea. Note that we have excluded the inactive neurons for the evaluations. If the inactive ones are included with their zero values, the covariance and correlation values will be different. , and the one with the best validation performance for each regularizer and condition was chosen for testing. We used the validation data for MNIST dataset and the last 10,000 samples of training data for CIFAR-10/100 datasets as validation data. After a loss weight was fixed, the validation data was merged back into the training data in the case of CIFAR-10/100. Five training trials were performed, and the means of the five trials were reported. Note that a mini-batch size was set to 100 for MNIST and CIFAR-10 tasks but to 500 for CIFAR-100 due to the class-wise statistic calculation.

In this work, we carried out all the experiments using TensorFlow 1.5.

D.2 ARCHITECTURE PARAMETERS:

We used a 6-layer MLP with 100 units per hidden layer for MNIST image classification tasks. We used a CNN with four convolutional layers and one fully-connected layer for CIFAR-10/100 image classification tasks. Architecture details are described in Table 7 . 

:

SPARSITY, DEAD UNIT, AND RANK Sparsity and dead unit have been considered as important representation characteristics. Rank of C l , however, has received much less attention so far. In this section, we investigate the three and show that rank might be the most fundamental characteristics.

Figures and Tables:

Figure 1: Visualization of the learned representations for a 6-layer MLP trained with MNIST dataset. A single unit's activation histogram (upper plots) and two randomly chosen units' activation scatter plots (lower plots) are shown for the fifth layer's representation, where each color corresponds to a different class. The plots were generated using 10,000 test samples of MNIST dataset. (Upper) It can be seen that the baseline has a large classwise variance and inter-class overlaps, and BN and CR (originally known as DeCov (Cogswell et al., 2015)) show similar properties. Dropout looks completely different where activation values are more spread out for the active classes. L1R (L1 Representation regularizer) typically allow only one or two classes to be activated per unit. (Lower) While the baseline shows modest linearity, the others show quite different representation characteristics depending on the choice of regularizer. Dropout shows an extremely high class-wise correlation, but BN shows very low correlation. CR shows almost no correlation. Since L1R increases sparsity on representation, a class is activated over at most one of the two randomly chosen units.

explains L1 representation regularization, called L1R in this work, that applies L1 penalty on activations. These representation regularization methods have distinct effects on representation characteristics, and examples of the learned representations are shown in Figure 1. The representation regularizers are described in Appendix B, and a quantitative analysis of representation characteristics are provided in Appendix C.

Figure 2: Effect of L1R (L1 Representation Regularizer): representation sparsity (P s ) and accuracy are shown as a function of L1R's loss weight. Each line corresponds to a different number of independent factors. While sparsity is well controlled, accuracy does not show any meaningful dependency on the number of independent factors used in the data generation process.

Figure 4: Mutual information and generalization errorcan observe that all the regularizers end up with almost the same I(z l ; y) value. But the bounds of I(z l ; x) can be seen to be strongly dependent on which regularizer is used, and the upper and lower bounds show a similar pattern as the generalization error's pattern. In fact, the correlation between the lower bound and generalization error can be calculated to be 0.84, and the correlation between the upper bound and generalization error can be calculated to be 0.78. Therefore, it can be surmised that the regularizers might be indirectly affecting the performance by influencing I(z l ; x).

EXPERIMENTAL CONDITIONSDefault conditions are shown in bold, and the full experimental conditions are listed below.• Training data size: 1k, 5k, 50k • Layer width: (MNIST) 2, 8, 100 / (CIFAR-10/100): 32, 128, 512 Table 7: Architecture hyperparameters of CIFAR-10/100 CNN model. Layer Parameter Convolutional layer-1 Number of filters=32, Filter size=3 × 3, Convolution stride=1 Convolutional layer-2 Number of filters=64, Filter size=3 × 3, Convolution stride=1 Max-pooling layer-1 Pooling size=2 × 2, Pooling stride=2 Convolutional layer-3 Number of filters=128, Filter size=3 × 3, Convolution stride=1 Max-pooling layer-2 Pooling size=2 × 2, Pooling stride=2 Convolutional layer-4 Number of filters=128, Filter size=3 × 3, Convolution stride=1 Max-pooling layer-3 Pooling size=2 × 2, Pooling stride=2 Fully connected layer Number of units=128 • Optimizer (CIFAR-10): Adam, Momentum (lr=0.01, momentum=0.9), RMSProp (lr=0.0001) • Number of classes (CIFAR-100): 16, 64, 100 D.4 EXPERIMENTAL RESULTS (MNIST, CIFAR-100)

Representation characteristics.

Table content: <table><tr><td>Characteristic</td><td>Symbol</td><td>Expression</td></tr><tr><td>ACTIVATION AMPLITUDE</td><td/><td/></tr></table>

Comparison of statistical characteristics for linear and ReLU layers. A 7-layer MLP was used with MNIST dataset, and only the sixth layer was linear, and the others were ReLU layers. Statistical characteristics of the first layer (ReLU), fifth layer (ReLU), and sixth layer (linear) are compared. It can be seen that representation characteristics of fifth and sixth are very similar because they are both located in the upper part of the network. Note that the characteristics of the sixth layer were calculated only using positive activation values for fair comparison.

Table content: <table><tr><td>Regularizer</td><td colspan="6">ACTIVATION AMPLITUDE COVARIANCE CORRELATION SPARSITY DEAD UNIT RANK</td></tr><tr><td>Baseline (1st)</td><td>1.16</td><td>0.08</td><td>0.14</td><td>0.00</td><td>0.00</td><td>2.22</td></tr><tr><td>Baseline (5th)</td><td>2.22</td><td>0.45</td><td>0.25</td><td>0.32</td><td>0.07</td><td>2.27</td></tr><tr><td>Baseline (6th)</td><td>2.75</td><td>0.62</td><td>0.25</td><td>0.47</td><td>0.00</td><td>2.78</td></tr><tr><td>Dropout (1st)</td><td>0.76</td><td>0.04</td><td>0.27</td><td>0.86</td><td>0.00</td><td>3.50</td></tr><tr><td>Dropout (5th)</td><td>2.19</td><td>1.23</td><td>0.70</td><td>0.53</td><td>0.00</td><td>2.19</td></tr><tr><td>Dropout (6th)</td><td>1.85</td><td>0.97</td><td>0.53</td><td>0.48</td><td>0.01</td><td>1.52</td></tr><tr><td>BN (1st)</td><td>0.80</td><td>0.03</td><td>0.10</td><td>0.49</td><td>0.00</td><td>4.07</td></tr><tr><td>BN (5th)</td><td>1.01</td><td>0.10</td><td>0.19</td><td>0.51</td><td>0.00</td><td>4.59</td></tr><tr><td>BN (6th)</td><td>1.39</td><td>0.19</td><td>0.23</td><td>0.48</td><td>0.00</td><td>4.26</td></tr></table>

Statistical characteristics of representations transformed by CPNs. The original network is 6-layer MLP on the MNIST dataset, and the 5th layer representations were transformed. To improve the performance, the weights to the output layer were fine tuned after applying Q.

Table content: <table><tr><td>Q matrix</td><td colspan="6">Error (%) ACTIVATION AMPLITUDE COVARIANCE CORRELATION SPARSITY DEAD UNIT</td></tr><tr><td>Identity</td><td>2.54</td><td>6.79</td><td>4.29</td><td>0.28</td><td>0.36</td><td>0.13</td></tr><tr><td>Random positive diagonal</td><td>2.54</td><td>3.37</td><td>1.04</td><td>0.28</td><td>0.36</td><td>0.13</td></tr><tr><td>Ones with random positive diagonal</td><td>2.76</td><td>158.88</td><td>723.55</td><td>0.80</td><td>0.00</td><td>0.00</td></tr><tr><td>Whitening</td><td>5.48</td><td>1.22</td><td>0.96</td><td>0.09</td><td>0.49</td><td>0.02</td></tr><tr><td colspan="2">4.1 ANALYTICAL RELATIONSHIP</td><td/><td/><td/><td/><td/></tr><tr><td colspan="2">In Table 1, sparsity is defined as</td><td/><td/><td/><td/><td/></tr></table>

Regularizer (RR) is designed to encourage a lower rank of representations and used during network training. Because the usual definition of rank can be very sensitive to the tiny singular values, we use stable rank of activation matrix Z = [z 1 l , . . . , z N M B

Table content: <table><tr><td>l</td></tr></table>

Condition experiment results for CIFAR-10 CNN model. The best performing regularizer in each condition (each column) is shown in bold, and other regularizers whose performance range overlaps with the best one are highlighted in green. For the default condition, the standard values of data size=50k and layer width=128 were used, and Adam optimizer was applied. For the other columns, all the conditions were the same as the default except for the condition indicated on the top part of the columns. Regularizers were applied to the fully-connected layer. ± 0.98 56.57 ± 0.22 44.87 ± 0.81 28.54 ± 0.30 27.79 ± 0.83 26.35 ± 0.54 28.02 ± 0.88 Dropout 26.38 ± 0.21 56.11 ± 0.83 44.78 ± 0.41 27.66 ± 0.51 28.43 ± 0.88 25.95 ± 0.57 27.69 ± 0.38 BN 31.97 ± 3.10 56.49 ± 0.32 43.75 ± 0.76 28.83 ± 0.47 28.20 ± 0.40 25.50 ± 0.55 28.38 ± 0.86 CR 24.96 ± 0.63 57.40 ± 2.11 45.16 ± 0.94 26.45 ± 0.22 28.65 ± 1.21 26.72 ± 0.61 27.94 ± 0.43 cw-CR 22.99 ± 0.58 53.50 ± 1.05 42.15 ± 0.64 26.40 ± 0.62 28.54 ± 1.01 25.93 ± 0.59 27.77 ± 0.88

Table content: <table><tr><td>Regularizer</td><td>Default</td><td>Data Size</td><td/><td colspan="2">Layer Width</td><td colspan="2">Optimizer</td></tr><tr><td/><td/><td>1k</td><td>5k</td><td>32</td><td>512</td><td>Momentum</td><td>RMSProp</td></tr><tr><td>Baseline</td><td colspan="7">26.64 ± 0.16 56.07 ± 0.36 43.95 ± 0.43 28.54 ± 0.63 28.52 ± 1.06 25.78 ± 0.37 28.52 ± 1.21</td></tr><tr><td>L1W</td><td colspan="7">26.46 ± 0.39 56.64 ± 0.91 44.32 ± 0.66 28.65 ± 1.14 27.96 ± 0.72 25.73 ± 0.40 28.30 ± 0.99</td></tr><tr><td colspan="8">L2W 25.71 VR 21.44 ± 0.88 53.90 ± 0.97 42.33 ± 0.57 24.96 ± 0.26 24.96 ± 0.26 24.96 ± 0.26 26.61 ± 0.47 25.01 ± 0.41 26.06 ± 0.72</td></tr><tr><td>cw-VR</td><td colspan="7">21.58 ± 0.21 51.93 ± 1.09 51.93 ± 1.09 51.93 ± 1.09 43.00 ± 0.95 25.81 ± 0.64 26.46 ± 0.25 26.46 ± 0.25 26.46 ± 0.25 24.42 ± 0.31 26.19 ± 1.35</td></tr><tr><td>L1R</td><td colspan="7">20.63 ± 0.50 20.63 ± 0.50 20.63 ± 0.50 52.39 ± 0.99 40.92 ± 0.33 40.92 ± 0.33 40.92 ± 0.33 25.49 ± 0.61 27.81 ± 0.43 25.13 ± 0.52 26.49 ± 0.96</td></tr><tr><td>RR</td><td colspan="7">26.46 ± 0.25 57.09 ± 1.08 44.35 ± 1.09 26.58 ± 0.66 26.87 ± 0.58 23.92 ± 0.37 23.92 ± 0.37 23.92 ± 0.37 25.80 ± 0.85 25.80 ± 0.85 25.80 ± 0.85</td></tr><tr><td>cw-RR</td><td colspan="7">26.29 ± 0.41 57.55 ± 0.46 44.71 ± 1.59 26.62 ± 0.77 27.12 ± 0.46 24.34 ± 0.27 26.10 ± 0.59</td></tr><tr><td>Best improvement</td><td>6.01</td><td>4.14</td><td>3.03</td><td>3.58</td><td>2.06</td><td>1.86</td><td>2.72</td></tr></table>

Penalty loss functions of representation regularizers.

Table content: <table><tr><td>Penalty loss function</td><td>Description on regularization term</td></tr></table>

Statistical characteristics of learned representations. ReLU activation function and Adam optimizer were used, and a learning rate was set to 0.0001. Validation performance was evaluated with different loss weights {0.001, 0.01, 0.1, 1, 10, 100}

Table content: <table><tr><td colspan="8">Regularizer Error ACTIVATION AMPLITUDE COVARIANCE CORRELATION SPARSITY DEAD UNIT RANK</td></tr><tr><td>Baseline</td><td>2.85</td><td>4.93</td><td>2.08</td><td>0.27</td><td>0.34</td><td>0.13</td><td>2.41</td></tr><tr><td>L1W</td><td>2.85</td><td>4.53</td><td>1.95</td><td>0.28</td><td>0.29</td><td>0.01</td><td>2.32</td></tr><tr><td>L2W</td><td>3.02</td><td>4.76</td><td>2.23</td><td>0.29</td><td>0.34</td><td>0.09</td><td>2.26</td></tr><tr><td>Dropout</td><td>2.70</td><td>2.72</td><td>0.87</td><td>0.42</td><td>0.58</td><td>0.06</td><td>2.75</td></tr><tr><td>BN</td><td>2.81</td><td>1.35</td><td>0.24</td><td>0.28</td><td>0.52</td><td>0.00</td><td>5.14</td></tr><tr><td>CR</td><td>2.50</td><td>0.50</td><td>0.01</td><td>0.19</td><td>0.40</td><td>0.03</td><td>7.12</td></tr><tr><td>cw-CR</td><td>2.49</td><td>0.63</td><td>0.02</td><td>0.31</td><td>0.51</td><td>0.07</td><td>3.60</td></tr><tr><td>VR</td><td>2.65</td><td>1.35</td><td>0.15</td><td>0.26</td><td>0.40</td><td>0.08</td><td>3.92</td></tr><tr><td>cw-VR</td><td>2.42</td><td>0.63</td><td>0.02</td><td>0.36</td><td>0.53</td><td>0.06</td><td>3.90</td></tr><tr><td>L1R</td><td>2.35</td><td>1.29</td><td>0.03</td><td>0.40</td><td>0.97</td><td>0.39</td><td>5.94</td></tr><tr><td>RR</td><td>2.81</td><td>7.23</td><td>226.2</td><td>0.90</td><td>0.43</td><td>0.18</td><td>1.00</td></tr><tr><td>cw-RR</td><td>2.57</td><td>10.31</td><td>96.3</td><td>0.91</td><td>0.31</td><td>0.22</td><td>1.00</td></tr><tr><td colspan="3">D EXPERIMENT DETAILS</td><td/><td/><td/><td/><td/></tr><tr><td colspan="3">D.1 DEFAULT SETTINGS</td><td/><td/><td/><td/><td/></tr><tr><td>By default,</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

Condition experiment results for MNIST MLP model. The best performing regularizer in each condition (each column) is shown in bold, and other regularizers whose performance range overlaps with the best one are highlighted in green. For the default condition, the standard values of data size=50k and layer width=100 were used and Adam optimizer was applied. For other columns, all the conditions were the same as the default except for the condition indicated on the top part of the columns. Regularizers were applied to the fifth hidden layer.

Table content: <table><tr><td>Regularizer</td><td>Default</td><td>Data Size</td><td>Layer Width</td></tr></table>

Peer review (iclr2019):
 This paper studies the characteristics of representations and their roles in neural network expressiveness. The results are overall not very impressive. 1. There are many characteristics of representations such as scaling, permutation, covariance, correlation, sparsity, dead units, rank. The papers discusses some (not surprising) theoretical properties relating to scaling, permutation, covariance, correlation, while making less efforts on the more interesting characteristics sparsity, dead units, rank, mutual information. Only some heuristic results are obtained for them without rigorous theory. It would be better if these heuristic arguments can be formed as theorems as well. 2. Probably the most interesting experimental finding of this paper is that the mutual information between z and output is constant, while the one between z and input strongly depends on the regularizers. That is, the dependence between z and y does not vary with regularizers but the one between z and x does. Is this a coincidence or a general phenomenon? Is there a theoretical explanation? 3.