Year: 2018

Abstract:
It has been common to argue or imply that a regularizer can be used to alter a statistical property of a hidden layer’s representation and thus improve generalization or performance of deep networks. For instance, dropout has been known to improve performance by reducing co-adaptation, and representational sparsity has been argued as a good characteristic because many data-generation processes have only a small number of factors that are independent. In this work, we analytically and empirically investigate the popular characteristics of learned representations, including correlation, sparsity, dead unit, rank, and mutual information, and disprove many of the conventional wisdom. We first show that infinitely many Identical Output Networks (IONs) can be constructed for any deep network with a linear layer, where any invertible affine transformation can be applied to alter the layer’s representation characteristics. The existence of ION proves that the correlation characteristics of representation can be either low or high for a well-performing network. Extensions to ReLU layers are provided, too. Then, we consider sparsity, dead unit, and rank to show that only loose relationships exist among the three characteristics. It is shown that a higher sparsity or additional dead units do not imply a better or worse performance when the rank of representation is fixed. We also develop a rank regularizer and show that neither representation sparsity nor lower rank is helpful for improving performance even when the data-generation process has only a small number of independent factors. Mutual information I(zl;x) and I(zl;y) are investigated as well, and we show that regularizers can affect I(zl;x) and thus indirectly influence the performance. Finally, we explain how a rich set of regularizers can be used as a powerful tool for performance tuning.

heading: 1 INTRODUCTION
text: A learned representation can significantly affect the performance of deep networks, and the representation’s distributed and deep natures are the essential elements for the success of deep learning (Bengio et al., 2013). As a consequence, deep networks have a greater expressiveness compared to the other machine learning algorithms (Hinton et al., 1986) or shallow networks (Montufar et al., 2014; Telgarsky, 2015; Eldan & Shamir, 2016; Raghu et al., 2016). Besides the distributed and deep natures that have been intensively studied, a hidden layer’s representation characteristics are considered to be important as well. Nonetheless, a relatively smaller number of studies have been completed on the topic, and the goal of this work is to understand the representation characteristics better. Therefore, the meaning of representation in this work is restricted to the activation vector of a single hidden layer and a unit refers to a neuron of the hidden layer.
A few previous studies considered manipulating statistical characteristics of representations such as reducing covariance among hidden units (Cogswell et al., 2015; Xiong et al., 2016), encouraging representational sparsity (Glorot et al., 2011), or forcing parsimonious representations via clustering (Liao et al., 2016). In some of the similar works, a popular argument has been that the representation regularization reduces the generalization error via altering a representation characteristic. This argument, however, has not been rigorously studied. Another popular argument has been the reduction of effective capacity via regularization. This argument has been recently disproved by Zhang et al. (2016) where they empirically show that explicit regularization methods like L2 weight decay and dropout cannot sufficiently limit the effective capacity of deep networks.
Since a novel information-theoretic analysis method was proposed for deep learning (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017), information theoretic characteristics of representation have become an important research topic. In their work, mutual information I(zl;x) and I(zl;y) are used to address the learning dynamics and generalization of deep learning, where zl is the hidden layer l’s representation, x is the input, and y is the output. It is further discussed that a good representation is the one that contains a minimal amount of information from the input while containing a sufficient amount of information from the output. In Achille & Soatto (2017), the Information Bottleneck Lagrangian (Tishby et al., 2000) is decomposed into the sum of a crossentropy term and a regularization term. The regularization term turns out to be I(zl;x) that needs to be minimized. Some of the recent works will be additionally addressed in Section 5.

heading: 2 REPRESENTATION CHARACTERISTICS
text: In this section, we briefly address the most popular statistical characteristics and information theoretic characteristics of representations. Consider a neural network NA whose architecture A is fixed and the weights for the lth layer are given by {Wl} and {bl} after training. Notation-wise, we simply write NA = (W,b) to define a network and y or NA(x) to refer to its deterministic output for a given input x. The layer index l is omitted when the meaning is obvious. The lth layer’s activation vector for the given input x is noted as zl(x) or simply zl, and the ith element of zl is noted as zl,i. The mean, variance, and standard deviation of zl,i are defined as µzl,i , vzl,i , and σzl,i , respectively. The covariance of zl is defined as Cl. Then, the basic representation characteristics can be summarized as in Table 1. Six of them are statistical characteristics, and the last two are information theoretic characteristics.
Previous studies on statistical characteristics are often based on regularizers. Srivastava et al. (2014) addresses dropout for preventing co-adaptation among hidden units by randomly putting zeros in a layer’s activation vector. Ioffe & Szegedy (2015) explains batch normalization (BN) that reduces internal covariate shift via normalizing activations of each unit to speed up network training. Cogswell et al. (2015) suggest DeCov that utilizes a penalty loss function to reduce activation covariance among hidden units. Choi & Rhee (2018) considers extension to class-wise regularization and provides four representation regularizers. Among them, CR (Covariance Regularizer) is equivalent to DeCov, and we adopt the notation of CR. Glorot et al. (2011) explains L1 representation regularization, called L1R in this work, that applies L1 penalty on activations. These representation regularization methods have distinct effects on representation characteristics, and examples of the learned representations are shown in Figure 1. The representation regularizers are described in Appendix B, and a quantitative analysis of representation characteristics are provided in Appendix C.
Because the true distribution of data is not accessible, the numerical results in the following sections are evaluated using the empirical distribution of the test dataset. Then, the expectations in Table 1 are with respect to the empirical distribution. For instance, Cl is calculated as the covariance matrix of N activation vectors {z1l , ..., zNl } where znl corresponds to the activation vector for the n’th test data sample, xn. Rank can be calculated by examining Cl, but often there are tiny eigenvalues that hinder a proper assessment of the rank. Therefore, we evaluate stable rank instead, and it will be explained further in Section 4. Two information-theoretical characteristics, I(zl;x) and I(zl;y), are estimated using upper and lower bounds explained in Kolchinsky & Tracey (2017); Kolchinsky et al. (2017). Further details are provided in Section 5. ReLU is the only activation function that is considered in this work. When ReLU is used, ACTIVATION AMPLITUDE, COVARIANCE, and CORRELATION are calculated using only the positive activation values such that the numerical evaluations can provide meaningful insights on what is happening to the non-zero representation values.

heading: 3 SCALING, PERMUTATION, COVARIANCE, AND CORRELATION
text: After training is completed for a deep networkNA, the output of the network becomes a deterministic function of the input x. Without an activation function, i.e. a linear layer, zl = WTl zl−1 +bl. When ReLU is applied to layer l, the activation vector becomes zl = ReLU(WTl zl−1 +bl) = max(WTl zl−1 +bl, 0). In this section, we investigate the most flexible affine transformation that can be applied to a layer’s representation zl without influencing the output NA(x) for any arbitrary input vector x. While complicated transformations over multiple layers can be explored, we limit our focus to manipulating only the weights of layer l and layer l + 1 for the analytical tractability. Because scaling and permutation are well known results, covariance and correlation are the main focus of this section.

heading: 3.1 IDENTICAL OUTPUT NETWORK (ION)
text: We first consider a linear layer l. For a linear layer, it turns out that any affine transformation can be applied as long as the transformation does not cause an information loss.
Theorem 1. (ION for a linear layer) For a deep network NA = (W, b) whose layer l is linear, there exists ÑA = (W̃, b̃) that satisfy the following conditions:
∀x, NA(x) = ÑA(x); (1) ∀x, z̃l = Q(zl−m), (2)
where Q is any nonsingular square matrix of a proper size and m is any vector of a proper size.
The first condition says that the two networks generate identical outputs for any input x. The second condition says that zl can be affinely transformed using any nonsingular matrix Q. The proof is straightforward and can be found in Appendix A.
While simple, Theorem 1 has significant implications on the representation characteristics of zl. Let’s inspect covariance and correlation (normalized version of covariance) first. If NA is a network that is globally optimal for a task and has at least one linear layer l, then NA’s covariance Cl can be whitened to have C̃l = I by choosing m as the expected mean and Q as a whitening matrix. The
resulting network ÑA will have zero correlation between any pair of units in layer l, but will be globally optimal, too. In fact, there are infinitely many globally optimal networks with different covariance characteristics, and one can easily construct an ION with an arbitrary covariance matrix C̃l as long as its rank is the same as Cl’s rank. With this result, it becomes unclear why one should pursue a lower correlation when training a deep network. Unless regularization for a low correlation somehow helps optimization to reach a better performing network, there seems to be no reason to pursue low (or high) correlation.
For dead neurons, a similar claim can be made. If globally optimal NA has no dead neurons in layer l and Cl is not full rank, one can make an affine transformation to align the null spaces of Cl to some of the neurons. Then, the resulting network ÑA will be still globally optimal, but with some dead neurons in layer l. For higher layers of classification tasks, typically the rank of Cl is close to the number of classes. For classification tasks with only 2∼10 classes, it is possible to construct an ION that has as many dead neurons as the size of Cl’s null space. This can be done without negatively affecting the performance, and the wisdom of ‘reduce the number of dead neurons’ becomes dubious.
For scaling and permutation, their influences are rather insignificant. As for the scaling that can affect activation amplitude, it often has no effect on the network’s performance. For instance, scaled activation amplitude can affect the probability of classification tasks when softmax is in the last layer, but the class with the highest probability remains the same anyway. When representation regularizers are used, often activation amplitude is squashed to reduce the cost of representation penalty function but the network can still perform well. As reported in Choi & Rhee (2018), such an activation squashing can make covariance much smaller, but the effect is removed when correlation is calculated. As for the permutation, it is considered to be meaningless because the index number itself is not important.
Before discussing further, a similar result is developed for ReLU layers. The resulting Q, however, is much more limited. The proof can be found in Appendix A as well. Theorem 2. (ION for a ReLU layer) For a deep network NA = (W, b) whose activation function of layer l is ReLU, there exists ÑA = (W̃, b̃) that satisfy the following conditions:
∀x, NA(x) = ÑA(x); (3) ∀x, z̃l = Qzl, (4)
where Q is any permuted positive diagonal matrix of a proper size. Furthermore, it can be shown that any Q that satisfy the above two conditions must be a permuted positive diagonal matrix.
Using a permuted positive diagonal matrix Q, covariance can be affected by independently scaling activation amplitudes of layer l’s units. As explained above, such a scaling is canceled out when calculating correlation and therefore a linear transformation cannot affect correlation while keeping the output identical. There are a few possibilities for overcoming the limitations of Theorem 2, and they are discussed in the following subsection.
For rank and mutual information, the invertible affine transformation has no effect. They are discussed in the following sections.

heading: 3.2 POSSIBLE EXTENSIONS OF ION
text: We discuss three possible extensions of ReLU’s ION.
Insertion of a linear layer One way to overcome the limitations of Q is to insert an extra linear layer near the target ReLU layer and to consider its implications. When representation characteristics are analyzed or interpreted, researchers do not care much about the layer’s activation function, regularization, etc. The activation vector’s representation characteristics are the essential components for understanding and assessing the representations. Therefore, one can apply the insights from Theorem 1 when the extra linear layer shows similar statistical properties as the ReLU layer. In Table 2, statistical properties of immediately neighboring ReLU and linear layers are compared. Compared to the representation characteristics of the first ReLU layer, the 5th ReLU layer and the inserted 6th linear layer show very similar characteristics. Then the correlation and dead unit characteristics are
not so important as the consequence of Theorem 1, and the same might be conjectured for the 5th ReLU layer.
Non-affine transformations over multiple layers In the ION derivations, we have considered only an affine transformation applied to the layers l and l+1 only. If we remove the constraints and borrow the results from expressivity of DNN and universal approximation theorem, it might be possible to derive more powerful and general results. In the extreme case, one can divide a deep network NA into two parts: NAlower and NAupper . Then, NAlower(x) = zl and NA(x) = NAupper(NAlower(x)). In theory, there exist ÑAlower and ÑAupper that can result in NA(x) = ÑAupper(ÑAlower(x)) while allowing z̃l to have a completely different characteristics compared to zl. Such ÑAlower and ÑAupper , however, might be infeasibly large or fail to learn in the way we desire. Therefore, it might be more practical to consider a reasonable extension of Theorem 1 and Theorem 2.
Comparable Performance Network (CPN) According to Theorem 2, only permuted positive diagonal matrices can form IONs. If we ignore the result and apply an affine transformation in the same way as in the ION of a linear layer, the resulting network ÑA will not form an ION but instead we might be able to find a Comparable Performance Network (CPN) that achieves a comparable performance while showing different representation characteristics. We tried this brute-force method, and two sample results along with the baseline and a positive diagonal matrix case are shown in Table 3. In the first row where Q is identity, the values are for the original network NA. In the next row, ‘Random positive diagonal’, uniformly random values between 0 and 1 (U(0, 1)) were used as the diagonal values. Note that this choice of Q satisfies Theorem 2, and therefore the error performance remains the same while affecting activation amplitude and covariance only. In the ‘Ones with random positive diagonal’, Q was chosen as a matrix of ones with its diagonals replaced with random values chosen from U(0, 1). We randomly generated 100 of such random matrices and selected the one that resulted in a higher correlation while showing a comparable performance. Despite of the very high correlation of 0.80, the selected network ÑA can perform comparably well. In the last row, we applied a whitening filter where Q and m were calculated while ignoring ReLU. The resulting network does not end up with zero correlation because the whitening is not perfect in the presence of ReLU. But the correlation is considerably reduced to 0.09 from 0.28 while achieving a slightly worse error rate of 5.48%.
To find the examples in Table 3, all we had to do was to construct a meaningful matrix Q or to try 100 random matrices and choose one. The fact that it is almost painless job to find a CPN also implies that the relevant representation characteristics might not be essential for achieving high performance.

heading: 4 SPARSITY, DEAD UNIT, AND RANK
text: Sparsity and dead unit have been considered as important representation characteristics. Rank of Cl, however, has received much less attention so far. In this section, we investigate the three and show that rank might be the most fundamental characteristics.

heading: 4.1 ANALYTICAL RELATIONSHIP
text: In Table 1, sparsity is defined as Ps = Ei,n[1(znl,i)]. This can be interpreted as the probability of znl,i (unit i’s activation for n’th test sample x n) being zero, because 1(znl,i) = 1 when z n l,i = 0 and 1(znl,i) = 0 when z n l,i 6= 0. Similarly, dead unit is defined as Pd = Ei[1(zl,i)] and it can be interpreted as the probability of zl,i (unit i’s activation) being always zero or at least for all M test samples. Because 1(zl,i) ≤ 1(znl,i) for any pair of (i, n), Pd ≤ Ps can be shown by taking expectations on both sides. The rank r in Table 1 is defined as the rank of Cl. For layer l with M units, this means that r out of M linearly independent dimensions are used by the codewords {z1l , ..., zNl } and that the other M − r dimensions form a null space of Cl. When dead units are considered, MPd units need to be constant zero by the definition of the dead unit and it implies that at least MPd dimensions need to be included in the null space. Therefore, MPd ≤M − r. These results can be summarized as below.
Pd ≤ Ps (5) MPd ≤M − r (6)
Between sparsity Ps and rank r, there is no clear relationship. The codeword znl for a test sample xn can be very sparse, and yet the set of codewords {z1l , ..., zNl } collectively might use all of the M dimensions. Conversely, rank r can be very small and yet Ps can also be very small when {znl,1, ..., znl,M} are strongly correlated and the basis vectors are not sparse over the M units. From the viewpoint of signal processing or information theory, sparsity is a property that is related to individual signals or individual codewords while rank is a property that is related to the total number of dimensions used by the set of signals or the entire codebook. Therefore, sparsity is not directly responsible for the efficiency of the signals or codebook while rank is directly responsible for the efficiency. From the viewpoint of deep learning, rank can be associated with the maximum number of latent factors that are independent. As for the dead unit, we know from equation 6 that it is upper bounded as a function of rank. When the bound is met, the value of Pd is merely an artifact of how the representation vectors {zl} are aligned with the eigenvectors of Cl. If each neuron is aligned to an eigenmode of Cl, then Pd = 1− r/M will be achieved. From our experience, however, such a perfect alignment never happens when using the backpropagation based learning process. This has been true even when L1R or other advanced representation regularizers were applied. ION, however, can easily meet the requirement for a linear layer.
Motivated by the above discussion, we have designed a rank regularizer and examined common wisdom that says ‘most of the data generation processes have a small number of independent factors and therefore increasing sparsity of representation can be helpful.’ For instance, see Bengio et al. (2013). We first explain the design of rank regularizer.

heading: 4.2 RANK REGULARIZER
text: In deep learning, a low-rank approximation of convolutional filters (Jaderberg et al., 2014; Lebedev et al., 2014; Tai et al., 2015) and weight matrices (Xue et al., 2013; 2014; Nakkiran et al., 2015; Masana et al., 2017) has been widely used for network compression and fast network training. Some of the works applied a singular value decomposition to weight matrices after network training ends but not to representations. In this work, as L1 representation regularizer was designed to encourage a higher sparsity by adding a penalty loss term ΩL1R =
∑ n ∑ i |znl,i|, Rank Regularizer (RR) is
designed to encourage a lower rank of representations and used during network training. Because the usual definition of rank can be very sensitive to the tiny singular values, we use stable rank of activation matrix Z = [z1l , . . . , z NMB l ]
T as a surrogate. Note that NMB instead of N activation vectors are used for each mini-batch. Stable rank of Z is defined as
ΩRR = ‖Z‖2F ‖Z‖22 =
∑ i s 2 i
maxi s2i , (7)
where‖Z‖F is the Forbenius norm,‖Z‖2 is the spectral norm, and {si} are the singular values of Z. From ∑ i s 2 i
maxi s2i , it can be clearly seen that stable rank is upper bounded by the usual rank that counts
strictly positive singular values. Because the spectral norm is based on a singular value decomposition, calculating stable rank’s derivative for every mini-batch is a computationally heavy operation. To reduce the computational burden, we introduce an approximation using a special case of Holder’s inequality.
ΩRR = ‖Z‖2F ‖Z‖22 = trace(ZT Z) ‖Z‖22 (8)
≥ trace(Z T Z)
‖Z‖1‖Z‖∞ =
∑ i,n(z n i ) 2
(maxi ∑NMB n=1 |zni |)(maxn ∑M i=1 |zni |) (9)
The inequality‖Z‖2 ≤ √ ‖Z‖1‖Z‖∞ was used where‖Z‖1 is the maximum absolute column sum of the matrix Z (sum of all activation values of unit i) and‖Z‖∞ is the maximum absolute row sum of the matrix Z (sum of all activation values of sample n). Then the gradient of approximation ΩRR can be as below.
∂ΩRR ∂zni
' ∂‖Z‖2F ∂zni ‖Z‖1‖Z‖∞ − ‖Z‖2F · ( ∂‖Z‖1 ∂zni ·‖Z‖∞ +‖Z‖1 · ∂‖Z‖∞ ∂zni ) ‖Z‖21‖Z‖ 2 ∞ , (10)
where ∂‖Z‖2F ∂zni = 2zni ,
∂‖Z‖1 ∂zni = 1(i=i∗) · sign(zni ), ∂‖Z‖∞ ∂zni = 1(n=n∗) · sign(zni ),
i∗ = arg max 1≤i≤M NMB∑ n=1 |zni |, and n∗ = arg max 1≤n≤NMB M∑ i=1 |zni |.

heading: 4.3 A CONTROLLED EXPERIMENT ON DATA GENERATION PROCESS
text: We have designed two datasets where the number of independent factors is fully controlled. The first dataset is a synthetic 10-class classification dataset that was created using Python scikit-learn library (Pedregosa et al., 2011). The number of independent Gaussian factors, d, was controlled to be 10, 50, 100, 250, and 500, and the independent factors were mixed using a randomly generated 1000× d rotation matrix. The second dataset is a PCA-controlled MNIST data that was created by including only the top 10, 50, 100, 250, and 500 dimensions of MNIST’s PCA dimensions.
For the two datasets, we have chosen NA to be the same 5-layer MLP as before and repeatedly performed training while applying either L1R or RR with different loss weights. The results are shown in Figure 2 and Figure 3. The sparsity and rank plots show that indeed sparsity is increased and rank is reduced by increasing the loss weight. The accuracy performance, however, does not show any meaningful dependency on d. For instance, even when d = 10 and there were only ten independent factors in the data generation process, strongly applying L1R or RR did not result in improved performance. On the contrary, the accuracy often suffered when loss weight was increased.
According to the discussion in subsection 4.1, it is not surprising that the level of learned representation’s sparsity does not affect the accuracy performance. Perhaps it is more surprising that even the level of learned representation’s rank does not affect the accuracy performance. We move on to the analysis of mutual information for a further discussion on this issue.

heading: 5 MUTUAL INFORMATION
text: So far, we have investigated popular statistical characteristics of representation zl where none of the statistical characteristics showed a strong and clear relationship to deep network’s performance. In this section, we examine two information-theoretical characteristics: I(zl;x) and I(zl; y). In the original and pioneering work of Shwartz-Ziv & Tishby (2017), the two characteristics of zl were used to explain the concept of information bottleneck on deep networks. Basically, the work shows that the task-relevant information should be maximized via I(zl; y) while the task-irrelevant information should be minimized via I(zl;x). A further development was made in Achille & Soatto (2017), where the Information Bottleneck Lagrangian L(p(zl |x)) = H(y | zl) + βI(zl;x) was explained - the first term is the usual cross entropy cost function, the second term is a penalty term on I(zl;x), and β is parameter for controlling a trade-off between sufficiency (the first term) and minimality (the second term). In Achille & Soatto (2018), they develop ‘information dropout’ method that implicitly minimize I(zl;x). In their limited performance experiments, they showed that information dropout can improve MNIST classification performance by about 0.25% for the best case. In another work by Kolchinsky et al. (2017), an upper bound derived using a non-parametric estimator of mutual information and a variational approximation is used to develop a gradient-based optimization method. They showed I(zl;x) and I(zl;y) are indeed reduced by the method, but did not report anything about performance.
In this work, we neither tried the aforementioned techniques nor explicitly implemented an I(zl;x) regularizer. Instead, we simply applied the twelve regularizers (including baseline) and calculated the upper and lower bounds of I(zl;x) and I(zl;y). The bounds can be calculated using the results of Kolchinsky & Tracey (2017) where a pairwise distance function between mixture components is used. They prove that the Chernoff α-divergence and the Kullback-Leibler divergence provide lower and upper bounds when they are chosen as the distance function, respectively. Figure 4 shows the results for the last hidden layer together with the generalization error where the same network and dataset as in Figure 1 were used, and regularizers were also applied to the last hidden layer. One
can observe that all the regularizers end up with almost the same I(zl;y) value. But the bounds of I(zl;x) can be seen to be strongly dependent on which regularizer is used, and the upper and lower bounds show a similar pattern as the generalization error’s pattern. In fact, the correlation between the lower bound and generalization error can be calculated to be 0.84, and the correlation between the upper bound and generalization error can be calculated to be 0.78. Therefore, it can be surmised that the regularizers might be indirectly affecting the performance by influencing I(zl;x).
When a mutual information regularizer is excluded, the rest of the representation regularizers fail to provide general reasoning on why any of the statistical characteristics should be pursued. In fact, one can argue that even a single-neuron in layer l (activation becomes a scalar) can be a sufficient condition for encoding to have a chance to achieve the maximum possible I(zl;y), i.e., lossless in terms of relevant information. Such an encoding on a scalar activation might be very inefficient, and a practical learning method might never reach such an encoding. Nonetheless, there is no reason why such an encoding should be impossible. Obviously, many of the statistical characteristics become meaningless for such a scalar representation, and it is high time to reconsider the so-called conventional wisdom on representation characteristics.

heading: 6 IMPROVING PERFORMANCE WITH REPRESENTATION REGULARIZERS
text: In the previous sections, we have investigated representation characteristics and their relationship to the performance. All the results, except for mutual information that is shown in Figure 4, indicate that there might be no firm ground to believe that zl’s representation characteristics are strongly related to the performance. But there have been numerous reports that performance was improved by utilizing newly designed regularizers. In this section, we investigate if (representation) regularizers can indeed consistently improve the performance for a given task condition. Here, a task condition means a learning task with small data size, a small layer width, a specific dataset, a large number of classes, or a specific optimizer. We perform experiments on MNIST, CIFAR-10, and CIFAR-100 datasets using twelve regularizers, and the representation regularizers are explained in Appendix B. The details of experimental settings and architecture parameters can be found in Appendix D.

heading: 6.1 CONSISTENTLY WELL PERFORMING REGULARIZER
text: We analyze if there is a consistent dependency between a regularizer and its effect on the performance when a particular regularizer is applied to a particular task condition. Our results, as shown in Table 4, Table 8, and Table 9 indicate that there is no consistently well-performing regularizer for a specific task condition. As an example, consider the entire CIFAR-10 dataset results in Table 4. While task conditions change over different columns, the data remains common for all the tasks. If there is a representation characteristic that fits the data-generation process well and one of the regularizers can match the representation well, it might have outperformed across all the columns. In the table, the best performing regularizer for each task (column) is marked in bold, and any other regularizer whose performance overlaps with the best one is highlighted in green. Looking at the bold and green-highlight patterns, one can easily conclude that there is no single regularizer that works well for all the tasks of CIFAR-10. A similar observation can be made for other task conditions. For instance, one can examine the data size of 1k. For the 1k columns of the three tables, there is no single regularizer that always performs distinctively well.
In fact, we have experimented many more settings than what are shown in this paper. The hope was to find a strong match between a task condition and a representation regularizer, but we have failed to find anything that looks consistent. Many of the previous works on regularizers have compared their regularizer with only a small number of known regularizers. When many regularizers are compared over many different tasks as in our work, one can easily conclude that there is no obvious relation to declare where a certain representation characteristic is advantageous.

heading: 6.2 PERFORMANCE IMPROVEMENT USING REGULARIZERS AS A SET
text: Even though no single representation characteristics consistently outperforms, it can be seen that one can improve performance by using the twelve regularizers as a set and by choosing the best performing regularizer for the given task. This is in line with the usual theme of tuning in many areas of deep learning. Looking into Table 4 more carefully, we can see that cw-VR and L1R often had the best performance for CIFAR-10 test cases. In our experiments, we have observed that one of representation regularizers often outperform weight regularizers (L1W, L2W), dropout, and BN. Even though representation regularizers do not seem to have a direct impact on the performance, they might have indirect effects on mutual information as we have seen in Section 5 or on the optimization process. When many representation regularizers are tried as a set, perhaps there is a larger chance of one of such indirect effects improving the performance.

heading: 7 DISCUSSION AND CONCLUSION
text: We have studied the most popular statistical characteristics and information theoretic characteristics of DNN representations. All the statistical characteristics that were studied failed to show any general or causal pattern for improving performance. Some of the conventional wisdom were analytically dismissed. Empirical results consistently showed that none of the studied statistical characteristics is a requirement for achieving good performance. While we could not identify any systematic pattern, the popular regularizers have been frequently observed to provide a healthy performance improvement. To understand this phenomenon, we have tried applying twelve different regularizers over many classification tasks with different task conditions. The results show that still no systematic pattern can be found, but the set of regularizers can be used as a very compelling tool for tuning the performance. In contrast to the statistical characteristics, information theoretic characteristic I(zl;x) showed a strong correlation with the performance of a classification task. Regularizers were able to affect the mutual information, and possibly they ended up affecting the performance as well. In this work, we have directly addressed and dismissed several conventional wisdom. However, perhaps the most important contribution of this work is to provide an early work on developing rigorous and general theories and methodologies that can be used to better understand the learned representations.

heading: A PROOFS
text: Theorem 1 For a deep network NA = (W, b) whose layer l is linear, there exists ÑA = (W̃, b̃) that satisfy the following conditions:
∀x, NA(x) = ÑA(x); ∀x, z̃l = Q(zl−m),
where Q is any nonsingular square matrix of a proper size and m is any vector of a proper size.
Proof. Proof is based on a simple construction. Choose the weights of ÑA as below.
W̃ T
l = QW T l
b̃l = Q(bl−m)
W̃ T
l+1 = W T l+1 Q
−1
b̃l+1 = bl+1 +W T l+1 m
For all the other layers, choose the same as NA’s weights. Then, clearly z̃l−1 = zl−1 and therefore z̃l = W̃ T l zl−1 +b̃l = Q(W T l zl−1 +bl−m) = Q(zl−m). Also, z̃l+1 = W̃ T l+1z̃l + b̃l+1 = WTl+1 Q −1 Q(zl−m) + bl+1 +WTl+1 m = zl+1. Because the activation vector of layer l + 1 is the same for NA and ÑA, the resulting outputs NA(x) and ÑA(x) are exactly the same as well.
Theorem 2 For a deep network NA = (W, b) whose activation function of layer l is ReLU, there exists ÑA = (W̃, b̃) that satisfy the following conditions:
∀x, NA(x) = ÑA(x); ∀x, z̃l = Qzl,
where Q is any permuted positive diagonal matrix of a proper size. Furthermore, it can be shown that any Q that satisfy the above two conditions must be a permuted positive diagonal matrix.
Proof. For simplicity, we denote a+ = ReLU(a) and a− = ReLU(−a). Then a = a+ + a−. We denote hl = WTl zl +bl, which is the representation before applying the activation function, such that zl = h+l . If we choose W̃ T l = QW T l and W̃ T l+1 = W T l+1 Q
−1, our focus is to find an invertible matrix Q that satisfy WTl+1 Q −1(Qhl) + + bl+1 = W T l+1 h + l +bl+1 for all x. This reduces down to finding Q that satisfies (Qhl)+ = Qh+l . We denote i’th row of Q as q T i , and the statement mentioned above can be written as: (qTi hl) + = qTi hl. For q T i that satisfy (q T i hl) ≥ 0, obviously (qTi hl) + = qTi h i l . If we substitute hl with h + l −h − l , then q T i h − l = 0. For q T i that satisfy qTi hl < 0, we can derive the condition q T i h + l = 0.
Now, we will show that Q should be a permuted positive diagonal matrix using the statements proved above. For a permuted positive diagonal matrix, {qTi } are linearly independent and each qTi has only one positive element. Because Q needs to be invertible (otherwise information loss occurs and NA(x) = ÑA(x) cannot be achieved), it is trivial that each qTi is linearly independent. To show each qTi has only one positive element, let’s assume q T i has more than one non-zero elements. If we denote qTik as the q T i ’s k-th element, and hlk as the hl’s k-th element, we can divide the element indexes as follow. A = {k|qTik 6= 0 and hlk > 0} B = {k|qTik 6= 0 and hlk < 0}
Then, we can consider hl such that A 6= ∅ and B 6= ∅. If qTi hl >= 0, then∑ j∈A qTij hlj > ∑ j∈B qTij hlj .
In this case, the right side should be zero because qTi h − l = 0. However, q T ij should be zero to satisfy qTi h − l = 0 because hl can be chosen arbitrary in the range that A 6= ∅ and B 6= ∅ and qTi hl >= 0. This is a contradiction due to the definition of B. We can prove the case of qTi hl < 0 similarly, which shows that each qTi has only one non-zero element. To show the q T i ’s one element is positive, we denotes the qTi ’s only one element as q T ij , in which j is the index of the non-zero element. When qTij < 0, we can consider hl such that hlj < 0. In this case, (q T i hl) > 0, but q T i h − l > 0, which contradicts the condition qTi h − l = 0. If we consider hl such that hlj > 0, (q T i hl) < 0, but qTi h + l > 0, which contradicts the condition q T i h + l > 0. Therefore, q T ij is positive.

heading: B REPRESENTATION REGULARIZERS
text: In this section, we briefly introduce the representation regularizers that are used in our experiments. Based on the notations provided in Section 2, we define class-wise statistics that are calculated using only class k’s samples out of a total of K labels in the mini-batch. Class-wise mean, covariance, and variance are defined as below.
µ (k) zl,i = En∈Sk [znl,i]. (11)
c(k)i,j = En∈Sk [(z n l,i − µ (k) zl,i )(znl,j − µ (k) zl,j )]. (12)
v(k)zl,i = c (k) i,i . (13)
Here, Sk is the set that contains indexes of the samples with class label k. Note that superscripts with and without parenthesis indicate class label and sample index, respectively. Penalty loss functions of the representation regularizers are summarized in Table 5.

heading: C REPRESENTATION CHARACTERISTICS
text: In this section, we investigate the statistical characteristics of the learned representations when different regularizers are applied. We used the same network and dataset as the ones used for
generating Figure 1. All the regularizers were applied only to the fifth layer, and the representation characteristics were calculated using the fifth layer as well. In Table 6, typical evaluation results of statistical characteristics are shown. We can confirm that the statistical characteristics targeted by each representation regularizer are indeed manipulated as expected (Bold). In particular, RR and cw-RR designed in this work to regularize the stable rank work as expected. The two weight regularizers (L1W: L1 Weight Regularizer, L2W: L2 Weight Regularizer) have similar characteristic values as the baseline’s, and this can be taken for granted because the regularizers do not directly regularize representations. A few conventional beliefs mentioned in the paper are quantitatively confirmed as well. A large number of dead units is known to be harmful because they do not contribute toward improving the performance of deep networks. Our result shows even 39% of DEAD UNIT caused by L1R does not hurt the performance, which is in line with our analysis in Section 4. For dropout, COVARIANCE is reduced as in Cogswell et al. (2015), but CORRELATION is actually increased compared to the baseline. In fact, COVARIANCE is reduced simply because ACTIVATION AMPLITUDE is reduced as mentioned in Section 3, and the correlation between two active units is actually made larger by applying dropout. Therefore, it cannot be said that the relationship between a pair of neurons becomes weaker by applying dropout. This is in contrary to the ‘reduction of coadaptation’ idea. Note that we have excluded the inactive neurons for the evaluations. If the inactive ones are included with their zero values, the covariance and correlation values will be different.

heading: D EXPERIMENT DETAILS
text: D.1 DEFAULT SETTINGS
By default, ReLU activation function and Adam optimizer were used, and a learning rate was set to 0.0001. Validation performance was evaluated with different loss weights {0.001, 0.01, 0.1, 1, 10, 100}, and the one with the best validation performance for each regularizer and condition was chosen for testing. We used the validation data for MNIST dataset and the last 10,000 samples of training data for CIFAR-10/100 datasets as validation data. After a loss weight was fixed, the validation data was merged back into the training data in the case of CIFAR-10/100. Five training trials were performed, and the means of the five trials were reported. Note that a mini-batch size was set to 100 for MNIST and CIFAR-10 tasks but to 500 for CIFAR-100 due to the class-wise statistic calculation. In this work, we carried out all the experiments using TensorFlow 1.5.
D.2 ARCHITECTURE PARAMETERS
We used a 6-layer MLP with 100 units per hidden layer for MNIST image classification tasks. We used a CNN with four convolutional layers and one fully-connected layer for CIFAR-10/100 image classification tasks. Architecture details are described in Table 7.
D.3 EXPERIMENTAL CONDITIONS
Default conditions are shown in bold, and the full experimental conditions are listed below.
• Training data size: 1k, 5k, 50k • Layer width: (MNIST) 2, 8, 100 / (CIFAR-10/100): 32, 128, 512
• Optimizer (CIFAR-10): Adam, Momentum (lr=0.01, momentum=0.9), RMSProp (lr=0.0001) • Number of classes (CIFAR-100): 16, 64, 100
D.4 EXPERIMENTAL RESULTS (MNIST, CIFAR-100)

Review summaries:
This paper makes concerted efforts to examine the existing beliefs about the significance of various statistical characteristics of hidden layer activations (or representations) in a DNN. In the past, many works have argued for encouraging the certain statistical behavior of these representations (e.g., sparsity, low correlation etc) in order to have better classification accuracy. However, this paper tries to argue that such efforts are not very useful as these statistical characteristics don't provide any systematic explanation for the performance of DNNs across different settings.