Question: How does the work of Mishkin and Matas (2016) as referenced by the authors support the idea that subnetworks are born of non-deterministic effects caused by weight initialization?
Evidence: The softmax function, a.k.a. softargmax, is a normalization function often used as the last activation function of a neural network
where q i ∈ [0, 1] and i q i = 1. Thus, the normalized output vector can be interpreted as marginal probabilities. The softmax output can be naturally combined with the cross entropy function J = − i p i log q i , where p i is the target probability. The derivative of J with respect to z i takes a simple form of q i − p i. The simple probabilistic interpretation and derivative computation make the combination of softmax normalization and cross entropy loss a pervasive choice for multinomial classification problems. However, potential issues using softmax normalization with the backpropagation (BP) algorithm has not been fully investigated.
Suppose a neural network G can be decomposed into two or more smaller subnetworks
The final activation Z is the superposition of the subnetwork activation before the softmax normalization in the output layer
where f m is the non-linear function representing subnetwork G m . The decomposition is done according to the final activation without considering intermediate hidden layers. The softmax normalization operation has the following properties regarding the relationship between subnetwork activations (see Appendix A).
1. If the subnetwork activations are linear offset versions of each other, such that Y
, the normalization operation is equivalent to applying the softmax function to the scaled principal subnetwork: Q = softmax(SY 0 ), where S = 1 + α 1 + α 2 + • • • . The softmax normalization allows proportional integration of information. A single subnetwork that has very strong activation (higher prediction probabilities) can dominate over other subnetworks with weak activations. If there are no dominant subnetworks, the total number of contributing subnetworks may be large and the whole network tends to be overparameterized.
In short, the softmax function can act as a super combinator for different modes of the neural network, summing and amplifying weak subnetwork activations. This could partially explain why deep neural networks are so expressive that they are suitable for diverse types of problems. However, when there are redundant subnetworks that produce linearly correlated activations, the softmax normalization function make them indistinguishable from each other. The linearly correlated subnetworks potentially lead to overfitting and overparameterization. We have the following hypothesis regarding the effects of such redundant subnetworks: Hypothesis 1: For deep neural networks, the existence of redundant subnetworks combining with softmax normalization can lead to overfitting and overparameterization when training with the backpropagation algorithm.
The derivative of the cross entropy loss is linear with regard to the softmax output Q and target P , and softmax normalization makes it impossible to differentiate between the effects of different subnetworks, the BP algorithm thus will fine-tune all the parameters without penalizing any individual subnetwork. Therefore, the initialization of weights may create redundant subnetworks that have non-deterministic effects on the training process. For example, demonstrated that initialization of weights can affect test accuracy. Such behaviors and the existence of redundant subnetworks will be validated from empirical results in Section 3.
The derivative of the cross entropy loss is linear with regard to the softmax output Q and target P , and softmax normalization makes it impossible to differentiate between the effects of different subnetworks, the BP algorithm thus will fine-tune all the parameters without penalizing any individual subnetwork. Therefore, the initialization of weights may create redundant subnetworks that have non-deterministic effects on the training process. For example, Mishkin & Matas (2016) demonstrated that initialization of weights can affect test accuracy. Such behaviors and the existence of redundant subnetworks will be validated from empirical results in Section 3.
Answer: The work of Mishkin and Matas (2016) is cited for the non-deterministic effects of weight initialization, which could be explained using the subnetwork hypothesis. Their work does not directly support the idea of subnetwork. We have revised the corresponding text in the paper.