In this task, you are given the abstract of a research paper. Your task is to generate a summary of this abstract. Your summary should not be very short, but it's better if it's not more than 30 words.

Input: Consider Input: Understanding the groundbreaking performance of Deep Neural Networks is one
 of the greatest challenges to the scientific community today. In this work, we
 introduce an information theoretic viewpoint on the behavior of deep networks
 optimization processes and their generalization abilities. By studying the Information
 Plane, the plane of the mutual information between the input variable and
 the desired label, for each hidden layer. Specifically, we show that the training of
 the network is characterized by a rapid increase in the mutual information (MI)
 between the layers and the target label, followed by a longer decrease in the MI
 between the layers and the input variable. Further, we explicitly show that these
 two fundamental information-theoretic quantities correspond to the generalization
 error of the network, as a result of introducing a new generalization bound that is
 exponential in the representation compression. The analysis focuses on typical
 patterns of large-scale problems. For this purpose, we introduce a novel analytic
 bound on the mutual information between consecutive layers in the network.
 An important consequence of our analysis is a super-linear boost in training time
 with the number of non-degenerate hidden layers, demonstrating the computational
 benefit of the hidden layers.

Output: Introduce an information theoretic viewpoint on the behavior of deep networks optimization processes and their generalization abilities


Input: Consider Input: Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers often with little or no drop in classification accuracy. However most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain classification accuracy. In this paper we propose a cheap pruning algorithm based on difference of convex (DC) optimisation. We also provide theoretical analysis for the growth in the Generalisation Error (GE) of the new pruned network. Our method can be used with any convex regulariser and allows for a controlled degradation in classification accuracy while being orders of magnitude faster than competing approaches. Experiments on common feedforward neural networks show that for sparsity levels above 90% our method achieves 10% higher classification accuracy compared to Hard Thresholding.

Output: A fast pruning algorithm for fully connected DNN layers with theoretical analysis of degradation in Generalisation Error.


Input: Consider Input: Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a ``forgetting event'' to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. Across several benchmark data sets, we find that: (i) certain examples are forgotten with high frequency, and some not at all; (ii) a data set's (un)forgettable examples generalize across neural architectures; and (iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.


Output: We show that catastrophic forgetting occurs within what is considered to be a single task and find that examples that are not prone to forgetting can be removed from the training set without loss of generalization.