TASK DEFINITION: In this task, you are given the abstract of a research paper. Your task is to generate a summary of this abstract. Your summary should not be very short, but it's better if it's not more than 30 words.
PROBLEM: Stochastic gradient descent (SGD) has been the dominant optimization method for training deep neural networks due to its many desirable properties. One of the more remarkable and least understood quality of SGD is that it generalizes relatively well
 on unseen data even when the neural network has millions of parameters. We hypothesize that in certain cases it is desirable to relax its intrinsic generalization properties and introduce an extension of SGD called deep gradient boosting (DGB). The key idea of DGB is that back-propagated gradients inferred using the chain rule can be viewed as pseudo-residual targets of a gradient boosting problem. Thus at each layer of a neural network the weight update is calculated by solving the corresponding boosting problem using a linear base learner. The resulting weight update formula can also be viewed as a normalization procedure of the data that arrives at each layer during the forward pass. When implemented as a separate input normalization layer (INN) the new architecture shows improved performance on image recognition tasks when compared to the same architecture without normalization layers. As opposed to batch normalization (BN), INN has no learnable parameters however it matches its performance on CIFAR10 and ImageNet classification tasks.

SOLUTION: What can we learn about training neural networks if we treat each layer as a gradient boosting problem?

PROBLEM: Most existing deep reinforcement learning (DRL) frameworks consider action spaces that are either
 discrete or continuous space. Motivated by the project of design Game AI for King of Glory
 (KOG), one the world’s most popular mobile game, we consider the scenario with the discrete-continuous
 hybrid action space. To directly apply existing DLR frameworks, existing approaches
 either approximate the hybrid space by a discrete set or relaxing it into a continuous set, which is
 usually less efficient and robust. In this paper, we propose a parametrized deep Q-network (P-DQN)
 for the hybrid action space without approximation or relaxation. Our algorithm combines DQN and
 DDPG and can be viewed as an extension of the DQN to hybrid actions. The empirical study on the
 game KOG validates the efficiency and effectiveness of our method.

SOLUTION: A DQN and DDPG hybrid algorithm is proposed to deal with the discrete-continuous hybrid action space.

PROBLEM: Modern deep neural networks (DNNs) require high memory consumption and large computational loads.   In order to deploy DNN algorithms efficiently on edge or mobile devices, a series of DNN compression algorithms have been explored, including the line of works on factorization methods. Factorization methods approximate the weight matrix of a DNN layer with multiplication of two or multiple low-rank matrices. However, it is hard to measure the ranks of DNN layers during the training process. Previous works mainly induce low-rank through implicit approximations or via costly singular value decomposition (SVD) process on every training step. The former approach usually induces a high accuracy loss while the latter prevents DNN factorization from efficiently reaching a high compression rate. In this work, we propose SVD training, which first applies SVD to decompose DNN's layers and then performs training on the full-rank decomposed weights. To improve the training quality and convergence, we add orthogonality regularization to the singular vectors, which ensure the valid form of SVD and avoid gradient vanishing/exploding. Low-rank is encouraged by applying sparsity-inducing regularizers on the singular values of each layer. Singular value pruning is applied at the end to reach a low-rank model. We empirically show that SVD training can significantly reduce the rank of DNN layers and achieve higher reduction on computation load under the same accuracy, comparing to not only previous factorization methods but also state-of-the-art filter pruning methods.

SOLUTION:
Efficiently inducing low-rank deep neural networks via SVD training with sparse singular values and orthogonal singular vectors.