instruction:
In this task, you are given the abstract of a research paper. Your task is to generate a summary of this abstract. Your summary should not be very short, but it's better if it's not more than 30 words.
question:
We conduct a mathematical analysis on the Batch normalization (BN) effect on gradient backpropagation in residual network training in this work, which is believed to play a critical role in addressing the gradient vanishing/explosion problem. Specifically, by analyzing the mean and variance behavior of the input and the gradient in the forward and backward passes through the BN and residual branches, respectively, we show that they work together to confine the gradient variance to a certain range across residual blocks in backpropagation. As a result, the gradient vanishing/explosion problem is avoided. Furthermore, we use the same analysis to discuss the tradeoff between depth and width of a residual network and demonstrate that shallower yet wider resnets have stronger learning performance than deeper yet thinner resnets.
answer:
Batch normalisation maintains gradient variance throughout training, thus stabilizing optimization.


question:
Kronecker-factor Approximate Curvature (Martens & Grosse, 2015) (K-FAC) is a 2nd-order optimization method which has been shown to give state-of-the-art performance on large-scale neural network optimization tasks (Ba et al., 2017).   It is based on an approximation to the Fisher information matrix (FIM) that makes assumptions about the particular structure of the network and the way it is parameterized. The original K-FAC method was applicable only to fully-connected networks, although it has been recently extended by Grosse & Martens (2016) to handle convolutional networks as well. In this work we extend the method to handle RNNs by introducing a novel approximation to the FIM for RNNs. This approximation works by modelling the covariance structure between the gradient contributions at different time-steps using a chain-structured linear Gaussian graphical model, summing the various cross-covariances, and computing the inverse in closed form. We demonstrate in experiments that our method significantly outperforms general purpose state-of-the-art optimizers like SGD with momentum and Adam on several challenging RNN training tasks.
answer:
We extend the K-FAC method to RNNs by developing a new family of Fisher approximations.


question:
Detecting communities or the modular structure of real-life networks (e.g. a social
 network or a product purchase network) is an important task because the way a
 network functions is often determined by its communities.
 The traditional approaches to community detection involve modularity-based approaches,
 which generally speaking, construct partitions based on heuristics that
 seek to maximize the ratio of the edges within the partitions to those between
 them. Node embedding approaches, which represent each node in a graph as a
 real-valued vector, transform the problem of community detection in a graph to
 that of clustering a set of vectors. Existing node embedding approaches are primarily
 based on first initiating uniform random walks from each node to construct
 a context of a node and then seeks to make the vector representation of
 the node close to its context. However, standard node embedding approaches do
 not directly take into account the community structure of a network while constructing
 the context around each node. To alleviate this, we explore two different
 threads of work. First, we investigate the use of biased random walks (specifically,
 maximum entropy based walks) to obtain more centrality preserving embedding
 of nodes, which we hypothesize may lead to more effective clusters in the embedded
 space. Second, we propose a community structure aware node embedding
 approach where we incorporate modularity-based partitioning heuristics into
 the objective function of node embedding. We demonstrate that our proposed approach
 for community detection outperforms a number of modularity-based baselines
 as well as K-means on a standard node-embedded vector space (specifically,
 node2vec) on a wide range of real-life networks of different sizes and densities.
answer:

A community preserving node embedding algorithm that results in more effective detection of communities with a clustering on the embedded space