Issue 1:
🐛 Bug

After register a backward hook myhook(module, grad_input, grad_output) of torch.nn.Linear, and call myhook, it seems that the grad_input has an unexpected order/existence of gradient tensors of (grad_of_linear_output, grad_of_linear_input, grad_of_linear_weight), which are respectively gradient tensors wrt outputs of Linear, inputs of Linear, and weights of Linear. This is unexpected and biases' gradient is missing. It is also inconsistent with Conv2d, BatchNorm2d and ReLU, whose grad_input return something like (grad_of_input, grad_of_parameter1, grad_of_parameter2).

To Reproduce

Steps to reproduce the behavior:

Code fragments

# define a hook
def myhook(m, grad_input, grad_output, name=None):
print(name, '======================')
print('--')
for grad in grad_input:
if grad is not None:
print(grad.size())
else:
print('None')
print('--')
for grad in grad_output:
if grad is not None:
print(grad.size())
else:
print('None')

# hook up
for idx, m in enumerate(model.named_modules()):
if isinstance(m[1], torch.nn.Linear):
logger.info('\t{} registering backward hook...'.format(m[0]))
h = m[1].register_backward_hook(hook=partial(myhook, name=m[0]))

# build a model with
linear = nn.Linear(64, 10)

# run with batch size 128 and get
linear ======================
--
(128, 10)
(128, 64)
(64, 10)
--
(128, 10)

Expected behavior

Based on the doc and consistency with Conv2d, the shapes should be

linear ======================
--
(128, 64)
(64, 10)
(10,)
--
(128, 10)

The shapes are normal for Conv2d, BatchNorm2d and ReLU

layer1.0.relu ======================
--
(128, 16, 32, 32)
--
(128, 16, 32, 32)
layer1.0.bn1 ======================
--
(128, 16, 32, 32)
(16,)
(16,)
--
(128, 16, 32, 32)
layer2.0.conv1 ======================
--
(128, 16, 32, 32)
(32, 16, 3, 3)
None # no biases in the tested conv
--
(128, 32, 16, 16) # stride is 2

For consistency, we should correct this or highlight it in the doc if compatibility is a concern.

Environment

Collecting environment information...
PyTorch version: 0.4.1.post2
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: Could not collect

Python version: 2.7
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration:
GPU 0: GeForce GTX 1080
GPU 1: GeForce GTX 1080
GPU 2: GeForce GTX 1080
GPU 3: GeForce GTX 1080

Nvidia driver version: 396.37
cuDNN version: Probably one of the following:
/usr/local/cuda-9.2/lib64/libcudnn.so
/usr/local/cuda-9.2/lib64/libcudnn.so.7
/usr/local/cuda-9.2/lib64/libcudnn.so.7.1.1
/usr/local/cuda-9.2/lib64/libcudnn.so.7.2.1
/usr/local/cuda-9.2/lib64/libcudnn_static.a

Versions of relevant libraries:
[pip] Could not collect
[conda] blas 1.0 mkl
[conda] mkl 2018.0.2 1
[conda] mkl-service 1.1.2 py27hb2d42c5_4
[conda] mkl_fft 1.0.1 py27h3010b51_0
[conda] mkl_random 1.0.1 py27h629b387_0
[conda] pytorch 0.4.1 py27__9.0.176_7.1.2_2 pytorch
[conda] torchvision 0.2.1 py27_1 pytorch

Issue 2:
Maxim Naumov @mnaumovfb sends in the following feedback:

I thought I would share some feedback from my recent work with register_backward_hook function. I will describe some some inconsistencies in it that may be worth addressing in the future.

Background

PyTorch supports different hook functions, including register_hook, register_forward_hook and register_backward_hook. The former is applied to a tensor variable, while the latter two are applied to a layer module. I will discuss the latest form here. It has the following signature https://pytorch.org/docs/stable/nn.html func(layer, grad_input, grad_output)

Details

It is extremely important to understand the meaning of gradients to register_backwards_hook. Let a layer be defined as

output z (grad_output)
____|____
|__layer__|
|
input x (grad_input)

with overall loss E, error gradient dE/dz^(k) and weight gradient dE/dw^(k).

First, let us assume the simplest case: a layer with no bias. Then,

grad_output= [dE/dz^(k)]
grad_input = [dE/dz^(k-1), dE/dw^(k)]

Inconsistencies

It seems that there are some inconsistencies in how gradients for different layers are handled by this function.

Shape

in convolution layers the weight gradient has the same shape as the weights

in fully connected layers the weight gradient is transpose of the weights

Bias

in convolution layers bias gradient are appended: grad_input = [dE/dz^(k-1), dE/dw^(k), dE/db^(k)]

in fully connected layers bias gradient are prepended: grad_input = [dE/db^(k), dE/dz^(k-1), dE/dw^(k)]

Batch size > 1

in convolution layers bias gradient corresponds to the gradient over the entire batch: grad_input = [dE/dz^(k-1), dE/dw^(k), dE/db^(k)]

in fully connected layers bias gradient corresponds to the gradient per data point j=1,...,r in the batch (therefore it needs to be added to get the gradient over the entire batch): grad_input = [[dE/db^(k,1),...,dE/db^(k,r)], dE/dz^(k-1), dE/dw^(k)]

These discrepancies can make handling of different layers, bias and batch sizes quite cumbersome in the code. It would help if they were done more consistently in the future.