Code change:
If a norm is zero, its gradient returns nan: Obviously just happening because the gradient divides by the norm, but the (sub)gradient here should probably be zero, or at least not nan, since that will propagate to make all updates nan. Probably low priority, as it's not going to be an issue in 99% of cases, but we're doing a few things with (exact) line searches where this caused a nan to appear, breaking everything downstream.
I'm encountering exactly the same issue! Spent hours on debugging, just to find PyTorch has a bug on such basic thing.
+1 just found this bug too
+1 for this bug. Temporarily changing my code to something like the following for the sake of debugging. x = Variable((1), requires_grad=True) y = x + 1e-16 y.norm().backward() print x.grad
The thing is that in the 2 norm, there is a square root, which has a gradient of at 0. The gradient gives you because you then multiply 0 and an infinity during the backward pass.
For a scalar, norm 2 is basically abs. But x.abs().backward() gives you 0 gradient. In this sense, it's not coherent.
I found this error, too
Alban fixed this behavior in
Hi, the norm function can give use the 0 gradients now. However, the following code still has the nan gradient problem
Ho The square root has no gradient at 0. This is expected behavior.
Hi, but the sub-gradient of the square root should be zero? Also, y = ( x * x ) should equal to x.norm(), why they have different gradient ( 0 and nan )?
I think was right. The left-side derivative of sqrt(x) at x=0 is undefined, so it doesn't even have a subgradient at x=0.
square root has no subgradient at 0. You could define a gradient by continuity but then it would be ... Given that pytorch is using autograd, and (equivalent to your ) are completely different: The first one is a single function that is convex and defined on R, it has a subgradient of 0 at 0. The second one is composed of two function, the first function is the square function which is differentiable and outputs values in . The second function is the square root that is not convex and even though it is defined on , it is only differentiable on and it's gradient in 0 in undefined. Given that, even though and will return the same value, their gradients may differ at points where it is not differentiable, this is because automatic differentiation looks at each step of the computation one by one and even though in some cases a subgradient exist (because we look at multiple operations as a single function), it is not always the case and the gradient remains undefined.
I think math is math. Any root's gradient at zero is either inf or undefined. This issue shall be handled by the users' themselves by adding a small value(as did), but an error(warning) message may be helpful since it is pretty hard to debug. Say: Infinite/Undefined gradient is detected at XfunctionX at line Y. Exit.
Agree, norm is not differentiable at 0 , the bandaid that Alban put there in is wrong (even in the limit sense the gradient at 0 should be 1 not 0), but it should not have been there at all. Norm is norm, if someone want to add epsilons to their norms (like batchrnorm, e.g.) they are welcome to do so in the user code. What would numpy do?
I've also run into a number of problems related to the change introduced in . Is there a reason the subgradient is set to 0, rather than the 1? (The limit as norm-0?) As a minimal example: Produces
Well any value between [-1, 1] is a valid subgradient for the 2-norm. More genereally, any vector in the 1 ball for the dual norm is a valid subgradient. This means that 0 is always going to be a subgradient, while 1 will not be for all p. Anyway, the theory says that any of them could be taken and subgradient descent will work. I'm sure that depending on the application, one will be better than the other. For example, the relu function will also give a 0 subgradient at 0, you could have given 1. The main point here was to remove nans that make your network give nan for everything which is not convenient.

Related discussion:
text: ctx.keepdim = False if keepdim is None else keepdim if dim is None: <del> ctx.norm = input.norm(p) ctx.save_for_backward(input) return input.new((ctx.norm,)) </del> <ins> norm = input.norm(p) output = input.new((norm,)) </ins> else: if keepdim is not None: output = input.norm(p, dim, keepdim=keepdim) else: output = input.norm(p, dim) <del> ctx.save_for_backward(input, output) return output </del> <ins> ctx.save_for_backward(input, output) return output </ins> @staticmethod def backward(ctx, grad_output): <del> if ctx.dim is None: input, = ctx.saved_variables if ctx.p == 2: scale_v = (grad_output / ctx.norm).expand_as(input) return input.mul(scale_v), None, None, None else: pow = input.abs().pow(ctx.p - 2) scale_v = (grad_output / ctx.norm ** (ctx.p - 1)).expand_as(input) return input.mul(pow).mul(scale_v), None, None, None </del> <ins> input, output = ctx.saved_variables if ctx.dim is not None and ctx.keepdim is False and input.dim() != 1: grad_output = grad_output.unsqueeze(ctx.dim) output = output.unsqueeze(ctx.dim) if ctx.p == 2: grad_input = input.mul(grad_output).div(output) </ins> else: <del> input, output = ctx.saved_variables </del> <ins> input_pow = input.abs().pow(ctx.p - 2) output_pow = output.pow(ctx.p - 1) grad_input = input.mul(input_pow).mul(grad_output).div(output_pow) </ins> <del> if ctx.keepdim is False and input.dim() != 1: grad_output = grad_output.unsqueeze(ctx.dim) output = output.unsqueeze(ctx.dim) </del> <ins> # Special case at 0 where we return a subgradient containing 0 grad_input.masked_fill_(output == 0, 0) </ins> <del> big_grad_output = grad_output.expand_as(input) if ctx.p == 2: big_output = output.expand_as(input) return input.mul(big_grad_output).div(big_output), None, None, None else: pow = input.abs().pow(ctx.p - 2) big_output = output.pow(ctx.p - 1).expand_as(input) return input.mul(pow).mul(big_grad_output).div(big_output), None, None, None </del> <ins> return grad_input, None, None, None </ins> # TODO: renorm