Proper way to do gradient clipping?

kim.seonghyeon: Is there a proper way to do gradient clipping, for example, with Adam? It seems like that the value of Variable.data.grad should be manipulated (clipped) before calling optimizer.step() method. I think the value of Variable.data.grad can be modified in-place to do gradient clipping. Is it safe to do? Also, Is there a reason that Autograd RNN cells have separated biases for input-to-hidden and hidden-to-hidden? I think this is redundant and has a some overhead.

apaszke: The reason for that is that it has a nice user facing API where you have both weight tensors exposed. Also, it opens up a possibility of doing batched matrix multiply on the inputs for all steps, and then only applying the hidden-to-hidden weights (it’s not yet added there). If you measure the overhead and prove us that it can be implemented in a clean and fast way, we’ll happily accept a PR or change it.

kim.seonghyeon: I have tested nn.LSTM against simple LSTM implementation and found almost no difference in the performance. Maybe I overestimated the overhead of the additional addition with simple guess. Thank you!

apaszke: If you’re running on GPU you’ll also likely see great speedups from using cuDNN LSTM implementation.

kim.seonghyeon: 

dmadeka1: In the sense, does optimizer itself call backward? In which case, the code below should pass optimizer to the clip function right? <SCODE>        optimizer.zero_grad()
        output, hidden = model(data, hidden)
        loss = criterion(output.view(-1, ntokens), targets)
        loss.backward()
        clipped_lr = lr * clip_gradient(model, clip)
        for p in model.parameters():
            p.data.add_(-clipped_lr, p.grad.data)
            
        optimizer.step()
<ECODE>

apaszke: 

mcleonard: Maybe I’m doing something wrong here, but using gradient clipping like <SCODE>nn.utils.clip_grad_norm(model.parameters(), clip)
for p in model.parameters():
    p.data.add_(-lr, p.grad.data)
<ECODE> <SCODE>Epoch: 1/10... Step: 10... Loss: 4.4288
Epoch: 1/10... Step: 20... Loss: 4.4274
Epoch: 1/10... Step: 30... Loss: 4.4259
Epoch: 1/10... Step: 40... Loss: 4.4250
Epoch: 1/10... Step: 50... Loss: 4.4237
Epoch: 1/10... Step: 60... Loss: 4.4223
Epoch: 1/10... Step: 70... Loss: 4.4209
Epoch: 1/10... Step: 80... Loss: 4.4193
Epoch: 1/10... Step: 90... Loss: 4.4188
Epoch: 1/10... Step: 100... Loss: 4.4174
<ECODE> And without gradient clipping, everything else equal: <SCODE>Epoch: 1/10... Step: 10... Loss: 3.2837
Epoch: 1/10... Step: 20... Loss: 3.1901
Epoch: 1/10... Step: 30... Loss: 3.1512
Epoch: 1/10... Step: 40... Loss: 3.1296
Epoch: 1/10... Step: 50... Loss: 3.1170
Epoch: 1/10... Step: 60... Loss: 3.0758
Epoch: 1/10... Step: 70... Loss: 2.9787
Epoch: 1/10... Step: 80... Loss: 2.9104
Epoch: 1/10... Step: 90... Loss: 2.8271
Epoch: 1/10... Step: 100... Loss: 2.6813
<ECODE> There is probably something I don’t understand, but I’m just switching out those two bits of code.

apaszke: Maybe you’re clipping them to very small values. It’s a possible effect

WendyShang: The one comes with nn.util clips in proportional to the magnitude of the gradients. Thus you’d like to make sure it is not too small for your particular model as Adam said (I think :p). The old-fashioned way of clipping/clampping is <SCODE>def gradClamp(parameters, clip=5):
    for p in parameters:
        p.grad.data.clamp_(max=clip)<ECODE>

Brando_Miranda: for people trying to just get an answer quickly: <SCODE>torch.nn.utils.clip_grad_norm(mdl_sgd.parameters(),clip)
<ECODE> or with in-place clamp: <SCODE>W.grad.data.clamp_(-clip,clip)
<ECODE> also similar Q:

ntubertchen: I thought nn.utils.clip_grad_norm(model.parameters(), clip) is supposed to finish the job. Can someone give a more explicit explain? Is it because after I use gradient clipping, I may not use adam optimizer?

Neta_Zmora: Regarding the code you ask about: <SCODE>for p in model.parameters():
    p.data.add_(-lr, p.grad.data)
<ECODE> <SCODE>p.data = p.data + (-lr * p.grad.data)
<ECODE> In other words, this performs a similar function as optimizer.step(), using the gradients to updates the model parameters, but without the extra sophistication of a torch.optim.Optimizer.  If you use the above code, then you should not use an optimizer (and vice-versa).

tom: Best regards Thomas

hughperkins: 

Jared_77: Thanks. Where does this go in relation to forward and backward propagation?

jukiewiczm: 

jukiewiczm: 

aaronlelevier: