AdamW in HuggingFace is different from AdamW in Pytorch
Author: songsuoyuan
Pull request: no
Labels: name: wontfix

# ❓ Question

I just noticed that the implementation of AdamW in HuggingFace is different from PyTorch. The previous AdamW first updates the gradient then apply the weight decay. However, in the paper (Decoupled Weight Decay Regularization, link: https://arxiv.org/abs/1711.05101) and the implementation of Pytorch, the AdamW first apply the weight decay then update the gradient. 

I was wondering if the two approaches are the same. Thanks! (In my opinion, they are not the same procedure.)

HuggingFace:
```python
for group in self.param_groups:
    for p in group["params"]:
        ...
        # Decay the first and second moment running average coefficient
        # In-place operations to update the averages at the same time
        exp_avg.mul_(beta1).add_(1.0 - beta1, grad)
        exp_avg_sq.mul_(beta2).addcmul_(1.0 - beta2, grad, grad)
        denom = exp_avg_sq.sqrt().add_(group["eps"])

        step_size = group["lr"]
        if group["correct_bias"]:  # No bias correction for Bert
            bias_correction1 = 1.0 - beta1 ** state["step"]
            bias_correction2 = 1.0 - beta2 ** state["step"]
            step_size = step_size * math.sqrt(bias_correction2) / bias_correction1

        p.data.addcdiv_(-step_size, exp_avg, denom)

        # Just adding the square of the weights to the loss function is *not*
        # the correct way of using L2 regularization/weight decay with Adam,
        # since that will interact with the m and v parameters in strange ways.
        #
        # Instead we want to decay the weights in a manner that doesn't interact
        # with the m/v parameters. This is equivalent to adding the square
        # of the weights to the loss with plain (non-momentum) SGD.
        # Add weight decay at the end (fixed version)
        if group["weight_decay"] > 0.0:
            p.data.add_(-group["lr"] * group["weight_decay"], p.data)
```
Pytorch:
```python
for group in self.param_groups:
    for p in group['params']:
        ...
        # Perform stepweight decay
        p.data.mul_(1 - group['lr'] * group['weight_decay'])

        exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
        if amsgrad:
            max_exp_avg_sq = state['max_exp_avg_sq']
        beta1, beta2 = group['betas']

        state['step'] += 1
        bias_correction1 = 1 - beta1 ** state['step']
        bias_correction2 = 1 - beta2 ** state['step']

        # Decay the first and second moment running average coefficient
        exp_avg.mul_(beta1).add_(1 - beta1, grad)
        exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
        if amsgrad:
            # Maintains the maximum of all 2nd moment running avg. till now
            torch.max(max_exp_avg_sq, exp_avg_sq, out=max_exp_avg_sq)
            # Use the max. for normalizing running avg. of gradient
            denom = (max_exp_avg_sq.sqrt() / math.sqrt(bias_correction2)).add_(group['eps'])
        else:
            denom = (exp_avg_sq.sqrt() / math.sqrt(bias_correction2)).add_(group['eps'])

        step_size = group['lr'] / bias_correction1

        p.data.addcdiv_(-step_size, exp_avg, denom)
```

Comments:
@songsuoyuan , if you notice the line:
        p.data.mul_(1 - group['lr'] * group['weight_decay'])
The multiplication factor is (1 - group['lr'] * group['weight_decay']) .
All subsequent first and second order moment calculations are not using p.data anymore.
This means we would get the same result if we had skipped that multiplication and introduce an addition operation at the end if weight decay > 0 with  p.data._mul(-group['lr'] * group['weight_decay']) and this what has been done in the hugginFace implementation as well.
So essentially both are same.

Also in the paper, in-fact the weight decay term is introduced at end ( line-12 :Algorithm-2).  Decay term in line-6 corresponds to L2 regularization which is not used here.
Therefore it looks to me both the implementation are the same and reflect what {ilya,fh}@ proposed in the paper.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I find this question too. Two codes are obviously different. 
Because the `p.data` in huggingface has changed through `p.data.addcdiv_(-step_size, exp_avg, denom)`
But I can't understand why.

*bump*

Update: they are indeed the same. PyTorch's implementation is just too confusing to understand.

They are not equivalent. This should be reported as a bug, but I see the huggingface AdamW has been deprecated.

bump again.
I see old code from researcher on github use AdamW with huggingface scheduler

```
from pytorch_transformers import AdamW, WarmupLinearSchedule
```

Should I replace AdamW of huggingface to AdamW of pytorch ?

```
from torch.optim import AdamW
from pytorch_transformers import WarmupLinearSchedule
```

Any advise ?