Regression bug in `torch.nn.ReLU6` and `torch.nn.Hardtanh` that `inplace=True` doesn't work in PyTorch 1.10.0~1.13.1
Labels: high priority, module: nn, module: memory usage, triaged, actionable

### 🐛 Describe the bug


From version 1.10.0 to version 1.13.1 of PyTorch, the `inplace` argument does not seem to work well on `ReLU6` and `Hardtanh`.
The change of `inplace=True` and `inplace=false` will not affect the memory usage of the model.
However, in versions 1.8 and 1.9, setting `inplace` to False can effectively reduce the GPU usage of the model.

The root causes of these bugs should be related to [#81548](https://github.com/pytorch/pytorch/issues/81548)

### To Reproduce
```python
import time
from torch import nn
import torch
torch.manual_seed(0)

func_cls=nn.Hardtanh
# nn.Hardsigmoid
# nn.Hardtanh
# nn.ReLU6
# nn.Hardswish

# replace `inplace=True` to `inplace=False` and run again to observe reuslts

class EasyModel(nn.Module):
    def __init__(self, num_classes=10):
        super(EasyModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, groups=1, bias=False, dilation=1)


        self.relu1= func_cls(inplace=True)

        self.conv2 = nn.Conv2d(64, 64, kernel_size=3, stride=2, padding=1, groups=1, bias=False, dilation=1)


        self.relu2= func_cls(inplace=True)

        self.deconv1 = nn.ConvTranspose2d(64, 64, kernel_size=3, stride=2, padding=1, groups=1, bias=False, dilation=1)


        self.derelu1= func_cls(inplace=True)
    
        self.conv3 = nn.Conv2d(64, 64, kernel_size=3, stride=2, padding=1, groups=1, bias=False, dilation=1)
        self.relu5 = func_cls(inplace=True)

        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        self.fc = nn.Linear(64, num_classes)
        self.model = nn.Sequential(self.conv1, self.relu1, self.conv2,  self.relu2, self.deconv1, self.derelu1,
                                    self.conv3,  self.relu5, self.avgpool)
    
    def forward(self, x):
        x = self.model(x)
        x = torch.flatten(x, 1)
        x = self.fc(x)

        return x



model = EasyModel()
model.cuda()
x = torch.randn(2,3,128,128).cuda()
y = model(x)
after_forward = torch.cuda.memory_allocated()

# 42659328 for nn.ReLU6\nn.Hardtanh\nn.Hardswish\nn.Hardsigmoid, no matter inplace=True or False

# 40431616 for nn.Hardtanh\nn.ReLU6 in 8,9+True; 21818368 in 8,9+False.
# 40431616 for nn.Hardsigmoid\nn.Hardswish, no matter inplace=True or False in 8-13.1

print(after_forward)
```

### Error Results

We show some erroneous results from the above code:

1. Version 1.13.1, using `ReLU6` and `Hardtanh`, `inplace` is True or False, `torch.cuda.memory_allocated` returns **40562176**.

2. Version 1.8.0/1.9.0, using `ReLU6` and `Hardtanh`, `inplace=True`,  `torch.cuda.memory_allocated` returns **40431616**; and `inplace=False` returns **21818368**.

In addition, we also found that `Hardsigmoid` and `Hardswish` have similar interesting behavior.

3. Version 1.8.0/1.9.0/1.10.0/1.11.0/1.12.0/1.13.0/1.13.1, using `Hardsigmoid`and `Hardswish`, `inplace` is True or False, `torch.cuda.memory_allocated` returns **40431616**.

But we are not yet sure whether this is an expected behavior.
Considering that the implementation of these two functions can also optionally do operation in-place, they may have similar bugs with `ReLU6` and `Hardtanh`.


### Expected behavior

`inplace=True` should provide a solution to save memory overhead (selectively doing operations in-place, like what `ReLU6` and `Hardtanh` do in PyTorch 1.8.0), but the existing performance does not seem to be able to selectively execute and save GPU memory.
I want to confirm whether this is a feature or an implementation bug.


### Versions


<details>
  <summary>pytorch 1.8.0</summary>
  <pre><code>
  [pip3] numpy==1.19.2
  [pip3] torch==1.8.0
  [pip3] torchelastic==0.2.2
  [pip3] torchtext==0.9.0
  [pip3] torchvision==0.9.0
  [conda] blas                      1.0                         mkl  
  [conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
  [conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
  [conda] mkl                       2020.2                      256  
  [conda] mkl-service               2.3.0            py38he904b0f_0  
  [conda] mkl_fft                   1.3.0            py38h54f3939_0  
  [conda] mkl_random                1.1.1            py38h0573a6f_0  
  [conda] numpy                     1.19.2           py38h54aff64_0  
  [conda] numpy-base                1.19.2           py38hfa32c7d_0  
  [conda] pytorch                   1.8.0           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
  [conda] torchelastic              0.2.2                    pypi_0    pypi
  [conda] torchtext                 0.9.0                      py38    pytorch
  [conda] torchvision               0.9.0                py38_cu111    pytorch</code></pre>
</details>

<details>
  <summary>pytorch 1.9.0</summary>
  <pre><code>
  [pip3] numpy==1.20.2
  [pip3] torch==1.9.0
  [pip3] torchelastic==0.2.0
  [pip3] torchtext==0.10.0
  [pip3] torchvision==0.10.0
  [conda] blas                      1.0                         mkl  
  [conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
  [conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
  [conda] mkl                       2021.2.0           h06a4308_296  
  [conda] mkl-service               2.3.0            py37h27cfd23_1  
  [conda] mkl_fft                   1.3.0            py37h42c9631_2  
  [conda] mkl_random                1.2.1            py37ha9443f7_2  
  [conda] numpy                     1.20.2           py37h2d18471_0  
  [conda] numpy-base                1.20.2           py37hfae3a4d_0  
  [conda] pytorch                   1.9.0           py3.7_cuda11.1_cudnn8.0.5_0    pytorch
  [conda] torchelastic              0.2.0                    pypi_0    pypi
  [conda] torchtext                 0.10.0                     py37    pytorch
  [conda] torchvision               0.10.0               py37_cu111    pytorch</code></pre>
</details>


<details>
  <summary>pytorch 1.10.0</summary>
  <pre><code>
  [pip3] numpy==1.21.2
  [pip3] torch==1.10.0
  [pip3] torch-tb-profiler==0.4.0
  [pip3] torchelastic==0.2.0
  [pip3] torchtext==0.11.0
  [pip3] torchvision==0.11.0
  [conda] blas                      1.0                         mkl  
  [conda] cudatoolkit               11.3.1               ha36c431_9    nvidia
  [conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
  [conda] mkl                       2021.3.0           h06a4308_520  
  [conda] mkl-service               2.4.0            py37h7f8727e_0  
  [conda] mkl_fft                   1.3.1            py37hd3c417c_0  
  [conda] mkl_random                1.2.2            py37h51133e4_0  
  [conda] numpy                     1.21.2           py37h20f2e39_0  
  [conda] numpy-base                1.21.2           py37h79a1101_0  
  [conda] pytorch                   1.10.0          py3.7_cuda11.3_cudnn8.2.0_0    pytorch
  [conda] pytorch-mutex             1.0                        cuda    pytorch
  [conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
  [conda] torchelastic              0.2.0                    pypi_0    pypi
  [conda] torchtext                 0.11.0                     py37    pytorch
  [conda] torchvision               0.11.0               py37_cu113    pytorch</code></pre>
</details>


<details>
  <summary>pytorch 1.11.0</summary>
  <pre><code>
  [pip3] numpy==1.21.6
  [pip3] pytorch-lightning==1.6.3
  [pip3] torch==1.11.0
  [pip3] torch-tb-profiler==0.4.0
  [pip3] torchaudio==0.11.0
  [pip3] torchmetrics==0.9.0
  [pip3] torchvision==0.12.0
  [conda] blas                      1.0                         mkl  
  [conda] cudatoolkit               11.3.1               h2bc3f7f_2  
  [conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
  [conda] mkl                       2021.4.0           h06a4308_640  
  [conda] mkl-service               2.4.0            py38h7f8727e_0  
  [conda] mkl_fft                   1.3.1            py38hd3c417c_0  
  [conda] mkl_random                1.2.2            py38h51133e4_0  
  [conda] numpy                     1.19.5                   pypi_0    pypi
  [conda] numpy-base                1.21.5           py38hf524024_1  
  [conda] pytorch                   1.11.0          py3.8_cuda11.3_cudnn8.2.0_0    pytorch
  [conda] pytorch-lightning         1.6.3                    pypi_0    pypi
  [conda] pytorch-mutex             1.0                        cuda    pytorch
  [conda] torch                     1.10.0                   pypi_0    pypi
  [conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
  [conda] torchaudio                0.11.0               py38_cu113    pytorch
  [conda] torchmetrics              0.9.0                    pypi_0    pypi
  [conda] torchvision               0.12.0               py38_cu113    pytorch</code></pre>
</details>

<details>
  <summary>pytorch 1.12.0</summary>
  <pre><code>
  [pip3] numpy==1.21.5
  [pip3] torch==1.12.0
  [pip3] torchelastic==0.2.0
  [pip3] torchtext==0.13.0
  [pip3] torchvision==0.13.0
  [conda] blas                      1.0                         mkl  
  [conda] cudatoolkit               11.3.1               ha36c431_9    nvidia
  [conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
  [conda] mkl                       2021.4.0           h06a4308_640  
  [conda] mkl-service               2.4.0            py37h7f8727e_0  
  [conda] mkl_fft                   1.3.1            py37hd3c417c_0  
  [conda] mkl_random                1.2.2            py37h51133e4_0  
  [conda] numpy                     1.21.5           py37he7a7128_2  
  [conda] numpy-base                1.21.5           py37hf524024_2  
  [conda] pytorch                   1.12.0          py3.7_cuda11.3_cudnn8.3.2_0    pytorch
  [conda] pytorch-mutex             1.0                        cuda    pytorch
  [conda] torchelastic              0.2.0                    pypi_0    pypi
  [conda] torchtext                 0.13.0                     py37    pytorch
  [conda] torchvision               0.13.0               py37_cu113    pytorch</code></pre>
</details>

<details>
  <summary>pytorch 1.13.0</summary>
  <pre><code>
  [pip3] numpy==1.22.3
  [pip3] torch==1.13.0
  [pip3] torchtext==0.14.0
  [pip3] torchvision==0.14.0
  [conda] blas                      1.0                         mkl  
  [conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
  [conda] mkl                       2021.4.0           h06a4308_640  
  [conda] mkl-service               2.4.0            py39h7f8727e_0  
  [conda] mkl_fft                   1.3.1            py39hd3c417c_0  
  [conda] mkl_random                1.2.2            py39h51133e4_0  
  [conda] numpy                     1.22.3           py39he7a7128_0  
  [conda] numpy-base                1.22.3           py39hf524024_0  
  [conda] pytorch                   1.13.0          py3.9_cuda11.6_cudnn8.3.2_0    pytorch
  [conda] pytorch-cuda              11.6                 h867d48c_0    pytorch
  [conda] pytorch-mutex             1.0                        cuda    pytorch
  [conda] torchtext                 0.14.0                     py39    pytorch
  [conda] torchvision               0.14.0               py39_cu116    pytorch</code></pre>
</details>

<details>
  <summary>pytorch 1.13.1</summary>
  <pre><code>
  [pip3] numpy==1.22.3
  [pip3] torch==1.13.1
  [pip3] torchelastic==0.2.2
  [pip3] torchtext==0.14.1
  [pip3] torchvision==0.14.1
  [conda] blas                      1.0                         mkl  
  [conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
  [conda] mkl                       2021.4.0           h06a4308_640  
  [conda] mkl-service               2.4.0           py310h7f8727e_0  
  [conda] mkl_fft                   1.3.1           py310hd6ae3a3_0  
  [conda] mkl_random                1.2.2           py310h00e6091_0  
  [conda] numpy                     1.22.3          py310hfa59a62_0  
  [conda] numpy-base                1.22.3          py310h9585f30_0  
  [conda] pytorch                   1.13.1          py3.10_cuda11.6_cudnn8.3.2_0    pytorch
  [conda] pytorch-cuda              11.6                 h867d48c_1    pytorch
  [conda] pytorch-mutex             1.0                        cuda    pytorch
  [conda] torchelastic              0.2.2                    pypi_0    pypi
  [conda] torchtext                 0.14.1                    py310    pytorch
  [conda] torchvision               0.14.1              py310_cu116    pytorch</code></pre>
</details>


cc @ezyang @gchanan @zou3519 @albanD @mruberry @jbschlosser @walterddr @saketh-are