LayerNorm == InstanceNorm?

username: zyh3826
cooked: <p>I found the result of torch.nn.LayerNorm equals torch.nn.InstanceNorm1d, why?<br>
testing code:</p>
<pre data-code-wrap="python"><code class="lang-python">batch_size, seq_size, dim = 2, 3, 4
x = torch.randn(batch_size, seq_size, dim)

#layer norm
layer_norm = torch.nn.LayerNorm(dim, elementwise_affine=False)
print('y_layer_norm: ', layer_norm(x))
print('=' * 30)

# custom instance norm
eps: float = 0.00001
mean = torch.mean(x, dim=-1, keepdim=True)
var = var = torch.var(x, dim=-1, keepdim=True, unbiased=False)
print('y_custom: ', (x - mean) / torch.sqrt(var + eps))
print('=' * 30)

# instance norm
instance_norm = torch.nn.InstanceNorm1d(dim, affine=False)
print('y_instance_norm', instance_norm(x))
print('=' * 30)

# follow the description in https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html
# For example, if normalized_shape is (3, 5) (a 2-dimensional shape),
# the mean and standard-deviation are computed over the last 2 dimensions of the input (i.e. input.mean((-2, -1)))
mean = torch.mean(x, dim=(-2, -1), keepdim=True)
var = torch.var(x, dim=(-2, -1), keepdim=True, unbiased=False)
print('y_custom1: ', (x - mean) / torch.sqrt(var + esp))
print('=' * 30)
</code></pre>
<p>results:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/7218b3364ae0f975af48f17e28f6415e56206dfb.png" data-download-href="https://discuss.pytorch.org/uploads/default/7218b3364ae0f975af48f17e28f6415e56206dfb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/2/7218b3364ae0f975af48f17e28f6415e56206dfb.png" alt="image" data-base62-sha1="ghllM9U5z8gqq81NyBLrzuAmCKL" width="487" height="500" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">666×683 16.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
In the results, I found LayerNorm equals InstanceNorm1d, and I custom the compute progress also found that the description in <a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html" rel="noopener nofollow ugc">LayerNorm doc</a> maybe not correct? Do I miss something or LayerNorm and InstanceNorm1d in pytorch are absolutely equal?<br>
Hope someone can answer this question, thanks!</p>

username: snio89
cooked: <p>This seems like a bug of <code>nn.InstanceNorm1d</code> when <code>affine=False</code>.</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/50134">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/50134" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/50134" target="_blank" rel="noopener nofollow ugc">nn.InstanceNorm should warn user if input channel is inconsistent with num_features, even when affine=False</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-01-06" data-time="09:02:40" data-timezone="UTC">09:02AM - 06 Jan 21 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/seungwonpark" target="_blank" rel="noopener nofollow ugc">
          <img alt="seungwonpark" src="https://avatars.githubusercontent.com/u/13394294?v=4" class="onebox-avatar-inline" width="20" height="20">
          seungwonpark
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: nn
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          needs design
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: norms and normalization
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## 🐛 Bug

When `nn.InstanceNorm1d` is used without affine transformation, it d<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">oes not warn the user even if the channel size of input is inconsistent with `num_features` parameter. Though the `num_features` won't matter on computing `InstanceNorm(num_features, affine=False)`, I think it should warn the user if the wrong argument/input is being given.

## To Reproduce &amp; Expected behavior

```python
import torch
import torch.nn as nn

# we define an InstanceNorm1d layer without affine transformation, where num_features=7
# note that affine is set False by default
m = nn.InstanceNorm1d(7)

# here, the input with the wrong channel size (3) is given.
input = torch.randn(2, 3, 5)

# the following operation should warn the user, although there's no problem with performing the computation.
output = m(input)
```

I'm not sure whether this is intended behavior or not.

## Environment

PyTorch version: 1.7.0+cu110
Is debug build: True
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: (MinGW.org GCC-8.2.0-3) 8.2.0
Clang version: Could not collect
CMake version: version 3.14.3

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2060
Nvidia driver version: 460.89
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.3
[pip3] pytorch-lightning==1.0.3
[pip3] pytorch-ranger==0.1.1
[pip3] torch==1.7.0+cu110
[pip3] torch-optimizer==0.0.1a15
[pip3] torchvision==0.8.1+cu110
[conda] Could not collect

cc @albanD @mruberry @jbschlosser</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><code>nn.InstanceNorm1d</code> should take an input of the shape <code>(batch_size, dim, seq_size)</code>.<br>
However, if <code>affine=False</code>, <code>nn.InstanceNorm1d</code> can take an input of the wrong shape <code>(batch_size, seq_size, dim)</code> and provides a LayerNorm-like result.</p>