What is nn.Identity() used for?

TheOraware: <p>I am trying to understand usage of nn.Identity , but unable to understand what pytorch document referring with following example <a href="https://pytorch.org/docs/stable/generated/torch.nn.Identity.html" rel="noopener nofollow ugc">nn.Identity</a></p>
<pre><code class="lang-auto">seed =4
m = nn.Identity()
input = torch.randn(4,4)
output = m(input)
print(output)
print(output.size())
</code></pre>
<p>What nn.Identity is doing here ?</p>
<pre><code class="lang-auto">tensor([[-0.2381, -0.0488,  1.2883, -2.0334],
        [-0.7297, -0.8721,  0.7086,  0.3899],
        [ 0.6550,  1.4832,  0.3744, -1.2825],
        [ 2.3327,  0.5004, -0.8785,  0.5100]])
torch.Size([4, 4])
</code></pre>
<p>Without identiy</p>
<pre><code class="lang-auto">seed =4
input = torch.randn(4,4)
output = input
print(output)
print(output.size())
</code></pre>
<pre><code class="lang-auto">tensor([[-0.0255, -0.5771, -0.5268, -0.6201],
        [ 1.0814,  0.4274,  0.6822,  0.2369],
        [ 0.0251, -1.1992, -0.9557,  0.9640],
        [-0.2789, -2.4326, -2.4736,  0.2777]])
torch.Size([4, 4])
</code></pre>
<pre><code class="lang-auto">
seed =4
m = nn.Identity()
input = torch.randn(4,4)
output = m(input)
print(f'input {input}')
print(f'output {output}')

</code></pre>
<pre><code class="lang-auto">input tensor([[-1.3704, -0.4062, -0.5499,  1.8572],
        [ 0.3674,  0.4486, -0.3362, -1.4035],
        [ 0.1322,  0.2116, -0.1157,  1.0715],
        [ 0.8215,  0.1549,  0.8935, -0.1930]])
output tensor([[-1.3704, -0.4062, -0.5499,  1.8572],
        [ 0.3674,  0.4486, -0.3362, -1.4035],
        [ 0.1322,  0.2116, -0.1157,  1.0715],
        [ 0.8215,  0.1549,  0.8935, -0.1930]])

</code></pre>

pain: <p><a class="mention" href="/u/theoraware">@TheOraware</a><br>
please, see this accepted answer in stack overflow, <a href="https://stackoverflow.com/questions/64229717/what-is-the-idea-behind-using-nn-identity-for-residual-learning" rel="noopener nofollow ugc">here</a></p>

TheOraware: <p><a class="mention" href="/u/pain">@pain</a> i read this and understand it , but in a very simple lay man example as above i posted what is it doing with and without nn.Identity ? What Identity playing role in above example</p>

pain: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> has a great experince. I hope he can help.</p>

TheOraware: <p><a class="mention" href="/u/pain">@pain</a> i think i got it what does it do is it remains keep intact of original input shape , as NN shapes change over many different layer , we can keep original input layer shape as a placeholder and use this to add on your other layer’s output for skip connection</p>
<pre><code class="lang-auto">a = torch.arange(4.)
print(f' "a" is {a} and its shape is {a.shape}')
m = nn.Identity()
input_identity = m(a)
# change shape of a
a= torch.reshape(a, (2, 2))
print(f' "a" shape is now changed {a.shape}')
print(f' due to identity it remains has same shape as time of input {input_identity.shape}')
</code></pre>
<pre><code class="lang-auto">"a" is tensor([0., 1., 2., 3.]) and its shape is torch.Size([4])
"a" shape is now changed torch.Size([2, 2])
due to identity it remains has same shape as time of input torch.Size([4])
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> please acknowledge my understanding is correct or not , if not then please help me to understand it</p>

ptrblck: <p>The <code>nn.Identity</code> module will just return the input without any manipulation and can be used to e.g. replace other layers. The source code can be found <a href="https://github.com/pytorch/pytorch/blob/58d1b3639bc07f9519de18e5a18e575f260c7eeb/torch/nn/modules/linear.py#L12-L32">here</a>.</p>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you please see my last post ? i tried to represent in code</p>

ptrblck: <p>Your code snippet shows that <code>nn.Identity</code> will just return its input, but doesn’t show that it’s a view.<br>
If you thus manipulate the input inplace, the output of <code>nn.Identity</code> will also be changed:</p>
<pre><code class="lang-python">a = torch.arange(4.)
m = nn.Identity()
input_identity = m(a)

print(a)
&gt; tensor([0., 1., 2., 3.])

print(input_identity)
&gt; tensor([0., 1., 2., 3.])

# manipulate inplace
a[0] = 2.
print(a)
&gt; tensor([2., 1., 2., 3.])

print(input_identity)
&gt; tensor([2., 1., 2., 3.])
</code></pre>

TheOraware: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="125457">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>print(input_identity)</code></p>
</blockquote>
</aside>
<p>how can i use nn.Identity() where it just return the input without any manipulation? What is the point of using it when i can’t achieve original input without influence of any subsequent manipulation? We can achieve this using clone () as below, what is the point of nn.Identity()?</p>
<pre><code class="lang-auto">a = torch.arange(4.)
input_identity = a.clone().detach()
print(a)
print(input_identity)
</code></pre>
<pre><code class="lang-auto">tensor([0., 1., 2., 3.])
tensor([0., 1., 2., 3.])
</code></pre>
<pre><code class="lang-auto">a[0] = 2.
print(a)
print(input_identity)
</code></pre>
<pre><code class="lang-auto">tensor([2., 1., 2., 3.])
tensor([0., 1., 2., 3.])
</code></pre>

ptrblck: <p>As mentioned before, <code>nn.Identity</code> will just return the input without any <code>clone</code> usage or manipulation of the input. The input and output would thus be the same.</p>
<aside class="quote no-group" data-username="TheOraware" data-post="9" data-topic="125457">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/theoraware/48/30315_2.png" class="avatar"> TheOraware:</div>
<blockquote>
<p>what is the point of nn.Identity()?</p>
</blockquote>
</aside>
<p>Yes, this “pass-through” layer can easily be written manually, which was also the reason feature requests were declined in the past.<br>
However, since a lot of users were rewriting the same layer to e.g. replace specific modules inside a larger model, this layer was introduced.</p>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for reply - sorry i am not getting your last paragraph</p>
<p>“Yes, this “pass-through” layer can easily be written manually, which was also the reason feature requests were declined in the past.<br>
However, since a lot of users were rewriting the same layer to e.g. replace specific modules inside a larger model, this layer was introduced.”</p>
<p>Could you please elaborate it more?</p>

ptrblck: <p>You might not want to use this layer as it’s not “doing anything” besides just returning the input.<br>
However, there are use cases where users needed exactly this (e.g. to replace another layer) and were manually creating custom modules to do so and asked for the <code>nn.Identity</code> layer in the PyTorch <code>nn</code> backend. Since more and more users were depending on it, it was created.<br>
However, as already said, this layer might not be interesting for you.</p>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> it means its best use is when we connect previous layer output to the output of new layer? For example in residual connection sometime we connect output of previous layer to output of new layer. If this is nn.Identity() is designed for then we can achieve the same as mentioned in my following code.</p>
<p><strong>I think i need to read about nn.Identity() more , maybe it will take time to me to understand its underline mechanism</strong></p>
<p>However , could you please confirm following code has skip connection <strong>res</strong> and <strong>res1</strong> which i used to avoid gradient vanishing, i joined output of layer1 (<strong>res</strong>) and output of layer 2 (<strong>res1</strong>) with output of layer6 <strong>F.relu(torch.cat([x,res,res1],1))</strong> .Is this is sort of way we join layers by skipping one to another in order to avoid gradient vanishing?</p>
<pre><code class="lang-auto">class multiNetA(nn.Module):
    def __init__(self, num_feature, num_class):
        super().__init__()

        self.lin1 = nn.Linear(num_feature,50)
        self.lin2 = nn.Linear(50, 30)
        self.lin6 = nn.Linear(30, 20)
        self.lin7 = nn.Linear(100, num_class)

        self.bn0 = nn.BatchNorm1d(num_feature)
        self.bn1 = nn.BatchNorm1d(50)
        self.bn2 = nn.BatchNorm1d(30)
        self.bn6 = nn.BatchNorm1d(20)
        self.bn7 = nn.BatchNorm1d(9)

    def forward(self, x):
        x = self.bn0(x)
        x = self.lin1(x) 
        x = self.bn1(x) 
        x = F.relu(x)
        res = x &lt;---- skip connection
        
        
        x = self.lin2(x) 
        x = self.bn2(x) 
        x = F.relu(x)
        
        res1 = x &lt;--- another skip connection
      

        x = self.lin6(x)
        x = self.bn6(x) 
        x = F.relu(torch.cat([x,res,res1],1)) &lt;-- joined res and res1 (input) with output of layer 6
        x = self.lin7(x) # output layer
        
        return x
</code></pre>

ptrblck: <p>Skip connections would usually a add activations instead of concatenating them as seen in the <a href="https://github.com/pytorch/vision/blob/ef711591a5db69d36f904ab5c39dec13627a58ad/torchvision/models/resnet.py#L68-L80">resnet example</a>.</p>
<p>An often used use case for <code>nn.Identity</code> would be to get the “features” of a pretrained model instead of the class logits.<br>
Here is an example:</p>
<pre><code class="lang-python">model = models.resnet18()
# replace last linar layer with nn.Identity
model.fc = nn.Identity()

# get features for input
x = torch.randn(1, 3, 224, 224)
out = model(x)
print(out.shape)
&gt; torch.Size([1, 512])
</code></pre>

Felix_Yu: <aside class="quote no-group" data-username="TheOraware" data-post="1" data-topic="125457">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/theoraware/48/30315_2.png" class="avatar"> TheOraware:</div>
<blockquote>
<pre><code class="lang-auto">seed =4
m = nn.Identity()
input = torch.randn(4,4)
output = m(input)
print(output)
print(output.size())
</code></pre>
</blockquote>
</aside>
<p>change seed=4 to torch.manual_seed(4), you will get same results.</p>