What pytorch means by buffers?

seyeeet: <p>Can someone explain to me what do we mean by buffers in pytorch.<br>
what is its characteristics and when we will use that and when we should not use that?<br>
If a buffer is basically the same as tensor, Why would I even need a buffer when I can simply create my tensor inside the module?<br>
an example would be appreciated!!<br>
thanks</p>

ptrblck: <p>Buffers are tensors, which are registered in the module and will thus be inside the <code>state_dict</code>.<br>
These tensors do not require gradients and are thus not registered as parameters.<br>
This is useful e.g. to track the mean and std in batchnorm layers etc. which should be stored and loaded using the <code>state_dict</code> of the module.</p>

seyeeet: <p>I understand, thanks.</p>
<ul>
<li>is there a way to know all the buffers in a model?</li>
<li>in case of bn that you mentioned how I can get the mean and std values? I reach to this point, but not sure how to get  mean and std</li>
</ul>
<pre><code class="lang-auto">import torchvision
import torch.nn as nn
import torch.nn.functional as F
resnet = torchvision.models.resnet50()
resnet.layer1[0].bn1.state_dict
</code></pre>

ptrblck: <p>You can get all buffers via <code>model.buffers()</code> or <code>model.named_buffers()</code> (same as with <code>.parameters()</code> and <code>.named_parameters()</code>).<br>
To access the buffers in a specific layer, you can access them directly:</p>
<pre><code class="lang-python">model.bn.running_mean
model.bn.running_var
</code></pre>

Konstantin_Burlachen: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="120266">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>named_parameters</p>
</blockquote>
</aside>
<p>Q1: Do I understand correctly that named_buffers() and buffers() is the same parameters?</p>
<p>Q2: Is Pytotrch automatically determinates the name of the parameter?<br>
Because ctor does not obtain this as argument (<a href="https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html" class="inline-onebox" rel="noopener nofollow ugc">Parameter — PyTorch 2.1 documentation</a>)</p>

ptrblck: <ol>
<li>
<p><code>named_buffers()</code> and <code>buffers()</code> returns the same buffers where the first operation returns the corresponding name for each buffer. I’m explicitly using “buffer” to avoid conflicting it with parameters, which are different. Both are registered to the <code>nn.Module</code> where parameters are trainable while buffers are not.</p>
</li>
<li>
<p>Yes, the name of the buffer or parameter is determined by its variable name (e.g. <code>self.my_param = nn.Parameter(...)</code> would return “my_param” as the name) or explicitly via <code>self.register_buffer(name, buffer)</code> or <code>self.register_parameter(name, param)</code>.</p>
</li>
</ol>

Konstantin_Burlachen: <p>It’s an interesting example of using Python reflection like behaviour in such way that there is no need to register named_pararamters with specific names:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class NN_Network(nn.Module):
    def __init__(self,in_dim,hid,out_dim):
        super(NN_Network, self).__init__()
        self.linear1 = nn.Linear(in_dim,hid)
        self.linear2 = nn.Linear(hid,out_dim)

    def forward(self, input_array):
        h = self.linear1(input_array)
        y_pred = self.linear2(h)
        return y_pred

in_d = 5
hidn = 2
out_d = 3
model = NN_Network(in_d, hidn, out_d)

for name, param in model.named_parameters():
     print(name, "[", type(name), "]", type(param), param.size())
</code></pre>

Konstantin_Burlachen: <p>I have updated a code snippet to play with parameters and buffers from public Python API side:</p>
<pre><code class="lang-auto">#!/usr/bin/env python3

import torch
import torch.nn as nn

class NN_Network(nn.Module):
    def __init__(self,in_dim,hid,out_dim):
        super(NN_Network, self).__init__()

        # https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=torch%20nn%20linear#torch.nn.Linear
        self.linear1 = nn.Linear(in_dim,hid)
        self.linear2 = nn.Linear(hid,out_dim)

        # https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html?highlight=torch%20nn%20parameter#torch.nn.parameter.Parameter
        self.my_param_a = nn.Parameter(torch.zeros(3,3))

        # https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=register_buffer#torch.nn.Module.register_buffer
        self.my_buffer_1 = self.register_buffer("A", torch.zeros(3,3))
        self.my_buffer_2 = self.register_buffer("B", torch.zeros(4,4))

    def forward(self, input_array):
        h = self.linear1(input_array)
        y_pred = self.linear2(h)
        return y_pred

in_d = 5
hidn = 2
out_d = 3
model = NN_Network(in_d, hidn, out_d)

print("\nNAMED PARAMS")
for name, param in model.named_parameters():
     print("    ", name, "[", type(name), "]", type(param), param.size())

print("\nNAMED BUFFERS")
for name, param in model.named_buffers():
     print("    ", name, "[", type(name), "]", type(param), param.size())

print("\nSTATE DICT (KEYS ONLY)")
for k, v in model.state_dict().items():
   print("    ", k)

print("\nSTATE DICT (KEYS VALUE PAIRS)")
for k, v in model.state_dict().items():
   print("        ", "(", k, "=&gt;", v, ")")


</code></pre>