What is the difference between register_buffer() and parameter.detach() in PyTorch?

jasonyoung: <p>I am writing a PositionalEmbedding() module which is an implementation based on “<a href="https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf" rel="nofollow noopener">Attention Is All You Need</a>” using PyTorch. According to the paper, there should be no learnable attribute in the module PositionalEmbedding().</p>
<p>The initialization of the Embedding weights is as follows:</p>
<pre><code class="lang-auto">    import math
    import torch
    
    class TrigonometricPositionalEmbedding(torch.nn.Module):
        def __init__(self, embedding_number, dimension, padding_idx):
            position = torch.arange(0, embedding_number).unsqueeze(1)
            sin_multiplicator = torch.exp(-(math.log(10000) / dimension) * 2 * torch.arange(0, dimension, 2))
            cos_multiplicator = torch.exp(-(math.log(10000) / dimension) * 2 * torch.arange(1, dimension, 2))
            sin_weight = torch.sin(position * sin_multiplicator)
            cos_weight = torch.cos(position * cos_multiplicator)
    
            weight = torch.zeros(embedding_number, dimension)
            weight[:, 0::2] = sin_weight
            weight[:, 1::2] = cos_weight
</code></pre>
<p>Next up, because the <code>weight</code> in the module should not be learnable, <code>detach()</code> the <code>weight</code> came into my mind. I think it may not be the most elegant way, so I made a further investigation. I found that there are two kinds of implementation method like what <a href="https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/modules/embeddings.py#L34" rel="nofollow noopener">OpenNMT</a> and <a href="https://github.com/pytorch/fairseq/blob/master/fairseq/modules/sinusoidal_positional_embedding.py#L104" rel="nofollow noopener">FAIRSeq</a> have done respectively.</p>
<p>I follow the method of <strong>OpenNMT</strong> and my implementation is as follows:</p>
<pre><code class="lang-auto">    class TrigonometricPositionalEmbedding(torch.nn.Module):
        def __init__(self, embedding_number, dimension, padding_idx):
            ...
            weight[:, 1::2] = cos_weight
            self.register_buffer('weight', weight)
    
        def forward(self, position):
            torch.index_select(self.weight, 0, position)
</code></pre>
<p>I follow the method of <strong>FAIRSeq</strong> and my implementation is as follows:</p>
<pre><code class="lang-auto">    class TrigonometricPositionalEmbedding(torch.nn.Module):
        def __init__(self, embedding_number, dimension, padding_idx):
            ...
            weight[:, 1::2] = cos_weight
            self.weight = weight
    
        def forward(self, position):
            torch.index_select(self.weight, 0, position).detach()
</code></pre>
<p>So I am curious about the difference between the two method, or what is the difference between <strong>registered buffer</strong> and <strong>detached parameter</strong>?</p>
<p>(The question was also posted in <a href="https://datascience.stackexchange.com/questions/76787/what-is-the-difference-between-register-buffer-and-parameter-detach-in-pytor" rel="nofollow noopener">StackOverflow</a> but no answer yet.)</p>

googlebot: <p>parameter is a tensor wrapped in nn.Parameter, this way an optimizer can find it and build a list of parameters to optimize.</p>
<p>buffer is a tensor that is serialized with the model, plus it participates in module.cuda() device transfer</p>
<p>and object attribute assignment is just that, tensors assigned this way are “unmanaged”</p>