Really basic question on declaring variables in pytorch classes

username: maxmatical
cooked: <p>I’m trying to learn the implementation of Transformers, using <a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html#model-architecture" rel="nofollow noopener">this tutorial</a>. I’ve noticed that when declaring a variable inside a class using <code>__init__</code>, everything is set as <code>self</code> such as</p>
<pre><code class="lang-auto">class EncoderDecoder(nn.Module):
    """
    A standard Encoder-Decoder architecture. Base for this and many 
    other models.
    """
    def __init__(self, encoder, decoder, src_embed, tgt_embed, generator):
        super(EncoderDecoder, self).__init__()
        self.encoder = encoder
        self.decoder = decoder
        self.src_embed = src_embed
        self.tgt_embed = tgt_embed
        self.generator = generator
        
    def forward(self, src, tgt, src_mask, tgt_mask):
        "Take in and process masked src and target sequences."
        return self.decode(self.encode(src, src_mask), src_mask,
                            tgt, tgt_mask)
    
    def encode(self, src, src_mask):
        return self.encoder(self.src_embed(src), src_mask)
    
    def decode(self, memory, src_mask, tgt, tgt_mask):
        return self.decoder(self.tgt_embed(tgt), memory, src_mask, tgt_mask)
</code></pre>
<p>But sometimes not everything is set as <code>self</code>, such as</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    "Core encoder is a stack of N layers"
    def __init__(self, layer, N):
        super(Encoder, self).__init__()
        self.layers = clones(layer, N)
        self.norm = LayerNorm(layer.size)
        
    def forward(self, x, mask):
        "Pass the input (and mask) through each layer in turn."
        for layer in self.layers:
            x = layer(x, mask)
        return self.norm(x)
</code></pre>
<p>Where <code>layer</code> and <code>N</code> are not set as <code>self</code></p>
<p>Previously when I implemented models I have been setting everything as <code>self</code>, but it’s really got me wondering if there is a difference, and if setting everything as <code>self</code> is wrong?</p>

username: JuanFMontesinos
cooked: <p>I’m sorry but that code is very ambiguous. As a python object you can set whatever as “self”. Self just indicates it’s an attribute of the object, this is not related to pytorch itself but python.</p>
<p>The pytorch way is to set nn.Module instances as attributes in <strong>init</strong> and call them during forward however this does not necessary mean that everything set as an attribute were a nn.Module.</p>

username: maxmatical
cooked: <p>I know self is not exclusive to pytorch, but I am curious why some of the declared variables in the object are set as <code>self</code> but others aren’t (like <code>layer</code> and <code>N</code> in the Encoder)</p>

username: JuanFMontesinos
cooked: <p>Sorry could you explain what layer is and what does clones do?<br>
Layer</p>

username: maxmatical
cooked: <p>The layer is the self-attention + feed forward layer for transformer networks, and the clone just makes N copies of the layer so in the encoder the data passes through N copies of the layer</p>

username: JuanFMontesinos
cooked: <p>So in that case I’m pretty sure that (if it’s been properly done) clones will generate a <code>torch.nn.</code> <code>ModuleList</code>, right? If it’s the case, those modules are properly registered and you can work with that attribute as a python list</p>