How to define arbitrary connections in the forward function when defining a model?

dangmanhtruong1995: <p>Hi everyone, so as we know, when we define a model class, in the <strong>init</strong> function we write the layer definition, and in the forward function, we specify the connections between the layers as well as getting the output. Example:</p>
<pre><code class="lang-auto">class TinyModel(torch.nn.Module):

    def __init__(self):
        super(TinyModel, self).__init__()

        self.linear1 = torch.nn.Linear(100, 200)
        self.activation = torch.nn.ReLU()
        self.linear2 = torch.nn.Linear(200, 10)
        self.softmax = torch.nn.Softmax()

    def forward(self, x):
        x = self.linear1(x)
        x = self.activation(x)
        x = self.linear2(x)
        x = self.softmax(x)
        return x
</code></pre>
<p>However, this is usually fixed, meaning that the connections between layers are known beforehand. If, for example, I have a dictionary called connection_dict, which stores the connections between layers, such as 1-&gt;2, 1-&gt;3, 2-&gt;4, etc., and another list called layer_encoding_list, which stores the layer definition, and I want to create a model based on the connections from these two variables. Then it would be written as follows:</p>
<pre><code class="lang-auto">class MyDLModel(torch.nn.Module):

    def __init__(self, connection_dict, layer_encoding_list):
        super(MyDLModel, self).__init__()

        self.connection_dict = connection_dict
        self.layer_encoding_list = layer_encoding_list
		
		self.layer_list = nn.ModuleList() 
		# Define the layers according to layer_encoding_list
		...
		
    def forward(self, x):
        # Define the connections according to self.connection_dict
		...
</code></pre>
<p>As you can see, the layer definitions can be done using nn.ModuleList(), however I’m not clear how to define things in the forward function so that there won’t be any problem with backprop. Should I put it into nn.ModuleList() ?  Or a nn.ModuleList() of nn.ModuleList()? And I have seen some examples where people put the outputs from layers to a list, then do torch.cat on it and it still works, however I’m not too sure on the details, since here connections can be arbitrary. Please help me, thank you very much.</p>

ptrblck: <aside class="quote no-group" data-username="dangmanhtruong1995" data-post="1" data-topic="204390">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dangmanhtruong1995/48/59870_2.png" class="avatar"> dangmanhtruong1995:</div>
<blockquote>
<p>however I’m not clear how to define things in the forward function so that there won’t be any problem with backprop.</p>
</blockquote>
</aside>
<p>As long as you are properly registering modules and parameters, Autograd won’t have a problem. E.g. use <code>nn.ModuleList</code> to create a list of modules instead of a plain Python <code>list</code>.</p>
<aside class="quote no-group" data-username="dangmanhtruong1995" data-post="1" data-topic="204390">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dangmanhtruong1995/48/59870_2.png" class="avatar"> dangmanhtruong1995:</div>
<blockquote>
<p>And I have seen some examples where people put the outputs from layers to a list, then do torch.cat on it and it still works, however I’m not too sure on the details, since here connections can be arbitrary.</p>
</blockquote>
</aside>
<p>This is a valid approach and will keep the gradient history. To double check if tensors were detached by a specific operation, you could check the <code>.grad_fn</code> attribute of intermediate tensors before and after the operation in question. If both are showing a valid function, the tensor was not detached.</p>