How to create multiple instances of a model with different config inside a wrapper function

username: AbraMunna
cooked: <p>class wrapper(nn.Module):<br>
def <strong>init</strong>(self, input_size, combined_input_size, hidden_size, num_classes, number_of_vars):<br>
super().<strong>init</strong>()<br>
self.m1 = var(input_size=input_size[0], hidden_size=hidden_size, num_classes=num_classes, p=0.0)<br>
self.m2 = var(input_size=input_size[1], hidden_size=hidden_size, num_classes=num_classes, p=0.0)<br>
self.m3 = var(input_size=input_size[2], hidden_size=hidden_size, num_classes=num_classes, p=0.0)<br>
self.m4 = var(input_size=input_size[3], hidden_size=hidden_size, num_classes=num_classes, p=0.0)<br>
self.m5 = var(input_size=input_size[4], hidden_size=hidden_size, num_classes=num_classes, p=0.0)</p>
<pre><code>def forward(self, var_list):
    x = F.normalize(self.m1(var_list[0]), p=2, dim=1)
    y = F.normalize(self.m2(var_list[1]), p=2, dim=1)
    z = F.normalize(self.m3(var_list[2]), p=2, dim=1)
    h = F.normalize(self.m4(var_list[3]), p=2, dim=1)
    w = F.normalize(self.m5(var_list[3]), p=2, dim=1)
    return torch.cat([x, y, z, h, w], dim=1)
</code></pre>
<p>How can I make all this calls inside a loop? or is there a better way to do this?</p>

username: AbraMunna
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, I would appreciate your suggestions. Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Bibhabasu_Mohapatra
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> is super hero of pytorch…<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for all the work you have done and the work you will do…<br>
Your work gets the world of DL  going…    Without any hinderances…</p>

username: AbraMunna
cooked: <p>Exactly, I am a big fan of <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

username: ptrblck
cooked: <p>Ha, thanks for the kind words <a class="mention" href="/u/abramunna">@AbraMunna</a> <a class="mention" href="/u/bibhabasu_mohapatra">@Bibhabasu_Mohapatra</a></p>
<p>I assume you would like to create <code>self.mX</code> in a loop and also use a loop in the forward?<br>
If <code>var</code> is returning a plain Python object, i.e. no PyTorch <code>nn.Module</code>, you could use <code>setattr/getattr</code> for both.<br>
However, based on the usage in <code>forward</code> I would guess <code>self.mX</code> are modules, so using an <code>nn.ModuleList</code> would be the right approach here. Something like this should work:</p>
<pre><code class="lang-python">class wrapper(nn.Module):
    def __init__(self, input_size, hidden_size, number_of_vars):
        super().__init__()
        self.module_list = nn.ModuleList()
        # assuming you want to create linear layers
        layer = nn.Linear
        for idx in range(number_of_vars):
            self.module_list.append(nn.Linear(input_size[idx], hidden_size))

    def forward(self, input_list):
        assert len(input_list)==len(self.module_list), "different lengths"
        out = [F.normalize(module(input)) for module, input in zip(self.module_list, input_list)]
        out = torch.cat(out, dim=1)
        return out

input_size = [2, 3, 4, 5, 6]
model = wrapper(input_size=input_size, hidden_size=16, number_of_vars=5)

batch_size = 4
input_list = [torch.randn(batch_size, s) for s in input_size]
out = model(input_list)
</code></pre>

username: AbraMunna
cooked: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a> . It is really helpful! I appreciate your gracious help</p>

username: AbraMunna
cooked: <p>Actually I am trying to create a different model (with linear layers) for different variables. That’s why I wrote one model and created k instances of that model where k is the number of variables.<br>
Its not one linear layer for one variable !</p>
<p>My var model look like the following,</p>
<p>class var(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size, num_classes, p=0.0, l=1):<br>
super().<strong>init</strong>()<br>
layers = []<br>
for i in range(l):<br>
if i == 0:<br>
layers.append(nn.Linear(input_size, hidden_size))<br>
layers.append(nn.Dropout(0.2))<br>
else:<br>
layers.append(nn.Linear(hidden_size, hidden_size))<br>
layers.append(nn.Dropout(0.2))<br>
<a href="http://self.net" rel="noopener nofollow ugc">self.net</a> = nn.Sequential(*layers)<br>
def forward(self, x):<br>
out = <a href="http://self.net" rel="noopener nofollow ugc">self.net</a>(x)<br>
return out</p>

username: ptrblck
cooked: <p>You could wrap the linear and dropout layer into an <code>nn.Sequential</code> module and append it to the <code>nn.ModuleList</code>. However, for more complicated constructs I would probably just use the for loop for the sake of readability. It’ll be executed once during the model initialization so I would claim no performance impact would be visible as it’s not in the hot path.</p>

username: AbraMunna
cooked: <p>Make sense! Thank you</p>

username: AbraMunna
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, If in the wrapper the models, instead of instantiating the models, I would like to load them from a saved model. How can I do?  The idea is each of the model will be trained separately and saved in the memory using ‘torch.save(model.state_dict(), m_path)’ this command. Now I want to load the saved models and remove the classifier layer. Because, I want to concat the embeddings (hidden layer) for further use. A code snipped of the idea would look like following,</p>
<pre><code class="lang-auto">class wrapper(nn.Module):
    def __init__(self, input_size, hidden_size, number_of_vars):
        super().__init__()
        self.module_list = nn.ModuleList()
        for idx in range(number_of_vars):
            model = var()
            model.load_state_dict(torch.load(m_path))
            model.eval()
            # remove last layer somehow don't know how? 
            self.module_list.append(model)
</code></pre>
<p>The forward would be as usual!  Thanks in advance</p>

username: ptrblck
cooked: <p>You can “remove” the last linear layer by replacing them with an <code>nn.Identity</code> layer via:</p>
<pre><code class="lang-python">model = var()
model.load_state_dict(torch.load(m_path))
model.last_linear_layer = nn.Identity()
...
</code></pre>
<p>Alternatively, you could also write a custom model implementation and skip this layer entirely, but based on your usage the replacement with <code>Identity</code> would be the simplest approach.</p>

username: AbraMunna
cooked: <p>Thank you very much for your gracious help <a class="mention" href="/u/ptrblck">@ptrblck</a></p>