How to train several network at the same time?

username: Dylon
cooked: <p>Hi, I am just a beginner of PyTorch, even a beginner of Python. So this question may too simple for you but I really want to get some help.<br>
I want to train a neural network which is included sub-networks. It looks something like that.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/754642de2628de39bc3d0edc8924ee73a3003b32.png" data-download-href="https://discuss.pytorch.org/uploads/default/754642de2628de39bc3d0edc8924ee73a3003b32" title="nnp.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/754642de2628de39bc3d0edc8924ee73a3003b32_2_418x500.png" width="418" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/754642de2628de39bc3d0edc8924ee73a3003b32_2_418x500.png, https://discuss.pytorch.org/uploads/default/original/1X/754642de2628de39bc3d0edc8924ee73a3003b32.png 1.5x, https://discuss.pytorch.org/uploads/default/original/1X/754642de2628de39bc3d0edc8924ee73a3003b32.png 2x" data-dominant-color="EBF3F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nnp.png</span><span class="informations">468×559 72.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In my opinion, this network could be implemented by constructing three simple fully-connection neural networks.<br>
My implementation is:</p>
<pre><code>#firstly, define the simple NN
class Net(nn.Module):
    def __init__(self,n_input,n_hidden1,n_hidden2):
        super(Net, self).__init__()
        self.hidden1 = nn.Linear(n_input,n_hidden1)
        self.hidden2 = nn.Linear(n_hidden1,n_hidden2)
        self.output  = nn.Linear(n_hidden2,1)

    def forward(self,x):
    x = F.tanh(self.hidden1(x))
    x = F.tanh(self.hidden2(x))
    x = self.output(x)
    return x

#construct 3 same nets
net = []
for i in range(3):
    net.append(Net(3,5,5))


# define loss function and optimizer
optimizer = []
for i in range(3):
    optimizer.append(torch.optim.SGD(net[i].parameters(), lr = 0.2))
loss_func = torch.nn.MSELoss()

#training neural network
for step in range(300):  #training circles
    for i in range(3):
        output[:, i] = net[i](input[i, :, :])
    prediction = torch.sum(output,1)

    loss = loss_func(prediction,target)
    for i in range(3):
         optimizer[i].zero_grad()
    loss.backward(retain_variables=True)
    for i in range(3):
        optimizer[i].step()
</code></pre>
<p>after several circles, all output value become to the same value.<br>
There must something wrong with my code. Could anyone give me some help?<br>
Or are there any better way to implement this network?<br>
Thanks a lot.</p>

username: hughperkins
cooked: <p>according to your diagram, you want to train one network that contains 3 sub networks?</p>
<p>but in your code, you are creating 3 separate optimizers.</p>
<p>I cant help wondering:</p>
<ol>
<li>why are you creating 3 optimiziers?</li>
<li>what happens if you use just one optimizer instead?</li>
</ol>

username: Dylon
cooked: <p>Oh, I have tried using only one optimizer. But<br>
net = []<br>
is a list but not torch.autograd.Variable, So there are no parameters in net. PyTorch reported as an error for one optimizer.</p>
<p>That’s why I used 3 optimizers.</p>

username: hughperkins
cooked: <p>sure. well, you have a couple of options. Because I htink you should be using only one optimizer</p>
<ul>
<li>
<p>the more idiomatic solution might be to create another nn.Module child class, to hold the three Net modules.  The rest should be straightforward</p>
</li>
<li>
<p>otherwise you can simply do something like:</p>
<p>parameters = set()<br>
for net_ in net:<br>
parameters |= set(net_.parameters())</p>
</li>
</ul>

username: Dylon
cooked: <p>Thanks for replying, I will try these solutions and will give you a feedback.</p>

username: Dylon
cooked: <p>Hi, I have just tried the second solution. It works well!</p>
<p>But the problem, output always become same value, still exists.</p>
<p>Fortunately, I solve this problem by adding an extra Batch_Normalization layer.</p>
<p>Thank you very much!</p>

username: McLawrence
cooked: <p>Would the network learn different weights when using these seperate optimizers which all optimize the same loss, or is it just a matter of style or performance?</p>

username: abragantini
cooked: <p>Hi, I would like to share my experience in case it might be helpful for other newbies (like me) around here.<br>
I created a bigger NN model (nn.Module child class) containing 2 smaller subnetworks.<br>
I just defined first the simple subnetwork as a nn.module child class.<br>
Then in the <strong>init</strong> method definition of the bigger model class, I just called/defined the 2 submodels.<br>
They appear of course also in the forward method (with same input x in my case).<br>
You can then add another layer to merge the solutions coming from subnetworks, in case you need.<br>
Training goes than as usual.</p>

username: OuisYasser
cooked: <p>Please can you share the code ? I’m beginner so I’m trying to understand codes and thank you</p>

username: erenovic
cooked: <p>Hello Andrea, I have tried implementing the solution you have offered but something is bugging me.</p>
<p>In my case, I have two seperate networks as subnetworks and their output form a loss function together. I have initialized those subnetworks in a larger nn.Module child class and included them in the forward.</p>
<p>However, my reason for applying this was that I have a single GPU which seems to be under allocated (4 GiB allocation in a 32 GiB GPU) and I don’t want to allocate another GPU. But it seems like I am having problems at parallelization of my subnetworks. I was hoping that PyTorch internally allows parallelization by building the computation graph but it seems like something is wrong. My training with two subnetworks takes more than twice duration. Have you observed a similar situation?</p>