Network architicture keep printed during training

username: Rami_Nasser
cooked: <p>using Jupyter notebook, the model architecture keep printed during training, how can I stop it ?</p>
<p>example output:</p>
<p>(0): FrNMFLayer(<br>
(fc1): Linear(in_features=39, out_features=39, bias=False)<br>
(fc2): Linear(in_features=96, out_features=39, bias=False)<br>
)<br>
(1): FrNMFLayer(<br>
(fc1): Linear(in_features=39, out_features=39, bias=False)<br>
(fc2): Linear(in_features=96, out_features=39, bias=False)<br>
)<br>
(2): FrNMFLayer(<br>
(fc1): Linear(in_features=39, out_features=39, bias=False)<br>
(fc2): Linear(in_features=96, out_features=39, bias=False)<br>
)<br>
(3): FrNMFLayer(<br>
(fc1): Linear(in_features=39, out_features=39, bias=False)<br>
(fc2): Linear(in_features=96, out_features=39, bias=False)<br>
)<br>
(4): FrNMFLayer(<br>
(fc1): Linear(in_features=39, out_features=39, bias=False)<br>
(fc2): Linear(in_features=96, out_features=39, bias=False)<br>
)<br>
(5): FrNMFLayer(<br>
(fc1): Linear(in_features=39, out_features=39, bias=False)<br>
(fc2): Linear(in_features=96, out_features=39, bias=False)</p>
<p>and so on …</p>

username: BaruchG
cooked: <p>Can you please post your training loop?</p>

username: Rami_Nasser
cooked: <pre><code>for i in range(train_iter):
    out = model(*inputs)
    loss = criterion(out, h_train)  # loss between predicted and truth

    optimizerADAM.zero_grad()
    loss.backward()
    optimizerADAM.step()

    model.apply(constraints)  # keep wieghts positive after gradient decent
    loss_values.append(loss.item())</code></pre>

username: BaruchG
cooked: <p>I would guess that you have something in your <code>constraints</code>  function which is causing it to print out.  That’s the most likely to me at least, everything else is pretty standard.<br>
For example this will print out on <code>apply</code>:</p>
<pre><code class="lang-auto">import torch

@torch.no_grad()
def init_weights(m):
  print(m)
  if type(m) == nn.Linear:
    m.weight.fill_(1.0)
    # print(m.weight)
net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))
n = net.apply(init_weights);
</code></pre>
<p>will output</p>
<pre><code class="lang-auto">Linear(in_features=2, out_features=2, bias=True)
Linear(in_features=2, out_features=2, bias=True)
Sequential(
  (0): Linear(in_features=2, out_features=2, bias=True)
  (1): Linear(in_features=2, out_features=2, bias=True)
)
</code></pre>
<p>which is similar to what you are seeing.  You would just need to get rid of the print statement and you should be good. Of course if anything in those other methods has been modified it could they could be the culprit as well.</p>

username: Rami_Nasser
cooked: <p>OK, so the apply function what causes the printing, how this can be turned off ?</p>

username: BaruchG
cooked: <p>It’s not the <code>apply</code> function itself, it’s probably the <code>constraints</code> function which you are passing into <code>model.apply()</code>, but without seeing it I can’t know for sure.  For example, in my code above the <code>init_weights</code> function is being passed into <code>apply</code>, you should have a similar function called <code>constraints</code> which is getting passed into <code>apply</code>.  You can see more information on how this works at <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.apply" rel="nofollow noopener">https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.apply</a></p>

username: Rami_Nasser
cooked: <p>This is my constrains func, simply a weight clipper</p>
<pre><code class="lang-auto">class WeightClipper(object):
    def __init__(self, lower=-inf, upper=inf):
        self.lower = lower
        self.upper = upper

    def __call__(self, module):
        # filter the variables to get the ones you want
        if hasattr(module, "weight"):
            w = module.weight.data
            w = w.clamp(min=self.lower, max=self.upper)
            module.weight.data = w
</code></pre>

username: BaruchG
cooked: <p>If it’s not in there then I’m not sure where it’s coming from.  You can troubleshoot it by commenting out all lines and then uncommenting one at a time to see what causes it.</p>