Tensorboard issue with self-defined forward function

promach: <p>How to get around the following <a href="https://gist.github.com/promach/b6f526c56e20f029d68e6f9041c3f5c0#file-gdas-py-L357" rel="noopener nofollow ugc">tensorboard issue with self-defined forward function</a> ?</p>
<pre><code class="lang-python">/home/phung/miniconda3/envs/py39/bin/python3.9 /home/phung/PycharmProjects/beginner_tutorial/gdas.py
Files already downloaded and verified
Files already downloaded and verified
run_num =  0

Error occurs, No graph saved
Traceback (most recent call last):
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 770, in &lt;module&gt;
    ltrain = train_NN(forward_pass_only=0)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 357, in train_NN
    writer.add_graph(graph, train_inputs)
  File "/home/phung/miniconda3/envs/py39/lib/python3.9/site-packages/torch/utils/tensorboard/writer.py", line 736, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose, use_strict_trace))
  File "/home/phung/miniconda3/envs/py39/lib/python3.9/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 297, in graph
    raise e
  File "/home/phung/miniconda3/envs/py39/lib/python3.9/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 291, in graph
    trace = torch.jit.trace(model, args, strict=use_strict_trace)
  File "/home/phung/miniconda3/envs/py39/lib/python3.9/site-packages/torch/jit/_trace.py", line 741, in trace
    return trace_module(
  File "/home/phung/miniconda3/envs/py39/lib/python3.9/site-packages/torch/jit/_trace.py", line 958, in trace_module
    module._c._create_method_from_trace(
  File "/home/phung/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/phung/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1098, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/phung/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 201, in _forward_unimplemented
    raise NotImplementedError
NotImplementedError

Process finished with exit code 1
</code></pre>

ptrblck: <p>Based on the error message it doesn’t look like a Tensorboard issue but it seems as if your <code>forward</code> method is undefined in the used model.</p>

promach: <p>yes and I added <a href="https://gist.github.com/promach/b6f526c56e20f029d68e6f9041c3f5c0#file-gdas-py-L298" rel="noopener nofollow ugc">dummy forward function</a> to every class hierarchy in my code which helps.</p>
<p>Now, I am facing the following Tensorboard error:</p>
<pre><code class="lang-auto">Error occurs, No graph saved
Traceback (most recent call last):
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 789, in &lt;module&gt;
    ltrain = train_NN(forward_pass_only=0)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 538, in train_NN
    writer.add_graph(graph, train_inputs)
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/utils/tensorboard/writer.py", line 736, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose, use_strict_trace))
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 295, in graph
    raise e
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 289, in graph
    trace = torch.jit.trace(model, args, strict=use_strict_trace)
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/jit/_trace.py", line 741, in trace
    return trace_module(
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/jit/_trace.py", line 958, in trace_module
    module._c._create_method_from_trace(
RuntimeError: Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions

Process finished with exit code 1
</code></pre>

ptrblck: <p>The error is raised as tracing the model fails. Check the error message and make sure the expected types are returned in the <code>forward</code> function.</p>

promach: <p><a href="https://gist.github.com/promach/b6f526c56e20f029d68e6f9041c3f5c0#file-gdas-py-L283" rel="noopener nofollow ugc">This new code</a> seems to have gotten around the self-defined forward function issue, but it has the following new error.  Should I create a new question for this new issue ?</p>
<p>Should I use <a href="https://discuss.pytorch.org/t/when-to-use-add-module-function/10534">add_module()</a> instead of  <a href="https://gist.github.com/promach/b6f526c56e20f029d68e6f9041c3f5c0#file-gdas-py-L279" rel="noopener nofollow ugc">nn.ModuleList()</a> in this case ?</p>
<pre><code class="lang-auto">Error occurs, No graph saved
Traceback (most recent call last):
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 794, in &lt;module&gt;
    ltrain = train_NN(forward_pass_only=0)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 551, in train_NN
    writer.add_graph(graph, NN_input)
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/utils/tensorboard/writer.py", line 736, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose, use_strict_trace))
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 295, in graph
    raise e
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 289, in graph
    trace = torch.jit.trace(model, args, strict=use_strict_trace)
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/jit/_trace.py", line 741, in trace
    return trace_module(
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/jit/_trace.py", line 958, in trace_module
    module._c._create_method_from_trace(
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1090, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 322, in forward
    self.cells[c].nodes[n].connections[
RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient
</code></pre>

Unity05: <p>Hi again,<br>
I think <code>ModuleList</code> should be fine, but the <code>combined_feature_map</code>, <code>output</code> and so on are the problem.</p>

promach: <p><a class="mention" href="/u/unity05">@Unity05</a> How shall I fix <code>combined_feature_map</code> ?</p>

Unity05: <p>I think here the problem really is that this tensor requires gradient but is no parameter. If you try it requires gradient set to False, there should be no problems regarding <code>combined_feature_map</code> and the graph tracing. However, I remember you’ve set requires gradient to True a while ago to solve another issue.</p>

promach: <p><a class="mention" href="/u/unity05">@Unity05</a> I have already set <code>requires_grad=False</code> for <code>combined_feature_map</code> , but still the same runtime error</p>

Unity05: <p>Yes, but that should be a problem because it calls the graph a second time.</p>

promach: <aside class="quote no-group quote-modified" data-username="Unity05" data-post="10" data-topic="140628">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/unity05/48/20944_2.png" class="avatar"> Unity05:</div>
<blockquote>
<p>it class the graph a second time.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/unity05">@Unity05</a>  What do you exactly mean by the above quoted sentence ?</p>

Unity05: <p>Sorry, I meant <code>calls</code> instead of <code>class</code>.</p>

promach: <p><a class="mention" href="/u/unity05">@Unity05</a> wait, why the <strong>second time</strong> ?</p>

Unity05: <p>That’s how even if you set <code>requires_grad=False</code> at initialization, the next calls it’s gonna be True.</p>

promach: <p><a class="mention" href="/u/unity05">@Unity05</a> How do I get around the issue with <strong>the next calls it’s gonna be True.</strong> ?</p>

Unity05: <p>You could for instance just reset the <code>combined_feature_map</code> after an epoch. But spoiler: <code>combined_feature_map</code> is not the only thing with that problem.</p>

promach: <p><a class="mention" href="/u/unity05">@Unity05</a> Thanks for your suggestion.  However, if you look at value of <code>NUM_EPOCHS</code> variable that I set for code development and debugging purpose, I am afraid that this is not really the solution to the runtime error that I am facing now.</p>
<p>In other words, the runtime error happens within 1 epoch itself, not between 2 epochs</p>

Unity05: <p>I’ve just tried it with resetting it in <a href="https://gist.github.com/promach/b6f526c56e20f029d68e6f9041c3f5c0#file-gdas-py-L472" rel="noopener nofollow ugc">line 472</a>, what did work.</p>

promach: <p><a class="mention" href="/u/unity05">@Unity05</a> Would you be able to share your reset modification coding ?</p>

Unity05: <p>If you just want to test it out quickly, try:</p>
<pre><code class="lang-auto">def reset(self):
        self.combined_feature_map = torch.zeros([BATCH_SIZE, NUM_OF_IMAGE_CHANNELS, IMAGE_HEIGHT, IMAGE_WIDTH],
                                                requires_grad=False)
        if USE_CUDA:
            self.combined_feature_map = self.combined_feature_map.cuda()
</code></pre>
<p>in line 211 and</p>
<pre><code class="lang-auto">            for c in range(NUM_OF_CELLS):
                for n in range(NUM_OF_NODES_IN_EACH_CELL):
                    # not all nodes have same number of Type-1 output connection
                    for cc in range(MAX_NUM_OF_CONNECTIONS_PER_NODE - n - 1):
                        self.cells[c].nodes[n].connections[cc].reset()
</code></pre>
<p>in line 472. Line numbers according to your gist. It’s not too pretty but should do its job on that part.</p>