Print Autograd Graph

mattyd2: <p>Is there a way to visualize the graph of a model similar to what Tensorflow offers?</p>

fmassa: <p>There will be tensorboard integration in the future.<br>
For the moment, you can use the <code>visualize</code> function from <a href="https://github.com/szagoruyko/functional-zoo/blob/master/visualize.py">https://github.com/szagoruyko/functional-zoo/blob/master/visualize.py</a><br>
and an example on how to use it can be found in <a href="https://github.com/szagoruyko/functional-zoo/blob/master/resnet-18-export.ipynb">here</a></p>

wangg12: <p>I’ve made a simpler example to visualize resnet-18 by using the <code>visualize.py</code> as <a class="mention" href="/u/fmassa">@fmassa</a> mentioned.<br>
See <a href="https://gist.github.com/wangg12/f11258583ffcc4728eb71adc0f38e832" rel="nofollow noopener">https://gist.github.com/wangg12/f11258583ffcc4728eb71adc0f38e832</a>.</p>

zym1010: <p>I tried your code snippet. However, it doesn’t seem to visualize ResNet correctly… I also tried AlexNet, VGG-19, and same story…</p>

wangg12: <p>What is your result? What is the problem specifically? <a class="mention" href="/u/zym1010">@zym1010</a></p>

zym1010: <p>code:</p>
<pre data-code-wrap="python"><code class="lang-python">%matplotlib inline
from graphviz import Digraph
import re
import torch
import torch.nn.functional as F
from torch.autograd import Variable
from torch.autograd import Variable
import torchvision.models as models


def make_dot(var):
    node_attr = dict(style='filled',
                     shape='box',
                     align='left',
                     fontsize='12',
                     ranksep='0.1',
                     height='0.2')
    dot = Digraph(node_attr=node_attr, graph_attr=dict(size="12,12"))
    seen = set()

    def add_nodes(var):
        if var not in seen:
            if isinstance(var, Variable):
                value = '('+(', ').join(['%d'% v for v in var.size()])+')'
                dot.node(str(id(var)), str(value), fillcolor='lightblue')
            else:
                dot.node(str(id(var)), str(type(var).__name__))
            seen.add(var)
            if hasattr(var, 'previous_functions'):
                for u in var.previous_functions:
                    dot.edge(str(id(u[0])), str(id(var)))
                    add_nodes(u[0])
    add_nodes(var.creator)
    return dot


inputs = torch.randn(1,3,224,224)
resnet18 = models.resnet18()
y = resnet18(Variable(inputs))
print(y)

g = make_dot(y)
g
</code></pre>
<p>result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/c7e0a44b7bcebfb41315b56f8418ce37f0adbfeb.png" data-download-href="https://discuss.pytorch.org/uploads/default/c7e0a44b7bcebfb41315b56f8418ce37f0adbfeb" title="Screen Shot 2017-04-16 at 12.38.08 PM.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/c7e0a44b7bcebfb41315b56f8418ce37f0adbfeb_2_415x500.png" width="415" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/c7e0a44b7bcebfb41315b56f8418ce37f0adbfeb_2_415x500.png, https://discuss.pytorch.org/uploads/default/optimized/1X/c7e0a44b7bcebfb41315b56f8418ce37f0adbfeb_2_622x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/1X/c7e0a44b7bcebfb41315b56f8418ce37f0adbfeb_2_830x1000.png 2x" data-dominant-color="F1F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2017-04-16 at 12.38.08 PM.png</span><span class="informations">952×1146 48.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>definitely, the result doesn’t seem like a ResNet… similar things happen for AlexNet, VGG, etc.</p>

jekbradbury: <p>Yes, the visualization code is currently broken for convnets because certain layers have C++ implementations that don’t expose the graph pointers to Python. It’ll be fixed after the autograd refactor.</p>

wangg12: <aside class="quote no-group" data-username="zym1010" data-post="6" data-topic="692">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zym1010/48/3443_2.png" class="avatar"> zym1010:</div>
<blockquote>
<p>g = make_dot(y)</p>
</blockquote>
</aside>
<p>This is my result for <code>resnet18()</code>.<br>
I can not find any problem. <a class="mention" href="/u/zym1010">@zym1010</a><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/aca25b6684190e89d3e148021d928bc0d9d53ae0.png" data-download-href="https://discuss.pytorch.org/uploads/default/aca25b6684190e89d3e148021d928bc0d9d53ae0" title="image.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/aca25b6684190e89d3e148021d928bc0d9d53ae0_2_690x119.png" width="690" height="119" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/aca25b6684190e89d3e148021d928bc0d9d53ae0_2_690x119.png, https://discuss.pytorch.org/uploads/default/optimized/1X/aca25b6684190e89d3e148021d928bc0d9d53ae0_2_1035x178.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/1X/aca25b6684190e89d3e148021d928bc0d9d53ae0_2_1380x238.png 2x" data-dominant-color="F4F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image.png</span><span class="informations">2226×384 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

zym1010: <p>maybe different pytorch version? <a class="mention" href="/u/jekbradbury">@jekbradbury</a></p>

varun-suresh: <p>I’m having the same problem as <a class="mention" href="/u/zym1010">@zym1010</a> . Is this because I’m using a different pytorch version? If not, can someone suggest how to go about fixing this?</p>

albanD: <p><a class="mention" href="/u/varun-suresh">@varun-suresh</a> this problem should be fixed in master.</p>

platero: <p>Same problem here. Any solutions?</p>

varun-suresh: <p>I built PyTorch from source. I now get a <code>KeyError</code> when I try to build the graph. I’m using <a href="https://github.com/szagoruyko/functional-zoo/blob/master/visualize.py" rel="nofollow noopener">this</a> script to build my graph. I’m trying to generate a graph for <a href="https://github.com/bamos/densenet.pytorch/blob/master/densenet.py" rel="nofollow noopener">this</a> network.</p>

lantiga: <p>What’s the key that fails?</p>

moskomule: <p>I built from the latest master and change <code>var.creator</code> to <code>var.grad_fn</code> (because there is no <code>creator</code> in master’s <code>Variable</code> according to <a href="https://github.com/szagoruyko/functional-zoo/blob/master/visualize.py" rel="nofollow noopener">https://github.com/szagoruyko/functional-zoo/blob/master/visualize.py</a>) but it still does not  draw like <a class="mention" href="/u/wangg12">@wangg12</a> shows.</p>

moskomule: <p>The script <a class="mention" href="/u/fmassa">@fmassa</a> showed “worked” with a bit modification bellow, though I think it is not what we need… (I mean not “Threshold…” but “Relu” etc are better.)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/1b224166ff911f22751905d43c1217eb052a1099.png" data-download-href="https://discuss.pytorch.org/uploads/default/1b224166ff911f22751905d43c1217eb052a1099" title="Digraph.gv.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/1b224166ff911f22751905d43c1217eb052a1099_2_219x499.png" width="219" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/1b224166ff911f22751905d43c1217eb052a1099_2_219x499.png, https://discuss.pytorch.org/uploads/default/optimized/1X/1b224166ff911f22751905d43c1217eb052a1099_2_328x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/1X/1b224166ff911f22751905d43c1217eb052a1099_2_438x998.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Digraph.gv.png</span><span class="informations">856×1950 132 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">from graphviz import Digraph
import torch
from torch.autograd import Variable


def make_dot(var, params):
    """ Produces Graphviz representation of PyTorch autograd graph
    
    Blue nodes are the Variables that require grad, orange are Tensors
    saved for backward in torch.autograd.Function
    
    Args:
        var: output Variable
        params: dict of (name, Variable) to add names to node that
            require grad (TODO: make optional)
    """
    param_map = {id(v): k for k, v in params.items()}
    print(param_map)
    
    node_attr = dict(style='filled',
                     shape='box',
                     align='left',
                     fontsize='12',
                     ranksep='0.1',
                     height='0.2')
    dot = Digraph(node_attr=node_attr, graph_attr=dict(size="12,12"))
    seen = set()
    
    def size_to_str(size):
        return '('+(', ').join(['%d'% v for v in size])+')'

    def add_nodes(var):
        if var not in seen:
            if torch.is_tensor(var):
                dot.node(str(id(var)), size_to_str(var.size()), fillcolor='orange')
            elif hasattr(var, 'variable'):
                u = var.variable
                node_name = '%s\n %s' % (param_map.get(id(u)), size_to_str(u.size()))
                dot.node(str(id(var)), node_name, fillcolor='lightblue')
            else:
                dot.node(str(id(var)), str(type(var).__name__))
            seen.add(var)
            if hasattr(var, 'next_functions'):
                for u in var.next_functions:
                    if u[0] is not None:
                        dot.edge(str(id(u[0])), str(id(var)))
                        add_nodes(u[0])
            if hasattr(var, 'saved_tensors'):
                for t in var.saved_tensors:
                    dot.edge(str(id(t)), str(id(var)))
                    add_nodes(t)
    add_nodes(var.grad_fn)
    return dot

from torchvision import models
inputs = torch.randn(1,3,224,224)
resnet18 = models.resnet18()
y = resnet18(Variable(inputs))
# print(y)

g = make_dot(y, resnet18.state_dict())
g.view()
</code></pre>

hyqneuron: <p>My visualization is cut off by <code>BatchNormBackward</code>,  which, like jekbradbury suggested, doesn’t properly expose the <code>Function</code> interface. I am using the wheel build, which apparently isn’t in sync with the source repo (I’m still using <code>var.creator</code> here).</p>
<p>Do things work they way they’re supposed to in the source? Should we consider adding a proper interface to all <code>Function</code> objects to support visualization?</p>

hyqneuron: <p>In case anyone’s still looking, <a href="https://gist.github.com/hyqneuron/caaf6087162f1c6361571ee489da260f#file-pytorch_visualize-py" rel="nofollow noopener">here</a> is another implementation that’s a bit more robust, as it does not depend on the autograd backward interface, which seems to be undergoing quite a bit of change. It also gets around the problem of BatchNorm C backend not supporting the python interface.</p>

elbamos: <p><a class="mention" href="/u/hyqneuron">@hyqneuron</a> I’ve had really good results with your vis implementation, but it runs into trouble with functions that can take a varying number of inputs.  Are you going to be maintaining and updating the code going forward?</p>

ahmedmazari: <p>Hello <a class="mention" href="/u/moskomule">@moskomule</a> &amp; <a class="mention" href="/u/zym1010">@zym1010</a>,</p>
<p>thank you for code.<br>
l’m wondering if l can do the same with my model ?<br>
l tried the following :</p>
<pre><code>model = crnn.CRNN(32, 1, 37,256, 1).cuda() # it's my model 
print(model) 

CRNN (
  (cnn): Sequential (
    (conv0): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu0): ReLU (inplace)
    (pooling0): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu1): ReLU (inplace)
    (pooling1): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (conv2): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (batchnorm2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
    (relu2): ReLU (inplace)
    (conv3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu3): ReLU (inplace)
    (pooling2): MaxPool2d (size=(2, 2), stride=(2, 1), dilation=(1, 1))
    (conv4): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (batchnorm4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
    (relu4): ReLU (inplace)
    (conv5): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu5): ReLU (inplace)
    (pooling3): MaxPool2d (size=(2, 2), stride=(2, 1), dilation=(1, 1))
    (conv6): Conv2d(512, 512, kernel_size=(2, 2), stride=(1, 1))
    (batchnorm6): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
    (relu6): ReLU (inplace)
  )
  (rnn): Sequential (
    (0): BidirectionalLSTM (
      (rnn): LSTM(512, 256, bidirectional=True)
      (embedding): Linear (512 -&gt; 256)
    )
    (1): BidirectionalLSTM (
      (rnn): LSTM(256, 256, bidirectional=True)
      (embedding): Linear (512 -&gt; 37)
    )
  )
)
</code></pre>
<p>However l fail to visualize the graph<br>
l followed exactly the same steps as you’ve defined them :</p>
<pre><code>inputs = torch.randn(1,3,224,224)
model = crnn.CRNN(32, 1, 37,256, 1).cuda()
**y = model(Variable(inputs)) # l got an error at this line** 
print(y)
g = make_dot(y)
g 
</code></pre>
<p>after executing <strong>y</strong>  l got the following error</p>
<pre><code>======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ff2608057e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7ff26080e37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7ff26081253c]
/home/ahmed/anaconda3/envs/cv/lib/python2.7/site-packages/../../libstdc++.so.6(_ZNSt15basic_stringbufIcSt11char_traitsIcESaIcEE8overflowEi+0x13b)[0x7ff24918604b]
/home/ahmed/anaconda3/envs/cv/lib/python2.7/site-packages/../../libstdc++.so.6(_ZNSt15basic_streambufIcSt11char_traitsIcEE6xsputnEPKcl+0x36)[0x7ff24918a1b6]
/home/ahmed/anaconda3/envs/cv/lib/python2.7/site-packages/torch/lib/libshm.so(_ZSt16__ostream_insertIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_PKS3_l+0x1c5)[0x7ff23b391235]
/home/ahmed/anaconda3/envs/cv/lib/python2.7/site-packages/torch/_C.so(+0x5d2842)[0x7ff23bc12842]
/home/ahmed/anaconda3/envs/cv/lib/python2.7/site-packages/torch/_C.so(+0x5d34ae)[0x7ff23bc134ae]
/home/ahmed/anaconda3/envs/cv/lib/python2.7/site-packages/torch/_C.so(_ZN5torch2nn33SpatialConvolutionMM_updateOutputEPN4thpp6TensorES3_S3_S3_S3_S3_iiiiii+0xb3)[0x7ff23bc271a3]
/home/ahmed/anaconda3/envs/cv/lib/python2.7/site-packages/torch/_C.so(+0x5caf27)[0x7ff23bc0af27]
/home/ahmed/anaconda3/envs/cv/lib/python2.7/site-packages/torch/_C.so(_ZN5torch8autograd11ConvForward5applyERKSt6vectorISt10shared_ptrINS0_8VariableEESaIS5_EE+0x17bf)[0x7ff23bc0f65f]
/home/ahmed/anaconda3/envs/cv/lib/python2.7/site-packages/torch/_C.so(+0x5c191b)[0x7ff23bc0191b]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7ff2614cee93]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x715d)[0x7ff26158180d]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7ff261583c3e]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8b47)[0x7ff2615831f7]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7ff261583c3e]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(+0x79b68)[0x7ff2614feb68]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7ff2614cee93]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x61d6)[0x7ff261580886]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7ff261583c3e]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(+0x79a61)[0x7ff2614fea61]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7ff2614cee93]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(+0x5c64f)[0x7ff2614e164f]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7ff2614cee93]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(+0xba2ac)[0x7ff26153f2ac]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7ff2614cee93]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x715d)[0x7ff26158180d]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7ff261583c3e]
/home/ahmed/anaconda3/envs/cv/bin/../lib/libpython2.7.so.1.0(+0x79b68)[0x7ff2614feb68]

7ff261a9e000-7ff261a9f000 rw-s 133756000 00:06 541                       /dev/nvidiactl
7ff261a9f000-7ff261aa0000 rw-s 9fee8000 00:06 542                        /dev/nvidia0
7ff261aa0000-7ff261aa1000 rw-s 13372f000 00:06 541                       /dev/nvidiactl
7ff261aa1000-7ff261aa2000 rw-s 9fee8000 00:06 542                        /dev/nvidia0
7ff261aa2000-7ff261aa3000 rw-s 133790000 00:06 541                       /dev/nvidiactl
7ff261aa3000-7ff261aa4000 rwxp 00000000 00:00 0 
7ff261aa4000-7ff261aa6000 rw-p 00000000 00:00 0 
7ff261aa6000-7ff261aa7000 r--p 00025000 08:01 2101713                    /lib/x86_64-linux-gnu/ld-2.23.so
7ff261aa7000-7ff261aa8000 rw-p 00026000 08:01 2101713                    /lib/x86_64-linux-gnu/ld-2.23.so
7ff261aa8000-7ff261aa9000 rw-p 00000000 00:00 0 
7fffff3a5000-7fffff3c8000 rwxp 00000000 00:00 0                          [stack]
7fffff3c8000-7fffff3ca000 rw-p 00000000 00:00 0 
7fffff3ed000-7fffff3ef000 r--p 00000000 00:00 0                          [vvar]
7fffff3ef000-7fffff3f1000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
</code></pre>
<p>Thank you for your help</p>