Question on CustomFuseGraph (graph_fuser). pytorch/tvm , pytorch/glow

username: vdantu
cooked: <p>Hi all,<br>
I have some basic questions on CustomFuseGraph and its relation to other projects. I see that <strong><a href="https://github.com/pytorch/tvm/blob/master/torch_tvm/fusion_pass.cpp#L119" rel="nofollow noopener">pytorch/tvm</a></strong> and <strong><a href="https://github.com/pytorch/glow/blob/master/torch_glow/src/GlowFuser.cpp#L131" rel="nofollow noopener">pytorch/glow</a></strong> are using different custom fuse passes and not using the <strong>CustomFuseGraph</strong>. Could someone explain why they do not use the graph_fuser provided by PyTorch? Sorry if this is a naive question.</p>
<p>Thanks</p>

username: tom
cooked: <p>Traditionally, the graph fuser in PyTorch has been operating on a per-<code>Block</code> basis. This is inherited (at least it used to a few weeks ago) by the CustomFuseGraph.<br>
PyTorch TVM (and probably also glow) are interested in fusing across blocks, e.g. to also fuse control flow nodes. This is why <a href="https://github.com/pytorch/tvm/pull/72" rel="nofollow noopener">PyTorch TVM has switched away from CustomFuseGraph</a>. Personally, I would expect that custom fusion mechanism will be folded back into PyTorch when the picture of what users need there becomes clearer (just like CustomFuseGraph was introduced when one thought that one could also make good use of that, I think I recall that originally we thought that the CustomFuseGraph might already work for PyTorch TVM).</p>
<p>Best regards</p>
<p>Thomas</p>

username: vdantu
cooked: <p><a class="mention" href="/u/tom">@tom</a>: Awesome. Thanks for taking the time and responding. I think that answers my question to a great extent. So, is the plan to upstream these changes to the CustomFuseGraph? Also, the torch/tvm’s custom fusion seems to ignore the control flow ops for now <a href="https://github.com/pytorch/tvm/blob/master/torch_tvm/fusion_pass.cpp#L30" rel="nofollow noopener">seen in fusion_pass.cpp</a>.</p>
<p>I was also experimenting a bit and see that fusing blocks of control flow ops is not currently handled in <strong>subgraph_utils</strong>, is this understanding right ?</p>
<p>Lets take this graph for example</p>
<pre><code class="lang-auto">graph(%x.1 : Float(*)):
  %18 : Float(1) = prim::Constant[value={1}]()
  %16 : int[] = prim::Constant[value=[1]]()
  %1 : None = prim::Constant()
  %2 : int = prim::Constant[value=1]() # script_module.py:64:22
  %3 : int = prim::Constant[value=2]() # script_module.py:68:20
  %4 : int = prim::Constant[value=3]() # script_module.py:71:18
  %ret.1 : Double(*) = aten::zeros(%16, %1, %1, %1, %1) # script_module.py:64:10
  %7 : Bool(*) = aten::eq(%x.1, %2) # script_module.py:65:7
  %8 : bool = aten::Bool(%7) # script_module.py:65:7
  %ret : Tensor(*) = prim::If(%8) # script_module.py:65:4
    block0():
      %12 : Tensor = aten::add_(%ret.1, %18, %2) # script_module.py:67:8
      %ret.4 : Double(*) = aten::mul(%ret.1, %3) # script_module.py:68:14
      %ret.7 : Double(*) = aten::add(%ret.4, %18, %2) # script_module.py:69:14
      -&gt; (%ret.7)
    block1():
      %ret.9 : Float(*) = aten::add(%x.1, %4, %2) # script_module.py:71:14
      -&gt; (%ret.9)
  return (%ret)
</code></pre>
<p>Here, the first node <code>aten::add_()</code> inside  <code>block0()</code> of  <code>prim::If</code> has one of its inputs <code>%ret.1</code> which is coming from outside the <code>prim::If</code>. But when cloning this node, during <code>mergeNodeIntoSubgraph</code> its not able to find the metadata of this input node. I think its because when cloning, the <code>value_map</code> only has the inputs in block scope and not the graph scope. I am not very familiar with the PT graph manipulation to understand the reason why only block’s inputs are added to value_map in <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/ir.cpp#L632" rel="nofollow noopener">ir.cpp</a> and only the prim::If’s inputs are added to the value map in <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/passes/utils/subgraph_utils.cpp#L70" rel="nofollow noopener">subgraph_utils</a> . The error seems to come because the <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/ir.cpp#L1586" rel="nofollow noopener">value_map(i) in ir.cpp</a> returns a NULL as the <code>ret.1</code> is not in the <code>value_map</code>.</p>
<p>Is there an example in code  where fusion of control flow ops is handled? Also, please correct me if my understanding is wrong.</p>

username: tom
cooked: <p>So I don’t know how to answer most of your questions and I haven’t looked in a while relative to the rate of change. Apparently, something is not quite there yet for the control flow in torch TVM.<br>
For the fusion in the fusion pass, blocks is not currently handled, and I am ignorant of whether the obstacle torch TVM hit was with the graph rearrangement itself or something else.<br>
to the best of my knowledge we don’t currently fuse inplace ops, so I would expect adventure when you try that. Maybe that is part of the problem you are seeing.</p>
<p>I don’t know any examples beyond the obvious users. In theory, I would expect the optimizations removing ifs with constant true/false result to give some idea of what it would take to move ops with blocks into a subgraph.</p>
<p>I’ll be very interested to hear from your progress.</p>
<p>Best regards</p>
<p>Thomas</p>

username: bwasti
cooked: <p>Yep, your understanding is correct. Control flow is currently unhandled.</p>

username: vdantu
cooked: <p><a class="mention" href="/u/bwasti">@bwasti</a>:  Thanks for your response. Could you let me know if my understanding <a href="https://discuss.pytorch.org/t/question-on-customfusegraph-graph-fuser-pytorch-tvm-pytorch-glow/58627/3">above</a> is correct? I just want to understand if fusion of control flow ops into the custom op node is even allowed by PT.</p>