[Documentation] FX module is for Pass Writers

Question:
I am going through the documentation of recently released FX module to Pytorch release 1.10.
Can someone explain what does “pass writers” mean in this context?

FX is a toolkit for pass writers to facilitate Python-to-Python transformation of nn.Module instances.

I tried looking up the term “pass writers” with no relevant result that can tie it back to FX module.

      github.com

    pytorch/pytorch/blob/master/torch/fx/OVERVIEW.md

    # FX Technical Overview (WIP)

FX is a toolkit for pass writers to facilitate Python-to-Python transformation of `nn.Module` instances. This toolkit aims to support a subset of Python language semantics—rather than the whole Python language—to facilitate ease of implementation of transforms. Currently, this feature is under a Beta release and its API may change.

## Table of Contents

<!-- toc -->

- [Introduction](#introduction)
  - [Motivation](#motivation)
  - [Use Cases](#use-cases)
  - [Technical Details](#technical-details)
- [Internal Structure](#internal-structure)
  - [Graph](#graph)
  - [GraphModule](#graphmodule)
- [Symbolic Tracing](#symbolic-tracing)
  - [Tracer](#tracer)
  - [Proxy](#proxy)
- [The FX IR](#the-fx-ir)
- [Transformation and Codegen](#transformation-and-codegen)

  This file has been truncated. show original

Thanks!

Context:
It’s just meant to reference a person (a “writer”) who writes some transformation (a “pass”) for the FX IR. E.g. relevant wiki page on a compiler that does multiple passes:

      en.wikipedia.org

Multi-pass compiler

A multi-pass compiler is a type of compiler that processes the source code or abstract syntax tree of a program several times. This is in contrast to a one-pass compiler, which traverses the program only once. Each pass takes the result of the previous pass as the input, and creates an intermediate output. In this way, the (intermediate) code is improved pass by pass, until the final pass produces the final code.
 Multi-pass compilers are sometimes called wide compilers,[citation needed] referrin...


Using torch.fx.symbolic_trace with view(*shape)

Question:
Hello,
I’m trying to use torch.fx.symbolic_trace to trace the bert.SelfAttention module. However, I’m running into the following error when torch.Tensor.view is called as x = x.view(*new_shape):
torch.fx.proxy.TraceError: Proxy object cannot be iterated. This can be attempted when used in a for loop or as a *args or **kwargs function argument.
Possibly caused due to *new_shape argument to view? I’ve been trying to write a custom Tracer class with a new iter method, but I’m not able to resolve the issue. Is there a workaround to fixing this without modifying the actual self-attention code?
Full code:
from torch.fx import symbolic_trace
from transformers.models.bert import modeling_bert as bert
from transformers.models.bert import BertConfig

bert_config = BertConfig(hidden_size=10, num_attention_heads=5)
attention_module = bert.BertSelfAttention(bert_config)
graph = symbolic_trace(attention_module).graph

Context:
Sometimes, these models can be involved, especially for a library with the genesis and target audience as transformers.
Chance has it that I released an initial commit of the toroidal transformers 1 that has an attention module that works well with symbolic tracing:
m = toroidal.models.Attention(786, 3)
gr = torch.fx.symbolic_trace(m)

or in a BERT model (the multilingual from the test directory):
gr = torch.fx.symbolic_trace(model_bert.blocks[0].attn)

print(gr)

gives
Attention(
  (qkv): Linear(in_features=768, out_features=2304, bias=True)
  (dropout_attn): Dropout(p=0.1, inplace=False)
  (proj): Linear(in_features=768, out_features=768, bias=True)
  (dropout_out): Dropout(p=0.1, inplace=False)
)

def forward(self, x):
    getattr_1 = x.shape
    getitem = getattr_1[0]
    getitem_1 = getattr_1[1]
    getitem_2 = getattr_1[2];  getattr_1 = None
    qkv = self.qkv(x);  x = None
    view = qkv.view(getitem, getitem_1, 3, 12, -1);  qkv = None
    unbind = view.unbind(dim = 2);  view = None
    getitem_3 = unbind[0]
    getitem_4 = unbind[1]
    getitem_5 = unbind[2];  unbind = None
    einsum = torch.functional.einsum('bthc,bshc->bhts', getitem_3, getitem_4);  getitem_3 = getitem_4 = None
    mul = einsum * 0.125;  einsum = None
    softmax = mul.softmax(-1);  mul = None
    dropout_attn = self.dropout_attn(softmax);  softmax = None
    einsum_1 = torch.functional.einsum('bhts,bshc->bthc', dropout_attn, getitem_5);  dropout_attn = getitem_5 = None
    reshape = einsum_1.reshape(getitem, getitem_1, getitem_2);  einsum_1 = getitem = getitem_1 = getitem_2 = None
    proj = self.proj(reshape);  reshape = None
    dropout_out = self.dropout_out(proj);  proj = None
    return dropout_out

and I guess this is the disadvantage for manipulating the matmuls: I used einsum instead.
If GPT works for you, you could check out minGPT, I’d imagine that A. Karpathy’s code might work more easily with the tracer.
Best regards
Thomas


Question:
Hi there,
I’m trying to run the fx profiling tutorial in tutorials/fx_profiling_tutorial.py at master · pytorch/tutorials · GitHub 1 on a single nn.TransformerEncoderLayer as opposed to the resnet in the example and I keep running into a StopIteration error. Why is this happening? All I did was replace the resnet with a transformer encoder layer. Here is the code:
# -*- coding: utf-8 -*-
"""
(beta) Building a Simple CPU Performance Profiler with FX
*******************************************************
**Author**: `James Reed <https://github.com/jamesr66a>`_

In this tutorial, we are going to use FX to do the following:

1) Capture PyTorch Python code in a way that we can inspect and gather
   statistics about the structure and execution of the code
2) Build out a small class that will serve as a simple performance "profiler",
   collecting runtime statistics about each part of the model from actual
   runs.

"""

######################################################################
# For this tutorial, we are going to use the torchvision ResNet18 model
# for demonstration purposes.

import torch
import torch.fx
import torchvision.models as models

model = torch.nn.TransformerEncoderLayer(d_model=512, nhead=8)
model.eval()

######################################################################
# Now that we have our model, we want to inspect deeper into its
# performance. That is, for the following invocation, which parts
# of the model are taking the longest?
input = torch.randn(10, 32, 512)
output = model(input)

######################################################################
# A common way of answering that question is to go through the program
# source, add code that collects timestamps at various points in the
# program, and compare the difference between those timestamps to see
# how long the regions between the timestamps take.
#
# That technique is certainly applicable to PyTorch code, however it
# would be nicer if we didn't have to copy over model code and edit it,
# especially code we haven't written (like this torchvision model).
# Instead, we are going to use FX to automate this "instrumentation"
# process without needing to modify any source.

######################################################################
# First, let's get some imports out of the way (we will be using all
# of these later in the code).

import statistics, tabulate, time
from typing import Any, Dict, List
from torch.fx import Interpreter

######################################################################
# .. note::
#     ``tabulate`` is an external library that is not a dependency of PyTorch.
#     We will be using it to more easily visualize performance data. Please
#     make sure you've installed it from your favorite Python package source.

######################################################################
# Capturing the Model with Symbolic Tracing
# -----------------------------------------
# Next, we are going to use FX's symbolic tracing mechanism to capture
# the definition of our model in a data structure we can manipulate
# and examine.

# traced_rn18 = torch.fx.symbolic_trace(rn18)
# print(traced_rn18.graph)

######################################################################
# This gives us a Graph representation of the ResNet18 model. A Graph
# consists of a series of Nodes connected to each other. Each Node
# represents a call-site in the Python code (whether to a function,
# a module, or a method) and the edges (represented as ``args`` and ``kwargs``
# on each node) represent the values passed between these call-sites. More
# information about the Graph representation and the rest of FX's APIs ca
# be found at the FX documentation https://pytorch.org/docs/master/fx.html.

######################################################################
# Creating a Profiling Interpreter
# --------------------------------
# Next, we are going to create a class that inherits from ``torch.fx.Interpreter``.
# Though the ``GraphModule`` that ``symbolic_trace`` produces compiles Python code
# that is run when you call a ``GraphModule``, an alternative way to run a
# ``GraphModule`` is by executing each ``Node`` in the ``Graph`` one by one. That is
# the functionality that ``Interpreter`` provides: It interprets the graph node-
# by-node.
#
# By inheriting from ``Interpreter``, we can override various functionality and
# install the profiling behavior we want. The goal is to have an object to which
# we can pass a model, invoke the model 1 or more times, then get statistics about
# how long the model and each part of the model took during those runs.
#
# Let's define our ``ProfilingInterpreter`` class:

class ProfilingInterpreter(Interpreter):
    def __init__(self, mod : torch.nn.Module):
        # Rather than have the user symbolically trace their model,
        # we're going to do it in the constructor. As a result, the
        # user can pass in any ``Module`` without having to worry about
        # symbolic tracing APIs
        gm = torch.fx.symbolic_trace(mod)
        super().__init__(gm)

        # We are going to store away two things here:
        #
        # 1. A list of total runtimes for ``mod``. In other words, we are
        #    storing away the time ``mod(...)`` took each time this
        #    interpreter is called.
        self.total_runtime_sec : List[float] = []
        # 2. A map from ``Node`` to a list of times (in seconds) that
        #    node took to run. This can be seen as similar to (1) but
        #    for specific sub-parts of the model.
        self.runtimes_sec : Dict[torch.fx.Node, List[float]] = {}

    ######################################################################
    # Next, let's override our first method: ``run()``. ``Interpreter``'s ``run``
    # method is the top-level entrypoint for execution of the model. We will
    # want to intercept this so that we can record the total runtime of the
    # model.

    def run(self, *args) -> Any:
        # Record the time we started running the model
        t_start = time.time()
        # Run the model by delegating back into Interpreter.run()
        return_val = super().run(*args)
        # Record the time we finished running the model
        t_end = time.time()
        # Store the total elapsed time this model execution took in the
        # ProfilingInterpreter
        self.total_runtime_sec.append(t_end - t_start)
        return return_val

    ######################################################################
    # Now, let's override ``run_node``. ``Interpreter`` calls ``run_node`` each
    # time it executes a single node. We will intercept this so that we
    # can measure and record the time taken for each individual call in
    # the model.

    def run_node(self, n : torch.fx.Node) -> Any:
        # Record the time we started running the op
        t_start = time.time()
        # Run the op by delegating back into Interpreter.run_node()
        return_val = super().run_node(n)
        # Record the time we finished running the op
        t_end = time.time()
        # If we don't have an entry for this node in our runtimes_sec
        # data structure, add one with an empty list value.
        self.runtimes_sec.setdefault(n, [])
        # Record the total elapsed time for this single invocation
        # in the runtimes_sec data structure
        self.runtimes_sec[n].append(t_end - t_start)
        return return_val

    ######################################################################
    # Finally, we are going to define a method (one which doesn't override
    # any ``Interpreter`` method) that provides us a nice, organized view of
    # the data we have collected.

    def summary(self, should_sort : bool = False) -> str:
        # Build up a list of summary information for each node
        node_summaries : List[List[Any]] = []
        # Calculate the mean runtime for the whole network. Because the
        # network may have been called multiple times during profiling,
        # we need to summarize the runtimes. We choose to use the
        # arithmetic mean for this.
        mean_total_runtime = statistics.mean(self.total_runtime_sec)

        # For each node, record summary statistics
        for node, runtimes in self.runtimes_sec.items():
            # Similarly, compute the mean runtime for ``node``
            mean_runtime = statistics.mean(runtimes)
            # For easier understanding, we also compute the percentage
            # time each node took with respect to the whole network.
            pct_total = mean_runtime / mean_total_runtime * 100
            # Record the node's type, name of the node, mean runtime, and
            # percent runtim
            node_summaries.append(
                [node.op, str(node), mean_runtime, pct_total])

        # One of the most important questions to answer when doing performance
        # profiling is "Which op(s) took the longest?". We can make this easy
        # to see by providing sorting functionality in our summary view
        if should_sort:
            node_summaries.sort(key=lambda s: s[2], reverse=True)

        # Use the ``tabulate`` library to create a well-formatted table
        # presenting our summary information
        headers : List[str] = [
            'Op type', 'Op', 'Average runtime (s)', 'Pct total runtime'
        ]
        return tabulate.tabulate(node_summaries, headers=headers)

######################################################################
# .. note::
#       We use Python's ``time.time`` function to pull wall clock
#       timestamps and compare them. This is not the most accurate
#       way to measure performance, and will only give us a first-
#       order approximation. We use this simple technique only for the
#       purpose of demonstration in this tutorial.

######################################################################
# Investigating the Performance of ResNet18
# -----------------------------------------
# We can now use ``ProfilingInterpreter`` to inspect the performance
# characteristics of our ResNet18 model;

interp = ProfilingInterpreter(model)
interp.run(input)
print(interp.summary(True))

######################################################################
# There are two things we should call out here:
#
# * MaxPool2d takes up the most time. This is a known issue:
#   https://github.com/pytorch/pytorch/issues/51393
# * BatchNorm2d also takes up significant time. We can continue this
#   line of thinking and optimize this in the Conv-BN Fusion with FX
#   tutorial TODO: link
#
#
# Conclusion
# ----------
# As we can see, using FX we can easily capture PyTorch programs (even
# ones we don't have the source code for!) in a machine-interpretable
# format and use that for analysis, such as the performance analysis
# we've done here. FX opens up an exiciting world of possibilities for
# working with PyTorch programs.
#
# Finally, since FX is still in beta, we would be happy to hear any
# feedback you have about using it. Please feel free to use the
# PyTorch Forums (https://discuss.pytorch.org/) and the issue tracker
# (https://github.com/pytorch/pytorch/issues) to provide any feedback
# you might have.

Context:
Here is the stack trace:
Traceback (most recent call last):
  File "test.py", line 131, in <module>
    profiler.run(input)
  File "test.py", line 28, in run
    return_val = super().run(*args)
  File "/home/keshav/anaconda3/envs/MolMPNN/lib/python3.7/site-packages/torch/fx/interpreter.py", line 116, in run
    self.env[node] = self.run_node(node)
  File "test.py", line 36, in run_node
    return_val = super().run_node(n)
  File "/home/keshav/anaconda3/envs/MolMPNN/lib/python3.7/site-packages/torch/fx/interpreter.py", line 142, in run_node
    return getattr(self, n.op)(n.target, args, kwargs)
  File "/home/keshav/anaconda3/envs/MolMPNN/lib/python3.7/site-packages/torch/fx/interpreter.py", line 169, in placeholder
    return next(self.args_iter)
StopIteration


Question:
Is there a problem with using hook and fx together？
when I use the torch.fx to get quantizated-module, but as the same as with hook to get output’s param, it return the Proxy type, why.
Thanks

Context: Is there any way to change Proxy to Tensor？


Question:
Hi all.
I’m trying to trace model with positional arguments *args in forward.
The simplified problem is Container that forwards positional arguments to its Module.
When I pass Container to fx.symbolic_trace I get the following error:
torch.fx.proxy.TraceError: Proxy object cannot be iterated. This can be attempted when used in a for loop or as a *args or **kwargs function argument.
This happened because all arguments with prefix * were replaced by Proxy.
What can be done to solve this problem?
from torch import Tensor
import torch.nn as nn
import torch.fx as fx

class Module(nn.Module):
    def forward(self, x):
        return x

class Container(nn.Module):
    def __init__(self):
        super().__init__()
        self._module = Module()

    def forward(self, *args):
        return self._module(*args)

model = Container()
model(Tensor())

trace = fx.symbolic_trace(model)  # Error.

Thanks.

Context:
from torch import Tensor
import torch.nn as nn
import torch.fx as fx

class M(nn.Module):
    def forward(self, x):
        return x

class Container(nn.Module):
    def __init__(self):
        super().__init__()
        self._module = M()

    def forward(self, *args):
        return self._module(args)  # set a variable to store

model = Container()
model(Tensor())

trace = fx.symbolic_trace(model)


Question:
import torch
x = torch.tensor([[1., -1.], [1., 1.]], requires_grad=True)
out = x.pow(2).sum()
out.backward()
x.grad

Context: What kind of issues are you seeing with this code?


Why transforms.ToTensor() behaves as a True when used in conditional statements?

Context:
I don’t think this behavior is specific to transforms.ToTensor(), but it seems that each valid object would return True in this condition:
if nn.Linear(1, 1):
    print('True')

if logging.Logger(name='test'):
    print('True')


[SOLVED} Calling symbolic_trace after inserting a call_module node creates problems

Question:
I am getting errors during a call to symbolic_trace on a GraphModule after having done modifications to it (inserting a call_module node).
This is the exact error::
/home/SERILOCAL/n.perto/.anaconda3/envs/test/lib/python3.7/site-packages/torch/fx/graph.py:606: UserWarning: Attempted to insert a call_module Node with no underlying reference in the owning GraphModule! Call GraphModule.add_submodule to add the necessary submodule
  warnings.warn("Attempted to insert a call_module Node with "
Traceback (most recent call last):
  File "pytorch_issue.py", line 44, in <module>
    new_mod = fx.symbolic_trace(mod)
  File "/home/SERILOCAL/n.perto/.anaconda3/envs/test/lib/python3.7/site-packages/torch/fx/symbolic_trace.py", line 859, in symbolic_trace
    graph = tracer.trace(root, concrete_args)
  File "/home/SERILOCAL/n.perto/.anaconda3/envs/test/lib/python3.7/site-packages/torch/fx/symbolic_trace.py", line 571, in trace
    self.create_node('output', 'output', (self.create_arg(fn(*args)),), {},
  File "<eval_with_key_2>", line 3, in forward
  File "/home/SERILOCAL/n.perto/.anaconda3/envs/test/lib/python3.7/site-packages/torch/fx/graph_module.py", line 513, in wrapped_call
    raise e.with_traceback(None)
AttributeError: 'MyOtherModule' object has no attribute 'conv'

And here is the code:
import torch
import torch.fx as fx
import torch.nn as nn

class MyModule(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x

class MyOtherModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(24, 32, 3)

    def forward(self, x):
        x = nn.functional.relu(self.conv(x))

def insert_mod_after(mod: fx.GraphModule, from_idx: int, insertme: fx.GraphModule):
    mod.add_submodule('inserted', insertme)
    cutoff_node: fx.Node = list(mod.graph.nodes)[from_idx]
    next_node: fx.Node = list(mod.graph.nodes)[from_idx+1]

    with mod.graph.inserting_after(cutoff_node):
        new_node = mod.graph.call_module('inserted', (cutoff_node,), {})

    next_node.replace_input_with(cutoff_node, new_node)

    mod.delete_all_unused_submodules()
    mod.graph.eliminate_dead_code()
    mod.recompile()
    mod.graph.lint()

    return mod

mod = fx.symbolic_trace(MyModule())
other = fx.symbolic_trace(MyOtherModule())
insert_mod_after(mod, 0, other)
new_mod = fx.symbolic_trace(mod)

It looks like it could be related to the warning, although I don’t understand why this warning is happening since I am calling add_submodule already… I created another topic 1 for the warning.
Thank you for reading

Context:
It seems that the problems come from the call to mod.delete_all_unused_submodules().
I think it deletes submodules  of the Sequential submodule added as they don’t appear in the graph even though the call_module node calling the Sequential submodule use them internally.
Putting it after the second symbolic_trace seems to solve the issue.
import torch
import torch.fx as fx
import torch.nn as nn

class MyModule(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x

class MyOtherModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(24, 32, 3)

    def forward(self, x):
        x = nn.functional.relu(self.conv(x))

def insert_mod_after(mod: fx.GraphModule, from_idx: int, insertme: fx.GraphModule):
    mod.add_submodule('inserted', insertme)
    cutoff_node: fx.Node = list(mod.graph.nodes)[from_idx]
    next_node: fx.Node = list(mod.graph.nodes)[from_idx+1]

    with mod.graph.inserting_after(cutoff_node):
        new_node = mod.graph.call_module('inserted', (cutoff_node,), {})

    next_node.replace_input_with(cutoff_node, new_node)

    mod.graph.eliminate_dead_code()
    mod.recompile()
    mod.graph.lint()
    mod = fx.symbolic_trace(mod)
    mod.delete_all_unused_submodules()
    return mod

mod = fx.symbolic_trace(MyModule())
other = fx.symbolic_trace(MyOtherModule())
new_mod = insert_mod_after(mod, 0, other)

I would like these type of ordering to appear in the documentation as it is not clear in which order to call Graph.eliminate_dead_code(), Graph.lint(), GraphModule.delete_all_unused_submodules(), GraphModule.recompile().


How to insert after a node with a set of ops at one time?

Question:
for example, for a computation flow:
z = a + b

i want to change it to:
z = torch.sum(torch.abs(a)) + b

for now, i only figure out a way to do it like this:
with gm.graph.inseting_after(node_a):
    abs_node = gm.graph.call_function(torch.abs)
    abs_node.args = (node_a,)

# without this second with statement, sum_node will be added before abs_node
with gm.graph.inserting_after(abs_node):
    sum_node = gm.graph.call_function(torch.sum)
    sum_node.args = (abs_node,)

node_a.replace_all_uses_with(sum_node)

the question is, how can i do it faster and more elegant if i have 10 nodes to insert after node_a?

Context:
solved.
pack all operations in a Module, then use call_module instead.


Question:
Hi,
Following this discussion 1 about inserting a block instead of a node, I would like to rebuild a model using Proxy.
However I am unable to build a model with calls to modules, here is an example:
import torch.nn as nn
import torch.fx as fx

module = nn.Module()
submodule = nn.Linear(10, 1)
module.add_module('sub', submodule)

graph = fx.Graph()
raw = graph.placeholder('x')
x = fx.Proxy(raw)
y = submodule(x)
output = graph.output(y.node)
gm = fx.GraphModule(module, graph)

result:
Traceback (most recent call last):

  File "/home/SERILOCAL/n.perto/.anaconda3/envs/automl/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-37-1d2b0a63ef08>", line 1, in <module>
    gm = fx.GraphModule(module, graph)

  File "/home/SERILOCAL/n.perto/.anaconda3/envs/automl/lib/python3.7/site-packages/torch/fx/graph_module.py", line 194, in __init__
    self.graph = graph

  File "/home/SERILOCAL/n.perto/.anaconda3/envs/automl/lib/python3.7/site-packages/torch/nn/modules/module.py", line 995, in __setattr__
    object.__setattr__(self, name, value)

  File "/home/SERILOCAL/n.perto/.anaconda3/envs/automl/lib/python3.7/site-packages/torch/fx/graph_module.py", line 217, in graph
    self.recompile()

  File "/home/SERILOCAL/n.perto/.anaconda3/envs/automl/lib/python3.7/site-packages/torch/fx/graph_module.py", line 297, in recompile
    cls.forward = _forward_from_src(self._code)

  File "/home/SERILOCAL/n.perto/.anaconda3/envs/automl/lib/python3.7/site-packages/torch/fx/graph_module.py", line 41, in _forward_from_src
    exec_with_source(src, gbls)

  File "/home/SERILOCAL/n.perto/.anaconda3/envs/automl/lib/python3.7/site-packages/torch/fx/graph_module.py", line 26, in exec_with_source
    exec(compile(src, key, 'exec'), globals)

  File "<eval_with_key_10>", line 3
    linear_1 = torch.nn.functional.linear(x, Parameter containing:
                                                                ^
SyntaxError: invalid syntax

When I print the graph I obtain this:
graph(x):
    %linear_1 : [#users=1] = call_function[target=torch.nn.functional.linear](args = (%x, Parameter containing:
tensor([[-0.1319, -0.2979,  0.2596, -0.1899,  0.1811, -0.1425,  0.1439, -0.0744,
         -0.0514,  0.1271]], requires_grad=True)), kwargs = {bias: None})
    return linear_1

Any help would be appreciated.
Thank you.

Context:
This seems to happen with modules containing parameters.
Is it a bug?
If not then how should I handle them?
The example in the github repository doesn’t showcase the use of proxy with these type of modules.
Thank you in advance.


Torch fx supports TFLite quantization format

Question: I am quite impressed by the current fx-based quantization workflow. However, the hardware I am using only supports TFLite-style quantization. Will the TFLite-style quantization format be supported in the future? Otherwise if I want to implement a new quantization backend for TFLite, is there any useful documentation that helps me understand the to-dos?

Context: Unfortunately, we do not have any support for directly porting models from pytorch to tflite format. However, the underlying quantization support is very similar. We support quantization per-tensor or per-row, much like TFLite and have very similar operator support.


Question:
I am trying to ramp up on torch.fx. I have a forward function with torch.no_grad() (example below).
def forward(self, x):
with torch.no_grad():
tmp_val = … do something …
out = torch.mul(x, tmp_val)
return out
I did symbolic trace then printed out the code.
traced = torch.fx.symbolic_trace(model)
print(traced.code)
Then I can’t see no_grad() in the traced code anymore. Is there a way to keep this in the traced model?

Context:
Hi @kyong,
FX does not support context managers (such as no_grad) in traced regions. Can you refactor the original code to something like:
class MyModel(torch.nn.Module):
  def forward(self, x):
    tmp_val = … do something …
    out = torch.mul(x, tmp_val)
    return out

class NoGradWrapper(torch.nn.Module):
  def __init__(self):
    self.mm = MyModel()

  def forward(self, x):
    with torch.no_grad():
      return self.mm(x)

Then you can apply FX transforms to NoGradWrapper.mm and still preserve the no-gradient behavior


Integrated Gradients with Captum in a sequential model

Question:
Hi, I am using Integrated Gradients with a NN model that has both sequential (LSTM) and non sequential layers (MLP). I was following the basic steps as given in the examples and ran into a weird error.
Code snippet:
test_tens.requires_grad_()
attr = ig.attribute(test_tens, target=1, return_convergence_delta=False)
attr = attr.detach().numpy()
error:
in line 2 of the above snippet:
" index 1 is out of bounds for dimension 1 with size 1 "
Probably some problem around the target thing, but not sure what to do

Context: Thank you for posting the issue, @key_crusher! What are the shapes of the output of your model and test_tens?


Using captum with nn.Embedding getting RuntimeError

Question:
I am using captum library and getting following error. Here is complete code to reproduce the error. Appreciate if someone could help me here.
Thank in advance. -OneQ-
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from captum.attr import IntegratedGradients

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

vocab_size = 1024
embedding_dim = 32
seq_len = 128
num_classes = 5
hidden_dim = 256

class predictor(nn.Module):
    def __init__(self):
        super().__init__()
        self.seq_len = seq_len
        self.num_classes = num_classes
        self.hidden_dim = hidden_dim
        self.vocab_size, self.embedding_dim = vocab_size, embedding_dim

        self.embedding = nn.Embedding(self.vocab_size, self.embedding_dim)
        self.linear = nn.Linear(self.seq_len*self.embedding_dim, self.num_classes)

    def forward(self, x):
        x = self.embedding(x.long())
        x = x.reshape(-1, self.seq_len*self.embedding_dim)
        x = F.relu(self.linear(x))
        return x

class wrapper_predictor(nn.Module):
    def __init__(self, model):
        super().__init__()
        self.model = model
    def forward(self, x):
        x = self.model(x)
        x = F.softmax(x, dim=1)
        return x

indexes = torch.Tensor(np.random.randint(0, vocab_size, (seq_len))).to(device)

model = predictor().to(device)
wrapper_model = wrapper_predictor(model).to(device)

ig = IntegratedGradients(wrapper_model)
attributions, delta = ig.attribute(inputs=indexes, target=0, n_steps=1, return_convergence_delta=True)

Context:
I resolved the issue with LayerIntegratedGradients.
Here is the link to read more to know other possible solutions. Captum · Model Interpretability for PyTorch
Sample code
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from captum.attr import IntegratedGradients, LayerIntegratedGradients
from torchsummary import summary

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

vocab_size = 1024
embedding_dim = 1
seq_len = 128
num_classes = 5
hidden_dim = 256

class predictor(nn.Module):
	def __init__(self):
		super(predictor, self).__init__()
		self.seq_len = seq_len
		self.num_classes = num_classes
		self.hidden_dim = hidden_dim
		self.vocab_size, self.embedding_dim = vocab_size, embedding_dim

		self.embedding = nn.Sequential(
			nn.Embedding(self.vocab_size, self.embedding_dim),
		)
		self.embedding.weight = torch.randn((self.vocab_size, self.embedding_dim), requires_grad=True)
		self.fc = nn.Sequential(
			nn.Linear(self.seq_len*self.embedding_dim, self.hidden_dim, device=device, bias=False),
			nn.Linear(self.hidden_dim, self.num_classes, device=device, bias=False),
		)
	def forward(self, x):
		x = self.embedding(x.long())
		x = x.view(-1, self.seq_len*self.embedding_dim)
		x = self.fc(x)
		return x

class wrapper_predictor(nn.Module):
	def __init__(self, model):
		super().__init__()
		self.model = model
	def forward(self, x):
		x = self.model(x)
		x = F.softmax(x, dim=1)
		return x

model = predictor().to(device)

indexes = torch.Tensor(np.random.randint(0, vocab_size, (seq_len))).to(device)
input_size = indexes.shape
summary(model=model, input_size=input_size, batch_size=-1, device='cuda')

wrapper_model = wrapper_predictor(model).to(device)

lig = LayerIntegratedGradients(model, model.embedding)
attributions, delta = lig.attribute(inputs=indexes, target=0, n_steps=1, return_convergence_delta=True)


Question:
How can I use a permutation-based importance feature using captum as an example
https://scikit-learn.org/stable/modules/permutation_importance.html 3

Context:
Yes, You can use captum.attr.FeaturePermutation

      captum.ai

Captum · Model Interpretability for PyTorch 6

  Model Interpretability for PyTorch


Question:
So I’m currently at odds with how to approach this. At first, I was only really using Layer Conductance on a document-by-document basis. However, I’m now trying to scale this a little bit as I’m using the attributions towards a paper. The set that I’m passing in for analysis of an already trained model, consists of 6658 documents. This is taking me, on average, 1 to 1.5 hours to process on a Titan RTX. So for around 120 models, it’s taking me just shy of about 3 days to process it if I split this across 4-6 Titan RTXs (I have GPUs available to me via my work cluster so it grabs how many are available).
The process I am currently following is the BERT SQUAD tutorial, slightly adapted for classification. I am using HuggingFace’s BERT Model as a base with dropout + linear layer on top for output.
I was wondering what optimization strategies were available to me, given my situation. I’ve checked the input tensors and they’re float32. I’ve noticed executing cpu().detach().numpy() slows things down a fair bit. Should I maybe be running this on a batch of documents at a time to avoid running the whole embedding process over and over?
I’ve attached my eval script for reference:
def run(docs):
    res = []
    for i, doc in enumerate(docs):
        print(f"Processing Doc #{i+1}/{len(docs)}")
        text_ids = tokenizer.encode(doc, add_special_tokens=False)
        input_ids = [tokenizer.cls_token_id] + text_ids + [tokenizer.sep_token_id]
        ref_input_ids = [tokenizer.cls_token_id] + [tokenizer.pad_token_id] * len(text_ids) + [tokenizer.sep_token_id]
        input_ids = torch.tensor([input_ids], device=DEVICE)
        ref_input_ids = torch.tensor([ref_input_ids], device=DEVICE)

        sep_id = len(text_ids)
        seq_len = input_ids.size(1)
        token_type_ids = torch.tensor(
            [[0 if i <= sep_id else 1 for i in range(seq_len)]],
            device=DEVICE
        )
        ref_token_type_ids = torch.zeros_like(token_type_ids, device=DEVICE)

        seq_length = input_ids.size(1)
        position_ids = torch.arange(seq_length, dtype=torch.long, device=DEVICE)
        position_ids = position_ids.unsqueeze(0).expand_as(input_ids)
        ref_position_ids = torch.zeros(seq_length, dtype=torch.long, device=DEVICE)
        ref_position_ids = ref_position_ids.unsqueeze(0).expand_as(input_ids)
        attention_mask = torch.ones_like(input_ids)

        indices = input_ids[0].detach().tolist()
        all_tokens = tokenizer.convert_ids_to_tokens(indices)

        input_embeds = model.base.embeddings(input_ids, token_type_ids=token_type_ids, position_ids=position_ids)
        ref_input_embeds = model.base.embeddings(ref_input_ids, token_type_ids=token_type_ids, position_ids=position_ids)

        def forward(input_embeds, attention_mask):
            logits = model(inputs_embeds=input_embeds, attention_mask=attention_mask)
            return torch.softmax(logits, dim=1)

        def summarize_attributions(attributions):
            attributions = attributions.sum(dim=-1).squeeze(0)
            attributions = attributions / torch.norm(attributions)
            return attributions

        layer_attrs = []
        for i in range(model.base.config.num_hidden_layers):
            lc = LayerConductance(forward, model.base.encoder.layer[i])
            layer_attributions = lc.attribute(
                inputs=input_embeds,
                baselines=ref_input_embeds,
                additional_forward_args=(attention_mask),
                target=1
            )
            layer_attrs.append(summarize_attributions(layer_attributions).numpy())

        res.append ({
            "doc_no": i,
            "tokens": all_tokens,
            "conductance": layer_attrs
        })
        torch.cuda.empty_cache()
    return res

Context:
Hi @ajhepburn, if you run layer conductance on a batch of documents it can definitely speed up things. Only you want to make sure that you have sufficient memory and perhaps create sub-batches combing multiple documents if you can’t fit everything into a memory. Note that the internal requirements of layer conductance is n_steps * #examples. n_steps is defaulted to 50 and you can change it if necessary.
As I can see, you are treating each document as one example, is that right ? If your documents aren’t too large you can definitely try to combine them.
Another option is to use DataParallels and GPUs.
More info about Dataparallels and Distributed Dataparallels can be found here:

      captum.ai

Captum · Model Interpretability for PyTorch 2

  Model Interpretability for PyTorch


Attributes to be considered while using model interpretability methods

Question:
Hello all,
I am using model interpretability methods such as IntegratedGradients, GradSHAP, DeepLift, etc. After executing the attributions method, I get a matrix of attributions corresponding to every feature in the dataset. My question is:
Which attributes should I consider as important attributes? The ones with high positive correlation, or the ones with negative correlation, or both? Maybe I want to ask how to assign the attribution values to important or not-important set?
Thank you.

Context: Both. Check plots in captum tutorials to get an intuition. Not sure if there is an universal recipe for importance thresholds…


Question: While using any model interpretability method, should we pass the train set or the test set as input parameter?

Context: As I understand it, base scenario is with the training set - you check which features are discovered as important during the training. There may be some advanced uses on a held-out dataset (e.g. comparision plots or exploring generalization problems), but that’s more serious scrutinizing than basic X “columnwise” importance checking.


Integrated gradients for negative labeled examples

Question: How do I compute the feature attributions for negative labeled examples (i.e., target = 0) when the model output is a scalar? Currently, integrated gradients gives me the feature attributions for all examples wrt target = 1.

Context:
I guess your problem is a binary classification and you are using sigmoid for your output layer.
Currently, it seems that specifying a specific output if you use sigmoid in the output layer is not supported yet. If you want integrated gradient (Captum) give you the feature attributions wrt a specific target, you should use softmax instead of sigmoid.


Question:
I am using pytorch lightning and I am want to visualize my validation data using GradCAM.I am using a pretrained Resnet 3d model and this is the code I wrote:
my_model.eval()
guided_gc = GuidedGradCam(my_model, my_model.model.fc)
input_image = next(iter(mri_dataset.val_dataloader()))
attribution = guided_gc.attribute(input_image, 1)
Error:
assert isinstance(
AssertionError: inputs must have type torch.Tensor but <class ‘dict’> found:
my val_dataloader is a list of dictionary and I am unable to pass it properly. Kindly suggest how should I go about it and how can I also view the output.

Context: Hi @Roshni_Pathak , inputs must be provided as a tensor or tuple of tensors to appropriately compute attributions. If the tensors are contained as values within the dictionary, then you can define a wrapper function which takes the tensors as input and constructs an appropriate dictionary and calls my_model with this dictionary. An example of this approach using a wrapper method with GuidedGradCam can be found in the discussion here 38. Hope this helps!


Occlusion Sensitivity-Sliding window shape

Question:
I am working with a tensor shape ([1,9]) and passing it as a tuple of bath size:2.
I am unable to pass the right sliding window shape for Occlusion Model.
Can you help me to know what would be the ideal input for sliding window shape and stride?
A snippet of my Occlusion model is here:
clinical=torch.stack(tuple(clinical), dim=1) #torch.Size([2, 9])
print(clinical.shape)
clinical = clinical[:, :].detach().cpu().numpy()
ablator = Occlusion(model)
attr = ablator.attribute(tuple(clinical), target=ds_item[‘Class’], sliding_window_shapes=(2,9))
occ_sens = OcclusionSensitivity(nn_module=my_model, n_batch=1, stride=(2,2))
occ_map, most_probable_class = occ_sens(clinical,
b_box=[-1, -1, 2, 3, -1, -1, -1, -1])
print(occ_map)
Error message:
in _format_and_verify_sliding_window_shapes
assert len(sliding_window_shapes) == len(
AssertionError: Must provide sliding window dimensions for each input tensor.

Context:
Hi @Roshni_Pathak , it looks like the input being passed to Occlusion here - tuple(clinical) - is a tuple of 2 numpy arrays. It is expected that the input should be a torch tensor.
For the sliding window shape, if your model’s input will still be 2 separate tensors, then the sliding window shape of each should be a separate tuple with one element e.g. ((9,), (9,)). This corresponds to one sliding window of length 9, which is equivalent to occluding each of the full input tensors together.
Alternatively, if the input to the model (or wrapper forward function) would be a tensor of shape 2x9 such as the initial clinical tensor, then the sliding window shape can just be provided as (9,). The stride for Occlusion is not relevant if the sliding window covers the entire tensor, since it will not be shifted. If you would like to occlude smaller segments of the tensor, a smaller sliding window can be selected, with stride defining the shift of the window at each step. For more info regarding the Occlusion parameters, you can refer to the documentation here 4.


Question:
I’ve been working with the EfficientNet architecture lately and I wanted to apply Captum’s GuidedGradCam on it but I am getting strange results:
This is the code that I’m using:
from models.efficientnet import EfficientNet
import torch
import PIL.Image
from matplotlib import pyplot as plt

from torchvision.transforms import ToTensor, Resize, Compose, ToPILImage

from dataset_fun import get_data

device = 'cpu' ## Let's just put this to CPU for simplicity
model = EfficientNet.from_pretrained('efficientnet-b2', num_classes=2)
model.load_state_dict(torch.load(
    '$PATH_TO_MODEL/pre_trained_model.pt'))

df = get_data() # Getting my data
df = df[df['label'] == 1]

transforms = Compose([Resize((224, 224)), ToTensor()])

model = model.to(device)
model.eval()

# Here I Instantiate the grad cam class as well as the desired layer to compute the gradients
grad_cam_layer = model._conv_head # Last convolutional layer of the EfficientNet
guided_fc = attr.GuidedGradCam(model, grad_cam_layer)
for filename in df['filename'].values.tolist():
        # Some due pre-processing
        im = PIL.Image.open(filename)
        im = im.convert('RGB')
        im = transforms(im).unsqueeze(0).to(device)

        # Obtain attributes
        attributes = guided_gc.attribute(im, 1)

        # Plot them on screen
        plt.imshow(ToPILImage()(attributes[0])) # We convert to PIL and show it on screen
        plt.show()
        plt.clf()  # will make the plot window empty

Below I show the photo that was sent to the model and the returned GradCam (I know the photos aren’t pretty, skin cancer isn’t beautiful). As it can be seen, the result doesn’t provide much information.
Dissecting the EfficientNet I’ve seen that it doesn’t use any nn.ReLU but rather a novel non-linear function named Swish. Could this be causing the error as in the Captum implementation 3 they mention that the ReLUs have to be of a specific type.
Any help on how to properly compute the Guided Gradient Cam of an image using Captum would be appreciated.
Caveat: I know the first image is bigger, but it gets resized in the transform.

Advances:
I managed to change the Swish to a nn.ReLU() with the following code:
for i in range(len(model._blocks)):
      model._blocks[i]._swish = nn.ReLU()
model._swish = nn.ReLU()

The problem is that it now outputs a complete black square.
Althought this is the recommended activation function, the model wasn’t trained with it.

Context:
Updates:
I retrained the model with nn.ReLU() instead of custom swish functions. The result is very similar.
Taking a look at this other post in the pytorch forum 4 one can see that their result for GuidedGradCam 7 is more informative.
As it can be seen in my code, the in which I plot the image is via first converting it to a PIL image. Has anyone used Captum’s GuidedGradCam on a CNN that might be able to shed some light into why I’m having this output?


Constructing reference/baseline for LSTM - Layer Integrated Gradients

Question:
`# compute attributions and approximation delta using layer integrated gradients
attributions_ig, delta = lig.attribute((input_indices,h), (reference_indices,h),
n_steps=500, return_convergence_delta=True)
`
When I run the above code, I get the following error.
AssertionError: baseline input argument must be either a torch.Tensor or a number however <class ‘tuple’> detected
But the documentation says that a tuple can be passed as baseline argument.
Am I doing this right?

Context: Similar problem here. Did you find a way out?


Passing hidden state through LayerIntegratedGradients

Question:
I was trying to replicate the CNN text model interpretability example using an LSTM. But the LayerIntegratedGradients throws an error about the missing hidden state.
forward() missing 1 required positional argument: ‘hidden’
What would be the best way of handling this?

Context:
create a function wrapper (partial function or lambda), hiding “auxiliary” parameters from captum
or use additional_forward_args argument (see docs)


Normal inference works fine, but captum (IntegratedGradients) throws a size missmatch

Question:
Hi,
I am trying to use captum for the frist time and would like to use integrated gradients a analyze my model.
Nomal inference works fine, but  trying to use integrated gradients with captum leads to an error:

RuntimeError: size mismatch, m1: [1 x 9800], m2: [196 x 512] at /opt/conda/conda-bld/pytorch_1579022027550/work/aten/src/TH/generic/THTensorMath.cpp:136

I have no clue why the size is 1 x 9800.
The tensor I use has a single dimension with 192 elements and the output of the fist model layer is 512 elements.
This code I tried:
# Initial imports
from captum.attr import IntegratedGradients
from captum.attr import LayerConductance
from captum.attr import NeuronConductance

import torch
import torch.nn as nn

# Define model
class MLP(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.model = nn.Sequential(
            # Add input layer
            nn.Linear(196, 512),
            # Add ReLU activation
            nn.ReLU(),
            # Add Another layer
            nn.Linear(512, 512),
            # Add ReLU activation
            nn.ReLU(),
            # Add Output layer
            nn.Linear(512, 12)
        )

    def forward(self, x):
        # Forward pass
        return self.model(x)

# Prepare data and model
sample = torch.rand(196)
label = 7
model  = MLP()

# Normal inference (works fine):
score = model(sample)
prob = nn.functional.softmax(score, dim=0)
y_pred =  prob.argmax()
print("Predicted class {} with probability {}. True label is: {}".format(y_pred, prob[y_pred], label))

# Usage of captum (does not work)
ig = IntegratedGradients(model)
sample.requires_grad_()
attr, delta = ig.attribute(sample,target=label, return_convergence_delta=True) # ERROR!
attr = attr.detach().numpy()

Has anyone an idea where the wrong dimension is coming from and how to solve this?
Thanks!

Context: I think you need to add a leading batch dimension to input. Additionally,  internal_batch_size argument may be needed to limit cuda memory requirements.


Question:
Hello everyone,
I just started to dig into Explainable ML/DNN field and was trying to play with “captum” a bit when I found a problem that I cannot explain.
N.B.
Since I can’t upload more than image and share more than 2 links (D:), ill share 2 images by links.
I’ve a custom model to perform classification on MNIST (it reaches 97% accuracy and since I use it as a playground im not interested to improve it) with this architecture:
MnistNet(
  (FeatureExtractor): Sequential(
    (0): Conv2d(1, 8, kernel_size=(2, 2), stride=(1, 1), padding=(1, 1))
    (1): ReLU()
    (2): Conv2d(8, 16, kernel_size=(2, 2), stride=(2, 2), padding=(1, 1))
    (3): ReLU()
    (4): Conv2d(16, 20, kernel_size=(2, 2), stride=(1, 1), padding=(1, 1))
    (5): ReLU()
    (6): Conv2d(20, 24, kernel_size=(2, 2), stride=(2, 2), padding=(1, 1))
    (7): ReLU()
  )
  (Classifier): Sequential(
    (0): Flatten()
    (1): Linear(in_features=1944, out_features=256, bias=True)
    (2): Dropout(p=0.3, inplace=False)
    (3): ReLU()
    (4): Linear(in_features=256, out_features=64, bias=True)
    (5): Dropout(p=0.3, inplace=False)
    (6): ReLU()
    (7): Linear(in_features=64, out_features=10, bias=True)
  )
)

Now I’m trying to perform GuidedGradCam against an image of a four which is rightfully classified with 0.999 probability.
After I perform a “run” of Guided GradCam using every possible target (0-9) i get these results:
Results of GuidedGradCam 24
Now, even without considering the poor general results, the real problem is the image for the target 4 (the true label and the real model output), in fact it’s “blank”!!
I interpreted this result as something caused by the final ReLU pass in the GradCam so to test it, i implemented my own version of GradCam (not GuidedGradCam!) and this is the result with the same input:
Result of custom GradCam 17
Not how the results are similar (target 4 and 9 still blank) with expected differences (its GradCam without matmul from DeepLift, so a “less resolution” is expected as it has written in the paper).
Finally i test what happens if i take out the ReLU from my own version of GradCam. In other words i modify the last line of code such that the return now is:
gradcam = torch.sum(A_k*alpha_k, dim=(0))

and not
gradcam = nn.ReLU()(torch.sum(A_k*alpha_k, dim=(0)))

not_relu_results1161×565 55.3 KB
Now target 4 and 9 have very “important” outputs with really low values (up to -50 and -14). It seems that the output of GradCam should be considered in its absolute value without performing ReLU.
Question
Given the above context and results, has anyone a good explanations of why this happens and why using ReLU make sense in the original GradCam?

Context:
Hi @Francesco_Grimaldi, this is a surprising result. To understand the cause of this further, I’d recommend looking at the activations and gradients separately of the last convolutional layer, since the product of these leads to GradCAM. You can do this with Captum using LayerActivation and dividing LayerGradientXActivation by LayerActivation for the gradient.
It could be that gradients of the target class of 4 with respect to the final conv output is mostly negative, potentially because the output is saturated, so the gradients are not representative of the “importance” of the corresponding activation. Looking at the gradients and activations separately will be helpful to further understand this.
Also, Captum does offer GradCAM itself as LayerGradCAM (since it technically outputs attributions with shape matching the layer output) with an argument relu_attributions to decide whether to apply a ReLU to the final attribution.


Captum to explain Albert-based mutli-class classification

Question: Hello everyone, I have trained a multi-class classification model with pre-trained Albert LM. I am now experimenting with Captum to see the attribution score of each class/label. I expect to see positive/max attribution score for model prediction class/label, and negative and smaller scores for the rest of classes/labels. However, I don’t really see such a correlation there. Not sure if I don’t use Captum correctly or my expectation is simply not correct. Anyone has experience with this? Thanks.

Context:
Hi Kevin, thank you for the question! Do you have any example code snippets that you could post here ?
Jupyter or google colab notebooks would be great too.


Question:
are there any global explanations algorithms embedded in captum?
is there any plan to add some anytime soon?
which ones?

Context:
Hi @ccarmel, We are currently working onTCAV - Testing with Concept Activation Vectors.
Besides that we can always aggregate sample level attributions to get a global view, but this of course it tied to the data.
Do you have any global explanation algorithms for Neural Networks in mind ?


Question: This category is for the Glow neural network accelerator compiler: https://github.com/pytorch/glow 63

Context: im thinking along the same lines. who created this category and what is it for?


Bias quantized with the same scale as the Convolution result

Question: Hey everyone. I’m building a new backend for Glow called NMP (from NeuroMorphic Processor). It is an embedded device to make inferences from convolutional neural networks. This device only works with quantized models (Int8 and Int16). It turns out that the external library that performs the convolution operations expects the Bias to be quantized on the same scale as the Convolution result (the same applies to FullyConnected operations). Any ideas on how can I specialize this type of quantization?

Context:
Hi @iviarcio, the NodeValue::setType() call does not change the actual payload of a Constant node, assuming this is what you are expecting to happen.
What you need to do is something like
template <typename T> static void rescaleBias(T *node, Function *F) {
  auto *bias = dyn_cast<Constant>(node->getBias());
  CHECK(bias) << "Expected bias would be Constant";
  Tensor floatBias = quantization::dequantizeTensor(bias->getPayload(),
                                                    ElemKind::FloatTy);
  auto outTy = node->getResult().getType();
  Tensor newBias = quantization::quantizeTensor(floatBias,
                                                {outTy->getScale(), outTy->getOffset()},
                                                outTy->getElementType());
  Constant *newConst = F->getParent()->createConstant(bias->getName().str() + "_converted",
                                                      std::move(newBias));
  bias->getOutput().replaceAllUsesOfWith(newConst->getOutput());
}


Question: Hey everyone. I’m building a new backend for Glow called NMP (from NeuroMorphic Processor). It is an embedded device to make inferences from convolutional neural networks. So far I’ve used stubs functions (libjit_nmp) to develop the code generator, similar to LLVMIRGen. When I replace the stubs functions with function declarations, the getFunction() called inside the NMPIRGen module always returns nullptr. Note that the definitions of the functions are external (libobj_nmp) as they need to be compiled with GCC because of the device-specific toolchain that was only developed for GCC. Basically what I have in libjit folder now is a header file containing the function declarations and a few variables and constants defined in the source file. What I need is for the clang to keep the declarations in libjit_nmp.bc so that the NMPIRGen module can generate code in the main function to perform their calls. These references would then be resolved at link time with the lib_nmp.o library. Any ideas how to solve this?

Context: Thanks, Roman L for your feedback. I worked around this issue by creating wrappers for the external functions. This adds indirection but with clang optimization, it doesn’t affect performance.


Question:
splitNode method throws segfault. I am in “Debug” mode, and using “resnet-runtime” network as the input.
In my case, the segfault occured when dealing Maxpool and AvgPool nodes.
for (auto &node : F->getNodes())
    {
      if (node.getUsers().size() == 0 || static_cast<std::string>(node.getKindName()) == "Convolution")
      {
        continue;
      }
      std::vector<glow::Node *> splitNodes;
      auto dims = node.dims(0);
      if(dims.size() == 4)
      {
        if(dims[3]/VW > 1)
        {
          std::cout << static_cast<std::string>(node.getName()) << "\n";
          ASSIGN_VALUE_OR_FATAL(splitNodes, glow::splitNode(&node, SplitNodeByNumChunks({ShapeNHWC::DimC},{dims[3]/VW})));
        }
      }
    }```

Context: Thanks for reporting this. Can you file an issue on Github with what you have here, and tag mciprian13, who is the original author of much of the splitNode infra?


Question: I am trying to build Glow on RHEL. Does it support?

Context: I think some OSS users do use it (or have in the past). But we don’t have active CI coverage of it on our end of the project, so any given master commit may have issues, no guarantees.


Question:
assertion when trying to create a maxpool node, I have created a mock backend and tried this in the transformPostLowering() function
mnist: /home/bucky/Desktop/compiler/glow/lib/Graph/NodeValue.cpp:10: glow::NodeValue::NodeValue(qsnn::Node *): Assertion `(!N || (N->getNumResults() == 1)) && "Constructing a value for a multi-res node"' failed.
Aborted (core dumped)

How can I over come this?

Context:
MaxPool has two outputs, Result and Argmax. For multi-output nodes you need to select a specific output when using it, for example maxPoolNode->getResult() or maxPoolNode->getArgmax(), instead of just using maxPoolNode as an input to another node.

      github.com

    pytorch/glow/blob/d5b32ee8a6f29166219963623648f9b15f1f94b8/tools/ClassGen/NodeGen.cpp#L161-L162 2

          .addResultFromCtorArg("Result")
          .addResultFromCtorArg("Argmax")


Question: Would it be possible to compile a model to LLVM-IR instead of the GLOW Low-Level IR?

Context:
We have a flag -dump-llvm-ir which you can use to dump the llvm IR to stdout. Note that this only works for llvm-based backends, e.g. our CPU backend. For example:
./bin/image-classifier tests/images/imagenet/cat_285.png -use-imagenet-normalization -image-mode=0to1 -m=resnet50 -model-input-name=gpu_0/data -dump-llvm-ir -cpu


Question: Hello I’m using Resnet18, Resnet50 as an AOT bundle now. When the test image is inputted, the inference time and the confidence can be confirmed to be output. Furthermore, I want to check memory usage or memory transfer. What should I do to check? Do I have to implement it directly on the code?

Context:
You should be able to use these structs that are autogenerated and should be included in the main function you write in in the C++ that drives inference:

      github.com

    pytorch/glow/blob/b0de69a7639567bfec9081ab6a8bcb792638cba9/lib/LLVMIRCodeGen/BundleSaver.cpp#L106-L135 3

          /// Header file common definitions for dynamic API.
          static const char *dynamicApiCommonDefines = R"RAW(
          // Type describing a symbol table entry of a generated bundle.
          struct SymbolTableEntry {
            // Name of a variable.
            const char *name;
            // Offset of the variable inside the memory area.
            uint64_t offset;
            // The number of elements inside this variable.
            uint64_t size;
            // Variable kind: 1 if it is a mutable variable, 0 otherwise.
            char kind;
          };

// Type describing the config of a generated bundle.
          struct BundleConfig {
            // Size of the constant weight variables memory area.
            uint64_t constantWeightVarsMemSize;
            // Size of the mutable weight variables memory area.
            uint64_t mutableWeightVarsMemSize;

  This file has been truncated. show original

E.g. using the BundleConfig you can understand the total memory usage of the network itself, such as the size of constant weights, etc. Aside from that if you need more info about data transfer then yes you would need to take a look at the actual data you’re passing into the network in the C++ file that drives inference.


Question:
Hello everyone,
I’m using the quantization-schema=symmetric_with_power2_scale and quantization-precision=Int8 for a new device. Except for a few operators like LRN, this schema works fine. But I’d like to have Int16 precision also. I’ve been studying the “base.h”, “quantization.h” & “quantization.cpp” modules but I’m a little confused on how to progress. Any help or guidance is welcome. Thanks

Context: So it looks like nothing is being quantized. One note here is that the default CPU backend doesn’t have any Int16 quantization support AFAIK. So unless you have added that support, this is probably why you aren’t seeing any int16 in the model – the quantization flow will skip it if Backend::isOpSupported() returns false for it. You will need to add support for those kernels in int16 to libjit, and update your backend’s isOpSupported() to return true for these newly supported ops w/ int16. And there may be other issues in LLVM IR gen, not sure.


Question:
I got such error and it stopped when i run ninja all when  building GLOW with Bundle enabling.
[95/318] Generating ../../../bundles/bundle_with_extra_objects/lenet_mnist.o
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0403 09:49:42.245189  8189 Graph.cpp:5876] Layout requirements checking is enabled
I0403 09:49:44.442871  8189 HostManager.cpp:235] Destroying host manager...
[103/318] Generating ../../../bundles/bundle_with_multiple_entries/testBundle.o
FAILED: bundles/bundle_with_multiple_entries/testBundle.o
cd /home/kittu28/chaitu/glow/build_Bundle/examples/bundles/bundle_with_multiple_entries && /home/kittu28/chaitu/glow/build_Bundle/bundles/bundle_with_multiple_entries/bundle_with_multiple_entriesBundleSaver -emit-bundle /home/kittu28/chaitu/glow/build_Bundle/bundles/bundle_with_multiple_entries -bundle-api=dynamic -g
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0403 09:49:49.055552  8229 Graph.cpp:5876] Layout requirements checking is enabled
bundle_with_multiple_entriesBundleSaver: ../lib/Backend/BackendUtils.cpp:534: void glow::allocateActivations(const glow::IRFunction::InstListTy &, glow::MemoryAllocator &, glow::runtime::SymbolTableTy &): Assertion `!symbolTable.count(std::string(TV->getName())) && "Allocation already made!"' failed.
Aborted (core dumped)
[107/318] Generating ../../../bundles/resnet50/resnet50.o
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0403 09:49:47.704874  8207 Graph.cpp:5876] Layout requirements checking is enabled
I0403 09:50:02.765214  8207 HostManager.cpp:235] Destroying host manager...
[108/318] Generating profile.yml
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0403 09:49:48.109372  8203 Graph.cpp:5876] Layout requirements checking is enabled
I0403 09:49:59.512897  8203 Partitioner.cpp:684] Profiling a model to be partitioned cross different backends. Each sub-network will be optimized and run on cpu backend.
I0403 09:50:05.110855  8203 ImageClassifier.cpp:239] Graph profiling is ON. Processing of output is disabled.
I0403 09:50:06.331144  8203 ImageClassifier.cpp:239] Graph profiling is ON. Processing of output is disabled.
I0403 09:50:07.494189  8203 ImageClassifier.cpp:239] Graph profiling is ON. Processing of output is disabled.
I0403 09:50:07.501137  8203 HostManager.cpp:235] Destroying host manager...
Model: /home/kittu28/chaitu/glow/build_Bundle/examples/bundles/resnet50/resnet50
Running 1 thread(s).
ninja: build stopped: subcommand failed.

Context:
The earlier one is the error i got when i ran “ninja all” after running “ninja image-classifier” first.
This is the error i got when i ran “ninja all” directly…
[497/722] Generating ../../../bundles/bundle_instrument/lenet_mnist.o
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0404 11:02:17.143170  5910 Graph.cpp:5876] Layout requirements checking is enabled
I0404 11:02:20.217358  5910 HostManager.cpp:235] Destroying host manager...
[501/722] Generating ../../../bundles/bundle_with_multiple_entries/testBundle.o
FAILED: bundles/bundle_with_multiple_entries/testBundle.o
cd /home/kittu28/chaitu/glow/build_Bundle/examples/bundles/bundle_with_multiple_entries && /home/kittu28/chaitu/glow/build_Bundle/bundles/bundle_with_multiple_entries/bundle_with_multiple_entriesBundleSaver -emit-bundle /home/kittu28/chaitu/glow/build_Bundle/bundles/bundle_with_multiple_entries -bundle-api=dynamic -g
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0404 11:02:23.937759  5938 Graph.cpp:5876] Layout requirements checking is enabled
bundle_with_multiple_entriesBundleSaver: ../lib/Backend/BackendUtils.cpp:534: void glow::allocateActivations(const glow::IRFunction::InstListTy &, glow::MemoryAllocator &, glow::runtime::SymbolTableTy &): Assertion `!symbolTable.count(std::string(TV->getName())) && "Allocation already made!"' failed.
Aborted (core dumped)
[505/722] Generating init_net.pb
--2021-04-04 10:56:05--  http://fb-glow-assets.s3.amazonaws.com/models/resnet50/init_net.pb
Resolving fb-glow-assets.s3.amazonaws.com (fb-glow-assets.s3.amazonaws.com)... 52.217.111.164
Connecting to fb-glow-assets.s3.amazonaws.com (fb-glow-assets.s3.amazonaws.com)|52.217.111.164|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 128070759 (122M) [application/octet-stream]
Saving to: ‘/home/kittu28/chaitu/glow/build_Bundle/examples/bundles/resnet50/resnet50/init_net.pb’

     0K .......... .......... .......... .......... ..........  0%  110K 18m59s
    50K .......... .......... .......... .......... ..........  0%  224K 14m8s
   100K .......... .......... .......... .......... ..........  0% 8.48M 9m30s
   150K .......... .......... .......... .......... ..........  0%  309M 7m7s
   200K .......... .......... .......... .......... ..........  0%  227K 7m32s
   250K .......... .......... .......... .......... ..........  0% 10.4M 6m18s
   300K .......... .......... .......... .......... ..........  0%  232K 6m41s
   350K .......... .......... .......... .......... ..........  0% 7.66M 5m53s
   400K .......... .......... .......... .......... ..........  0% 12.0M 5m14s
   450K .......... .......... .......... .......... ..........  0%  224K 5m38s
   500K .......... .......... .......... .......... ..........  0% 36.7M 5m8s
   550K .......... .......... .......... .......... ..........  0% 52.0M 4m42s
   600K .......... .......... .......... .......... ..........  0% 34.8M 4m21s
   650K .......... .......... .......... .......... ..........  0% 47.2M 4m2s
   700K .......... .......... .......... .......... ..........  0% 1.59M 3m51s
   750K .......... .......... .......... .......... ..........  0% 1.93M 3m40s
   800K .......... .......... .......... .......... ..........  0%  308K 3m51s
   850K .......... .......... .......... .......... ..........  0% 5.83M 3m39s
   900K .......... .......... .......... .......... ..........  0%  120M 3m28s
   950K .......... .......... .......... .......... ..........  0% 93.4M 3m17s
  1000K .......... .......... .......... .......... ..........  0% 58.2M 3m8s
  1050K .......... .......... .......... .......... ..........  0% 42.0M 3m0s
  1100K .......... .......... .......... .......... ..........  0% 52.1M 2m52s
  1150K .......... .......... .......... .......... ..........  0% 64.2M 2m45s
  1200K .......... .......... .......... .......... ..........  0% 55.5M 2m38s
  1250K .......... .......... .......... .......... ..........  1% 2.17M 2m34s
  1300K .......... .......... .......... .......... ..........  1% 36.0M 2m28s
  1350K .......... .......... .......... .......... ..........  1% 32.6M 2m23s
  1400K .......... .......... .......... .......... ..........  1% 2.02M 2m20s
  1450K .......... .......... .......... .......... ..........  1%  313K 2m29s
  1500K .......... .......... .......... .......... ..........  1%  160M 2m24s
  1550K .......... .......... .......... .......... ..........  1% 11.4M 2m20s
  1600K .......... .......... .......... .......... ..........  1% 17.0M 2m16s
  1650K .......... .......... .......... .......... ..........  1% 5.75M 2m12s
  1700K .......... .......... .......... .......... ..........  1% 76.5M 2m8s
  1750K .......... .......... .......... .......... ..........  1% 5.05M 2m5s
  1800K .......... .......... .......... .......... ..........  1% 69.3M 2m2s
  1850K .......... .......... .......... .......... ..........  1% 51.9M 1m59s
  1900K .......... .......... .......... .......... ..........  1% 61.1M 1m56s
  1950K .......... .......... .......... .......... ..........  1% 40.7M 1m53s
  2000K .......... .......... .......... .......... ..........  1% 48.0M 1m50s
  2050K .......... .......... .......... .......... ..........  1% 43.9M 1m48s
  2100K .......... .......... .......... .......... ..........  1%  269M 1m45s
  2150K .......... .......... .......... .......... ..........  1% 24.7M 1m43s
  2200K .......... .......... .......... .......... ..........  1% 26.2M 1m40s
  2250K .......... .......... .......... .......... ..........  1% 56.8M 98s
  2300K .......... .......... .......... .......... ..........  1%  100M 96s
  2350K .......... .......... .......... .......... ..........  1% 95.5M 94s
  2400K .......... .......... .......... .......... ..........  1% 44.7M 92s
  2450K .......... .......... .......... .......... ..........  1% 22.0M 91s
  2500K .......... .......... .......... .......... ..........  2% 46.4M 89s
  2550K .......... .......... .......... .......... ..........  2% 3.84M 88s
  2600K .......... .......... .......... .......... ..........  2%  150M 86s
  2650K .......... .......... .......... .......... ..........  2%  102M 84s
  2700K .......... .......... .......... .......... ..........  2% 5.76M 83s
  2750K .......... .......... .......... .......... ..........  2% 17.4M 82s
  2800K .......... .......... .......... .......... ..........  2%  323K 87s
  2850K .......... .......... .......... .......... ..........  2% 22.2M 85s
  2900K .......... .......... .......... .......... ..........  2% 82.4M 84s
  2950K .......... .......... .......... .......... ..........  2% 11.4M 83s
  3000K .......... .......... .......... .......... ..........  2% 35.1M 81s
  3050K .......... .......... .......... .......... ..........  2%  221M 80s
  3100K .......... .......... .......... .......... ..........  2% 17.1M 79s
  3150K .......... .......... .......... .......... ..........  2% 7.27M 78s
  3200K .......... .......... .......... .......... ..........  2% 37.6M 77s
  3250K .......... .......... .......... .......... ..........  2% 17.3M 76s
  3300K .......... .......... .......... .......... ..........  2% 27.0M 75s
  3350K .......... .......... .......... .......... ..........  2%  318M 73s
  3400K .......... .......... .......... .......... ..........  2% 2.26M 73s
  3450K .......... .......... .......... .......... ..........  2% 11.1M 72s
  3500K .......... .......... .......... .......... ..........  2% 68.3M 71s
  3550K .......... .......... .......... .......... ..........  2% 15.0M 70s
  3600K .......... .......... .......... .......... ..........  2% 37.5M 69s
  3650K .......... .......... .......... .......... ..........  2% 5.01M 69s
  3700K .......... .......... .......... .......... ..........  2% 30.5M 68s
  3750K .......... .......... .......... .......... ..........  3% 36.6M 67s
  3800K .......... .......... .......... .......... ..........  3% 24.0M 66s
  3850K .......... .......... .......... .......... ..........  3% 65.7M 65s
  3900K .......... .......... .......... .......... ..........  3% 40.8M 64s
  3950K .......... .......... .......... .......... ..........  3% 5.19M 64s
  4000K .......... .......... .......... .......... ..........  3%  202M 63s
  4050K .......... .......... .......... .......... ..........  3% 70.8M 62s
  4100K .......... .......... .......... .......... ..........  3% 84.8M 61s
  4150K .......... .......... .......... .......... ..........  3%  104M 61s
  4200K .......... .......... .......... .......... ..........  3%  109M 60s
  4250K .......... .......... .......... .......... ..........  3% 81.4M 59s
  4300K .......... .......... .......... .......... ..........  3%  142M 59s
  4350K .......... .......... .......... .......... ..........  3%  191M 58s
  4400K .......... .......... .......... .......... ..........  3%  183M 57s
  4450K .......... .......... .......... .......... ..........  3%  185M 57s
  4500K .......... .......... .......... .......... ..........  3%  167M 56s
  4550K .......... .......... .......... .......... ..........  3%  155M 55s
  4600K .......... .......... .......... .......... ..........  3% 74.5M 55s
  4650K .......... .......... .......... .......... ..........  3% 88.6M 54s
  4700K .......... .......... .......... .......... ..........  3% 75.7M 54s
  4750K .......... .......... .......... .......... ..........  3% 70.1M 53s
  4800K .......... .......... .......... .......... ..........  3% 76.6M 52s
  4850K .......... .......... .......... .......... ..........  3% 99.4M 52s
  4900K .......... .......... .......... .......... ..........  3% 69.3M 51s
  4950K .......... .......... .......... .......... ..........  3% 71.3M 51s
  5000K .......... .......... .......... .......... ..........  4%  108M 50s
  5050K .......... .......... .......... .......... ..........  4% 93.1M 50s
  5100K .......... .......... .......... .......... ..........  4% 79.9M 49s
  5150K .......... .......... .......... .......... ..........  4% 76.7M 49s
  5200K .......... .......... .......... .......... ..........  4% 96.1M 48s
  5250K .......... .......... .......... .......... ..........  4% 83.6M 48s
  5300K .......... .......... .......... .......... ..........  4% 84.5M 47s
  5350K .......... .......... .......... .......... ..........  4% 79.2M 47s
  5400K .......... .......... .......... .......... ..........  4% 99.0M 47s
  5450K .......... .......... .......... .......... ..........  4%  405K 49s
  5500K .......... .......... .......... .......... ..........  4% 24.0M 48s
  5550K .......... .......... .......... .......... ..........  4% 3.96M 48s
  5600K .......... .......... .......... .......... ..........  4% 49.1M 48s
  5650K .......... .......... .......... .......... ..........  4% 44.4M 47s
  5700K .......... .......... .......... .......... ..........  4% 51.8M 47s
  5750K .......... .......... .......... .......... ..........  4% 46.5M 47s
  5800K .......... .......... .......... .......... ..........  4% 44.2M 46s
  5850K .......... .......... .......... .......... ..........  4% 34.4M 46s
  5900K .......... .......... .......... .......... ..........  4% 51.7M 45s
  5950K .......... .......... .......... .......... ..........  4% 6.71M 45s
  6000K .......... .......... .......... .......... ..........  4%  114M 45s
  6050K .......... .......... .......... .......... ..........  4% 82.2M 44s
  6100K .......... .......... .......... .......... ..........  4%  113M 44s
  6150K .......... .......... .......... .......... ..........  4%  182M 44s
  6200K .......... .......... .......... .......... ..........  4%  124M 43s
  6250K .......... .......... .......... .......... ..........  5%  164M 43s
  6300K .......... .......... .......... .......... ..........  5%  185M 43s
  6350K .......... .......... .......... .......... ..........  5%  176M 42s
  6400K .......... .......... .......... .......... ..........  5%  184M 42s
  6450K .......... .......... .......... .......... ..........  5% 18.5M 42s
  6500K .......... .......... .......... .......... ..........  5% 81.4M 41s
  6550K .......... .......... .......... .......... ..........  5% 52.0M 41s
  6600K .......... .......... .......... .......... ..........  5% 70.4M 41s
  6650K .......... .......... .......... .......... ..........  5% 60.7M 40s
  6700K .......... .......... .......... .......... ..........  5% 1.61M 41s
  6750K .......... .......... .......... .......... ..........  5% 55.1M 40s
  6800K .......... .......... .......... .......... ..........  5% 76.2M 40s
  6850K .......... .......... .......... .......... ..........  5% 67.0M 40s
  6900K .......... .......... .......... .......... ..........  5% 44.8M 39s
  6950K .......... .......... .......... .......... ..........  5% 56.9M 39s
  7000K .......... .......... .......... .......... ..........  5% 82.7M 39s
  7050K .......... .......... .......... .......... ..........  5% 40.8M 39s
  7100K .......... .......... .......... .......... ..........  5% 71.1M 38s
  7150K .......... .......... .......... .......... ..........  5% 60.2M 38s
  7200K .......... .......... .......... .......... ..........  5% 64.4M 38s
  7250K .......... .......... .......... .......... ..........  5% 43.2M 38s
  7300K .......... .......... .......... .......... ..........  5% 28.1M 37s
  7350K .......... .......... .......... .......... ..........  5%  214M 37s
  7400K .......... .......... .......... .......... ..........  5% 26.3M 37s
  7450K .......... .......... .......... .......... ..........  5% 12.5M 37s
  7500K .......... .......... .......... .......... ..........  6%  252M 36s
  7550K .......... .......... .......... .......... ..........  6%  184M 36s
  7600K .......... .......... .......... .......... ..........  6%  263M 36s
  7650K .......... .......... .......... .......... ..........  6% 15.3M 36s
  7700K .......... .......... .......... .......... ..........  6%  206M 35s
  7750K .......... .......... .......... .......... ..........  6%  284M 35s
  7800K .......... .......... .......... .......... ..........  6%  347M 35s
  7850K .......... .......... .......... .......... ..........  6%  250M 35s
  7900K .......... .......... .......... .......... ..........  6%  207M 34s
  7950K .......... .......... .......... .......... ..........  6%  264M 34s
  8000K .......... .......... .......... .......... ..........  6% 13.1M 34s
  8050K .......... .......... .......... .......... ..........  6% 61.6M 34s
  8100K .......... .......... .......... .......... ..........  6% 55.7M 34s
  8150K .......... .......... .......... .......... ..........  6% 66.4M 33s
  8200K .......... .......... .......... .......... ..........  6% 73.2M 33s
  8250K .......... .......... .......... .......... ..........  6% 48.8M 33s
  8300K .......... .......... .......... .......... ..........  6% 63.8M 33s
  8350K .......... .......... .......... .......... ..........  6% 58.6M 33s
  8400K .......... .......... .......... .......... ..........  6% 77.8M 32s
  8450K .......... .......... .......... .......... ..........  6% 63.4M 32s
  8500K .......... .......... .......... .......... ..........  6% 58.1M 32s
  8550K .......... .......... .......... .......... ..........  6% 43.0M 32s
  8600K .......... .......... .......... .......... ..........  6% 60.8M 32s
  8650K .......... .......... .......... .......... ..........  6% 10.1M 32s
  8700K .......... .......... .......... .......... ..........  6%  191M 31s
  8750K .......... .......... .......... .......... ..........  7% 37.4M 31s
  8800K .......... .......... .......... .......... ..........  7% 16.9M 31s
  8850K .......... .......... .......... .......... ..........  7%  166M 31s
  8900K .......... .......... .......... .......... ..........  7%  220M 31s
  8950K .......... .......... .......... .......... ..........  7%  137M 30s
  9000K .......... .......... .......... .......... ..........  7%  138M 30s
  9050K .......... .......... .......... .......... ..........  7%  184M 30s
  9100K .......... .......... .......... .......... ..........  7%  308M 30s
  9150K .......... .......... .......... .......... ..........  7%  336M 30s
  9200K .......... .......... .......... .......... ..........  7% 23.8M 30s
  9250K .......... .......... .......... .......... ..........  7% 15.4M 29s
  9300K .......... .......... .......... .......... ..........  7% 22.7M 29s
  9350K .......... .......... .......... .......... ..........  7% 21.0M 29s
  9400K .......... .......... .......... .......... ..........  7% 11.8M 29s
  9450K .......... .......... .......... .......... ..........  7% 20.5M 29s
  9500K .......... .......... .......... .......... ..........  7% 19.6M 29s
  9550K .......... .......... .......... .......... ..........  7% 16.2M 29s
  9600K .......... .......... .......... .......... ..........  7% 14.9M 29s
  9650K .......... .......... .......... .......... ..........  7% 17.8M 28s
  9700K .......... .......... .......... .......... ..........  7% 14.6M 28s
  9750K .......... .......... .......... .......... ..........  7% 29.9M 28s
  9800K .......... .......... .......... .......... ..........  7% 13.0M 28s
  9850K .......... .......... .......... .......... ..........  7% 23.1M 28s
  9900K .......... .......... .......... .......... ..........  7% 13.0M 28s
  9950K .......... .......... .......... .......... ..........  7% 16.5M 28s
 10000K .......... .......... .......... .......... ..........  8%  223K 30s
 10050K .......... .......... .......... .......... ..........  8% 12.8M 30s
 10100K .......... .......... .......... .......... ..........  8% 14.2M 30s
 10150K .......... .......... .......... .......... ..........  8% 41.6M 30s
 10200K .......... .......... .......... .......... ..........  8% 22.4M 30s
 10250K .......... .......... .......... .......... ..........  8% 27.4M 29s
 10300K .......... .......... .......... .......... ..........  8% 23.8M 29s
 10350K .......... .......... .......... .......... ..........  8%  233K 32s
 10400K .......... .......... .......... .......... ..........  8% 33.7M 31s
 10450K .......... .......... .......... .......... ..........  8% 58.8M 31s
 10500K .......... .......... .......... .......... ..........  8% 17.1M 31s
 10550K .......... .......... .......... .......... ..........  8% 51.0M 31s
 10600K .......... .......... .......... .......... ..........  8% 16.0M 31s
 10650K .......... .......... .......... .......... ..........  8% 22.4M 31s
 10700K .......... .......... .......... .......... ..........  8%  234K 33s
 10750K .......... .......... .......... .......... ..........  8% 9.89M 33s
 10800K .......... .......... .......... .......... ..........  8% 46.2M 33s
 10850K .......... .......... .......... .......... ..........  8% 50.8M 32s
 10900K .......... .......... .......... .......... ..........  8% 39.4M 32s
 10950K .......... .......... .......... .......... ..........  8% 36.0M 32s
 11000K .......... .......... .......... .......... ..........  8% 23.9M 32s
 11050K .......... .......... .......... .......... ..........  8% 25.3M 32s
 11100K .......... .......... .......... .......... ..........  8% 56.5M 32s
 11150K .......... .......... .......... .......... ..........  8%  109M 32s
 11200K .......... .......... .......... .......... ..........  8%  236K 34s
 11250K .......... .......... .......... .......... ..........  9% 45.3M 33s
 11300K .......... .......... .......... .......... ..........  9% 21.8M 33s
 11350K .......... .......... .......... .......... ..........  9% 12.6M 33s
 11400K .......... .......... .......... .......... ..........  9% 38.0M 33s
 11450K .......... .......... .......... .......... ..........  9% 43.8M 33s
 11500K .......... .......... .......... .......... ..........  9% 38.7M 33s
 11550K .......... .......... .......... .......... ..........  9% 28.5M 33s
 11600K .......... .......... .......... .......... ..........  9% 39.9M 32s
 11650K .......... .......... .......... .......... ..........  9% 30.5M 32s
 11700K .......... .......... .......... .......... ..........  9% 20.7M 32s
 11750K .......... .......... .......... .......... ..........  9% 36.5M 32s
 11800K .......... .......... .......... .......... ..........  9% 47.0M 32s
 11850K .......... .......... .......... .......... ..........  9% 3.75M 32s
 11900K .......... .......... .......... .......... ..........  9%  265K 33s
 11950K .......... .......... .......... .......... ..........  9% 20.7M 33s
 12000K .......... .......... .......... .......... ..........  9% 20.0M 33s
 12050K .......... .......... .......... .......... ..........  9% 33.8M 33s
 12100K .......... .......... .......... .......... ..........  9% 40.9M 33s
 12150K .......... .......... .......... .......... ..........  9% 18.6M 33s
 12200K .......... .......... .......... .......... ..........  9% 17.4M 33s
 12250K .......... .......... .......... .......... ..........  9% 21.4M 33s
 12300K .......... .......... .......... .......... ..........  9% 36.7M 32s
 12350K .......... .......... .......... .......... ..........  9% 99.2M 32s
 12400K .......... .......... .......... .......... ..........  9% 14.0M 32s
 12450K .......... .......... .......... .......... ..........  9% 54.2M 32s
 12500K .......... .......... .......... .......... .......... 10% 32.8M 32s
 12550K .......... .......... .......... .......... .......... 10% 14.2M 32s
 12600K .......... .......... .......... .......... .......... 10% 52.2M 32s
 12650K .......... .......... .......... .......... .......... 10% 32.0M 32s
 12700K .......... .......... .......... .......... .......... 10% 24.6M 31s
 12750K .......... .......... .......... .......... .......... 10% 33.1M 31s
 12800K .......... .......... .......... .......... .......... 10%  126M 31s
 12850K .......... .......... .......... .......... .......... 10%  153M 31s

 **(deliberately cut here...the process carried out here ...)**

124050K .......... .......... .......... .......... .......... 99% 53.0M 3s
124100K .......... .......... .......... .......... .......... 99%  162K 3s
124150K .......... .......... .......... .......... .......... 99% 40.9M 3s
124200K .......... .......... .......... .......... .......... 99% 94.5K 3s
124250K .......... .......... .......... .......... .......... 99%  180M 2s
124300K .......... .......... .......... .......... .......... 99%  173K 2s
124350K .......... .......... .......... .......... .......... 99%  197K 2s
124400K .......... .......... .......... .......... .......... 99%  195K 2s
124450K .......... .......... .......... .......... .......... 99% 76.2M 2s
124500K .......... .......... .......... .......... .......... 99%  163K 2s
124550K .......... .......... .......... .......... .......... 99%  222K 1s
124600K .......... .......... .......... .......... .......... 99% 43.9M 1s
124650K .......... .......... .......... .......... .......... 99%  224K 1s
124700K .......... .......... .......... .......... .......... 99%  185K 1s
124750K .......... .......... .......... .......... .......... 99% 30.4M 1s
124800K .......... .......... .......... .......... .......... 99%  165K 1s
124850K .......... .......... .......... .......... .......... 99%  255M 1s
124900K .......... .......... .......... .......... .......... 99%  219K 0s
124950K .......... .......... .......... .......... .......... 99%  176K 0s
125000K .......... .......... .......... .......... .......... 99% 27.1M 0s
125050K .......... .........                                  100% 62.7K=6m22s

2021-04-04 11:02:27 (327 KB/s) - ‘/home/kittu28/chaitu/glow/build_Bundle/examples/bundles/resnet50/resnet50/init_net.pb’ saved [128070759/128070759]

[506/722] Building CXX object tests/unittests/CMakeFiles/CPUOperatorTest.dir/OperatorTest.cpp.o
ninja: build stopped: subcommand failed.


Question:
resnet-runtime: /home/bucky/Desktop/compiler/glow/include/qsnn/Support/Error.h:186: void glow::detail::CheckState<true>::ensureChecked() const: Assertion `checked_ && "Unchecked Error or Expected"' failed.
Aborted (core dumped)

I am facing this error and after a little debugging I understood that below line of code is responsible for the error. But couldnt figure out how to get rid of this error.
Also, I am only facing this error in debug mode, it runs fine in the release mode and produce correct results.
glow::splitNode(Node, SplitNodeByNumChunks({1}, {2}));

Context:
splitNode returns Expected<std::vector<Node *>> . So you need something like:
std::vector<Node *> splitNodes;
ASSIGN_VALUE_OR_FATAL(splitNodes, glow::splitNode(Node, SplitNodeByNumChunks({1}, {2})))

Even if you don’t need splitNodes. Or else you can handle the error using your own custom code, check Error.h for ASSIGN_VALUE_OR_FATAL and other options.


Question:
Hi Everyone.
My objective is to obfuscate my model so that the clients can’t look into model and the inference code. If I package my model and code to standalone executables using Glow, will this objective be achieved? I did see that Glow supports AOT compilation. Will this solve the problem?

Context: It should do a decent job, you’ll end up with a binary for running the model itself along with binary weights to load. Assuming you strip out any debug info then they’d have to understand the model based on machine code.


NCHW layout as default for a new accelerator device

Question:
Hello everyone,
I’m trying to build a backend for a new accelerator device in Glow.
It happens that this device only works with the NCHW layout (Convolution, etc.). I used OpenCL Backend as base but I still see in HIR, after optimizations, transpose operations as well as convolution operation in NHWC layout. What am I not doing? The models I’m using (lenet-mnist, resnet50) come from ONNX, in NCHW layout.

Context:
Hi @iviarcio, the OCL backend does this transformation in the below location for convolutions, did you also copy this transforms file over for your backend? Perhaps it needs to be tweaked for your usecase. I’d suggest debugging why the below code isn’t being run on your backend.

      github.com

    pytorch/glow/blob/778d44eff16f8dfac25503e0f917531d71bc950b/lib/Backends/OpenCL/Transforms.cpp#L62 1

            if (CN->getLayout() == NCHW) {
              continue;
            }

  // If there is no compiler controlled local memory on the device,
            // try to avoid kernels that use (additional) copies to local memory.
            if (devInfo != nullptr && devInfo->availableLocalMemory == 0) {
              continue;
            }

  auto *NR = convertConvToNCHWConv(CN, F);
            CN->getResult().replaceAllUsesOfWith(NR);
            changed = true;
            continue;
          }
          if (auto *PMN = dyn_cast<MaxPoolNode>(&node)) {
            if (cctx.compMode == CompilationMode::Train) {
              continue;
            }

  if (PMN->getLayout() == NCHW) {


Question:
Hello
I have a specific question about how resize instruction is implemented in the OpenCL backend. A resize instruction is not apparently supported by the OpenCL backend. It is not in the list of isOpSupported.
How is such type of unsupported operation by the OpenCL backend executed? I guess resize is executed in the host environment and the output is exchanged back and force with the device. I’m not sure which part of Glow is responsible for that.
It would be great if someone could give me a pointer to the code I should look into.
Best

Context:
It is not in the list of isOpSupported.

It is supported, perhaps you just missed it, it’s listed here:

      github.com

    pytorch/glow/blob/778d44eff16f8dfac25503e0f917531d71bc950b/lib/Backends/OpenCL/OpenCL.cpp#L1931

                  {ElemKind::FloatTy}, {},
                  {ConvolutionGradNode::GradOfInputNamedInputIdx});

  case Kinded::Kind::SoftMaxGradNodeKind:
              return NI.allInputsAndOutputsHaveSameElemKind(
                         {ElemKind::FloatTy}, {SoftMaxGradNode::SelectedIdx},
                         {SoftMaxGradNode::GradOfInputNamedSelectedIdx}) &&
                     (NI.getInElemTy(SoftMaxGradNode::SelectedIdx) == ElemKind::Int64ITy);

  case Kinded::Kind::SaveNodeKind:
            case Kinded::Kind::ReshapeNodeKind:
            case Kinded::Kind::OCLBatchedReduceAddNodeKind:
            case Kinded::Kind::TraceEventNodeKind:
              // These work regardless of the underlying type.
              return true;

  default:
              return false;
            }
          }


Question:
I’d like to ask if there is any way (or plan for) adding to Glow ability to use NCHW by default instead of NHWC? For now it is said in some places that Glow’s default is NHWC, but at the same time, many times methods to generate NCHW layout are used.
I strongly doubt that all of the backends will use NHWC, doing conversion instead - for example convertConvToNCHWConv. Such approach will significantly affect performance by introducing all of those Transpose operations. Instead it should be allowed from the very beginning to choose which format will be used. E.g. Tensorflow allows to chose how formatting will be done.

Context: Yep, that’s exactly right!  The OpenCL approach is a good place to look for the NCHW.  And please do let us know if you’re seeing transposes that aren’t eliminated with that approach; it’s always possible to improve the optimizer.


Steps/ideas for layer-wise quantization implementation

Question:
Hi,
I am interested in implementing layer-wise quantization in glow.
I have looked through the issue just about this topic ([Quantization] layer-wise quantization · Issue #4982 · pytorch/glow · GitHub 3), but it seems to be left untouched.
My question is, given node list of type glow::Node (or node name list e.g. Conv_conv_1__2) in a model, and each node has its quantization options respectively (for example, these options can be enableChannelwiseOpt, quantizationCalibrationOpt, keepOriginalPrecisionForNodesOpt, …), how do we incorporate these info to the code? Below I will explain a little more backgrounds.
Model profiling or model compiling, whichever it is, glow takes roughly three steps for the model loader:

load the model (loader.loadModel())
get compilation settings (loader.getCompilationContext(QuantizationMode))
compile the model using the CompilationContext config (loader.compile(cctx))

For simplicity, I think the first thing to do is load and add quantization options to all of the nodes in a model, in 2.. To contain these info, we can add a map to precConfig of type PrecisionConfiguration. For example, the map’s key is the node names, and the value is also a map, with the key quantization options and the value the actual preferences. This is done by modifying Loader::getCompilationContext function in Loader.cpp.
Then, modify transformForPrecisionMode function in GraphOptimizer.cpp, as mentioned at the issue link above. This function is called in optimizeFunction in the same file, which is called in Loader::compile function in step 3. (I suppose).
In transformForPrecisionMode, if we conduct profiling, profileQuantization function in GraphOptimizer/Quantization.cpp will be called. Or if we want to execute quantization, quantizeFunction function in quantization/Quantization.cpp will be called.
Either way, I am not sure exactly where to relate the above layer-wise config to these functions. I don’t see modification points in the former case (profiling), because profiling nodes themselves being inserted are not related to configs like calibration method and precision selection, and these are for executing quantization. The latter, however, since the quantizeFunction function creates an FunctionQuantizer instance inside it, I know there are some member functions to be modified in that class. Although I looked at the class, it"s a little complex and I didn’t manage to find where to implement.
Also, glow::lower function in Lower.cpp needs some expansion, since KindSet doNotLowerKinds is included there. What do you think?
To sum up, it is not clear to me that what functions called in/after transformForPrecisionMode should be modified, and how. Additionally if any functions other than transformForPrecisionMode needs to be changed, I would like to know what they are.
Any hints or thoughts?

Context: We actually have https://github.com/pytorch/glow/pull/5145 7 which is pretty similar and I should land soon. Does it help solve what you’re looking for?