Dealing with unsupported operations with NNAPI

username: julien-thierry
cooked: <p>Hi,</p>
<p>I am trying to run a model on android with NNAPI, however I have to deal with some unsupported simple constructions:</p>
<ul>
<li>Addition between constant and tensor (e.g. <code>X + 1</code>)</li>
<li>Division of tensor by a constant (e.g. <code>X / 4</code>)</li>
</ul>
<p>For the first one, is there any easy way to convert it to be supported by NNAPI? (I tried creating a torch.ones() to replace the constant, but creating new tensor while running on NNAPI doesn’t appear to be supported)</p>
<p>For the second one, I understand that dividing tensors is not currently supported for NNAPI. Is there a clean way to call a non-NNAPI module/function from a module running through NNAPI in order to still perform these operation?</p>
<p>Thanks,</p>
<p>Julien</p>

username: julien-thierry
cooked: <p>Any suggestions on the matter?</p>

username: axitkhurana
cooked: <p>Hi there, apologies for the delay.</p>
<p>For addition, that seems to be something we need to fix. Looks like X + constant (tensor or scalar) is failing. X + Y works where Y can be torch.ones(1).</p>
<p>Division op is in the works! Dividing by constant might have the same issue but I’ll take a look.</p>

username: julien-thierry
cooked: <p>Hi,</p>
<p>Thanks for the reply. The torch.ones(1) does actually seems to work but only for non-quantized models:</p>
<pre><code class="lang-auto">class TestNNAPI(torch.nn.Module):
    def __init__(self):
        super(TestNNAPI, self).__init__()
        self.conv = torch.nn.Conv2d(1, 6,  kernel_size=3, padding=1, stride=1)
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.dequant(x)
        res = x + torch.ones(1)
        return res
</code></pre>
<p>Tracing and converting this to NNAPI seems to work smoothly, however if quantization (using ‘qnnpack’ backend at least) is done on the model I get the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "./test-constant.py", line 36, in &lt;module&gt;
    traced_nnapi_model = torch.backends._nnapi.prepare.convert_model_to_nnapi(traced_nnapi_model, testinput)
  File "&lt;install&gt;/lib/python3.6/site-packages/torch/backends/_nnapi/prepare.py", line 85, in convert_model_to_nnapi
    ser_model, used_weights, inp_mem_fmts, out_mem_fmts, shape_compute_lines, retval_count = serialize_model(model, inputs)
  File "&lt;install&gt;/lib/python3.6/site-packages/torch/backends/_nnapi/serializer.py", line 1739, in serialize_model
    return _NnapiSerializer(config).serialize_model(module, inputs)
  File "&lt;install&gt;/lib/python3.6/site-packages/torch/backends/_nnapi/serializer.py", line 628, in serialize_model
    self.add_node(node)
  File "&lt;install&gt;/lib/python3.6/site-packages/torch/backends/_nnapi/serializer.py", line 793, in add_node
    adder(self, node)
  File "&lt;install&gt;/lib/python3.6/site-packages/torch/backends/_nnapi/serializer.py", line 750, in &lt;lambda&gt;
    self.add_add_sub_op(node, NNAPI_OperationCode.ADD, NNAPI_FuseCode.FUSED_NONE),
  File "&lt;install&gt;/lib/python3.6/site-packages/torch/backends/_nnapi/serializer.py", line 1119, in add_add_sub_op
    self._do_add_binary(node, opcode, fuse_code)
  File "&lt;install&gt;/.local/lib/python3.6/site-packages/torch/backends/_nnapi/serializer.py", line 1074, in _do_add_binary
    in1_id, in1_oper = self.get_tensor_operand_by_jitval(node.inputsAt(1))
  File "&lt;install&gt;/lib/python3.6/site-packages/torch/backends/_nnapi/serializer.py", line 456, in get_tensor_operand_by_jitval
    operand_id = self.jitval_operand_map[jitval]
KeyError: 31 defined in (%31 : Float(1, strides=[1], requires_grad=0, device=cpu) = prim::Constant[value={1}]()
)
</code></pre>
<p>Which I find a bit surprising since the ones is not even used in quantized context. (And if I try to have a quantized torch.ones, I get the error <code> Could not run 'aten::empty.memory_format' with arguments from the 'QuantizedCPU' backend.</code></p>
<p>Is this a known limitation?</p>

username: David_Reiss
cooked: <p>Can you post the full script you’re using to reproduce this?</p>

username: julien-thierry
cooked: <p>Yes, here’s the reproducer:</p>
<pre><code class="lang-auto">#! /usr/bin/python3                                                                                                                                                                                                                    

import torch
import torch.backends._nnapi.prepare

class TestNNAPI(torch.nn.Module):
    def __init__(self):
        super(TestNNAPI, self).__init__()
        self.conv = torch.nn.Conv2d(1, 6,  kernel_size=3, padding=1, stride=1)
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.dequant(x)
        res = x + torch.ones(1)
        return res

nnapi_model = TestNNAPI()
nnapi_model.eval()
nnapi_model.qconfig = torch.quantization.get_default_qconfig('qnnpack')
torch.backends.quantized.engine = 'qnnpack'

testinput = torch.rand(6, 1, 600, 800)
testinput.nnapi_nhwc = True

torch.quantization.prepare(nnapi_model, inplace=True)
nnapi_model(testinput)
torch.quantization.convert(nnapi_model, inplace=True)

traced_nnapi_model = torch.jit.trace(nnapi_model, testinput)
traced_nnapi_model = torch.backends._nnapi.prepare.convert_model_to_nnapi(traced_nnapi_model, testinput)
</code></pre>
<p>If I remove the quantization.prepare/convert , the script runs fine.</p>

username: julien-thierry
cooked: <p>Hi David,</p>
<p>Do you have any update regarding the reproducer? Is there a workaround for this issue?</p>
<p>Thanks,</p>
<p>Julien</p>

username: David_Reiss
cooked: <p>Sorry for the late reply.  I’ll try to add support for this.</p>

username: David_Reiss
cooked: <p>I was able to get this working (without broadcasting) with this change: <a href="https://gist.github.com/dreiss/c5a0a6c093c3b798e4c1a4bc3c79b3ce" class="inline-onebox">change.patch · GitHub</a></p>
<p>It’s a bit half-baked, though.  I’ll do some cleanup and should get this landed in time for the next release.</p>