How does pytorch determine which backend a function supports

Ofir_Zafrir: <p>Hi,</p>
<p>I wrote a c++ function that receives a quantized tensor and outputs a fp32 tensor. everything compiled without errors and the python binding also works but when i try to use the function i get the following error:</p>
<p><code>RuntimeError: Could not run 'quantized::linear_my' with arguments from the 'QuantizedCPU' backend. 'quantized::linear_my' is only available for these backends: [].</code></p>
<p>so my question is how does pytorch determine which backends does my function support?<br>
how can i fix this state where my function doesnt support any backend?</p>
<p>Thanks,<br>
Ofir</p>

tom: <p>Are you using <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_classes.html" rel="nofollow noopener">TorchBind</a> rather than PyBind11?</p>

Ofir_Zafrir: <p>I am not sure, I used the same method that was used in the torch library, I modified code inside the library.<br>
I resolved the problem in the end, it was a compilation problem, cleaning the project resolved it.</p>

jerryzh168: <p>you can take a look at how we implement quantized ops:<br>
we need to declare the signature:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/library.cpp#L23" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/library.cpp#L23" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/library.cpp#L23</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="13" style="counter-reset: li-counter 12 ;">
<li>extern template torch::class_&lt;ConvPackedParamsBase&lt;2&gt;&gt; register_conv_params&lt;2&gt;();</li>
<li>extern template torch::class_&lt;ConvPackedParamsBase&lt;3&gt;&gt; register_conv_params&lt;3&gt;();</li>
<li>torch::class_&lt;EmbeddingPackedParamsBase&gt; register_embedding_params();</li>
<li>
</li><li>TORCH_LIBRARY(quantized, m) {</li>
<li>  register_linear_params();</li>
<li>  register_conv_params&lt;2&gt;();</li>
<li>  register_conv_params&lt;3&gt;();</li>
<li>  register_embedding_params();</li>
<li>
</li><li class="selected">  m.def(TORCH_SELECTIVE_SCHEMA("quantized::add(Tensor qa, Tensor qb, float scale, int zero_point) -&gt; Tensor qc"));</li>
<li>  m.def(TORCH_SELECTIVE_SCHEMA("quantized::add.out(Tensor qa, Tensor qb, Tensor(a!) out) -&gt; Tensor(a!) out"));</li>
<li>  m.def(TORCH_SELECTIVE_SCHEMA("quantized::add.Scalar(Tensor qa, Scalar b) -&gt; Tensor qc"));</li>
<li>  m.def(TORCH_SELECTIVE_SCHEMA("quantized::add.Scalar_out(Tensor qa, Scalar b, Tensor(a!) out) -&gt; Tensor(a!) out"));</li>
<li>  m.def(TORCH_SELECTIVE_SCHEMA("quantized::add_relu(Tensor qa, Tensor qb, float scale, int zero_point) -&gt; Tensor qc"));</li>
<li>  m.def(TORCH_SELECTIVE_SCHEMA("quantized::add_relu.Scalar(Tensor qa, Scalar b) -&gt; Tensor qc"));</li>
<li>  m.def(TORCH_SELECTIVE_SCHEMA("quantized::add_relu.out(Tensor qa, Tensor qb, Tensor(a!) out) -&gt; Tensor(a!) out"));</li>
<li>  m.def(TORCH_SELECTIVE_SCHEMA("quantized::add_relu.Scalar_out(Tensor qa, Scalar b, Tensor(a!) out) -&gt; Tensor(a!) out"));</li>
<li>  // deprecated functions, kept for backward compatibility</li>
<li>  m.def(TORCH_SELECTIVE_SCHEMA("quantized::add_out(Tensor qa, Tensor qb, Tensor(a!) out) -&gt; Tensor(a!) out"));</li>
<li>  m.def(TORCH_SELECTIVE_SCHEMA("quantized::add_relu_out(Tensor qa, Tensor qb, Tensor(a!) out) -&gt; Tensor(a!) out"));</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>and then implement it:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qadd.cpp#L268" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qadd.cpp#L268" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qadd.cpp#L268</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="258" style="counter-reset: li-counter 257 ;">
<li>}</li>
<li>
</li><li>// `torch.jit.trace` will trace Scalar as Tensor</li>
<li>// This can be removed after broadcast is supported and</li>
<li>// all variations of `quantized::add` is merged into `quantized::add`</li>
<li>template &lt;bool ReLUFused = false&gt;</li>
<li>Tensor qadd_scalar_tensor_out(Tensor qa, Tensor b, Tensor out) {</li>
<li>  return qadd_scalar_out(qa, b.item(), out);</li>
<li>}</li>
<li>
</li><li class="selected">TORCH_LIBRARY_IMPL(quantized, QuantizedCPU, m) {</li>
<li>  m.impl(TORCH_SELECTIVE_NAME("quantized::add"),                 TORCH_FN(qadd&lt;/*ReLUFused=*/false&gt;));</li>
<li>  m.impl(TORCH_SELECTIVE_NAME("quantized::add.out"),             TORCH_FN(qadd_out&lt;/*ReLUFused=*/false&gt;));</li>
<li>  m.impl(TORCH_SELECTIVE_NAME("quantized::add.Scalar"),          TORCH_FN(qadd_scalar&lt;/*ReLUFused=*/false&gt;));</li>
<li>  m.impl(TORCH_SELECTIVE_NAME("quantized::add.Scalar_out"),      TORCH_FN(qadd_scalar_out&lt;/*ReLUFused=*/false&gt;));</li>
<li>  m.impl(TORCH_SELECTIVE_NAME("quantized::add_relu"),            TORCH_FN(qadd&lt;/*ReLUFused=*/true&gt;));</li>
<li>  m.impl(TORCH_SELECTIVE_NAME("quantized::add_relu.out"),        TORCH_FN(qadd_out&lt;/*ReLUFused=*/true&gt;));</li>
<li>  m.impl(TORCH_SELECTIVE_NAME("quantized::add_relu.Scalar"),     TORCH_FN(qadd_scalar&lt;/*ReLUFused=*/true&gt;));</li>
<li>  m.impl(TORCH_SELECTIVE_NAME("quantized::add_relu.Scalar_out"), TORCH_FN(qadd_scalar_out&lt;/*ReLUFused=*/true&gt;));</li>
<li>  // deprecated functions, kept for backward compatibility</li>
<li>  m.impl(TORCH_SELECTIVE_NAME("quantized::add_out"),             TORCH_FN(qadd_out&lt;/*ReLUFused=*/false&gt;));</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>