Could not run 'quantized::cat' with arguments from the 'QuantizedCUDA' backend

MrOCW: <p>Hi, I have replaced torch.cat with self.ff.cat as I encountered an error with torch.cat</p>
<p>After replacing, I got another error which is:</p>
<blockquote>
<p>NotImplementedError: Could not run ‘quantized::cat’ with arguments from the ‘QuantizedCUDA’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘quantized::cat’ is only available for these backends: [QuantizedCPU, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, Tracer, Autocast, Batched, VmapMode].</p>
</blockquote>
<p>and so, i tried adding x.to(“cpu”) since QuantizedCPU is available but i got YET another error pointing to x.to(“cpu”):</p>
<blockquote>
<p>NotImplementedError: Could not run ‘aten::empty_strided’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::empty_strided’ is only available for these backends: [CPU, CUDA, Meta, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].</p>
</blockquote>

jerryzh168: <aside class="quote no-group quote-modified" data-username="MrOCW" data-post="1" data-topic="133510">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/90db22/48.png" class="avatar"> MrOCW:</div>
<blockquote>
<blockquote>
<p>NotImplementedError: Could not run ‘aten::empty_strided’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::empty_strided’ is only available for these backends: [CPU, CUDA, Meta, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].</p>
</blockquote>
</blockquote>
</aside>
<p>we do not have any operators for QuantizedCUDA backend currently, although the other path (x.to(“cou”)) should work, which pytorch version are you using? or are you using master. cc <a class="mention" href="/u/hdcharles">@HDCharles</a> could you take a look?</p>

MrOCW: <p>sorry it points to x.to(‘cpu’) as i did not reassign the tensor. x = x.to(‘cpu’) helped solve the aten::empty_strided issue</p>

MrOCW: <p>in this case, would it be better to<br>
dequant → torch.cat with CUDA → quant<br>
or<br>
forward pass with ff.cat QuantizedCPU?</p>

jerryzh168: <p>I feel probably run everything on CPU would be better, running one op in CUDA sounds a bit weird…</p>
<p>not sure about your goal though, if you find running torch.cat on CUDA is faster and you need to squeeze every bit of perf then maybe that is OK as well</p>