Pytorch inference on Raspberry pi?

Gautham_Kumaran: <p>I’m working on an object detection model to be deployed on Raspberry pi.<br>
I usually stick to tflite for this use case. But I really want to use PyTorch if possible.</p>
<p>I’ll be running inference on Raspberry pi. Currently, what is the best way to go about doing this?</p>
<p>Will torchscript help?<br>
Most discussions in the forums give an arm_whl file to setup PyTorch in raspberry pi, is this the best method  - <a href="https://discuss.pytorch.org/t/installing-pytorch-on-raspberry-pi-3/25215/21" class="inline-onebox">Installing pytorch on raspberry pi 3</a>?</p>
<p>Any resource on this topic will be very much appreciated. Thanks.</p>

maltequast: <p>Hi Gautham,</p>
<p>in my opinion it is the best to install via whl file, otherwise it takes ages to be installed.</p>
<p>Might be a good idea to install pytorch &gt;= 1.5.0, because you (already mentioned) torchscript. If it is not possible on raspi3 you can also go for flask and a wdsgi as backend.</p>
<p>A great pretrained transformer is the DETR object detector and you can already download it from facebooks github for pytorch.<br>
Others like faster r-cnn might be a little to huge for a raspi3.</p>
<p>Hope it helps &amp; have fun!</p>

Gautham_Kumaran: <p>Thanks <a class="mention" href="/u/maltequast">@maltequast</a> , I’ll proceed with the whl file and pytorch &gt;= 1.5.0.</p>
<p>Regarding performance, is there any benchmarking data for pytorch vs tensorflow lite on raspberry pi? or in your opinion is the performance comparable, given that tflite was purpose-built for this?</p>

maltequast: <p>Haven’t used tflite, but I know since pytorch 1.3 you can use pytorch on mobile devices / arm architectures for experimental usage.<br>
In Pytorch 1.5 you can use dynamic quanization for better latency.<br>
<a href="https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html" rel="nofollow noopener">Here</a> is the tutorial.<br>
Also huge advantage of detr object detector is the model size. Check it out <a href="https://github.com/facebookresearch/detr" rel="nofollow noopener">here</a>.</p>

tom: <p>I put up <a href="http://mathinf.com/pytorch/arm64/" rel="nofollow noopener">PyTorch 1.6/TorchVision 0.7 wheels for Raspberry Pi OS 64bit</a> in the hope that they’ll be useful.</p>

mheriyanto: <p>Thanks <a class="mention" href="/u/tom">@tom</a> for sharing PyTorch’s wheel files.</p>

tom: <p>Incidentally, I made <a href="http://mathinf.eu/static/pytorch/arm64/2021-01/" rel="noopener nofollow ugc">new ones for a new course</a>.<br>
They are from PyTorch master, but if you have convs and your model is feeling slow, the added (incomplete) PR helps a lot.</p>

d4l3k: <p>PyTorch provides official wheels for arm64/aarch64 so if you install the newly released RPi OS 64-bit you can just run:</p>
<pre><code class="lang-auto">pip install torch torchvision torchaudio
</code></pre>
<p>I wrote up a tutorial on how to do efficient real time inference on a Raspberry Pi at <a href="https://pytorch.org/tutorials/intermediate/realtime_rpi.html" class="inline-onebox" rel="noopener nofollow ugc">Real Time Inference on Raspberry Pi 4 (30 fps!) — PyTorch Tutorials 1.10.1+cu102 documentation</a></p>

surak: <p>Hey, thanks for the tutorial!</p>
<p>The last part which prints the actual classes is missing (it’s not a copy-paste from the text). Although it’s trivial to import the linked file, it would be nice to have a self-contained example there!</p>
<p>This part:</p>
<pre><code class="lang-auto">top = list(enumerate(output[0].softmax(dim=0)))
top.sort(key=lambda x: x[1], reverse=True)
for idx, val in top[:10]:
    print(f"{val.item()*100:.2f}% {classes[idx]}")
</code></pre>
<p>Will not know what is the object classes, obviously.</p>
<p>Also, your example video shows a real-time update of the classes on the terminal, and I can’t really reproduce that: using <code>os.system(“clear”)</code> or <code>print(chr(27) + "[2J”)</code> makes the screen blink a lot.</p>
<p>Do you have the whole example in a working condition in some repo?</p>
<p>PS: I am not sure if it’s because of Arducam, but I don’t get more than 7fps.</p>