Object Detection with Transformers - Pix2Seq implementation in PyTorch

Moein_Shariatnia: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cb763b0d469cded377f2cf5a4b3c46d983b8880.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5cb763b0d469cded377f2cf5a4b3c46d983b8880" title="pix2seq - framework"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cb763b0d469cded377f2cf5a4b3c46d983b8880_2_690x388.jpeg" alt="pix2seq - framework" data-base62-sha1="decPUyRbsj0XGN4HkEgsJdeu3EQ" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cb763b0d469cded377f2cf5a4b3c46d983b8880_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cb763b0d469cded377f2cf5a4b3c46d983b8880_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cb763b0d469cded377f2cf5a4b3c46d983b8880_2_1380x776.jpeg 2x" data-dominant-color="DFE0E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pix2seq - framework</span><span class="informations">2999×1687 452 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve implemented the <a href="https://arxiv.org/abs/2109.10852" rel="noopener nofollow ugc">“<strong>Pix2seq</strong>: A Language Modeling Framework for Object Detection”</a> paper in PyTorch and written an in-depth <strong>tutorial</strong> on it.</p>
<p>Here’s the link to the <a href="https://pub.towardsai.net/easy-object-detection-with-transformers-simple-implementation-of-pix2seq-model-in-pytorch-fde3e7162ce7" rel="noopener nofollow ugc">blog</a> on <strong>Towards AI</strong>.</p>
<p>You can find the whole project on my <a href="https://github.com/moein-shariatnia/Pix2Seq" rel="noopener nofollow ugc"><strong>GitHub</strong></a></p>
<p>Also, the codes and tutorials are also available as <a href="https://colab.research.google.com/drive/1UeYIZ6_GHNwCHSi8nNV5dVc3oUrM5-BA?usp=sharing" rel="noopener nofollow ugc">Colab Notebook</a> and <a href="https://www.kaggle.com/code/moeinshariatnia/object-detection-w-transformers-pix2seq-pytorch/notebook" rel="noopener nofollow ugc">Kaggle Notebook</a>.</p>
<p><a href="https://colab.research.google.com/drive/1UeYIZ6_GHNwCHSi8nNV5dVc3oUrM5-BA?usp=sharing" rel="noopener nofollow ugc"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" width="117" height="20"></a></p>
<p><a href="https://www.kaggle.com/code/moeinshariatnia/object-detection-w-transformers-pix2seq-pytorch/notebook" rel="noopener nofollow ugc"><img src="https://kaggle.com/static/images/open-in-kaggle.svg" alt="Kaggle" width="111" height="20"></a></p>
<p>I hope you like it!</p>

Etienne_Perot: <p>hi! i read the code, i think that noisy inputs are missing. also i suspect that evaluation should be done in autoregressive fashion otherwise you are always using the ground truth of previous timesteps instead of your own prediction</p>

Moein_Shariatnia: <p>Hey,</p>
<p>Yes, for the sake of simplicity of my implementation and tutorial, I skipped the augmentation technique they used and that’s one reason why I don’t reach the good performance of the paper.</p>
<p>Regarding the evaluation, I am using “teacher forcing” in my implementation and the mask in decoder self-attention layers ensures the models does not peak on future tokens. You can refer to the issues of the repo where I explain in more depth.</p>

Etienne_Perot: <aside class="quote no-group" data-username="Moein_Shariatnia" data-post="3" data-topic="160368">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/moein_shariatnia/48/30185_2.png" class="avatar"> Moein_Shariatnia:</div>
<blockquote>
<p>explain</p>
</blockquote>
</aside>
<p>I understand the ‘teacher forcing’ technique. But imagine in a true evaluation mode where you do not have the ground truth, what do you feed the network with? For a true evaluation we need the autoregressive setup, otherwise you are using the ground truth as input sequence of previous timesteps instead of your prediction (being with argmax or nucleus sampling or whatever technique)</p>