Project: Unsupervised learning of Object Landmarks through Conditional Image Generation

DuaneNielsen: <p>Hi PyTorch community,</p>
<p>I’d like to announce a working PyTorch implementation of the paper <a href="http://www.robots.ox.ac.uk/~vgg/research/unsupervised_landmarks/" rel="noopener nofollow ugc">Unsupervised learning of Object Landmarks through Conditional Image Generation</a></p>
<p>This allows the generation of keypoints on 2D images, using unsupervised learning with <strong>no labels</strong>!  Works for both still images and video.  (Possibly easier/better on video).</p>
<p>If you are at all interested in unsupervised learning, it’s a great paper.  One of the best papers on unsupervised learning IMHO, due to the simplicity of the technique.</p>
<p>Check it out at …</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/duanenielsen/keypoints">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/duanenielsen/keypoints" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/a06e5a0a52d4b02f12e7c9aa779e1cbb694081a994cdc058441570a986ea410c/DuaneNielsen/keypoints" class="thumbnail" width="" height=""></div>

<h3><a href="https://github.com/duanenielsen/keypoints" target="_blank" rel="noopener nofollow ugc">GitHub - DuaneNielsen/keypoints: Unsupervised learning of Object Landmarks...</a></h3>

  <p>Unsupervised learning of Object Landmarks through Conditional Image Generation - GitHub - DuaneNielsen/keypoints: Unsupervised learning of Object Landmarks through Conditional Image Generation</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I also implemented transporter network from <a href="https://arxiv.org/abs/1906.11883" rel="noopener nofollow ugc">Unsupervised Learning of Object Keypoints for Perception and Control</a> and can confirm that it also works very well on Atari images.</p>
<p>Here are a few of the results.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/1/712004c728265a916fdfea4899cf045b53f69068.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/712004c728265a916fdfea4899cf045b53f69068" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/712004c728265a916fdfea4899cf045b53f69068_2_625x500.jpeg" alt="image" data-base62-sha1="g8KykVC4S2esMWdDBBCbELSleEE" width="625" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/712004c728265a916fdfea4899cf045b53f69068_2_625x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/712004c728265a916fdfea4899cf045b53f69068_2_937x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/712004c728265a916fdfea4899cf045b53f69068_2_1250x1000.jpeg 2x" data-dominant-color="867771"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2560×2048 785 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Enjoy!</p>
<p>Duane</p>

ptrblck: <p>Thanks for sharing this Duane! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I haven’t touched a lot of unsupervised methods yet, so this seems to be a great starter!</p>

DuaneNielsen: <p>Yeah, I think it is.  Super simple architecture, no adversarial losses, works with a simple reconstruction loss.</p>
<p>There were really 2 tricks to make it work.</p>
<ol>
<li>The data augmentation requires some tuning.  Too much distortion and you don’t learn, not enough and there is no contrast or prediction gradient to learn.</li>
<li>For still images, After distorting the input, you have to “loss mask” the parts of the image that fall outside the boundaries of the original image.  Otherwise the algorithm “cheats” by using the boundary to learn the parameters of your distortion!</li>
</ol>

Raj1: <p>Hello <a class="mention" href="/u/duanenielsen">@DuaneNielsen</a> ,<br>
I have one doubt, regarding the dataset.<br>
Let’s say we are using a dataset which contains both original image and small perturbed image of the original images or two consecutive frames of a video.<br>
So, In this case there is no need of using TPS function to create a dummy distorted image.</p>
<p>What should be our loss mask in this when we have both original and perturbed image, which you have used even in loss function?</p>
<p>And also, is this model architecture applicable to images having size different than 128 x 128 let’s say 448 x 256?</p>