Gumbel-Max implementation

username: promach
cooked: <p>Could anyone explain how this <a href="https://medium.com/swlh/on-the-gumbel-max-trick-5e340edd1e01" rel="noopener nofollow ugc">Gumbel-Max</a> pytorch <a href="https://github.com/D-X-Y/AutoDL-Projects/blob/main/lib/models/cell_searchs/search_model_gdas.py#L115-L129" rel="noopener nofollow ugc">code implementation</a> works ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8054cc3febd8999246ebce67f77c56750969a6f.png" data-download-href="https://discuss.pytorch.org/uploads/default/f8054cc3febd8999246ebce67f77c56750969a6f" title="Screenshot from 2021-05-15 19-14-49"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f8054cc3febd8999246ebce67f77c56750969a6f_2_690x266.png" alt="Screenshot from 2021-05-15 19-14-49" data-base62-sha1="zo5LAA4VLSy2Q25xRiMnTjgxoKj" width="690" height="266" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f8054cc3febd8999246ebce67f77c56750969a6f_2_690x266.png, https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8054cc3febd8999246ebce67f77c56750969a6f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8054cc3febd8999246ebce67f77c56750969a6f.png 2x" data-dominant-color="FAF7E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-05-15 19-14-49</span><span class="informations">936×361 38 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Unity05
cooked: <p>Hello,</p>
<p>the standart forumla for the Gumbel - Max is<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d90d5f54dba634490a8aaf1d6b3f9421748d46f.png" alt="image" data-base62-sha1="mtTgRRh7Qf08Zu2s5TK9Tk4TVWT" width="326" height="92"></p>
<p>The problem is that argmax is not differentiable.<br>
Your medium source derives this correlation:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90f147bd57a9046fac303901fcc2fb96797d8414.png" alt="image" data-base62-sha1="kGdHaq8A1oKfx3uiXYpPWPq9h3u" width="303" height="74"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed70f53efd55493ea0509401b496192f3c5df854.png" alt="image" data-base62-sha1="xSvaEHzEUvzIbjIkBcCyJUglBxG" width="176" height="62"></p>
<p>So I would usually use softargmax as a nice approximation for argmax, but this implementation you’ve found does something quite similar.</p>
<p>It basically calculates this formula:</p>
<pre><code class="lang-auto">probs = exp{ ((gumbel + log(pi)) / tau) } / sum[j] { exp{ ((gumbel_j + log(pi_j)) / tau) } }
</code></pre>
<p><code>tau</code> is a scaling / temparature parameter where if you decrease it, it will approximate the argmax, and when you increase it the probs will become more uniform.</p>
<p>Last but not least, the max value is taken so that a one - hot - encoding can be made.</p>
<p>(I still find the softargmax nicer as you would matmul the probs with a range tensor (you would choose a relatively small <code>tau</code> (I guess with softargmax it’s called <code>beta</code> and would be chosen high while not being used as a denominator).<br>
Please let me know if there is a performance or other reason why one should do it the way it is done in this implementation.)</p>

username: promach
cooked: <p>For the <a href="https://github.com/D-X-Y/AutoDL-Projects/issues/99#issuecomment-835802887" rel="noopener nofollow ugc">NAS multigraph</a> and equation (7) of <a href="https://arxiv.org/pdf/1910.04465.pdf#page=4" rel="noopener nofollow ugc">GDAS paper</a> , how to do backpropagation across <em><strong>multiple parallel edges</strong></em> between two nodes ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37b59ead1a75bf3ef27e1f1aa5ff2b8f5be65b15.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/37b59ead1a75bf3ef27e1f1aa5ff2b8f5be65b15" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37b59ead1a75bf3ef27e1f1aa5ff2b8f5be65b15_2_480x500.jpeg" alt="image" data-base62-sha1="7WPnx2ArKZsIYbwcKH8COHTzDgx" width="480" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37b59ead1a75bf3ef27e1f1aa5ff2b8f5be65b15_2_480x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37b59ead1a75bf3ef27e1f1aa5ff2b8f5be65b15_2_720x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37b59ead1a75bf3ef27e1f1aa5ff2b8f5be65b15_2_960x1000.jpeg 2x" data-dominant-color="ECF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">976×1015 285 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>