Indexing a variable with a variable

username: ncullen93
cooked: <p>How do you go about indexing a variable with another variable? For instance, it’s not clear how you could do a spatial transformer network, since the output of the transformer layer would be a Variable. Does calling <code>idx.data</code> work, or will that cause the graph to be disconnected?</p>
<p>Example</p>
<pre><code class="lang-python">import torch
from torch.autograd import Variable
x = Variable(torch.randn(3,3))
idx = Variable(torch.LongTensor([0,1]), requires_grad=True)

# doesnt work
print(x[idx])

# works, but can you cant call .backward?
print(x[idx.data])

t = torch.sum(x[idx.data])
t.backward() # gives an error about no graph nodes require gradients
</code></pre>

username: colesbury
cooked: <p>You can use <code>index_select</code>:</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable
x = Variable(torch.randn(3,3), requires_grad=True)
idx = Variable(torch.LongTensor([0,1]))
print(x.index_select(0, idx))
</code></pre>
<p>Note that the index variable (idx) can’t have requires_grad set to True. The variable being indexed (x) can have requires_grad=True.</p>
<p><a href="http://pytorch.org/docs/tensors.html#torch.Tensor.index_select" class="onebox" target="_blank">http://pytorch.org/docs/tensors.html#torch.Tensor.index_select</a></p>

username: ncullen93
cooked: <p>Ok, thank you for that explanation… But then it seems that a STN isn’t possible under that condition. I know this is very broad, but is there any way to index or grab values from a variable tensor with another index-like variable that <b>does</b> have require_grad=True besides using <code>detach()</code>?  <code>torch.gather</code> doesn’t work either… Is <code>scatter</code> an option? It seems straight-forward in TF to so-called “differentiate the index”… is this just a limitation of define-by-run?</p>
<p>Here’s a fairly straight-forward <a href="https://gist.github.com/ncullen93/3b8fbf3358db318eaf3fb30993c72874" rel="nofollow noopener">STN gist</a> I made showing how it <i>should</i> work except for the error in the indexing. And <a href="https://github.com/daviddao/spatial-transformer-tensorflow/blob/master/sgitatial_transformer.py" rel="nofollow noopener">here is</a> a tensorflow version of the transformer layer.</p>
<p>No worries though - if anyone else stumbles upon this and has insight, let me know <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>EDIT: eh, I guess I see in the TF example above and the <a href="https://github.com/oeway/pytorch-deform-conv/blob/master/torch_deform_conv/deform_conv.py#L54-L63" rel="nofollow noopener">defomable conv pytorch</a> example how they still propagate the gradient through other means than the index… still unclear if differentiating the index is an inherent limitation of pytorch or in general.</p>

username: colesbury
cooked: <p>Your tensorflow link is to a private repository so I can’t view it.</p>
<p>The index operation doesn’t have any gradients defined w.r.t. the index variable. That’s not a limitation of “define by run”, that’s a property of the operation: it has integer domain.</p>
<p>You need a differentiable sampling operation for spatial transformer networks. You can implement STN in PyTorch in roughly the same ways as in Tensorflow: sample<a href="https://github.com/tensorflow/models/blob/7c42a178f67676467ca6e71d0628954a22396271/transformer/spatial_transformer.py#L84-L87"> <code>floor(idx)</code> and <code>floor(idx) + 1</code></a> and linearly interpolate between the two. Note that the sampling doesn’t produce a meaningful gradient, it’s the interpolation that produces a useful gradient:</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

torch.manual_seed(0)
x = Variable(torch.randn(3,3), requires_grad=True)
idx = Variable(torch.FloatTensor([0,1]), requires_grad=True)

i0 = idx.floor().detach()
i1 = i0 + 1

y0 = x.index_select(0, i0.long())
y1 = x.index_select(0, i1.long())

Wa = (i1 - idx).unsqueeze(1).expand_as(y0)
Wb = (idx - i0).unsqueeze(1).expand_as(y1)

out = Wa * y0 + Wb * y1

print(out)
out.sum().backward()
print(idx.grad)
</code></pre>
<p>(You probably want to use gather instead of index_select and will need to interpolate in two dimensions instead of just one)</p>

username: ncullen93
cooked: <p>Ok, that explanation really makes it click. I got it to work with interpolation. the TF repository I linked was just using bilinear interpolation. Thanks a billion for taking the time.</p>

username: vadimkantorov
cooked: <p>Is the need for explicit <code>index_select</code> a “feature” or a “bug”? Should the indexing operator call <code>index_select</code> automatically for the Variables?</p>

username: colesbury
cooked: <p>A “bug” (or at least a missing feature)</p>

username: pcshih
cooked: <p>Hello everyone,</p>
<p>I have the problem about indexing…</p>
<p>In this <a href="http://openaccess.thecvf.com/content_ECCV_2018/papers/Mrigank_Rochan_Video_Summarization_Using_ECCV_2018_paper.pdf" rel="noopener nofollow ugc">paper</a> section 3.3</p>
<blockquote>
<p>We first select Y frames (i.e. keyframes) based on the prediction scores from the<br>
decoder.</p>
</blockquote>
<p>The decoder output is [2,320], which means non-keyframe score and key frame score of the 320 frames. We want to find a 0/1 vector according to the decoder output but the process of [2,320] → 0/1 vector seems not differentiable…</p>
<p>How to implement this in pytorch?</p>
<p>Thank you very much.</p>