Problem about the function torch.index_select(input, dim, index, out=None)

pcshih: <p>Is this function differentiable?</p>
<p>Is there any implementation code behind this function? I cannot find the source code of this function in the pytorch website.</p>
<p>I have implemented a <a href="https://github.com/pcshih/pytorch-VSLUD" rel="noopener nofollow ugc">paper</a><br>
using this function but the result is quite weird.</p>
<p>Other Question:<br>
I have the problem about index differentiating below.</p>
<p>In this <a href="http://openaccess.thecvf.com/content_ECCV_2018/papers/Mrigank_Rochan_Video_Summarization_Using_ECCV_2018_paper.pdf" rel="noopener nofollow ugc">paper</a> section 3.3</p>
<blockquote>
<p>We first select Y frames (i.e. keyframes) based on the prediction scores from the<br>
decoder.</p>
</blockquote>
<p>The decoder output is [2,320], which means non-keyframe score and key frame score of the 320 frames. We want to find a 0/1 vector according to the decoder output but the process of [2,320] → 0/1 vector seems not differentiable…</p>
<p>How to implement this in pytorch?</p>
<p>Thanks for anyone pointing out the reason.</p>

Tony-Y: <p>I have found its source code.</p>
<p>CPU:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/10c60b601a63710b17e2d14e8384c7c9bb51f9ff/aten/src/TH/generic/THTensorEvenMoreMath.cpp#L203-L207" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/10c60b601a63710b17e2d14e8384c7c9bb51f9ff/aten/src/TH/generic/THTensorEvenMoreMath.cpp#L203-L207" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/10c60b601a63710b17e2d14e8384c7c9bb51f9ff/aten/src/TH/generic/THTensorEvenMoreMath.cpp#L203-L207</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="203" style="counter-reset: li-counter 202 ;">
<li>void THTensor_(indexSelect)(THTensor *tensor, THTensor *src, int dim, THLongTensor *index)</li>
<li>{</li>
<li>  ptrdiff_t i, numel;</li>
<li>  THTensor *tSlice, *sSlice;</li>
<li>  int64_t *index_data;</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>CUDA:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/2e37ab85afa1b3b7b05f1ebe24c220809a05de9b/aten/src/THC/generic/THCTensorIndex.cu#L395-L400" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/2e37ab85afa1b3b7b05f1ebe24c220809a05de9b/aten/src/THC/generic/THCTensorIndex.cu#L395-L400" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/2e37ab85afa1b3b7b05f1ebe24c220809a05de9b/aten/src/THC/generic/THCTensorIndex.cu#L395-L400</a></h4>
<pre class="onebox"><code class="lang-cu"><ol class="start lines" start="395" style="counter-reset: li-counter 394 ;">
<li>void THCTensor_(indexSelect)(THCState *state, THCTensor *dst, THCTensor *src, int dim, THCudaLongTensor *indices)</li>
<li>{</li>
<li>  THCAssertSameGPU(THCTensor_(checkGPU)(state, 3, dst, src, indices));</li>
<li>
</li>
<li>  int dims = THCTensor_(nDimensionLegacyNoScalars)(state, dst);</li>
<li>  THArgCheck(dims &lt;= MAX_CUTORCH_DIMS, 2, CUTORCH_DIM_WARNING);</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

pcshih: <p>Thank you Tony-Y.<br>
Do you think this is a differentiable function?</p>

Tony-Y: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/6dfecc7e01842bb7e5024794fdce94480de7bb3a/tools/autograd/derivatives.yaml#L415-L417" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/6dfecc7e01842bb7e5024794fdce94480de7bb3a/tools/autograd/derivatives.yaml#L415-L417" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/6dfecc7e01842bb7e5024794fdce94480de7bb3a/tools/autograd/derivatives.yaml#L415-L417</a></h4>
<pre class="onebox"><code class="lang-yaml"><ol class="start lines" start="415" style="counter-reset: li-counter 414 ;">
<li>- name: index_select(Tensor self, int dim, Tensor index) -&gt; Tensor</li>
<li>  self: at::zeros(self.sizes(), grad.options()).index_add_(dim, index, grad)</li>
<li>  index: non_differentiable</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>It is non-differentiable with respect to index.</p>

pcshih: <p>Thank you very much, Tony-Y.</p>
<p>What do you think of the pytorch implementation of the “select” action in <a href="https://github.com/pcshih/pytorch-VSLUD" rel="nofollow noopener">“we select k key frames to form the predicted summary video”</a> ?</p>
<p>I use torch.index_select first but I know the function cannot be differentiable now.<br>
Do you have any implementation suggestion on “select” action?</p>

Tony-Y: <p>You use <code>index_select</code> 4 times in your code:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/SK.py#L38-L40" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/SK.py#L38-L40" target="_blank" rel="nofollow noopener">pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/SK.py#L38-L40</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="38" style="counter-reset: li-counter 37 ;">
<li>index = torch.tensor(column_mask, device=torch.device('cuda:0'))</li>
<li>h_select = torch.index_select(h, 3, index)</li>
<li>x_select = torch.index_select(x_temp, 3, index)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/training_set_preparation.py#L42-L47" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/training_set_preparation.py#L42-L47" target="_blank" rel="nofollow noopener">pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/training_set_preparation.py#L42-L47</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="42" style="counter-reset: li-counter 41 ;">
<li>gt_summary = torch.from_numpy(dataset[key]["gtsummary"][...]).to(device)</li>
<li>column_index = gt_summary.nonzero().view(-1)</li>
<li>feature_summary_cuda = torch.from_numpy(dataset[key]["features"][...]).to(device)</li>
<li>feature_summary_cuda = feature_summary_cuda.transpose(1,0).view(1,1024,1,feature_summary_cuda.shape[0])</li>
<li>feature_summary_cuda = torch.index_select(feature_summary_cuda, 3, column_index)</li>
<li>attributes["summary_features"] = feature_summary_cuda; #print(torch.isnan(feature_summary_cuda).nonzero().view(-1))</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/train.py#L195-L198" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/train.py#L195-L198" target="_blank" rel="nofollow noopener">pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/train.py#L195-L198</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="195" style="counter-reset: li-counter 194 ;">
<li>index = torch.tensor(column_mask, device=device)</li>
<li>select_vd = torch.index_select(vd, 3, index)</li>
<li>reconstruct_loss = torch.norm(S_K_summary-select_vd, p=2)**2</li>
<li>reconstruct_loss /= len(column_mask)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Where is the problem?</p>

pcshih: <p>Thank you very much, Tony-Y.</p>
<p>The index_select function cannot be diff. so the gradient cannot backprop. to the previous S_K architecture.</p>
<p>My problem is how do I implement  “select” action in pytorch instead of using the index_select function to implement it?</p>

Tony-Y: <p>Do you want a derivative with respect to the index rather than the source tensor?</p>

pcshih: <p>I want a derivative with respect to the source tensor[index] → the tensor on the “index” location.</p>
<p>Because the output tensor of <a href="https://github.com/pcshih/pytorch-VSLUD" rel="noopener nofollow ugc">FCSN</a> architecture is in the shape of [1,2,1,<span class="hashtag-raw">#frame</span>]. This tensor means whether frames are selected or not.</p>
<p>The algo. of this <a href="https://github.com/pcshih/pytorch-VSLUD" rel="noopener nofollow ugc">paper</a> is below:</p>
<ul>
<li>
<p>Downsampling every video in 2fps and pre-processing every downsampled training video to [1,1024, 1, T] (video) and [1,1024,1,S] (summary) through pre-trained googlenet.</p>
</li>
<li>
<p>for every pre-processed downsample video’s feature(in the format [1,1024,1,T]) and real summary video’s feature(in the format [1,1024,1,S]):  -&gt;T,S may differ in each video</p>
</li>
<li>
<ul>
<li>Put [1,1024,1,T] into FCSN and get the index_mask(this index_mask is constructed from the output of FCSN [1,2,1,T] which means which frame should be selected)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/78c661c955038481d704c6db036acffd4956256d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/78c661c955038481d704c6db036acffd4956256d" title="S__44130656"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/78c661c955038481d704c6db036acffd4956256d_2_333x249.jpeg" alt="S__44130656" data-base62-sha1="heqkknwbvid9bP2fqnBoZMeXHLn" width="333" height="249" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/78c661c955038481d704c6db036acffd4956256d_2_333x249.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/7/78c661c955038481d704c6db036acffd4956256d_2_499x373.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/7/78c661c955038481d704c6db036acffd4956256d_2_666x498.jpeg 2x" data-dominant-color="C8C5BF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">S__44130656</span><span class="informations">1478×1108 190 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ul>
</li>
<li>
<ul>
<li>Select K key outputs of FCSN according to index_mask and get the output in format [1,2,1,K].</li>
</ul>
</li>
<li>
<ul>
<li>Put the selected K key outputs of FCSN([1,2,1,K]) into the 1x1 conv to get the [1,1024,1,K].</li>
</ul>
</li>
<li>
<ul>
<li>Add K key features[1,1024,1,K] (Pick the K key features from original video feature according to index_mask) to the 1x1 conv output[1,1024,1,K] to do skip connection(this is the output of S_K).</li>
</ul>
</li>
<li>
<ul>
<li>Pick the K key features from original video feature according to index_mask and calculate the reconstruction loss with previous step output.</li>
</ul>
</li>
<li>
<ul>
<li>Calculate the diversity loss.</li>
</ul>
</li>
<li>
<ul>
<li>Calculate the adv. loss by putting the output of S_K in to S_D and set target score 1 to get the adv. loss</li>
</ul>
</li>
<li>
<ul>
<li>Update S_K.</li>
</ul>
</li>
<li>
<ul>
<li>Put Real summary videos’ features [1,1024,1,S] into S_D to calculate and set target score 1 to get the adv. loss.</li>
</ul>
</li>
<li>
<ul>
<li>Put Fake summary videos’ features [1,1024,1,K] come from S_K in S_D to calculate and set target score 0 to get adv. loss.</li>
</ul>
</li>
<li>
<ul>
<li>Update S_D.</li>
</ul>
</li>
<li>
<p>end</p>
</li>
</ul>
<p>Thank you very much, Tony-Y.</p>

Tony-Y: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4f77777e2b33ea9924c5fb77a0592f089e0b9a9e.png" data-download-href="https://discuss.pytorch.org/uploads/default/4f77777e2b33ea9924c5fb77a0592f089e0b9a9e" title="rec%20loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4f77777e2b33ea9924c5fb77a0592f089e0b9a9e_2_345x59.png" alt="rec%20loss" data-base62-sha1="bkZFWBOKz0XyekIy291KaPht2lU" width="345" height="59" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4f77777e2b33ea9924c5fb77a0592f089e0b9a9e_2_345x59.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4f77777e2b33ea9924c5fb77a0592f089e0b9a9e_2_517x88.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4f77777e2b33ea9924c5fb77a0592f089e0b9a9e_2_690x118.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">rec%20loss</span><span class="informations">798×138 7.24 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This reconstruction loss can be calculated by the weighted mean where the index_mask is used as the weights and k is the sum of the index_mask.</p>

pcshih: <p>But the the frame with larger index will get more weight, am I right?</p>

Tony-Y: <p>Since index_select is not used, the frame numbers of SK and v are the same.</p>

pcshih: <p>Sorry for my poor English understanding.</p>
<p>Could you please set an example?</p>
<p>Thank you very much, Tony-Y.</p>

Tony-Y: <pre><code class="lang-auto">import torch
index_mask = torch.Tensor([0.0, 0.0, 1.0, 1.0, 0.0])
v = torch.randn(3,5)
sk = torch.randn(3,5)
torch.sum((sk-v)**2 * index_mask) / torch.sum(index_mask)
</code></pre>
<p>where the feature size is 3 and the number of frames is 5.</p>

pcshih: <p>I have followed your idea but the loss is the same quite weird.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c09dc68bb6b59745e5ff42177e475714e0c4c6df.png" data-download-href="https://discuss.pytorch.org/uploads/default/c09dc68bb6b59745e5ff42177e475714e0c4c6df" title="loss_index_mask"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c09dc68bb6b59745e5ff42177e475714e0c4c6df_2_690x275.png" alt="loss_index_mask" data-base62-sha1="rtXHPpVdyOaRvJ3XocXXuuiYBZ5" width="690" height="275" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c09dc68bb6b59745e5ff42177e475714e0c4c6df_2_690x275.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c09dc68bb6b59745e5ff42177e475714e0c4c6df_2_1035x412.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c09dc68bb6b59745e5ff42177e475714e0c4c6df_2_1380x550.png 2x" data-dominant-color="F9F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_index_mask</span><span class="informations">1446×577 50.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pcshih/pytorch-VSLUD/blob/20a91393c040cd6a005c1b01ce77ca3db82efd25/SK.py#L24-L56">
  <header class="source">

      <a href="https://github.com/pcshih/pytorch-VSLUD/blob/20a91393c040cd6a005c1b01ce77ca3db82efd25/SK.py#L24-L56" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pcshih/pytorch-VSLUD/blob/20a91393c040cd6a005c1b01ce77ca3db82efd25/SK.py#L24-L56" target="_blank" rel="noopener nofollow ugc">pcshih/pytorch-VSLUD/blob/20a91393c040cd6a005c1b01ce77ca3db82efd25/SK.py#L24-L56</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="24" style="counter-reset: li-counter 23 ;">
          <li>h = x</li>
          <li>x_temp = x</li>
          <li></li>
          <li>h = self.FCSN(h)</li>
          <li></li>
          <li>values, indices = h.max(1, keepdim=True)</li>
          <li></li>
          <li>###old###</li>
          <li># # 0/1 vector, we only want key(indices=1) frame</li>
          <li># column_mask = (indices==1).view(-1).nonzero().view(-1).tolist() </li>
          <li></li>
          <li># # if S_K doesn't select more than one element, then random select two element(for the sake of diversity loss)</li>
          <li># if len(column_mask)&lt;2:</li>
          <li>#     print("S_K does not select anything, give a random mask with 2 elements")</li>
          <li>#     column_mask = random.sample(list(range(h.shape[3])), 2)</li>
          <li></li>
          <li># index = torch.tensor(column_mask, device=torch.device('cuda:0'))</li>
          <li># h_select = torch.index_select(h, 3, index)</li>
          <li># x_select = torch.index_select(x_temp, 3, index)</li>
          <li>###old###</li>
      </ol>
    </code></pre>


  This file has been truncated. <a href="https://github.com/pcshih/pytorch-VSLUD/blob/20a91393c040cd6a005c1b01ce77ca3db82efd25/SK.py#L24-L56" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox githubblob" data-onebox-src="https://github.com/pcshih/pytorch-VSLUD/blob/20a91393c040cd6a005c1b01ce77ca3db82efd25/train.py#L202-L214">
  <header class="source">

      <a href="https://github.com/pcshih/pytorch-VSLUD/blob/20a91393c040cd6a005c1b01ce77ca3db82efd25/train.py#L202-L214" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pcshih/pytorch-VSLUD/blob/20a91393c040cd6a005c1b01ce77ca3db82efd25/train.py#L202-L214" target="_blank" rel="noopener nofollow ugc">pcshih/pytorch-VSLUD/blob/20a91393c040cd6a005c1b01ce77ca3db82efd25/train.py#L202-L214</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="202" style="counter-reset: li-counter 201 ;">
          <li>###new reconstruct###</li>
          <li>reconstruct_loss = torch.sum((S_K_summary-vd)**2 * index_mask) / torch.sum(index_mask)</li>
          <li>###new reconstruct###</li>
          <li></li>
          <li></li>
          <li># diversity</li>
          <li>S_K_summary_reshape = S_K_summary.view(S_K_summary.shape[1], S_K_summary.shape[3])</li>
          <li>norm_div = torch.norm(S_K_summary_reshape, 2, 0, True)</li>
          <li>S_K_summary_reshape = S_K_summary_reshape/norm_div</li>
          <li>loss_matrix = S_K_summary_reshape.transpose(1, 0).mm(S_K_summary_reshape)</li>
          <li>diversity_loss = loss_matrix.sum() - loss_matrix.trace()</li>
          <li>#diversity_loss = diversity_loss/len(column_mask)/(len(column_mask)-1)</li>
          <li>diversity_loss = diversity_loss/(torch.sum(index_mask))/(torch.sum(index_mask)-1)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>p.s. The torch.index_select in below is just for training set preparation so I do not change.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/training_set_preparation.py#L42-L47">
  <header class="source">

      <a href="https://github.com/pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/training_set_preparation.py#L42-L47" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/training_set_preparation.py#L42-L47" target="_blank" rel="noopener nofollow ugc">pcshih/pytorch-VSLUD/blob/b3ed7aba9332d2e0a21a66e84eae3654c9e254af/training_set_preparation.py#L42-L47</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="42" style="counter-reset: li-counter 41 ;">
          <li>gt_summary = torch.from_numpy(dataset[key]["gtsummary"][...]).to(device)</li>
          <li>column_index = gt_summary.nonzero().view(-1)</li>
          <li>feature_summary_cuda = torch.from_numpy(dataset[key]["features"][...]).to(device)</li>
          <li>feature_summary_cuda = feature_summary_cuda.transpose(1,0).view(1,1024,1,feature_summary_cuda.shape[0])</li>
          <li>feature_summary_cuda = torch.index_select(feature_summary_cuda, 3, column_index)</li>
          <li>attributes["summary_features"] = feature_summary_cuda; #print(torch.isnan(feature_summary_cuda).nonzero().view(-1))</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thank you very much, Tony-Y.</p>

Tony-Y: <p>By the way, is the random selection a valid approach?</p>

pcshih: <p>The selection is based on the output of the FCSN architecture (i.e. [1,1024,1,T] tensor)</p>
<p>Thank you very much, Tony-Y.</p>

Tony-Y: <blockquote>
<p>if S_K doesn’t select more than one element, then random select two element(for the sake of diversity loss)</p>
</blockquote>
<p>What’s this?</p>

pcshih: <p>Because the calculation of diversity loss must have at least two frames.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b2626732dba2cd9eef88539d91f57aff7b5ca2f6.png" alt="diversity_loss" data-base62-sha1="ps3OGhDuMjEEBpNZW4lCiwLN8ge" width="614" height="102"></p>

pcshih: <p>I implement the diversity loss through below concept, where the feature size is 2 and the number of frames is 3.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/332b28df9d2fd222cf247bae7984ce44ac9a3746.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/332b28df9d2fd222cf247bae7984ce44ac9a3746" title="diversity"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/332b28df9d2fd222cf247bae7984ce44ac9a3746_2_374x500.jpeg" alt="diversity" data-base62-sha1="7iEOQCUiP8K1l5G7bECex2eQfwa" width="374" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/332b28df9d2fd222cf247bae7984ce44ac9a3746_2_374x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/3/332b28df9d2fd222cf247bae7984ce44ac9a3746_2_561x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/3/332b28df9d2fd222cf247bae7984ce44ac9a3746_2_748x1000.jpeg 2x" data-dominant-color="C5C1B8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">diversity</span><span class="informations">1108×1478 167 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>