Feed custom labels which can be out of range

Beck: <p>We are going to work with millions of labels and store them in redis.<br>
<strong>To reduce amount of memory</strong> used by currently unused labels/classes which we are feeding, we are filling them with None, but still each None is eating 16 bits of memory.</p>
<p>I have tried to fool pytorch and feed only current batch of labels instead of all millions of them, but it doesn’t work, because pytorch is throwing error about lengths mismatch of class and provided class_to_idx.<br>
One of builtin assertions is failing.</p>
<pre><code class="lang-auto">/opt/conda/conda-bld/pytorch_1646756402876/work/aten/src/ATen/native/cuda/Loss.cu:257: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [0,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
</code></pre>
<p>How we are changing dataset:</p>
<pre><code class="lang-auto">def change_dataset(dataset, redis_class_to_idx):
  len = max(redis_class_to_idx.values()) + 1
  actual_classes = [None] * len
  for i,(path, class_idx) in enumerate(dataset.samples):
    sample_class = dataset.classes[class_idx]
    if sample_class not in redis_class_to_idx:
      continue
    actual_class_idx = redis_class_to_idx[sample_class]
    current_class_idx = dataset.class_to_idx[sample_class]
    current_class = dataset.classes[current_class_idx]
    actual_classes[actual_class_idx] = current_class
    dataset.samples[i] = (path, actual_class_idx)
    dataset.targets[i] = actual_class_idx

  dataset.classes = actual_classes
  dataset.class_to_idx = redis_class_to_idx
  return dataset
</code></pre>
<p>Is there a way to make it work?<br>
For example, we are feeding batch of data with labels/classes</p>
<pre><code class="lang-auto">[None, None, None, None, None, None, None, None, None, 'b980bc1d-e7ef-11ec-97ee-b025aa41e7a9' , 'bbc22780-e8b2-11ec-8fea-0242ac120002', 'bbc22ad2-e8b2-11ec-8fea-0242ac120002']
</code></pre>
<p>so that their actual <code>class_to_idx</code> will be for example:</p>
<pre><code class="lang-auto">{
   'b980bc1d-e7ef-11ec-97ee-b025aa41e7a9': 112378568,
   'bbc22780-e8b2-11ec-8fea-0242ac120002': 112378569,
   'bbc22ad2-e8b2-11ec-8fea-0242ac120002': 112378570
}
</code></pre>
<p>and to feed it properly, we need a list with length 112378570 filled with None and only last 3 with real labels/classes.</p>

Beck: <p>Basically we keep whole actual list of uuid_to_index hash in redis, but to avoid keeping them in app runtime memory and avoid feeding them all in each training cycle, we are modifying it like that.</p>

nivek: <aside class="quote no-group" data-username="Beck" data-post="1" data-topic="153829">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/beck/48/49955_2.png" class="avatar"> Beck:</div>
<blockquote>
<p>I have tried to fool pytorch and feed only current batch of labels instead of all millions of them, but it doesn’t work, because pytorch is throwing error about lengths mismatch of class and provided class_to_idx.</p>
</blockquote>
</aside>
<p>I don’t really follow this part. Can you elaborate on it and provide some minimally run-able code example?</p>
<p>Aside from that, is it possible for you to read the data iterable-style? It seems like it should be possible but I can’t tell without knowing how your <code>Dataset</code> is defined.</p>
<p>If so, you can checkout <a href="https://pytorch.org/data/beta/torchdata.datapipes.iter.html" rel="noopener nofollow ugc"><code>IterDataPipe</code></a> from our new library <a href="https://github.com/pytorch/data" rel="noopener nofollow ugc"><code>torchdata</code></a>.</p>
<p>You can also use <a href="https://pytorch.org/docs/stable/data.html#iterable-style-datasets" rel="noopener nofollow ugc"><code>IterableDataset</code></a> within <code>torch</code>.</p>

Beck: <p>Thank you. I’m pretty new to python, so issue was pretty silly and was easily solved by replacing array with an object…</p>