Questions about prototype builtin datasets

Ren_Pang: <p>Hi all, I’m currently exploring builtin datasets with new standards:</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/vision/tree/main/torchvision/prototype/datasets">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/vision/tree/main/torchvision/prototype/datasets" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/vision/tree/main/torchvision/prototype/datasets" target="_blank" rel="noopener nofollow ugc">vision/torchvision/prototype/datasets at main · pytorch/vision</a></h3>

  <p><a href="https://github.com/pytorch/vision/tree/main/torchvision/prototype/datasets" target="_blank" rel="noopener nofollow ugc">main/torchvision/prototype/datasets</a></p>

  <p><span class="label1">Datasets, Transforms and Models specific to Computer Vision</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Let’s take <a href="https://github.com/pytorch/vision/blob/main/torchvision/prototype/datasets/_builtin/cifar.py" rel="noopener nofollow ugc">Cifar10</a> as an example. I have several questions:</p>
<ol>
<li>Why are all datasets constructed as <code>iter</code> rather than <code>map</code> style? When I have an index (e.g., <code>2331</code>), I can no longer use <code>dataset[2331]</code> like the old <code>CIFAR10</code>.<br>
In this case, how to <code>get_item</code> for the new format dataset? Do I have to use <code>IterToMapConverter</code>? That’ll be quite strange because raw data format is <code>map</code>, I make it <code>iter</code> and traverse to change back to <code>map</code>.</li>
<li>What does <code>hint_shuffling</code> do?<pre data-code-wrap="python3"><code class="lang-plaintext">def hint_shuffling(datapipe: IterDataPipe[D]) -&gt; Shuffler[D]:
    return Shuffler(datapipe, buffer_size=INFINITE_BUFFER_SIZE).set_shuffle(False)
</code></pre>
It’s used in all prototype datasets. It seems to wrap datapipe with a <code>shuffler</code> but <code>set_shuffle(False)</code>. That seems doing nothing?</li>
<li>When to use <code>Decompressor</code> and set <code>resource.preprocess='decompress' or 'extract'</code>?<br>
What’s the difference among <code>Decompressor</code>, <code>resource.preprocess='decompress'</code>, <code>resource.preprocess='extract'</code> and using nothing?
<ul>
<li>
<code>Cifar10</code> resource is a <code>cifar-10-python.tar.gz</code> and sets nothing. It will default call <code>_guess_archive_loader</code> in <code>OnlineResource.load</code> to generate a <code>TarArchiveLoader</code>
</li>
<li>
<code>MNIST</code> resource is a <code>train-images-idx3-ubyte.gz</code> and uses a <code>Decompressor</code>
</li>
<li>
<code>cub200</code> resource is a <code>CUB_200_2011.tgz</code> uses <code>decompress=True</code>
</li>
</ul>
</li>
<li>How to use Transform in the new dataset API? such as <code>AutoAugment</code> or <code>RandomCrop</code>? Append corresponding augment Datapipe to current Dataset pipe?</li>
<li>For dataset that each image is stored in encoded image format (the old <code>ImageFolder</code> type. e.g., ImageNet, GTSRB),the output image format is <code>EncodedImage -&gt; EncodedData -&gt; Datapoint</code>. For dataset stored in binary (e.g., MNIST and CIFAR), the output image format is <code>Image -&gt; Datapoint</code>. Why are they different? I see most transform V2 APIs are conducted on <code>Image</code>. Why is <code>EncodedImage</code> used here?</li>
</ol>