Creating subset of dataset using a metric?

samarendra109: <p>I am trying to do a pytorch implementation of the following paper,</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://openreview.net/forum?id=UmvSlP-PyV">
  <header class="source">
      <img src="https://openreview.net/favicon.ico" class="site-icon" width="16" height="16">

      <a href="https://openreview.net/forum?id=UmvSlP-PyV" target="_blank" rel="noopener nofollow ugc">OpenReview</a>
  </header>

  <article class="onebox-body">
    <img src="https://openreview.net/images/openreview_logo_512.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://openreview.net/forum?id=UmvSlP-PyV" target="_blank" rel="noopener nofollow ugc">Beyond neural scaling laws: beating power law scaling via data pruning</a></h3>

  <p>We show in theory and practice that power law scaling of error with respect to dataset size can be improved via intelligent data pruning.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This paper uses various metrics to prune the dataset into smaller sizes. Is there a way to achieve this in pytorch elegantly. I am currently using CIFAR10 and CIFAR100 datasets for this experiment so I was looking for something that could work with it, but I would love to learn other options for other kinds of datasets.</p>
<p>In general, the bottleneck is, the datasets can be very big.</p>

tom: <p>I think you’d need to be more precise which of their experiments you want to implement.<br>
For example, the finetuning one form section 6 would involve taking a pretrained-imagenet model (those are available in TorchVision or TIMM), computing embeddings and then cluster the embeddings (using Scikit Learn or so).</p>
<p>Best regards</p>
<p>Thomas</p>

samarendra109: <p>I want to implement the pruning of CIFAR10 dataset to 50% of it’s original size. To be exact the experiment mentioned in section B of the suplimentary pdf.</p>
<p>But I don’t understand why the answer to the qu depends on the experiment. Basically it I have a pytorch dataset, I want to know how I can generate a subset of this dataset based on some formula which is datapoint specific.</p>

ptrblck: <p>In section B I see:</p>
<blockquote>
<p>CIFAR-10 and SVHN. CIFAR-10 and SVHN model training was performed using a standard<br>
ResNet-18 through the PyTorch library. Each model was trained …</p>
</blockquote>
<blockquote>
<p>Data pruning for transfer learning. To assess the effect of pruning downstream finetuning data<br>
on transfer learning performance, vision transformers (ViTs) pre-trained on ImageNet21k were<br>
fine-tuned on different pruned subsets of CIFAR-10. Pre-trained models were obtained from …</p>
</blockquote>
<p>which does not mention how the CIFAR10/100 datasets were pruned.</p>
<aside class="quote no-group" data-username="samarendra109" data-post="3" data-topic="176508">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samarendra109/48/32204_2.png" class="avatar"> samarendra109:</div>
<blockquote>
<p>Basically it I have a pytorch dataset, I want to know how I can generate a subset of this dataset based on some formula which is datapoint specific.</p>
</blockquote>
</aside>
<p>You could use <code>torch.utils.data.Subset</code> and calculate the <code>indices</code> based on the target from the dataset and your pruning logic.<br>
For <code>CIFAR10</code> you can directly access the targets via <code>dataset.targets</code>.</p>