How to use Skorch for data that does not fit into memory?

Thabang_Lukhetho: <p>How to use Skorch if your data does not fit into memory for grid-search? I have a dataloader that returns mini-batches and I want to do a grid-search to find the best hyper-parameters for my model.</p>

ottonemo: <p>As you may know, you can use datasets or even your own dataloader with skorch but this is a bit problematic in conjunction with <code>GridSearchCV</code> or other parameter searches since they expect indexable inputs (torch datasets aren’t). For this reason skorch has a <a href="https://skorch.readthedocs.io/en/latest/user/helper.html#slicedataset" rel="nofollow noopener"><code>SliceDataset</code></a>. You can use it as follows:</p>
<pre><code class="lang-python">gs = GridSearchCV(mySkorchNet, params, ...)

ds = MyCustomDataset()
X_sl = SliceDataset(ds, idx=0)
y_sl = SliceDataset(ds, idx=1)

gs.fit(X_sl, y_sl)
</code></pre>
<p>What <code>SliceDataset</code> does is to emulate index operations so that <code>GridSearchCV</code> can compute the train/validation split and slice the data properly <em>without</em> loading all of the data, so basically a torch dataset you can slice.</p>

Thabang_Lukhetho: <p>Thanks, I managed to make it work using the sliceDataset</p>

Muhammad_Izaz: <p>hey how do you perform grid search please guide me.</p>

Leockl: <p>Hi <a class="mention" href="/u/thabang_lukhetho">@Thabang_Lukhetho</a>, I tried following the solution provided above but it didn’t work for me. Can I ask how did you managed to make it work using sliceDataset?</p>

Thabang_Lukhetho: <p>Can you post the error message or problem you are getting?</p>

Leockl: <p>Hi <a class="mention" href="/u/thabang_lukhetho">@Thabang_Lukhetho</a>, many thanks for your reply.</p>
<p>I have actually posted my question here in PyTorch Forums but haven’t found a solution yet. You can see my code along with the error message for this here: <a href="https://discuss.pytorch.org/t/how-to-use-pytorchs-dataloader-together-with-skorchs-gridsearchcv/99586" class="inline-onebox">How to use PyTorch's DataLoader together with skorch's GridSearchCV</a></p>

Thabang_Lukhetho: <p><a class="mention" href="/u/muhammad_izaz">@Muhammad_Izaz</a>    have a look at this tutorial, I also followed it,</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://miro.medium.com/1*m-R_BkNf1Qjr1YbyOIJY2w.png" class="site-icon" width="32" height="32">
      <a href="https://medium.com/datadriveninvestor/train-a-cnn-using-skorch-for-mnist-digit-recognition-53d7d2f971c7" target="_blank" rel="noopener nofollow ugc" title="02:35AM - 17 December 2020">Medium – 17 Dec 20</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/495;"><img src="https://miro.medium.com/max/1179/1*WpnjVPb3etddKwn_YX7Qyg.png" class="thumbnail" width="690" height="495"></div>

<h3><a href="https://medium.com/datadriveninvestor/train-a-cnn-using-skorch-for-mnist-digit-recognition-53d7d2f971c7" target="_blank" rel="noopener nofollow ugc">Train a CNN using Skorch for MNIST digit recognition</a></h3>

<p>A library that combines the flexibility of scikit-learn and the power of PyTorch</p>

  <p><span class="label1">Reading time: 5 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>