What is the best way to apply k-fold cross validation in CNN?

Neo: <p>Hello,</p>
<p>How can I apply k-fold cross validation with CNN. I do not want to make it manually; for example, in leave one out, I might remove one item from the training set and train the network then apply testing with the removed item. Could you please help me to make this in a standard way.</p>
<p>Regards,</p>

ptrblck: <p>Have a look at <a href="https://github.com/dnouri/skorch" rel="nofollow noopener">Skorch</a>. It’s a scikit-learn compatible wrapper for PyTorch.<br>
scikit itself offers a lot of <a href="http://scikit-learn.org/stable/modules/cross_validation.html" rel="nofollow noopener">cross-validation methods</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

Neo: <p>Thank you. This helped me a lot. Another question, How can I reset the network across the folds? to avoid data leakage.</p>

ptrblck: <p>You could re-initialize the weights of the model.</p>
<pre><code class="lang-auto">def weights_init(m):
    if isinstance(m, nn.Conv2d):
        xavier(m.weight.data)
        xavier(m.bias.data)

model.apply(weights_init)
</code></pre>
<p>I haven’t used skorch yet, but the model reset should be implemented somewhere.</p>

ptrblck: <p>It seems a call to <code>.fit</code> re-initializes the model when <code>warm_start</code> is set to false. The model reset should therefore be performed automatically in grid searches.</p>

Neo: <p>Sorry, I am not able to make it. What is m and how can I import xavier.</p>

ptrblck: <p><code>m</code> is each <code>nn.Module</code> or your model.<br>
You can specify different initializers for each type of layer with the mentioned condition.</p>
<p>You will find all initializers in <a href="http://pytorch.org/docs/0.3.1/nn.html#torch-nn-init" rel="nofollow noopener">torch.nn.init</a>.</p>

alejandrodebus: <p>Hi,</p>
<p>I have developed a function to perform cross-validation in medical images, but it is easily adaptable to other types of problems.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/alejandrodebus/Pytorch-Utils" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/3844860?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/alejandrodebus/Pytorch-Utils" target="_blank" rel="nofollow noopener">alejandrodebus/Pytorch-Utils</a></h3>

<p>Pytorch-Utils - Useful functions to work with PyTorch. At the moment, there is a function to work with cross validation.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Regards</p>

Lian_Jiang: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for the clue of using skorch. I am new to skorch and pytorch. My understanding is that skorch is a pytorch wrapper for sklearn. I found sklearn seems not to support CNN by searching online. For example, cross_val_score need sklearn estimators. sklearn model_selection fit function need X: array-like, shape = [n_samples, n_features] instead of images. Please help me understand how skorch can help cross validation for CNN. Thanks.</p>

ptrblck: <p>Skorch tries to add exactly the missing compatibility of scikit-learn for PyTorch.<br>
The <a href="https://skorch.readthedocs.io/en/latest/?badge=latest">docs</a> have some good examples.<br>
E.g. using the <code>NeuralNet</code> class you can call methods like <code>fit</code> and <code>predict</code> using PyTorch models in the background.</p>

Tsakunelson: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Is there specific way to put our X, y data (generated as  training and validation Loaders already) in the grid search fit method directly? For instance what is the preferred X, y input for the fit function in this scenario? It fails when I Provide the generated training data Loader (images directly from directory) in the grid search fit function. some linky examples if there are, will be quite helpful. Thanks</p>

BarryBA: <p>A question for cross-validation.<br>
Firstly, we divide all the data into training samples and test samples, such as the proportion of 80% and 20%.<br>
Then, we divide the training samples into five groups, four of which used as train data (64%) and one group used as validate data (16%). The 5-fold cross-validation can be carried out to find the suitable parameters of the CNN.<br>
So after the 5-fold cross validation, what should we do next for the testing samples? Should we use the parameters of CNN that determined in the cross validation stage, to train the network again on the 80% samples, and test on the 20% samples?<br>
I am a little confused. Thank you.</p>

ptrblck: <p>Based on the <a href="https://skorch.readthedocs.io/en/stable/net.html#skorch.net.NeuralNet.fit">.fit() docs</a> the fallback should be <code>skorch.dataset.Dataset</code> in case all other inputs won’t work:</p>
<blockquote>
<p>If this doesn’t work with your data, you have to pass a <code>Dataset</code> that can deal with the data.</p>
</blockquote>

ptrblck: <p><a class="mention" href="/u/rasbt">@rasbt</a> explains CV beautifully in <a href="https://sebastianraschka.com/blog/2016/model-evaluation-selection-part3.html" rel="nofollow noopener">his blog post</a> as well as his <a href="https://github.com/rasbt/stat479-machine-learning-fs18/tree/master/10_eval-cv" rel="nofollow noopener">lecture notes</a>. It’s always my source for a quick reassurance if I’m in doubt. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

Tsakunelson: <p>Thanks for the reply <a class="mention" href="/u/ptrblck">@ptrblck</a>. It actually works with my dataset (dataSet from ImageFolder with transformations applied), but in the forward(X) method of the neural net, It passes a List of X, y tensors, instead of just X tensors, which is weird. I do Grid Search with Skorch, and here is the sample code:</p>
<pre><code class="lang-auto">start = time.time()
with torch.no_grad():

    net = NeuralNet(model, criterion= nn.CrossEntropyLoss,
                max_epochs = args.epochs,
                lr = args.learning_rate,
                batch_size = 32,
                optimizer=optim.SGD,
                optimizer_momentum = 0.09,
                iterator_train__shuffle = True,
                iterator_train__num_workers = 4,
                iterator_valid__shuffle = True,
                iterator_valid__num_workers = 4,
                train_split= predefined_split(valid_dataset),
                callbacks= [ lr_scheduler, epoch_acc, checkpoint],
                device = 'cuda')

#pipe = Pipeline([('scale',StandardScaler()),
#                 ('net',net)])
#pipe.fit()
params = {
        'lr':[0.01,0.02],
        'max_epochs':[10,20],
        'module_num_units': [10,20],
        }
gs = GridSearchCV(net, params,refit=False, cv=5, scoring='accuracy')

gs.fit(train_dataset, y=None)
end = time.time()
print("Total training time: "+ str(end - start))
</code></pre>
<p>Is there a specific way I need to specify to send just the corresponding X tensor in my forward function? Thanks</p>

ptrblck: <p>The <code>skorch.dataset.Dataset</code> implementation is not the same as <code>torch.utils.data.Dataset</code> as described <a href="https://skorch.readthedocs.io/en/stable/user/dataset.html#id1" rel="nofollow noopener">here</a>. Could you try to wrap your data in the <code>skorch</code> implementation and pass it to the <code>fit</code> function?</p>

BarryBA: <p><img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=6" title=":+1:" class="emoji" alt=":+1:"> Thank you very much!</p>

avh77: <aside class="quote no-group" data-username="ptrblck" data-post="13" data-topic="15035">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Dataset</p>
</blockquote>
</aside>
<p>An example of such transformation, for instance for MNist dataset would be very usefull.<br>
Thank you in advance!</p>

ptrblck: <p>Which transformation do you mean?</p>

avh77: <p>IIt is unclear for me yet of how to use Pytorch<code>s DataLoader and Sklearn</code>s Cross validation? All I can find in Inet is: it is impossible, Skorch`s <a href="https://skorch.readthedocs.io/en/stable/user/dataset.html#id1" rel="nofollow noopener">explanation</a>  is not perspicuously for me.<br>
Could you give an advise of wrapping DataLoader in Skorch in this way?<br>
Thank you a lot in advance!</p>