Hyperparameter tuning using Bayesian optimization

username: Wafaa_Wardah
cooked: <p>I am using CNN for a binary classification problem and need to use Bayesian optimization to tune parameters like learning rate, number of hidden layers, optimizers, etc</p>
<p>I have built my model with fixed learning rate and number of epochs and Adam optimizer. Can you please give me some guidance on where to go from here in order to have Bayesian optimization implemented for tuning these parameters? And how would I deal with my 3 data sets (training, validation and test)?</p>
<p>I will really appreciate your help.<br>
Thanks.</p>

username: rasbt
cooked: <p>You don’t need to do anything special to perform bayesian optimization for your hyperparameter tuning when using pytorch. You could just setup a script with command line arguments like <code>--learning_rate</code>, <code>--num_layers</code> for the hyperparameters you want to tune and maybe have a second script that calls this script with the diff. hyperparameter values in your bayesian parameter optimization loop.</p>
<p>Conceptually, you can do sth like this</p>
<pre><code class="lang-auto">for round i in max_rounds:
     output = subprocess.call('model.py', '--learning_rate', 'some_val', '--num_layers',  'some_number')
     score = # parse score from output string (e.g., your loss output)
     #optimize p(score | hyperparam values)
     # get some_val, some_number for next round
</code></pre>
<p>But it is probably more efficient if you use a library like hyperopt (<a href="https://github.com/hyperopt/hyperopt" rel="nofollow noopener">https://github.com/hyperopt/hyperopt</a>)</p>
<p>Maybe sth like</p>
<pre><code class="lang-auto">
def run_model(learning_rate):
    # your model init here
    # your training here
    return loss


import numpy as np
from hyperopt import hp, tpe, fmin

# Single line bayesian optimization of polynomial function
best = fmin(fn=lambda x: run_model(x),
            space=hp.normal('x', 0.00001, 0.9),
            algo=tpe.suggest, 
            max_evals=2000)
</code></pre>
<p>But given how expensive typical model training for DL is, it may take a long time …</p>

username: eph_adm
cooked: <p>I went through similar problem only mine was hyperparameter tuning for feedforward neural networks. Then I ran into this very helpful article, you should check it:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://blog.valcarce.fr/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://blog.valcarce.fr/feedforward-neural-network/" target="_blank" title="02:35PM - 01 October 2018" rel="nofollow noopener">Xavier Valcarce – 1 Oct 18</a>
  </header>
  <article class="onebox-body">
    

<h3><a href="https://blog.valcarce.fr/feedforward-neural-network/" target="_blank" rel="nofollow noopener">Feedforward neural network &amp; Optimization of Hyperparameters using...</a></h3>

<p>0. Overview Here is shown how to create a Feedforward neural network (FNN) using Pytorch -- Facebook's AI framework. We'll go through the creation of a FNN, with a user set number of layer/neurones. Making a custom Dataloader is also discussed, so...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: aksg87
cooked: <p>Looks like this link is down <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: Naveen_Kumar
cooked: <p>Yup the link is down FYI <a class="mention" href="/u/eph_adm">@eph_adm</a></p>

username: saba
cooked: <p>Hi All,</p>
<p>I want to do hyper parameter tuning for CNN layers ( 2 or 3 layers), number of filters for CNN, FC layers ( 2 or 3 layers) and number of neurons ([100:10:100]) , batch size {100,200}, LR {10^-4,10^-5}, Dropout{0.3,0.5,0.7}.</p>
<p>Do you know any Pytorch package to implement as grid search?</p>

username: Elahe_Heydari
cooked: <p>you can use <a href="https://github.com/skorch-dev/skorch" rel="nofollow noopener">skorch</a>  for hyper parameter tuning.</p>

username: saba
cooked: <p>Thanks Elahe for your help<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji only-emoji" alt=":slight_smile:"></p>