What is the best practice for finding the best set of hyperparameters in PyTorch?

Capo_Mestre: <p>What is the best practice for finding the best set of hyperparameters in PyTorch?</p>
<p>It feels that the parameter space is so huge that one could get lost while trying to manually adjust them.</p>
<p>Using grid-approach for hyperparameters is extensively long procedure.<br>
It also feels that random grid, where a set of hyperparameters randomly sampled would also take a lot of time for, say, 20 different hyperparameters, especially when datasets are big, so one would wait quite a while while different versions of a neural network are being trained for each of the hyperparameter sets.</p>
<p>Is there a systematic approach (or the best accepted practice) to this problem in the cases of big datasets and a large hyperparameter space, so it would still be time efficient?</p>

suraj.pt: <p>You might want to check <a href="https://docs.ray.io/en/latest/tune/index.html" rel="nofollow noopener">Ray Tune</a></p>

Capo_Mestre: <p><a class="mention" href="/u/suraj.pt">@suraj.pt</a> Thanks! I will check that out!</p>

Capo_Mestre: <p>I discovered three practices so far:</p>
<ul>
<li>Bayesian optimization</li>
<li>Random search</li>
<li>Genetic algorithm</li>
</ul>
<p>Any more that can be added to the list?</p>

KaiHoo: <p>Maybe this paper can help:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.2.5/images/icons/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://arxiv.org/abs/1812.01187" target="_blank" rel="nofollow noopener">arXiv.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="16" height="16">

<h3><a href="https://arxiv.org/abs/1812.01187" target="_blank" rel="nofollow noopener">Bag of Tricks for Image Classification with Convolutional Neural Networks</a></h3>

<p>Much of the recent progress made in image classification research can be
credited to training procedure refinements, such as changes in data
augmentations and optimization methods. In the literature, however, most
refinements are either briefly...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Capo_Mestre: <p>Other suggestions include:</p>
<ul>
<li>PyTorch Ecosystem <a href="https://medium.com/pytorch/accelerate-your-hyperparameter-optimization-with-pytorchs-ecosystem-tools-bc17001b9a49" rel="nofollow noopener">https://medium.com/pytorch/accelerate-your-hyperparameter-optimization-with-pytorchs-ecosystem-tools-bc17001b9a49</a>
</li>
<li>Hyperband</li>
<li>RayTune</li>
<li>BoTorch</li>
<li>Ax</li>
<li>Optuna</li>
<li>HyperOpt</li>
<li>scikit-optimize</li>
<li>Spearmint.</li>
</ul>
<p>Might be useful:</p>
<ul>
<li>Comparison between hyperparameter optimizers: <a href="https://medium.com/@ramrajchandradevan/comparison-among-hyper-parameter-optimizers-cd37483cd47" rel="nofollow noopener">https://medium.com/@ramrajchandradevan/comparison-among-hyper-parameter-optimizers-cd37483cd47</a>
</li>
</ul>

Capo_Mestre: <p>Thanks for this interesting resource!</p>

Capo_Mestre: <p>One more suggestion is to use</p>
<ul>
<li>Optuna</li>
</ul>

Capo_Mestre: <p>Optunity is yet another library for haperparameter-tuning optimizers: <a href="https://optunity.readthedocs.io/en/latest/index.html" class="inline-onebox" rel="noopener nofollow ugc">Example — Optunity 1.1.0 documentation</a></p>