What is the best way to perform hyper parameter search in PyTorch?

Zhiqiu_Lin: <p>^^What is the best way to perform hyper parameter search in PyTorch? Are there frameworks that can ease this process?</p>

ptrblck: <p><a class="mention" href="/u/kevinzakka">@kevinzakka</a> has implemented <a href="https://github.com/kevinzakka/hypersearch">hypersearch</a>.<br>
There are still some TODOs, so alternatively you could have a look at <a href="https://github.com/dnouri/skorch">Skorch</a> which allows you to use the scikit-learn grid search / random search.</p>

dedeco: <p>An example:</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self):
        '''
        A feedForward neural network.
        Argurmets:
            n_feature: How many of features in your data
            n_hidden:  How many of neurons in the hidden layer
            n_output:  How many of neuros in the output leyar (defaut=1)
        '''
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(D_in, H, bias=True)   # hidden layer
        self.predict = torch.nn.Linear(H, D_out, bias=True)   # output layer
        self.n_feature, self.n_hidden, self.n_output = D_in, H, D_out
    def forward(self, x,**kwargs):
        '''
        Argurmets:
            x: Features to predict
        '''
        torch.nn.init.constant_(self.hidden.bias.data,1)
        torch.nn.init.constant_(self.predict.bias.data,1)
        x = torch.sigmoid(self.hidden(x))      # activation function for hidden layer
        x = torch.sigmoid(self.predict(x))     # linear output
        return x
</code></pre>
<pre><code class="lang-auto">from skorch import NeuralNetRegressor
net = NeuralNetRegressor(Net
                         , max_epochs=100
                         , lr=0.001
                         , verbose=1)
</code></pre>
<pre><code class="lang-auto">X_trf = X
y_trf = y.reshape(-1, 1)
print(X_trf.shape,y_trf.shape)
</code></pre>
<pre><code class="lang-auto">from sklearn.model_selection import GridSearchCV

params = {
    'lr': [0.001,0.005, 0.01, 0.05, 0.1, 0.2, 0.3],
    'max_epochs': list(range(500,5500, 500))
}

gs = GridSearchCV(net, params, refit=False, scoring='r2', verbose=1, cv=10)

gs.fit(X_trf, y_trf)
</code></pre>

saba: <p>Hi Ptrblck,</p>
<p>I hope you are doing well. Sorry to take your time. I want to do hyper parameter tuning for CNN layers ( 2 or 3 layers), number of filters for CNN, FC layers ( 2 or 3 layers) and number of neurons ([100:10:100]) , batch size {100,200}, LR {10^-4,10^-5}, Dropout{0.3,0.5,0.7}.</p>
<p>Would you please tell me what is your suggestion? Is there any function to use in Pytorch? Or it is better to do grid search for all combinations?</p>

FelipeVW: <p>I am a bit skeptical of methods like grid and random search. It is nice to try them but I think experience is key in hyperparameter fine-tunning. These methods are not that good when your training takes 1 week and you do not have a server with 100’s of gpus.<br>
For example, taking a better optimizer that converges faster is a cheaper and better way to optimize your training. Also, take for instances the batch size, a 32 batch size in a CNN will tend to perform better than a 4 or 8 batch size (at least in the dataset I am working on).<br>
Experience plays a big role I guess.</p>

Haxxardoux: <p><a class="mention" href="/u/felipevw">@FelipeVW</a></p>
<p>In my opinion, you are 75% right, In the case of something like a CNN, you can scale down your model procedurally so it takes much less time to train, THEN do hyperparameter tuning.  <a href="https://arxiv.org/pdf/1905.11946.pdf" rel="nofollow noopener">This paper</a> found that a grid search to obtain the best accuracy possible, THEN scaling up the complexity of the model led to superior accuracy. Probably would not work for all cases, but definitely a good application for grid searches.</p>

Bttxy: <p>Hello,<br>
I know it has been along time since this post.<br>
It has been days since I tried to find a solution to find a better hyperparameters for my model. I tried many solutions but with no good results <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"><br>
I am working with your proposed solution here with a classification bert model developed under the Pytorch lightning. I couldn’t find what to put in the <code>fit</code> method. i tried a tensor dataloader but It doesn’t work so I tried the result of the tokeniser like this :</p>
<pre><code class="lang-auto">input_ids_train = encoded_data_train['input_ids'] # Xs of my models
labels_train = torch.tensor(label.values) # the labels of my data
</code></pre>
<p>I got a <code>TypeError: forward() missing 1 required positional argument: 'attention_mask'</code> and a <code>FitFailedWarning</code> said that I need to reshape my data.</p>
<p>Could you please tell me how can I prepare my data (input_ids, attention_mask and labels) so they can fit the <code>fit</code> methods, please ?</p>

RAFAIL_MAHAMMADLI: <p>Hello,</p>
<p>How can we perform GridSearchCV with pretrained models in pytorch?</p>
<p>Thanks</p>

ptrblck: <p><a href="https://github.com/skorch-dev/skorch"><code>skorch</code></a> is compatible with these scikit-learn methods so you might want to check it out.</p>

RAFAIL_MAHAMMADLI: <p>I will check it out. Thank you very much</p>

RAFAIL_MAHAMMADLI: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I attempted applying GridSearch to optimize hyperparameters, but . Affter search.fit(train_ds, y=None)<br>
and  search.best_score_<br>
I wanted to see best score. It demonstrate nan.</p>
<p>What am I doing wrong for applying GridSearch?</p>
<p>My codes is following</p>
<p>Thank you</p>
<p>data_dir = ‘/content/drive/MyDrive/raphcatr’<br>
train_transforms = transforms.Compose([<br>
transforms.RandomResizedCrop(224),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406],<br>
[0.229, 0.224, 0.225])<br>
])<br>
val_transforms = transforms.Compose([<br>
transforms.Resize(256),<br>
transforms.CenterCrop(224),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406],<br>
[0.229, 0.224, 0.225])<br>
])</p>
<p>train_ds = datasets.ImageFolder(<br>
os.path.join(data_dir, ‘train’), train_transforms)<br>
val_ds = datasets.ImageFolder(<br>
os.path.join(data_dir, ‘val’), val_transforms)</p>
<p>class PretrainedModel(nn.Module):<br>
def <strong>init</strong>(self, output_features):<br>
super().<strong>init</strong>()<br>
model = models.resnet18(pretrained=True)<br>
num_ftrs = model.fc.in_features<br>
model.fc = nn.Linear(num_ftrs, output_features)<br>
classifier = nn.Sequential(nn.Dropout(p=0.5),<br>
nn.Linear(num_ftrs, output_features))</p>
<pre><code>    model.fc=classifier
    self.model = model
    
def forward(self, x):
    return self.model(x)
</code></pre>
<p>params = {<br>
‘lr’: [0.01, 0.02],<br>
‘max_epochs’: [10, 20],<br>
}</p>
<p>net = NeuralNetClassifier(<br>
PretrainedModel,<br>
criterion=nn.CrossEntropyLoss,<br>
lr=0.01,<br>
batch_size=16,<br>
max_epochs=10,<br>
optimizer=optim.SGD,<br>
optimizer__momentum=0.9,<br>
iterator_train__shuffle=True,<br>
callbacks=[freezer, lrscheduler, checkpoint],<br>
device=‘cuda’ # comment to train on cpu<br>
)</p>
<p>gs = GridSearchCV(net, params, refit=False, cv=3, scoring=‘accuracy’, verbose=2)<br>
gs.fit(train_ds, y=None)</p>

ptrblck: <p>Could you check, if you could see all scores and see if some of the runs created a NaN loss due to a bad hyperparameter set?<br>
If so, you might want to select the highest score which is not a NaN. I’m not familiar with the internal implementation of <code>skorch</code>’ grid search but would assume that invalid runs (yielding NaNs) would be removed from the best score.</p>

ChunkRaymond: <p>You could try raytune+pytorch-lighyning. As for me, that’s better than handle or sktorch.</p>