Hyperparameter Search for MultiOutputRegression

birjon: <p>Hello everybody,</p>
<p>just wondering if there are methods for hyperparameter search when using multioutput regression?<br>
I tried skorch but its not possible to pass more than 1 output, so it makes not really sense for my project…</p>
<p>Thanks for the help</p>

ptrblck: <p>Not sure, if this is really a limitation of Skorch, but <a class="mention" href="/u/ottonemo">@ottonemo</a> might know. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

ottonemo: <p>Can you please elaborate what you mean with</p>
<blockquote>
<p>its not possible to pass more than 1 output</p>
</blockquote>
<p>It is certainly possible to do a multi-output regression with skorch alongside <code>GridSearchCV</code>. Minimal example:</p>
<pre data-code-wrap="python"><code class="lang-python">X = ...
y = ...
assert y.shape[1] == 2

regressor = skorch.toy.make_regressor(output_units=y.shape[1])
net = skorch.NeuralNetRegressor(regressor)
gs = sklearn.model_selection.GridSearchCV(net, params={'module__dropout': [0, 0.1, 0.2]}, scoring='neg_mean_squared_error')
gs.fit(X, y)
</code></pre>

Yang_Shen: <p>Hi Ottonemo,</p>
<p>My case is kind of similar like the one you discussed. But I want to wrap the LSTM model from PyTorch which is a multi-output regression. If that, how can I realize it?</p>
<p>Thanks</p>

ottonemo: <p>Yes, of course <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>See this example:</p>
<pre><code class="lang-auto"># timeful multi regression
getX = lambda i: np.linspace(0+i, 10+i, 10)[None, :, None].astype(np.float32)
X = np.vstack([getX(i) for i in range(10)])
y = np.vstack([
    np.concatenate([Xi[None]**2, Xi[None]*3], axis=-1)
    for Xi in X
])

assert X.shape == (10, 10, 1)
assert y.shape == (10, 10, 2)

class RNNRegressor(torch.nn.Module):
    def __init__(self, dropout=0.):
        super().__init__()
        self.rnn = torch.nn.LSTM(input_size=1, hidden_size=10, batch_first=True)
        self.rnn.reset_parameters()
        self.dop = torch.nn.Dropout2d(p=dropout)
        self.out = torch.nn.Linear(10, 2)
        self.out.reset_parameters()
        
    def forward(self, X):
        out_rnn, _hid = self.rnn(X)
        out_lin = self.out(out_rnn)
        return out_lin
    
net = skorch.NeuralNetRegressor(RNNRegressor)
net.fit(X, y)
</code></pre>
<p>Of course, for grid search, you will need to define a custom split as with any other time-series data. But that’s not related to skorch and varies depending on what your data looks like.</p>

Yang_Shen: <p>Hi Ottonemo,</p>
<p>Thank you very much for the quick response. My training dataset is like this:</p>
<p>tensor([[ 50.,  50.,  50.,  50.],<br>
[ 50.,  50.,  50.,  50.],<br>
[ 50.,  50.,  50.,  51.],<br>
…,<br>
[169., 169., 169., 170.],<br>
[169., 169., 170., 170.],<br>
[169., 170., 170., 170.]])<br>
tensor([[ 50.,  50.,  50.,  50.],<br>
[ 50.,  50.,  50.,  51.],<br>
[ 50.,  50.,  51.,  52.],<br>
…,<br>
[169., 169., 170., 170.],<br>
[169., 170., 170., 170.],<br>
[170., 170., 170., 170.]])</p>
<p>My main code is like this:</p>
<h1><a name="pytorch-lstm-regressor-1" class="anchor" href="#pytorch-lstm-regressor-1"></a>PyTorch LSTM regressor</h1>
<p>class AirModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.lstm = nn.LSTM(input_size=1, hidden_size=50, num_layers=1, batch_first=True)<br>
self.linear = nn.Linear(50, 1)<br>
def forward(self, x):<br>
x, _ = self.lstm(x)<br>
x = self.linear(x)<br>
return x</p>
<h1><a name="creat-the-skorch-wrapper-2" class="anchor" href="#creat-the-skorch-wrapper-2"></a>creat the skorch wrapper</h1>
<p>model_skorch=NeuralNetRegressor(<br>
module=AirModel,<br>
criterion=nn.MSELoss,<br>
optimizer=optim.Adam,<br>
verbose=True)</p>
<p>param_grid={<br>
‘batch_size’:[8,10],<br>
‘max_epochs’:[500],<br>
}</p>
<h1><a name="grid-search-in-scikit-learn-3" class="anchor" href="#grid-search-in-scikit-learn-3"></a>grid search in Scikit-learn</h1>
<p>grid = GridSearchCV(estimator=model_skorch, param_grid=param_grid, n_jobs=-1, cv=3,error_score=‘raise’)<br>
grid_result = grid.fit(X_train, y_train)</p>
<p>But there is always a error in the last row of code: “input.size(-1) must be equal to input_size. Expected 1, got 4”</p>
<p>I appreciate any comments about it</p>