My model is erroring out on validation data but not test data

username: Jordan_Howell
cooked: <p>Hello,</p>
<p>I have trained and tested a model on categorical and numerical data.  Now I’m trying to run my validation data through the model with the same script as my test data but it’s erroring out.</p>
<p>Here is the script I’m running for the validation run:</p>
<pre><code class="lang-auto">#instantiate the model class
model = Data_Only_Model(embedding_size=train_categorical_embedding_sizes
                              ,num_numerical_cols=val_numerical_data.shape[1], output_size = 2, layers = [256,128,64,32])
#load the model from the hard drive
model.load_state_dict(torch.load('D:\\CIS inspection images 0318\\self_build\\data_only_model.pt'))
model = model.eval()
model = model.cuda()

#make predictions
val_preds = []
with torch.no_grad():
    val_categorical_data = val_categorical_data.cuda()
    val_numerical_data = val_numerical_data.cuda()
    val_target = val_target.cuda()
    y_val = model(val_categorical_data, val_numerical_data)
    y_val = y_val.data
    val_preds.append(np.exp(y_val.cpu().data.numpy()))
    loss = criterion(y_val, val_target)
print(f'Loss: {loss:.8f}')
</code></pre>
<p>Here is the error traceback:</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-181-72bf422e9737&gt; in &lt;module&gt;
      4     val_numerical_data = val_numerical_data.cuda()
      5     val_target = val_target.cuda()
----&gt; 6     y_val = model(val_categorical_data, val_numerical_data)
      7     y_val = y_val.data
      8     val_preds.append(np.exp(y_val.cpu().data.numpy()))

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--&gt; 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

&lt;ipython-input-157-ea50eaf0e182&gt; in forward(self, x_categorical, x_numerical)
     57         #concatenating numerical and categorical columns
     58         x = torch.cat([x, x_numerical], 1)
---&gt; 59         x = self.layers(x)
     60         x = F.log_softmax(x)
     61         return x

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--&gt; 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
     90     def forward(self, input):
     91         for module in self._modules.values():
---&gt; 92             input = module(input)
     93         return input
     94 

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--&gt; 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     85 
     86     def forward(self, input):
---&gt; 87         return F.linear(input, self.weight, self.bias)
     88 
     89     def extra_repr(self):

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1368     if input.dim() == 2 and bias is not None:
   1369         # fused op is marginally faster
-&gt; 1370         ret = torch.addmm(bias, input, weight.t())
   1371     else:
   1372         output = input.matmul(weight.t())

RuntimeError: CUDA error: device-side assert triggered```

I'm not sure what I'm doing wrong.</code></pre>

username: ptrblck
cooked: <p>Could you run the code on the CPU as this might be a clearer error message?</p>

username: Jordan_Howell
cooked: <p>When I run this on cpu, I get the following:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/1/011a48b947ff5d9c9d388f4e90b4bf8ecc811fef.png" alt="image" data-base62-sha1="9KN20LbMkc2crKW3ZTnqRd1nCT" width="362" height="225"></p>
<p>I’ve tried this 3 times with the same result.</p>

username: Jordan_Howell
cooked: <p>Ok. I think I’ve found something.  When I try to load the model and the parameters, I have to load the model with the training categorical embedding sizes.  When I don’t load the model via <code>model.Load_state_dict()</code>, and just run the validation data through a instantiated model with the validation embedding sizes, it works (via CPU, I haven’t tried GPU).  But I can’t load a <code>state_dict</code> with validation data embedding sizes and can’t run validation data through with training embedding training sizes.</p>
<p>Does that make sense?</p>
<p>If it makes sense, I guess this brings up a question as to how to train a model that will run with new categories inside the categorical variables and thus, produce different categorical embedding sizes that could be in new data brought in.</p>