Model.eval() giving 'out of bounds' error

Colby_Tunick: <p>Hi everyone! Preparing to deploy a trained model to AWS (code below). Not sure how to configure Model.eval() to take in simulated user-input and provide an output.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/519b102a55d65064c9d33d748f938f76cbed34f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/519b102a55d65064c9d33d748f938f76cbed34f6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/519b102a55d65064c9d33d748f938f76cbed34f6.png" alt="image" data-base62-sha1="bDUTzCKqoBUICjUclru1ctKOA98" width="690" height="88" data-dominant-color="464646"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">719×92 4.72 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is what I’ve got so far:</p>
<pre><code class="lang-auto">model.eval()
with torch.no_grad():
  y_val = model(inputColumns, inputNumColumns)
  loss = loss_function(y_val, test_outputs)
  print(f'Loss: {loss:.8f}')
</code></pre>
<p>inputColumns are categorical and inputNumColumns are numerical.</p>
<pre><code class="lang-auto">print(inputColumns.shape)
print(inputNumColumns.shape)
torch.Size([3, 1])
torch.Size([3, 1])
</code></pre>
<p>I am getting this error:</p>
<pre><code class="lang-auto">IndexError: index 1 is out of bounds for dimension 1 with size 1
</code></pre>
<p>Any help would be much appreciated!</p>
<p>This is the model:</p>
<pre><code class="lang-auto">class Model(nn.Module):

    def __init__(self, embedding_size, num_numerical_cols, output_size, layers, p=0.4):
        super().__init__()
        self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])
        #ni is number of embeddings
        #nf is the embedding dimensions(or number of features)
        self.embedding_dropout = nn.Dropout(p) # p is the dropout rate
        self.batch_norm_num = nn.BatchNorm1d(num_numerical_cols) #Only applied to the numerical column

        all_layers = []
        num_categorical_cols = sum((nf for ni, nf in embedding_size))
        input_size = num_categorical_cols + num_numerical_cols

        for i in layers:
            all_layers.append(nn.Linear(input_size, i))
            all_layers.append(nn.ReLU(inplace=False))
            all_layers.append(nn.BatchNorm1d(i))
            all_layers.append(nn.Dropout(p))
            input_size = i

        all_layers.append(nn.Linear(layers[-1], output_size))

        self.layers = nn.Sequential(*all_layers)

    def forward(self, x_categorical, x_numerical):
        embeddings = []
        for i,e in enumerate(self.all_embeddings):
            embeddings.append(e(x_categorical[:,i]))
        x = torch.cat(embeddings, 1)
        x = self.embedding_dropout(x)

        x_numerical = self.batch_norm_num(x_numerical) #Inputs the batch normalized numbers
        x = torch.cat([x, x_numerical], 1)
        x = self.layers(x)
        return x
</code></pre>

ptrblck: <p>I assume the error is thrown in these lines of code:</p>
<pre><code class="lang-python">for i,e in enumerate(self.all_embeddings):
    embeddings.append(e(x_categorical[:,i]))
</code></pre>
<p>Could you check the shape of <code>self.all_embeddings</code>?<br>
Based on your input, if should be only 1, to be able to index <code>x_categorical</code> at dim1.</p>

Colby_Tunick: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for taking a look! I just recently started working with embeddings so I want to make sure I understand your comment.</p>
<p>In the model.train() section (below) the</p>
<pre><code class="lang-auto">for i,e in enumerate(self.all_embeddings):
    embeddings.append(e(x_categorical[:,i]))
</code></pre>
<p>takes in embeddings of these sizes<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e06853016e7809e94e5a040a580fa939924803a2.png" data-download-href="https://discuss.pytorch.org/uploads/default/e06853016e7809e94e5a040a580fa939924803a2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e06853016e7809e94e5a040a580fa939924803a2.png" alt="image" data-base62-sha1="w1csXhzphWTG2HT2MDsv4wyvlYu" width="690" height="169" data-dominant-color="373737"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1275×313 6.21 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
In the model.eval() section, the embeddings are this size</p>
<pre><code class="lang-auto">torch.Size([3, 1])
</code></pre>
<p>Is there a different way to check the shape of <code>self.all_embeddings</code>?</p>

ptrblck: <p>Based on the code <code>x_categorical</code> should have the shape <code>[batch_size, len(self.all_embeddings), *]</code>, since you are indexing this tensor in dim1.<br>
If you are passing <code>x_categorical</code> as a tensor with the shape <code>[3, 1]</code>, <code>self.all_embeddings</code> can only contain a single <code>nn.Embedding</code> layer.</p>
<p>Could you explain your use case a bit more and how you would like to use the for loop?</p>

Colby_Tunick: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I have used this <a href="https://stackabuse.com/introduction-to-pytorch-for-classification/#disqus_thread" rel="nofollow noopener">open source model</a> as the basis for mine. From the description:<br>
Inside the constructor, a few variables are initialized. Firstly, the <code>all_embeddings</code> variable contains a list of <code>ModuleList</code> objects for all the categorical columns. The <code>embedding_dropout</code> stores the dropout value for all the layers. Finally, the <code>batch_norm_num</code> stores a list of <code>BatchNorm1d</code> objects for all the numerical columns.</p>
<p>Next, to find the size of the input layer, the number of categorical and numerical columns are added together and stored in the <code>input_size</code> variable. After that, a <code>for</code> loop iterates and the corresponding layers are added into the <code>all_layers</code> list. The layers added are:</p>
<ul>
<li>
<code>Linear</code> : Used to calculate the dot product between the inputs and weight matrixes</li>
<li>
<code>ReLu</code> : Which is applied as an activation function</li>
<li>
<code>BatchNorm1d</code> : Used to apply batch normalization to the numerical columns</li>
<li>
<code>Dropout</code> : Used to avoid overfitting</li>
</ul>
<p>After the <code>for</code> loop, the output layer is appended to the list of layers. Since we want all of the layers in the neural networks to execute sequentially, the list of layers is passed to the <code>nn.Sequential</code> class.</p>
<p>Next, in the <code>forward</code> method, both the categorical and numerical columns are passed as inputs. The embedding of the categorical columns takes place in the following lines.</p>
<pre><code class="lang-auto">embeddings = []
for i, e in enumerate(self.all_embeddings):
    embeddings.append(e(x_categorical[:,i]))
x = torch.cat(embeddings, 1)
x = self.embedding_dropout(x)
</code></pre>
<p>The batch normalization of the numerical columns is applied with the following script:</p>
<pre><code class="lang-auto">x_numerical = self.batch_norm_num(x_numerical)
</code></pre>
<h2>Finally, the embedded categorical columns <code>x</code> and the numeric columns <code>x_numerical</code> are concatenated together and passed to the sequential <code>layers</code> .</h2>
<p>This main issue is that this model was not designed with a model.eval() mode. Thank you in advance for your help!</p>

Colby_Tunick: <p>So, the solution is: came from the shape of your tensors being correct. In order for the tensor to be valid it must have a valid length, and in this case the same columns as the training data. So in our case we would want the tensor for categorical data to be (1,7) and (1,8) for numerical data</p>
<p>This is the data I used to mock it, keeping the columns we had in the original training dataset that was stacked, but just setting them to 0 and null</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dcb9de56087ff39b9626ac65278e23911c3b413.png" data-download-href="https://discuss.pytorch.org/uploads/default/5dcb9de56087ff39b9626ac65278e23911c3b413" title="image (2)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dcb9de56087ff39b9626ac65278e23911c3b413_2_690x497.png" alt="image (2)" data-base62-sha1="dnKEp6QISTo0eT5bXWFFXaaAKMX" width="690" height="497" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dcb9de56087ff39b9626ac65278e23911c3b413_2_690x497.png, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dcb9de56087ff39b9626ac65278e23911c3b413.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dcb9de56087ff39b9626ac65278e23911c3b413.png 2x" data-dominant-color="282625"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image (2)</span><span class="informations">844×608 63.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Thanks for the update.<br>
Based on your description you are missing some categorical features during evaluation.<br>
Replacing them with a zero or invalid value might yield bad performance or invalid outputs (e.g. NaNs).<br>
You could try to replace the missing values with the mean or median from your training set for this particular feature.</p>

Colby_Tunick: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> - duly noted! Since last post, I’ve made alot of progress (saved my models weights using  torch.save(model.state_dict(), ‘.pth’) and have been able to load it in .eval() in a new file).</p>
<p>I am confused how to change my eval (formerly training loop) to iterate over the new data set just one time to produce a binary prediction? Any advice you could give is greatly appreciated. Cheers!</p>
<pre><code class="lang-auto">epochs = 1
aggregated_losses = []

for x in range(epochs):
    x += 1
    y_pred = model(inputCatColumns, inputNumColumns)
    single_loss = loss_function(y_pred, train_outputs)
    aggregated_losses.append(single_loss)

    if x%25 == 1:
        print(f'epoch: {x:3} loss: {single_loss.item():10.8f}')

    optimizer.zero_grad()
    single_loss.backward()
    optimizer.step()
    #scheduler.step(x)

print(f'epoch: {x:3} loss: {single_loss.item():10.10f}')
</code></pre>
<pre><code class="lang-auto">with torch.no_grad():
  y_val = model(inputCatColumns, inputNumColumns)
  loss = loss_function(y_val, test_outputs)
  print(f'Loss: {loss:.8f}')
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Colby_Tunick" data-post="8" data-topic="72422">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/colby_tunick/48/21448_2.png" class="avatar"> Colby_Tunick:</div>
<blockquote>
<p>Since last post, I’ve made alot of progress (saved my models weights using torch.save(model.state_dict(), ‘.pth’) and have been able to load it in .eval() in a new file).</p>
</blockquote>
</aside>
<p>That sounds like a great progress!</p>
<aside class="quote no-group" data-username="Colby_Tunick" data-post="8" data-topic="72422">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/colby_tunick/48/21448_2.png" class="avatar"> Colby_Tunick:</div>
<blockquote>
<p>how to change my eval (formerly training loop) to iterate over the new data set just one time to produce a binary prediction?</p>
</blockquote>
</aside>
<p>I assume your model outputs a single logit, since you are dealing with a binary prediction?<br>
If that’s the case, this dummy code might be a good starter:</p>
<pre data-code-wrap="python"><code class="lang-python">model.eval()
preds = []
with torch.no_grad():
  y_val = model(inputCatColumns, inputNumColumns)
  preds.append(y_val &gt; 0.) # if y_val are logits
  loss = loss_function(y_val, test_outputs)
  print(f'Loss: {loss:.8f}')
preds = torch.stack(preds)
</code></pre>
<p>Let me know, if that helps or if I misunderstood your question.</p>

Colby_Tunick: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> That dummy code looks like a great start. When I run it, I get this an error. Should I not be converting my new data set to tensors like I had to for the model training? If so, would I just pass the values in ‘raw’ so to speak?</p>
<pre><code class="lang-auto">['POL_STATUS', 'P1_POLICY_REFUSED', 'FLOODING', 'Zipcode', 'Combined FIPS', 'State', 'Name']

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

&lt;ipython-input-62-c9aa9c799384&gt; in &lt;module&gt;()
      1 preds = []
      2 with torch.no_grad():
----&gt; 3   y_val = model(inputCatColumns, inputNumColumns)
      4   preds.append(y_val &gt; 0.) # if y_val are logits
      5   loss = loss_function(y_val, test_outputs)

1 frames

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

&lt;ipython-input-35-5bbeaf977447&gt; in forward(self, x_categorical, x_numerical)
     28         print(x_categorical)
     29         for i,e in enumerate(self.all_embeddings):
---&gt; 30           print(x_categorical[:,i])
     31           embeddings.append(e(x_categorical[:,i]))
     32 

TypeError: list indices must be integers or slices, not tuple


</code></pre>

ptrblck: <p>Ah, yeah sure. You would have to process the data in the same way.</p>

Colby_Tunick: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Great, thats what I am currently doing. The only difference between training the model, and eval mode is now I am only passing in a single row of data (but still preprocessing the same way). Is this the right construction of the eval loop or should I be doing it differently?</p>
<p>Also, to load in the single row, should I be using the DataLoader?</p>
<p>Cheers!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79fa37d70bfe8af5ad183378fd23bf96e8134d55.png" data-download-href="https://discuss.pytorch.org/uploads/default/79fa37d70bfe8af5ad183378fd23bf96e8134d55" title="pic 1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79fa37d70bfe8af5ad183378fd23bf96e8134d55_2_690x126.png" alt="pic 1" data-base62-sha1="hp3Ryo8GkNDzGa6rZijDcI65p77" width="690" height="126" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79fa37d70bfe8af5ad183378fd23bf96e8134d55_2_690x126.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79fa37d70bfe8af5ad183378fd23bf96e8134d55_2_1035x189.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79fa37d70bfe8af5ad183378fd23bf96e8134d55_2_1380x252.png 2x" data-dominant-color="232323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pic 1</span><span class="informations">3594×657 95.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The code looks good.<br>
Yes, you could use the <code>DataLoader</code> as a convenient method.</p>

Colby_Tunick: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for that suggestion! The new prediction block is running into this error - I found <a href="https://www.discoverbits.in/1057/python-typeerror-list-indices-must-integers-slices-not-tuple" rel="nofollow noopener">this solve for it</a> but am confused because the data being used has already been converted into a numpy array</p>
<pre><code class="lang-auto">name = userInput['Name'].cat.codes.values

inputColumns = np.stack([name, etc...], 1)
</code></pre>
<p>and then converted over to tensors.</p>
<pre><code class="lang-auto">inputColumns = torch.tensor(inputColumns, dtype=torch.int64)
</code></pre>
<p>Is there something wrong in this process that would cause this error? The same process worked for the<br>
<code>model.train()</code> function.</p>
<pre><code class="lang-auto">['POL_STATUS', 'P1_POLICY_REFUSED', 'FLOODING', 'Zipcode', 'Combined FIPS', 'State', 'Name']

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

&lt;ipython-input-62-4fec13edb637&gt; in &lt;module&gt;()
      1 preds = []
      2 with torch.no_grad():
----&gt; 3   newY_val = model(inputCatColumns, inputNumColumns)
      4   preds.append(newY_val &gt; 0.) # if y_val are logits
      5   loss = loss_function(newY_val, test_outputs)

1 frames

&lt;ipython-input-35-5bbeaf977447&gt; in forward(self, x_categorical, x_numerical)
     28         print(x_categorical)
     29         for i,e in enumerate(self.all_embeddings):
---&gt; 30           print(x_categorical[:,i])
     31           embeddings.append(e(x_categorical[:,i]))
     32 

TypeError: list indices must be integers or slices, not tuple
</code></pre>

ptrblck: <p>What does <code>print(x_categorical)</code> return?<br>
The indexing should work, if it’s a tensor.</p>