Validation set problem

username: dreamer
cooked: <p>Hi everyone, I design my nn but i got error about different sizes. My training set size is [77,768] but my validation set size is [77,1,3] how can i fix this problem:<br>
My loops are:</p>
<pre><code class="lang-auto">class Module(nn.Module):
        def __init__(self, D_in, H1, H2, D_out):
                        super().__init__()
                        self.linear1 = nn.Linear(D_in, H1)
                        self.linear2 = nn.Linear(H1, H2)
                        self.linear3 = nn.Linear(H2, D_out)
        def forward(self, x):
                        x = F.relu(self.linear1(x))  
                        x = F.relu(self.linear2(x))
                        x = self.linear3(x)
                        return x
model=Module(768,600,360,256)
for e in range(epochs):
  
        running_loss = 0.0
        running_corrects = 0.0
        val_running_loss = 0.0
        val_running_corrects = 0.0
  
        for inputs,out in train_generator:
                
                #inputs=torch.squeeze(inputs)
                inputs = inputs.view(inputs.shape[0], -1)
                output=model(inputs)
                print(inputs.size())
                #output = torch.squeeze(output)
                out=out.view(77,256)
                loss = criterion(output,out)
        
                preds,_=torch.max(output,1)
                #        
                outputss.append(preds.max().detach().numpy())
                losses.append(loss)
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
                                                                        
                #outputss.append(output.detach().numpy())
                #print(loss.item())
        '''
        else:
         with torch.no_grad():
                for val_inputs, val_labels in valid_generator:
                        val_inputs = val_inputs.view(val_inputs.shape[0], -1)
                        val_outputs = model(val_inputs)
                        val_loss = criterion(val_outputs, val_labels)
                        
                        _, val_preds = torch.max(val_outputs, 1)
                        val_running_loss += val_loss.item()
                        val_running_corrects += torch.sum(val_preds == val_labels.data)
</code></pre>

username: Nikronic
cooked: <p>Hi,</p>
<p>Usually a validation set is a subset of a whole dataset which means the structure of both training and validation should be consistent as you know. Could you please show a sample of your training set and validation set to track the issue? As you commented in your validation code, we can play with <code>view</code> ,etc but first, you need to make sure the data is transformed reliably.</p>

username: dreamer
cooked: <p>Firstly I get datas from .txt file and datasets are storing in ‘DATA’ array and then<br>
My manipulations are:</p>
<pre><code class="lang-auto">val_range = int(data_x.shape[0]/100) * 15
val_x = data_x[0:val_range, :,:]
train_x = data_x[val_range:None, :,:]

val_y = data_y[0:val_range, :]
train_y = data_y[val_range:None, :]
train_x=torch.Tensor(train_x)
#train_x=train_x.view(-1,1,3)
train_x=train_x.view(77,1,16,16,3)
train_y=torch.Tensor(train_y)
train_y=train_y.view(77,1,16,16)
val_x=torch.Tensor(val_x)
val_y=torch.Tensor(val_y)
</code></pre>
<p>If you asked me the original size of data:</p>
<pre><code class="lang-auto">train_x=[19172,1,3] and train_y=[19172,1]
</code></pre>

username: DoubtWang
cooked: <p>I can not understand why</p>
<blockquote>
<p>training set size is [77,768] but validation set size is [77,1,3]</p>
</blockquote>
<p>because the following code  make the shape of the train and valid set the same:</p>
<pre><code class="lang-auto">val_range = int(data_x.shape[0]/100) * 15
val_x = data_x[0:val_range, :,:]
train_x = data_x[val_range:None, :,:]
</code></pre>

username: dreamer
cooked: <p>Hi, it cannot be same because when I did the calculation, outputs is that:</p>
<pre><code class="lang-auto">train x size: torch.Size([77, 1, 16, 16, 3])
train y size:  torch.Size([77, 1, 16, 16])
val x size:  torch.Size([3465, 1, 3])
val y size:  torch.Size([3465, 1])
</code></pre>

username: DoubtWang
cooked: <p>generally, we use the entire (partly) trained model to run on the val sets.<br>
if the input shape of the train and val set have difference, u can change it to be consistent.<br>
this is my suggestion.<br>
hope that others give a good solution for this problem.</p>

username: Nikronic
cooked: <aside class="quote no-group" data-username="dreamer" data-post="3" data-topic="55143">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dreamer/48/15574_2.png" class="avatar"> dreamer:</div>
<blockquote>
<p>train_x=torch.Tensor(train_x) #train_x=train_x.view(-1,1,3) train_x=train_x.view(77,1,16,16,3) train_y=torch.Tensor(train_y) train_y=train_y.view(77,1,16,16)</p>
</blockquote>
</aside>
<p>Based on the above code, you changing the sizes, before this, val and train are consistent. If you want to extract <code>val</code> dataset, first do the size changing then extract <code>val</code>.</p>
<aside class="quote no-group" data-username="dreamer" data-post="5" data-topic="55143">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dreamer/48/15574_2.png" class="avatar"> dreamer:</div>
<blockquote>
<p>train x size: torch.Size([77, 1, 16, 16, 3]) train y size: torch.Size([77, 1, 16, 16]) val x size: torch.Size([3465, 1, 3]) val y size: torch.Size([3465, 1])</p>
</blockquote>
</aside>
<p>First, just separate your <code>x</code> and <code>y</code>, then do the dimension changes as you want, the extract <code>val</code> using list comprehension which you already have done. In this situation the sizes should be consistent.</p>
<p>Note that we always extract <code>train</code>, <code>test</code> and <code>val</code> set from same source so they all have to be same thing.</p>