Training 1D CNN in PyTorch

Tehreem: <p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F</p>
<p>class  CharCNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(CharCNN, self).<strong>init</strong>()<br>
self.conv1 = nn.Sequential(<br>
nn.Conv1d(num_channels, depth_1, kernel_size=kernel_size_1, stride=stride_size),<br>
nn.ReLU(),<br>
nn.MaxPool1d(kernel_size=kernel_size_1, stride=stride_size),<br>
nn.Dropout(0.1),<br>
)</p>
<pre><code>    self.conv2 = nn.Sequential(
        nn.Conv1d(depth_1, depth_2, kernel_size=kernel_size_2, stride=stride_size),
        nn.ReLU(),
        nn.MaxPool1d(kernel_size=3, stride=3),
        nn.Dropout(0.25)
    )            
        
         
    self.fc1 = nn.Sequential(
        nn.Linear(depth_2*kernel_size_2, num_hidden),
        nn.ReLU(),
        nn.Dropout(0.5)
    )
    
    self.fc2 = nn.Sequential(
        nn.Linear(num_hidden, 11),
        nn.ReLU(),
        nn.Dropout(0.5)
    )
    
    

def forward(self, x):
    x = self.conv1(x)
    x = self.conv2(x)

    # collapse
    x = x.view(x.size(0), -1)
    # linear layer
    x = self.fc1(x)
    # output layer
    x = self.fc2(x)
    x = self.log_softmax(x)
    
    return x
</code></pre>
<p>Hello,<br>
I am trying to train this network in pytorch. I am litlle confused regarding the training of 1D CNN network. We generally make train and test loaders in pytorch. But i want to train my network without data loader. Please help me how i can train this network.</p>

ptrblck: <p>You don’t need to use a <code>DataLoader</code> and should only make sure to pass the input in the right shape.<br>
<code>nn.Conv1d</code> modules expect an input in the shape <code>[batch_size, channels, seq_length]</code>.</p>

Tehreem: <p>Thank you for your response. I am training my network like this, but i am getting the error given below. Please how I can resolve this issue.</p>
<p>opt = torch.optim.Adam(model.parameters(),lr=learning_rate)<br>
criterion = nn.CrossEntropyLoss()</p>
<p>for e in range(training_epochs):<br>
if(train_on_gpu):<br>
net.cuda()<br>
train_losses = []</p>
<pre><code>for batch in iterate_minibatches(train_x, train_y, batch_size):
    x, y = batch
    inputs, targets = torch.from_numpy(x), torch.from_numpy(y)
    if(train_on_gpu):
        inputs, targets = inputs.cuda(), targets.cuda()
    opt.zero_grad()   
    output = model(inputs, batch_size)
        
    loss = criterion(output, targets.long())
    train_losses.append(loss.item())
    loss.backward()
    opt.step()
val_losses = []
accuracy=0
f1score=0
print("Epoch: {}/{}...".format(e+1, training_epochs),
              "Train Loss: {:.4f}...".format(np.mean(train_losses)))
</code></pre>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
14             inputs, targets = inputs.cuda(), targets.cuda()<br>
15         opt.zero_grad()<br>
—&gt; 16         output = model(inputs, batch_size)<br>
17<br>
18         loss = criterion(output, targets.long())</p>
<p>~\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\module.py in <strong>call</strong>(self, *input, **kwargs)<br>
530             result = self._slow_forward(*input, **kwargs)<br>
531         else:<br>
–&gt; 532             result = self.forward(*input, **kwargs)<br>
533         for hook in self._forward_hooks.values():<br>
534             hook_result = hook(self, input, result)</p>
<p>TypeError: forward() takes 2 positional arguments but 3 were given</p>

ptrblck: <p>Your model’s <code>forward</code> method takes one input, while you are passing two into it.<br>
Try to remove <code>batch_size</code> from the <code>model(inputs, batch_size)</code> call and it should work.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging a bit easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Tehreem: <p>I solved this issue then i am stuck in new problem. Now code is giving the following error:<br>
ValueError: Expected input batch_size (1) to match target batch_size (64).</p>

ptrblck: <p>What is your batch size?<br>
It seems the model output gets reshaped in a wrong way or you are creating the target in the wrong shape.</p>
<p>Could you post your code, which creates this error?</p>

Tehreem: <p>Thanks for your reply again. My batch size is 64.  This is the part of code which is causing this eeror.</p>
<pre><code>      ValueError                                Traceback (most recent call last)
    &lt;ipython-input-178-d3334fef0b32&gt; in &lt;module&gt;
  18         output = model(inputs)
  19 
   ---&gt; 20         loss = criterion(output, targets.long())
  21         train_losses.append(loss.item())
  22         loss.backward()</code></pre>

Tehreem: <p>Is there anyway i can share my complete code with you?. It’s very important. I would really appreciate your help.</p>

Tehreem: <p><strong>My complete code is here: I am following githubs code for accomplishing my task.</strong></p>
<pre><code>def windowz(data, size):
   start = 0
   while start &lt; len(data):
       yield start, start + size
      start += (size // 2)


 def segment_pa2(x_train,y_train,window_size):
segments = np.zeros(((len(x_train)//(window_size//2))-1,window_size,52))
labels = np.zeros(((len(y_train)//(window_size//2))-1))
i_segment = 0
i_label = 0
for (start,end) in windowz(x_train,window_size):
    if(len(x_train[start:end]) == window_size):
        m = stats.mode(y_train[start:end])
        segments[i_segment] = x_train[start:end]
        labels[i_label] = m[0]
        i_label+=1
        i_segment+=1

  return segments, labels
  print ('starting...')
 start_time = time.time()

dataset = sys.argv[1]
 path = '/Users/tehreem/Desktop/PAMAP2/PAMAP2_Dataset/pamap2.h5'

f = h5py.File(path, 'r')
print(f)

   x_train = f.get('train').get('inputs')[()]
   y_train = f.get('train').get('targets')[()]

  x_test = f.get('test').get('inputs')[()]
  y_test = f.get('test').get('targets')[()]


 print ("x_train shape =",x_train.shape)
 print ("y_train shape =",y_train.shape)
 print ("x_test shape =" ,x_test.shape)
 print ("y_test shape =",y_test.shape)


x_train = x_train[::3,:]
y_train = y_train[::3]
x_test = x_test[::3,:]
y_test = y_test[::3]
print ("x_train shape(downsampled) = ", x_train.shape)
print ("y_train shape(downsampled) =",y_train.shape)
print ("x_test shape(downsampled) =" ,x_test.shape)
print ("y_test shape(downsampled) =",y_test.shape)

 print (np.unique(y_train))
 print (np.unique(y_test))
unq = np.unique(y_test)

input_width = 52
print("segmenting signal...")
train_x, train_y = segment_pa2(x_train,y_train,input_width)
test_x, test_y = segment_pa2(x_test,y_test,input_width)
print ("signal segmented.")

train = pd.get_dummies(train_y)
test = pd.get_dummies(test_y)
train, test = train.align(test, join='inner', axis=1)

train_y = np.asarray(train)
test_y = np.asarray(test)

input_height = 1
input_width = input_width 
num_labels = 11  
num_channels = 52 

   batch_size = 64
   stride_size = 2
   kernel_size_1 = 7
  kernel_size_2 = 3
  kernel_size_3 = 1
  depth_1 = 128
 depth_2 = 128
depth_3 = 128
num_hidden = 512 

 dropout_1 = 0.1 #0.1
 dropout_2 = 0.25 #0.25
 dropout_3 = 0.5 #0.5

learning_rate = 0.0005
training_epochs = 50
total_batches = train_x.shape[0] // batch_size


  train_x = train_x.reshape(len(train_x),1,input_width,num_channels) 
  test_x = test_x.reshape(len(test_x),1,input_width,num_channels) 
 print ("test_x_reshaped = " , test_x.shape)
 print ("train_x shape =",train_x.shape)
 print ("train_y shape =",train_y.shape)
 print ("test_x shape =",test_x.shape)
 print ("test_y shape =",test_y.shape)

 train_x = train_x.reshape(-1,input_width,num_channels) 
 test_x = test_x.reshape(-1,input_width,num_channels) 

 def init_weights(m):
if type(m) == nn.LSTM:
    for name, param in m.named_parameters():
        if 'weight_ih' in name:
            torch.nn.init.orthogonal_(param.data)
        elif 'weight_hh' in name:
            torch.nn.init.orthogonal_(param.data)
        elif 'bias' in name:
            param.data.fill_(0)
  elif type(m) == nn.Conv1d or type(m) == nn.Linear:
    torch.nn.init.orthogonal_(m.weight)
    m.bias.data.fill_(0)
 model.apply(init_weights) 

 import torch
 import torch.nn as nn
import torch.nn.functional as F

  class  CharCNN(nn.Module):
     def __init__(self):
     super(CharCNN, self).__init__()
    
    self.conv1 = nn.Sequential(
        nn.Conv1d(num_channels, depth_1, kernel_size=kernel_size_1, stride=stride_size),
        nn.ReLU(),
        nn.MaxPool1d(kernel_size=kernel_size_1, stride=stride_size),
        nn.Dropout(0.1),
    )

    self.conv2 = nn.Sequential(
        nn.Conv1d(depth_1, depth_2, kernel_size=kernel_size_2, stride=stride_size),
        nn.ReLU(),
        nn.MaxPool1d(kernel_size=kernel_size_2, stride=stride_size),
        nn.Dropout(0.25)
    )            
        
         
    self.fc1 = nn.Sequential(
        nn.Linear(128*64, num_hidden),
        nn.ReLU(),
        nn.Dropout(0.5)
    )
    
    self.fc2 = nn.Sequential(
        nn.Linear(num_hidden, 11),
        nn.ReLU(),
        nn.Dropout(0.5)
    )
    
    

def forward(self, x):
    out = self.conv1(x)
    out = self.conv2(out)

    # collapse
    out = out.reshape(-1,128*64)
    #out = out.view(out.size(0), -1) 
    # linear layer
    out = self.fc1(out)
    # output layer
    out = self.fc2(out)
    #out = self.log_softmax(x,dim=1)
    
    return out

 model =  CharCNN()
 print(model)

def iterate_minibatches(inputs, targets, batch_size, shuffle=True):
assert len(inputs) == len(targets)
if shuffle:
    indices = np.arange(len(inputs))
    np.random.shuffle(indices)
for start_idx in range(0, len(inputs) - batch_size + 1, batch_size):
    if shuffle:
        excerpt = indices[start_idx:start_idx + batch_size]
    else:
        excerpt = slice(start_idx, start_idx + batch_size)
    yield inputs[excerpt], targets[excerpt]

 criterion = nn.CrossEntropyLoss()    
opt = torch.optim.Adam(model.parameters(),lr=learning_rate)


for e in range(training_epochs):
   if(train_on_gpu):
     net.cuda()
   train_losses = []    

for batch in iterate_minibatches(train_x, train_y, batch_size):
    x, y = batch
    inputs, targets = torch.from_numpy(x), torch.from_numpy(y)
    if(train_on_gpu):
        inputs, targets = inputs.cuda(), targets.cuda()
        #inputs= inputs.view(batch_size,  128*64)
        #targets = targets.view(batch_size)
    opt.zero_grad()   
    output = model(inputs)
        
    loss = criterion(output, targets.long())
    train_losses.append(loss.item())
    loss.backward()
    opt.step()
val_losses = []
accuracy=0
f1score=0
print("Epoch: {}/{}...".format(e+1, training_epochs),
              "Train Loss: {:.4f}...".format(np.mean(train_losses)))</code></pre>

ptrblck: <p>Thanks for the code. Using the provided model definition, and trying out some input shapes, this code seems to work fine and gives the desired output batch size:</p>
<pre><code class="lang-python">num_channels = 52
depth_1 = 128
kernel_size_1 = 7
stride_size = 3
depth_2 = 128
kernel_size_2 = 3
num_hidden = 512
model = CharCNN()

x = torch.randn(64, 52, 300)
out = model(x)
print(out.shape)
&gt; torch.Size([64, 11])
</code></pre>

Tehreem: <p>Thank you very much for your help. I just want to clarify what changes should i make in my code for achieving this??</p>
<p>Regards</p>

ptrblck: <p>I copy-pasted your model definition and used a random input tensor, which yields the desired output shape. I don’t know, if I had to change something else, but if so, it might be a typo.</p>
<p>Is my code snippet not running with your code?</p>

Tehreem: <p>No, this code is not working for me. I am getting this error with my code.</p>
<pre><code> ValueError: Expected input batch_size (1) to match target batch_size (64).</code></pre>

Tehreem: <p>When i print the output it gives me the following output: torch.Size([1, 11]). This is without squeezing. What I understand from this is that my model is not taking the batch size properly. What should i change to fix this.</p>

ptrblck: <p>I copy-pasted your model from your original post and added used my code snippet from the previous post.<br>
I needed to define <code>self.log_softmax</code> as <code>nn.LogSoftmax(1)</code>, as it was missing.<br>
Here is the complete code:</p>
<pre><code class="lang-python">class CharCNN(nn.Module):
    def __init__(self):
        super(CharCNN, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv1d(num_channels, depth_1, kernel_size=kernel_size_1, stride=stride_size),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=kernel_size_1, stride=stride_size),
            nn.Dropout(0.1),
        )
        
        self.conv2 = nn.Sequential(
            nn.Conv1d(depth_1, depth_2, kernel_size=kernel_size_2, stride=stride_size),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3, stride=3),
            nn.Dropout(0.25)
        )            
            
        self.fc1 = nn.Sequential(
            nn.Linear(depth_2*kernel_size_2, num_hidden),
            nn.ReLU(),
            nn.Dropout(0.5)
        )
        
        self.fc2 = nn.Sequential(
            nn.Linear(num_hidden, 11),
            nn.ReLU(),
            nn.Dropout(0.5)
        )
        self.log_softmax = nn.LogSoftmax(1)
        
    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
    
        # collapse
        x = x.view(x.size(0), -1)
        # linear layer
        x = self.fc1(x)
        # output layer
        x = self.fc2(x)
        x = self.log_softmax(x)
        
        return x

num_channels = 52
depth_1 = 128
kernel_size_1 = 7
stride_size = 3
depth_2 = 128
kernel_size_2 = 3
num_hidden = 512
model = CharCNN()

x = torch.randn(64, 52, 300)
out = model(x)
print(out.shape)
&gt; torch.Size([64, 11])
</code></pre>
<p>Most likely you’ve changed something in the code, so I would recommend to reuse your original code, as it keeps the batch size.</p>

Tehreem: <p><strong>Thank you very much for your reply. I used my original code with the changes you suggested. I got this error after that.</strong><br>
<code>RuntimeError: Given input size: (128x1x2). Calculated output size: (128x1x0). Output size is too small</code></p>

Tehreem: <p>Please also suggest any tutorial which will helps in calculating the parameters of each layer in CNN.</p>

ptrblck: <p>Did you just copy-paste my code snippet and did receive this error or did you modify something?</p>
<aside class="quote no-group" data-username="Tehreem" data-post="17" data-topic="83525" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tehreem/48/41058_2.png" class="avatar"> Tehreem:</div>
<blockquote>
<p>Please also suggest any tutorial which will helps in calculating the parameters of each layer in CNN.</p>
</blockquote>
</aside>
<p>I would recommend to have a look at <a href="https://cs231n.github.io/convolutional-networks/#conv">CS231n</a>, which explains CNNs quite well. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Tehreem: <p>I just copy pasted your snippet and received this error.</p>

Tehreem: <p><strong>Hi. Thanks for your help. At last i am able to solve the size issues with my code. But now i have new issue, this code is just giving me the training loss, not the test results. I am running this code. This give me the result for training loss. But when i print test loss and accuracy it returns ‘Nan’ as output. Please guide me why this code is not giving me the test results. Please guide if this code is giving me the training results, then it should give the test results also.</strong></p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class  CharCNN(nn.Module):
    def __init__(self):
        super(CharCNN, self).__init__()
        
        self.conv1 = nn.Sequential(
            nn.Conv1d(num_channels, depth_1, kernel_size=kernel_size_1, stride=stride_size),
            nn.ReLU(),
            #nn.MaxPool1d(kernel_size=2, stride=stride_size),
            #nn.Dropout(0.1),
        )
        
        
        self.conv2 = nn.Sequential(
            nn.Conv1d(depth_1, depth_2, kernel_size=kernel_size_2, stride=stride_size),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=stride_size),
            #nn.Dropout(0.25)
        )            
               
        self.fc1 = nn.Sequential(
            nn.Linear((depth_2)*6, num_hidden),
            nn.ReLU(),
            nn.Dropout(0.5)
        )
        
        self.fc2 = nn.Sequential(
            nn.Linear(num_hidden, num_labels),
            nn.ReLU(),
            #nn.Dropout(0.5)
        )
        
        self.log_softmax = nn.LogSoftmax(1)

    def forward(self, x):
        
        x = self.conv1(x)
        
        x = self.conv2(x)
        
        #y= x.size()
        #print (y)
        
        #x = x.reshape(-1,16384)
        #x = x.view(x.size(0), -1) 
        x = x.view(x.size(0),-1)
        # linear layer
        x = self.fc1(x)
        # output layer
        x = self.fc2(x)
        #out = self.log_softmax(x,dim=1)
        x = self.log_softmax(x)
        return x

    model =  CharCNN()


criterion = nn.CrossEntropyLoss()    
opt = torch.optim.Adam(model.parameters(),lr=learning_rate)

    
for e in range(training_epochs):
    if(train_on_gpu):
        net.cuda()
    train_losses = []    
    
    for batch in iterate_minibatches(train_x, train_y, batch_size):
        x, y = batch
        
        #x= torch.randn(64,52,300)
        
        inputs, targets = torch.from_numpy(x), torch.from_numpy(y)
        if(train_on_gpu):
            inputs, targets = inputs.cuda(), targets.cuda()
            #inputs= inputs.view(batch_size,  128*64)
            #targets = targets.view(batch_size)
        opt.zero_grad()   
        
        
        output = model(inputs)
        
        #print(output.size())
        
        #output=output.squeeze(0)
            
        #loss = criterion(output.float(), targets.long())
        loss = criterion(output, torch.max(targets, 1)[1])
        train_losses.append(loss.item())
        loss.backward()
        opt.step()
        
    print("Epoch: {}/{}...".format(e+1, training_epochs),
        "Train Loss: {:.4f}...".format(np.mean(train_losses)))
    
    
    
test_losses = []              

model = model.eval()
with torch.no_grad():
    accuracy=0
    f1score=0
    for batch in iterate_minibatches(test_x, test_y, batch_size):
        x1, y1 = batch     
        inputs, targets = torch.from_numpy(x1), torch.from_numpy(y1)
    
        if(train_on_gpu):
            inputs, targets = inputs.cuda(), targets.cuda()
            output = model(inputs)
            
            test_loss = criterion(output, torch.max(targets, 1)[1])
            
            
            test_losses.append(test_loss.item())
            

            top_p, top_class = output.topk(1, dim=1)
            equals = top_class == targets.view(*top_class.shape).long()
            accuracy += torch.mean(equals.type(torch.FloatTensor))
            f1score += metrics.f1_score(top_class.cpu(), targets.view(*top_class.shape).long().cpu(), 
            average='weighted')        
                
print("Epoch: {}/{}...".format(e+1, training_epochs),
        "test Loss: {:.4f}...".format(np.mean(test_losses)),
       "test Acc: {:.4f}...".format(accuracy/(len(test_x)//batch_size)),
       "F1-Score: {:.4f}...".format(f1score/(len(test_x)//batch_size)))```</code></pre>