Stack expects each tensor to be equal size, but got [0] at entry 0 and [3] at entry 1

krishna511: <p>Hi all, I have pre-processed my dataset to obtained three sets as train test and validation. The shapes and type of each of them are as follows.</p>
<pre><code class="lang-auto">Shape of X_train:  (3441, 7, 1, 128, 128)
type(X_train): numpy.ndarray
Shape of X_val:  (143, 7, 1, 128, 128)
type(X_val): numpy.ndarray
Shape of X_test:  (150, 7, 1, 128, 128)
type(X_test): numpy.ndarray
</code></pre>
<p>The <code>class Dataset</code> is created as given:</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import Dataset

class Dataset(Dataset):
    def __init__(this, X=None, y=None,  mode="train"):
        this.mode = mode
        this.X = X
        if mode == "train":

            this.y = y
            

    def __len__(this):
        return this.X.shape[0]

    def __getitem__(this, idx):
        if this.mode == "train":
            return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])
        else:
            return torch.FloatTensor(this.X[idx])
</code></pre>
<p>Then I have use <code>PytorchDataloader</code> as</p>
<pre><code class="lang-auto">train_set = Dataset(X=X_train, y=Y_train,mode="train")
tr_loader = DL(train_set, batch_size=32, shuffle=True)

test_set  = Dataset(X=X_test, y=Y_test,mode="train")
ts_loader = DL(test_set, batch_size=32, shuffle=False)
</code></pre>
<p>Everything is working fine uptill now, (I think so :)). Then for training I used the code below.</p>
<pre><code class="lang-auto">verbose=True
Losses = []
Accuracies = []
epochs=10
DLS = {"train": tr_loader, "valid": ts_loader}

start_time = time.time()
for e in range(epochs):
    epochLoss = {"train": 0, "valid": 0}
    epochAccs = {"train": 0, "valid": 0}

    for phase in ["train", "valid"]:
        if phase == "train":
            model.train()
        else:
            model.eval()

        lossPerPass = []
        accuracy = []

        for X, y in DLS[phase]:
            X, y = X.to(device), y.to(device).view(-1)

            optimizer.zero_grad()
            alpha=1.0
            beta=1.0
            with torch.set_grad_enabled(phase == "train"):
                
                pred_emo= model(X)
                emotion_loss = criterion(pred_emo,y)
               
                total_loss = alpha*emotion_loss
                if phase == "train":
                    total_loss.backward()
                    optimizer.step()
            lossPerPass.append(total_loss.item())
            accuracy.append(accuracy_score(torch.argmax(torch.exp(pred_emo.detach().cpu()), dim=1), y.cpu()))
            torch.save(model.state_dict(),"E:/Python_On_All_Dataset/IEMO/long_codes/Kosta.jo/model_checkpoint_spec/Epoch_{}.pt".format(e+1))  
        epochLoss[phase] = np.mean(np.array(lossPerPass))
        epochAccs[phase] = np.mean(np.array(accuracy))
      
    Losses.append(epochLoss)
    Accuracies.append(epochAccs)



    if verbose:
        print("Epoch : {} | Train Loss : {:.5f} | Valid Loss : {:.5f} \
| Train Accuracy : {:.5f} | Valid Accuracy : {:.5f}".format(e + 1, epochLoss["train"], epochLoss["valid"],
                                                        epochAccs["train"], epochAccs["valid"]))
</code></pre>
<p>Then I am getting this error traceback:</p>
<p>File “”, line 21, in <br>
for X, y in DLS[phase]:</p>
<p>File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 517, in <strong>next</strong><br>
data = self._next_data()</p>
<p>File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 557, in _next_data<br>
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration</p>
<p>File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py”, line 47, in fetch<br>
return self.collate_fn(data)</p>
<p>File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 83, in default_collate<br>
return [default_collate(samples) for samples in transposed]</p>
<p>File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 83, in <br>
return [default_collate(samples) for samples in transposed]</p>
<p>File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 55, in default_collate<br>
return torch.stack(batch, 0, out=out)</p>
<p><strong>RuntimeError: stack expects each tensor to be equal size, but got [0] at entry 0 and [3] at entry 1</strong></p>
<p>Can someone comment on this, Is there any problem with class<code> Dataset</code> or in <code>training</code>. The tittle of the problem can be changed on suggestions.<br>
please guide<br>
Regards</p>

ptrblck: <p>I cannot reproduce the issue using:</p>
<pre><code class="lang-python">X_train = torch.randn(3441, 7, 1, 128, 128)
y_train = torch.randint(0, 10, (3441,))

dataset = Dataset(X_train, y_train)
loader = DataLoader(dataset, batch_size=16)

for data, target in loader:
    print(data.shape)
    print(target.shape)
</code></pre>
<p>Could you compare my code snippet to yours and check, what the difference might be?</p>

krishna511: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> sir my Y_train shape is <code>(3441,)</code>, is this the problem? If yes what to do:<br>
also when I run your code on my class <code>dataset</code> it says:</p>
<p><strong>slice() cannot be applied to a 0-dim tensor.</strong> in line (in class Dataset)<br>
return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])</p>
<p>I have changed the numpy array into tensor using this:</p>
<pre><code class="lang-auto">X_test=torch.from_numpy(X_test)
</code></pre>
<p>then I got an error in line:<br>
return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])</p>
<p>TypeError: expected Float (got Double)<br>
Sir plz tell me if I need to share the model also. I think the whole problem is in loading the dataset .</p>
<p>Regards</p>

ptrblck: <aside class="quote no-group" data-username="krishna511" data-post="3" data-topic="123375">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishna511/48/30486_2.png" class="avatar"> krishna511:</div>
<blockquote>
<p>sir my Y_train shape is <code>(3441,)</code>, is this the problem?</p>
</blockquote>
</aside>
<p>This shouldn’t be a problem, since my code snippet uses the same shape for <code>y_train</code> as well as your <code>Dataset</code> definition.</p>
<aside class="quote no-group" data-username="krishna511" data-post="3" data-topic="123375">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishna511/48/30486_2.png" class="avatar"> krishna511:</div>
<blockquote>
<p>then I got an error in line:<br>
return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])</p>
</blockquote>
</aside>
<p><code>numpy</code> uses <code>float64</code> by default, so you might want to transform the tensor to <code>float32</code> via:</p>
<pre data-code-wrap="python"><code class="lang-python">X_test = torch.from_numpy(X_test).float()
</code></pre>
<aside class="quote no-group" data-username="krishna511" data-post="3" data-topic="123375">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishna511/48/30486_2.png" class="avatar"> krishna511:</div>
<blockquote>
<p>Sir plz tell me if I need to share the model also. I think the whole problem is in loading the dataset .</p>
</blockquote>
</aside>
<p>A minimal and executable code snippet would be helpful to further debug the issue.</p>

krishna511: <aside class="quote no-group" data-username="krishna511" data-post="3" data-topic="123375">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishna511/48/30486_2.png" class="avatar"> krishna511:</div>
<blockquote>
<p><strong>slice() cannot be applied to a 0-dim tensor.</strong> in line (in class Dataset)</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> sir what about this? I have changed the data to float() and targets as long(). Now it showing the above error at<br>
<code>return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])</code></p>

ejguan: <p>I would try to use the original numpy array and converted into torch tensor here:</p>
<pre><code class="lang-auto">class Dataset(Dataset):
    def __getitem__(this, idx):
        if this.mode == "train":
            return torch.from_numpy(this.X[idx]).float(), torch.from_numpy(this.y[idx]).long()
        else:
            return torch.from_numpy(this.X[idx]).float()
</code></pre>
<p>Your original back tracing is interesting</p>
<aside class="quote no-group quote-modified" data-username="krishna511" data-post="5" data-topic="123375" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishna511/48/30486_2.png" class="avatar"> krishna511:</div>
<blockquote>
<p>File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py”, line 47, in fetch<br>
return self.collate_fn(data)</p>
<p>File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 83, in default_collate<br>
return [default_collate(samples) for samples in transposed]</p>
<p>File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 83, in<br>
return [default_collate(samples) for samples in transposed]</p>
</blockquote>
</aside>
<p>It falls into the logic here <a href="https://github.com/pytorch/pytorch/blob/94cc681fc2e5218c10493938a8ca01272c3c6fc0/torch/utils/data/_utils/collate.py#L77-L84" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/94cc681fc2e5218c10493938a8ca01272c3c6fc0/torch/utils/data/_utils/collate.py#L77-L84</a>, which is not expected since you are returning tuple. Can you verify your output by printing out the data like the following code.</p>
<pre><code class="lang-auto">class Dataset(Dataset):
    def __getitem__(this, idx):
        if this.mode == "train":
            res =  torch.from_numpy(this.X[idx]).float(), torch.from_numpy(this.y[idx]).long()
            print(res)
            return res
        else:
            return torch.from_numpy(this.X[idx]).float()
</code></pre>

krishna511: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/ejguan">@ejguan</a> Sir<br>
I am trying to explain the complete code here. The processed dataset is</p>
<pre><code class="lang-auto">Shape of X_train:  (1147, 7, 1, 128, 128)
Shape of X_val:  (143, 7, 1, 128, 128)
Shape of X_test:  (150, 7, 1, 128, 128)
</code></pre>
<pre><code class="lang-auto">b,t,c,h,w = X_train.shape
X_train = np.reshape(X_train, newshape=(b,-1))
X_train = scaler.fit_transform(X_train)
X_train = np.reshape(X_train, newshape=(b,t,c,h,w))

b,t,c,h,w = X_test.shape
X_test = np.reshape(X_test, newshape=(b,-1))
X_test = scaler.transform(X_test)
X_test = np.reshape(X_test, newshape=(b,t,c,h,w))

b,t,c,h,w = X_val.shape
X_val = np.reshape(X_val, newshape=(b,-1))
X_val = scaler.transform(X_val)
X_val = np.reshape(X_val, newshape=(b,t,c,h,w))
</code></pre>
<p>The model is</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
# BATCH FIRST TimeDistributed layer
class TimeDistributed(nn.Module):
    def __init__(self, module):
        super(TimeDistributed, self).__init__()
        self.module = module

    def forward(self, x):

        if len(x.size()) &lt;= 2:
            return self.module(x)
        # squash samples and timesteps into a single axis
        elif len(x.size()) == 3: # (samples, timesteps, inp1)
            x_reshape = x.contiguous().view(-1, x.size(2))  # (samples * timesteps, inp1)
        elif len(x.size()) == 4: # (samples,timesteps,inp1,inp2)
            x_reshape = x.contiguous().view(-1, x.size(2), x.size(3)) # (samples*timesteps,inp1,inp2)
        else: # (samples,timesteps,inp1,inp2,inp3)
            x_reshape = x.contiguous().view(-1, x.size(2), x.size(3),x.size(4)) # (samples*timesteps,inp1,inp2,inp3)
            
        y = self.module(x_reshape)
        
        # we have to reshape Y
        if len(x.size()) == 3:
            y = y.contiguous().view(x.size(0), -1, y.size(1))  # (samples, timesteps, out1)
        elif len(x.size()) == 4:
            y = y.contiguous().view(x.size(0), -1, y.size(1), y.size(2)) # (samples, timesteps, out1,out2)
        else:
            y = y.contiguous().view(x.size(0), -1, y.size(1), y.size(2),y.size(3)) # (samples, timesteps, out1,out2, out3)
        return y
    
    
class HybridModel(nn.Module):
    def __init__(self,num_emotions):
        super().__init__()
        # conv block
        self.conv2Dblock = nn.Sequential(
            # 1. conv block
            TimeDistributed(nn.Conv2d(in_channels=1,
                                   out_channels=16,
                                   kernel_size=3,
                                   stride=1,
                                   padding=1
                                  )),
            TimeDistributed(nn.BatchNorm2d(16)),
            TimeDistributed(nn.ReLU()),
            TimeDistributed(nn.MaxPool2d(kernel_size=2, stride=2)),
            TimeDistributed(nn.Dropout(p=0.4)),
            # 2. conv block
            TimeDistributed(nn.Conv2d(in_channels=16,
                                   out_channels=32,
                                   kernel_size=3,
                                   stride=1,
                                   padding=1
                                  )),
            TimeDistributed(nn.BatchNorm2d(32)),
            TimeDistributed(nn.ReLU()),
            TimeDistributed(nn.MaxPool2d(kernel_size=4, stride=4)),
            TimeDistributed(nn.Dropout(p=0.4)),
            # 3. conv block
            TimeDistributed(nn.Conv2d(in_channels=32,
                                   out_channels=64,
                                   kernel_size=3,
                                   stride=1,
                                   padding=1
                                  )),
            TimeDistributed(nn.BatchNorm2d(64)),
            TimeDistributed(nn.ReLU()),
            TimeDistributed(nn.MaxPool2d(kernel_size=4, stride=4)),
            TimeDistributed(nn.Dropout(p=0.4)),
            # 4. conv block
            TimeDistributed(nn.Conv2d(in_channels=64,
                                   out_channels=128,
                                   kernel_size=3,
                                   stride=1,
                                   padding=1
                                  )),
            TimeDistributed(nn.BatchNorm2d(128)),
            TimeDistributed(nn.ReLU()),
            TimeDistributed(nn.MaxPool2d(kernel_size=4, stride=4)),
            TimeDistributed(nn.Dropout(p=0.4))
        )
        # LSTM block
        hidden_size = 64
        self.lstm = nn.LSTM(input_size=128,hidden_size=hidden_size,bidirectional=False, batch_first=True) 
        self.dropout_lstm = nn.Dropout(p=0.3)
        # Linear softmax layer
        self.out_linear = nn.Linear(hidden_size,num_emotions)
    def forward(self,x):
        conv_embedding = self.conv2Dblock(x)
        conv_embedding = torch.flatten(conv_embedding, start_dim=2) # do not flatten batch dimension and time
        lstm_embedding, (h,c) = self.lstm(conv_embedding)
        lstm_embedding = self.dropout_lstm(lstm_embedding)
        # lstm_embedding (batch, time, hidden_size)
        lstm_output = lstm_embedding[:,-1,:] 
        output_logits = self.out_linear(lstm_output)
        output_softmax = nn.functional.softmax(output_logits,dim=1)
        return output_logits, output_softmax    
</code></pre>
<p>The train test definition are</p>
<pre><code class="lang-auto">def loss_fnc(predictions, targets):
    return nn.CrossEntropyLoss()(input=predictions,target=targets)    

def make_train_step(model, loss_fnc, optimizer):
    def train_step(X,Y):
        # set model to train mode
        model.train()
        # forward pass
        output_logits, output_softmax = model(X)
        predictions = torch.argmax(output_softmax,dim=1)
        accuracy = torch.sum(Y==predictions)/float(len(Y))
        # compute loss
        loss = loss_fnc(output_logits, Y)
        # compute gradients
        loss.backward()
        # update parameters and zero gradients
        optimizer.step()
        optimizer.zero_grad()
        return loss.item(), accuracy*100
    return train_step


def make_validate_fnc(model,loss_fnc):
    def validate(X,Y):
        with torch.no_grad():
            model.eval()
            output_logits, output_softmax = model(X)
            predictions = torch.argmax(output_softmax,dim=1)
            accuracy = torch.sum(Y==predictions)/float(len(Y))
            loss = loss_fnc(output_logits,Y)
        return loss.item(), accuracy*100, predictions
    return validate
</code></pre>
<p>and final training previously was</p>
<pre><code class="lang-auto">EPOCHS=700
DATASET_SIZE = X_train.shape[0]
BATCH_SIZE = 32
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print('Selected device is {}'.format(device))
model = HybridModel(num_emotions=len(EMOTIONS)).to(device)
print('Number of trainable params: ',sum(p.numel() for p in model.parameters()) )
OPTIMIZER = torch.optim.SGD(model.parameters(),lr=0.01, weight_decay=1e-3, momentum=0.8)

train_step = make_train_step(model, loss_fnc, optimizer=OPTIMIZER)
validate = make_validate_fnc(model,loss_fnc)
losses=[]
val_losses = []
for epoch in range(EPOCHS):
    # schuffle data
    ind = np.random.permutation(DATASET_SIZE)
    X_train = X_train[ind,:,:,:,:]
    Y_train = Y_train[ind]
    epoch_acc = 0
    epoch_loss = 0
    iters = int(DATASET_SIZE / BATCH_SIZE)
    for i in range(iters):
        batch_start = i * BATCH_SIZE
        batch_end = min(batch_start + BATCH_SIZE, DATASET_SIZE)
        actual_batch_size = batch_end-batch_start
        X = X_train[batch_start:batch_end,:,:,:,:]
        Y = Y_train[batch_start:batch_end]
        X_tensor = torch.tensor(X,device=device).float()
        Y_tensor = torch.tensor(Y, dtype=torch.long,device=device)
        loss, acc = train_step(X_tensor,Y_tensor)
        epoch_acc += acc*actual_batch_size/DATASET_SIZE
        epoch_loss += loss*actual_batch_size/DATASET_SIZE
        print(f"\r Epoch {epoch}: iteration {i}/{iters}",end='')
    X_val_tensor = torch.tensor(X_val,device=device).float()
    Y_val_tensor = torch.tensor(Y_val,dtype=torch.long,device=device)
    val_loss, val_acc, _ = validate(X_val_tensor,Y_val_tensor)
    losses.append(epoch_loss)
    val_losses.append(val_loss)
    print('')
    print(f"Epoch {epoch} --&gt; loss:{epoch_loss:.4f}, acc:{epoch_acc:.2f}%, val_loss:{val_loss:.4f}, val_acc:{val_acc:.2f}%")

</code></pre>
<p>Now this last step I need to change as the dataset is huge for my GPU so I m suggested to use pytorch dataloaders. I hope I am clear now.<br>
Now along with previous dataset class whatelse I need to change please suggest.</p>
<p>Regards</p>

ejguan: <p>My suggestion when you start to use DataLoader. You should use it without worker (by setting <code>num_workers=0</code>) to validate your dataset. Then, you can start to use multiple workers to accelerate your data loading speed.</p>

krishna511: <p><a class="mention" href="/u/ejguan">@ejguan</a> sir where is the option to set num_workers??</p>
<p>I have tried without data augmentation too, but still getting same error</p>
<p><strong>IndexError:: slice() cannot be applied to a 0-dim tensor.</strong></p>

ejguan: <p>It’s an argument of DataLoader. (<a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 2.1 documentation</a>)</p>
<aside class="quote no-group" data-username="krishna511" data-post="9" data-topic="123375">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishna511/48/30486_2.png" class="avatar"> krishna511:</div>
<blockquote>
<p>IndexError:: slice() cannot be applied to a 0-dim tensor.</p>
</blockquote>
</aside>
<p>Can you please copy the whole traceback of this Error?</p>

krishna511: <p><a class="mention" href="/u/ejguan">@ejguan</a> Sir , I have made the change by putting num_workers=0 also, that made no difference here. Sir Now I think this whole code is executable, Plz check once.<br>
This is the complete traceback</p>
<pre><code class="lang-auto">  File "E:\Python_On_All_Dataset\IEMO\long_codes\Kosta.jo\stacked _cnn.py", line 471, in &lt;module&gt;
    for X, y in DLS[phase]:

  File "C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 517, in __next__
    data = self._next_data()

  File "C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration

  File "C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]

  File "C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]

  File "E:\Python_On_All_Dataset\IEMO\long_codes\Kosta.jo\Dataset.py", line 26, in __getitem__
    rec= torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])

IndexError: slice() cannot be applied to a 0-dim tensor.
</code></pre>

ejguan: <p>The problem comes from here:</p>
<aside class="quote no-group" data-username="krishna511" data-post="1" data-topic="123375">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishna511/48/30486_2.png" class="avatar"> krishna511:</div>
<blockquote>
<pre><code class="lang-auto">   def __getitem__(this, idx):
        if this.mode == "train":
            return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])
        else:
            return torch.FloatTensor(this.X[idx])
</code></pre>
</blockquote>
</aside>
<p>Please try to replace <code>torch.FloatTensor(this.X[idx])</code> by <code>torch.tensor(this.X[idx], dtype=torch.float)</code> and replace <code>torch.LongTensor(this.y[idx])</code> by <code>torch.tensor(this.y[idx], dtype=torch.long)</code></p>

krishna511: <p>Thanks <a class="mention" href="/u/ejguan">@ejguan</a> I have tried this.  Now how to replace this training part with dataloader? Now I am using the code below for model training.</p>
<pre><code class="lang-auto">verbose=True
Losses = []
Accuracies = []
epochs=10
DLS = {"train": tr_loader, "valid": ts_loader}

start_time = time.time()
for e in range(epochs):
    epochLoss = {"train": 0, "valid": 0}
    epochAccs = {"train": 0, "valid": 0}

    for phase in ["train", "valid"]:
        if phase == "train":
            model.train()
        else:
            model.eval()

        lossPerPass = []
        accuracy = []

        for X, y in DLS[phase]:
            X, y = X.to(device), y.to(device).view(-1)

            optimizer.zero_grad()
            alpha=1.0
            beta=1.0
            with torch.set_grad_enabled(phase == "train"):
                
                pred_emo= model(X)
                emotion_loss = criterion(pred_emo,y)
                
                total_loss = alpha*emotion_loss
                if phase == "train":
                    total_loss.backward()
                    optimizer.step()
            lossPerPass.append(total_loss.item())
            accuracy.append(accuracy_score(torch.argmax(torch.exp(pred_emo.detach().cpu()), dim=1), y.cpu()))
            torch.save(model.state_dict(),"E:/Python_On_All_Dataset/IEMO/long_codes/Kosta.jo/model_checkpoint_spec/Epoch_{}.pt".format(e+1))  
        epochLoss[phase] = np.mean(np.array(lossPerPass))
        epochAccs[phase] = np.mean(np.array(accuracy))
        # Epoch Checkpoint // All or Best
    Losses.append(epochLoss)
    Accuracies.append(epochAccs)

    # if scheduler:  # or use, if scheduler_1 or scheduler_2: // Use correct call method
    #     scheduler.step(epochLoss["valid"])
    #     #scheduler.step()

    if verbose:
        print("Epoch : {} | Train Loss : {:.5f} | Valid Loss : {:.5f} \
| Train Accuracy : {:.5f} | Valid Accuracy : {:.5f}".format(e + 1, epochLoss["train"], epochLoss["valid"],
                                                        epochAccs["train"], epochAccs["valid"]))

print("Time Taken [{} Epochs] : {:.2f} minutes".format(epochs, (time() - start_time) / 60))

print("Training Complete")

        
        
</code></pre>
<p>But in line</p>
<pre><code class="lang-auto"> emotion_loss = criterion(pred_emo,y)
</code></pre>
<p>it throws attribute <strong>error : ‘tuple’ object has no attribute 'log_softmax’</strong><br>
I know the model returning two values, as in class  <code>HybridModel()</code></p>
<p><strong>return output_logits, output_softmax</strong><br>
How can I resolve this. Also am I using correct code for training sir?<br>
As previously without dataloaders. There three functions are used for training validation etc.</p>

nikhil6041: <aside class="quote no-group" data-username="krishna511" data-post="13" data-topic="123375">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishna511/48/30486_2.png" class="avatar"> krishna511:</div>
<blockquote>
<p>‘tuple’ object has no attribute 'log_softmax’</p>
</blockquote>
</aside>
<p>This is occuring because your model’s forward gives u two outputs output_logits, output_softmax  , the loss criterion which is crossnetropyloss here expects one of those as a tensor but currently its a tuple just change the line</p>
<blockquote>
<p>pred_emo= model(X)</p>
</blockquote>
<p>to</p>
<blockquote>
<p>logits,softmax = model(X)</p>
</blockquote>
<p>and pass accordingly to ur criterion and u r good to go</p>

krishna511: <p>That’s done , Thank you<br>
But My main question is I want to replicate the model training as before data loaders, How it is to be done, that needs a high level understanding of pytorch training probably.</p>

nikhil6041: <p>Can you tell what do you mean by “replicate the model training as before data loaders”?</p>

krishna511: <p>Hi <a class="mention" href="/u/nikhil6041">@nikhil6041</a> if you read the querry since starting you may get it. But lemme brief you again.<br>
That code was without dalaloaders but not working for me due to cuda memory issues, the whole code is here before loaders above. Then I used pytorch dataloaders, now I want the same training as before. If you go through previous code here you gt the idea for sure.</p>
<p>Thanks</p>

nikhil6041: <p><a class="mention" href="/u/krishna511">@krishna511</a>  I dont see much change in the difference you are training your model by the earlier approach for the expected gpu error there r several ways u can avoid it for example by calling</p>
<blockquote>
<p>torch.cuda.empty_cache()</p>
</blockquote>
<p>after each epoch will help you probably.</p>

krishna511: <p>No <a class="mention" href="/u/nikhil6041">@nikhil6041</a> I set batch size of 1,its not executing even for one epoch then how to empty cashe</p>

nikhil6041: <p><a class="mention" href="/u/krishna511">@krishna511</a>  okay can u paste the entire code snippet which is throwing errors here once as a reply?</p>