Accuracy is not calculated?

username: Leon_Lopez
cooked: <p>Hello, I want to classificate isolated music notes.<br>
My dataset has the shape: input[252, seq_len], target[88, seq_len]<br>
Dataset is normalized.<br>
How seq_len is different for each note, I had to create my custom collate_fn:</p>
<pre><code class="lang-auto">def music_collate_fn(batch):
    data = np.zeros((252 , 0), dtype=np.float)
    target = np.zeros((88,0), dtype=np.int)
    for item in batch:
        data = np.concatenate((data, item[0]), axis=1)
        target = np.concatenate((target, item[1]), axis = 1)
    
    #to tensor
    data = torch.from_numpy(data).float()
    target = torch.from_numpy(target).float()
    #Transpose
    data = data.t()
    target = target.t()
    #Cuda
    data = data.to(device)
    target = target.to(device) 
    return data, target
</code></pre>
<p>The target is a binary classification. I have 88 musical notes. If a note is the number 0, I have 1 in the row 0 As many times as input data represents the note 0. If the note is the number 77, I have a 1 in the row 77. I think I am in the correct way. For example, target of note 0 before concatenate is like:</p>
<pre><code class="lang-auto">trainSet[0][1]
array([[1, 1, 1, ..., 1, 1, 1],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       ...,
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0]])
</code></pre>
<p>I am trying to calculate loss in my program:<br>
Train code:</p>
<pre><code class="lang-auto">mlp = rn.MLP(input_dim, hidden_dim, output_dim).to(device)
optimizer = torch.optim.SGD(mlp.parameters(), lr = 0.001, momentum=0.5)
def train(model, train_loader, optimizer, criterion):
    model.train()
    temp_loss = []
    i = 0
    for batch_idx, (input,target) in enumerate(train_loader):      
        outputs = model(input) 
        loss = criterion(outputs, target)
        temp_loss.append(loss.item())
        optimizer.zero_grad()           #Gradientes a cero
        loss.backward()                 #Back propagation
        optimizer.step()                #Actualizar parámetros.
        i +=1
        if(i % 5):
            print('----------TRAIN--------------PERDIDA: ', loss)
            print("Loss: {}".format(np.mean(temp_loss)))
    
    return temp_loss
</code></pre>
<p>The test code is:</p>
<pre><code class="lang-auto">def test( model, test_loader, optimizer, criterion):
    model.eval()
    test_loss = 0
    correct = 0
    temp_loss = []
    i=0
    with torch.no_grad():
        for data, target in test_loader:

            outputs = model(data)
            loss = criterion(outputs, target)
            temp_loss.append(loss.item())
            if(i == 3):
                print('\n----------TEST--------------\nPERDIDA: ', loss.item())
                i = 0
            pred = outputs.data.max(1, keepdim=True)[1]
            target = target.data.max(1,keepdim = True)[1]
            correct += pred.eq(target).sum()
            i += 1
    
    print('\nTest set: Avg. loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
    loss, correct, target.shape[0],
    correct / target.shape[0]))
    
    return temp_loss
</code></pre>
<p>I am trying to calculate accuracy, but prediction is always a low value, like 0,0001%. So the model never learn.</p>
<p>Furthermore, I am ussing F.binary_cross_entropy instead nn.BCELoss, because is the only way to have results. If I use nn.CrossEntropyLoss, the problem returned is:</p>
<pre><code class="lang-auto">
  File "&lt;ipython-input-9-5c699a4fd8ff&gt;", line 1, in &lt;module&gt;
    losses += train(mlp, train_loader, optimizer, criterion)

  File "&lt;ipython-input-7-072d93cf4aa0&gt;", line 9, in train
    loss = criterion(outputs, target)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 898, in __init__
    super(CrossEntropyLoss, self).__init__(weight, size_average, reduce, reduction)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 23, in __init__
    super(_WeightedLoss, self).__init__(size_average, reduce, reduction)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 16, in __init__
    self.reduction = _Reduction.legacy_get_string(size_average, reduce)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\_reduction.py", line 42, in legacy_get_string
    if size_average and reduce:

RuntimeError: bool value of Tensor with more than one value is ambiguous
</code></pre>

username: ptrblck
cooked: <p>Are you dealing with a multi-label classification use case, i.e. for each time step more than a single note can be active?<br>
If so, could you print the shapes of the data and target as I think I didn’t understand it properly.<br>
Anyway, here is a small dummy example:</p>
<pre><code class="lang-python">batch_size = 10
nb_features = 256
nb_classes = 88
seq_len = 5

x = torch.randn(batch_size, seq_len, nb_features)
target = torch.randint(0, 2, (batch_size, seq_len, nb_classes)).float()

model = nn.Sequential(
    nn.Linear(nb_features, nb_classes),
    nn.Sigmoid(),
)
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-0)

for epoch in range(1000):
    optimizer.zero_grad()
    output = model(x)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    print('Epoch {}, loss {}'.format(epoch, loss.item()))
</code></pre>

username: Leon_Lopez
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> and thanks so much for answer!</p>
<p>I am not sure if the classification is a multi-label, because I have to recognize notes, but the target is represented with 0 and 1.  I have a Dataset with data of 88 notes. When I create a batch in my custom collate function, I join as many notes as batch size size. If I have a batch of 10, I collect 10 notes, so it have to recognize that 10 notes. In the next batch, it will have to recognize 10 notes more.</p>
<p>I print you the shape of input, target and outputs while debug in the train function in the first and second batch. I should say seq_len is different in each epoch.</p>
<pre><code class="lang-auto">First batch: 
Input.shape -&gt; torch.Size([28471,252])
target.shape -&gt; torch.Size([28471,88])
outputs.shape-&gt; torch.Size([28471,88])

Second batch: 
Input.shape -&gt; torch.Size([28146,252])
target.shape -&gt; torch.Size([28146,88])
outputs.shape-&gt; torch.Size([28146,88])
</code></pre>
<p>I am not sure if I am in the correct way. I am new in Pytorch. Thanks so much.</p>

username: ptrblck
cooked: <p>Thanks for the information.<br>
Let me rephrase it: for each row of the target tensor, is there only one active note or multiple ones?<br>
I.e. are you using only single notes or chords as well?</p>

username: Leon_Lopez
cooked: <p>For each row of the target tensor, there is just an active note.</p>
<p>There is a simple example. If batch is size 2 I have two notes.<br>
If note 0 has a shape like [88,5] I have a target like:</p>
<pre><code class="lang-auto">[[1,1,1,1,1], # row 0
[0,0,0,0,0] # row 1
.
.
.[0,0,0,0,0]] #row 87
</code></pre>
<p>if note 1 has a shape like [88,3] I have a target like:</p>
<pre><code class="lang-auto">[[0,0,0], #row 0
[1,1,1], # row 1
.
.
.
.[0,0,0]] #row 87
</code></pre>
<p>After concatenate arrays:</p>
<pre><code class="lang-auto">[[1,1,1,1,1,0,0,0],
[0,0,0,0,0,1,1,1]
[0,0,0,0,0,0,0,0]
.
.
.
[0,0,0,0,0,0,0]]

</code></pre>
<p>This representation is before transpose.</p>

username: ptrblck
cooked: <p>In that case, you could use <code>nn.CrossEntropyLoss</code> and the target tensor should only contain the class indices.<br>
Given the target shape of <code>[batch_size, nb_classes]</code>, you can just call</p>
<pre><code class="lang-python">target = torch.argmax(target, 1)
</code></pre>
<p>to get the class indices.<br>
Also note that <code>nn.CrossEntropyLoss</code> expects logits as the model output, so remove any non-linearity for the last layer in case you are using some.</p>

username: Leon_Lopez
cooked: <p>I tried that, but it returns me an error.</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss
#criterion = F.binary_cross_entropy
def train( model, train_loader, optimizer, criterion):
    model.train()
for batch_idx, (input,target) in enumerate(train_loader):
        optimizer.zero_grad()           #Gradientes a cero  
        outputs = model(input) 
        target = torch.argmax(target, 1)
        loss = criterion(outputs, target)
        temp_loss.append(loss.item())
 
        loss.backward()                 #Back propagation
        optimizer.step()                #Actualizar parámetros.

</code></pre>
<p>I obtain the next error:</p>
<pre><code class="lang-auto">
  File "&lt;ipython-input-17-1aec7df1fd6b&gt;", line 69, in &lt;module&gt;
    train_losses += train(epoch, mlp, train_loader, optimizer, criterion)

  File "&lt;ipython-input-17-1aec7df1fd6b&gt;", line 15, in train
    loss = criterion(outputs, target)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 898, in __init__
    super(CrossEntropyLoss, self).__init__(weight, size_average, reduce, reduction)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 23, in __init__
    super(_WeightedLoss, self).__init__(size_average, reduce, reduction)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 16, in __init__
    self.reduction = _Reduction.legacy_get_string(size_average, reduce)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\_reduction.py", line 42, in legacy_get_string
    if size_average and reduce:

RuntimeError: bool value of Tensor with more than one value is ambiguous
</code></pre>

username: ptrblck
cooked: <p>You have to create an instance of <code>nn.CrossEntropyLoss</code>, i.e. you have to call the constructor using parentheses:</p>
<pre><code class="lang-python">criterion = nn.CrossEntropyLoss()
</code></pre>

username: Leon_Lopez
cooked: <p>Hello again <a class="mention" href="/u/ptrblck">@ptrblck</a>. I ask here because I don’t know if is a good idea create a new topic. If you think I should create a new topic, I will do it.</p>
<p>Reciently, I had studied my dataset, and I find I didn’t load correctly.</p>
<p>Now, I concatenate all examples correctly, and all is working almost correctly.</p>
<p>I have six dimensional matrixes, two for trains, two for test, and two for validation.</p>
<p>I deleted music_collate_fn because now I can use default function.<br>
I make some changes in my code. Now, dataset class is much more easy:</p>
<pre><code class="lang-auto">class MusicDataSet(Dataset):
    def __init__(self, input_matrix, target_matrix, function=None):
        self.ms = input_matrix
        self.target = target_matrix

    def __len__(self):
        return len(self.ms)

    def __getitem__(self, idx):
        inp = torch.from_numpy(self.ms[idx].astype(np.float))
        target = torch.from_numpy(self.target[idx].astype(np.double))

        return inp, target
</code></pre>
<p>And the model is very simple:</p>
<pre><code class="lang-auto">class MLP(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(MLP, self).__init__()

        self.dense1 = nn.Linear(input_dim, hidden_dim[0]) 
        self.dense2 = nn.Linear(hidden_dim[0], hidden_dim[1])  
        self.dense3 = nn.Linear(hidden_dim[1], hidden_dim[2])      
        self.dense4 = nn.Linear(hidden_dim[2], output_dim)
        self.relu = nn.ReLU()
        self.sigm = nn.Sigmoid()
        self.dropout = nn.Dropout(0.05)
     
    def forward(self, x):
        x = self.dense1(x)
        x = self.relu(x)
        x = self.dense2(x)
        x = self.relu(self.dropout(x))
        x = self.dense3(x)
        x = self.relu(self.dropout(x))
        x = self.dense4(x)
        x = self.sigm(x)
        return x
</code></pre>
<p>Preprocess code:</p>
<pre><code class="lang-auto">X_train, Y_train, X_val, Y_val , X_test, Y_test = sd.loadDatasets()

trainSet = mds.MusicDataSet(X_train, Y_train)
valSet = mds.MusicDataSet(X_val, Y_val)
testSet = mds.MusicDataSet(X_test, Y_test)
train_loader = torch.utils.data.DataLoader(trainSet, batch_size=batch_size_train, shuffle=False)
test_loader = torch.utils.data.DataLoader(testSet, batch_size=batch_size_test, shuffle=False)
val_loader = torch.utils.data.DataLoader(valSet, batch_size=batch_size_val, shuffle=False)

input_dim = 252
hidden_dim = (500, 600, 100)
output_dim = 88
if CUDA:
   mlp = rn.MLP(input_dim, hidden_dim, output_dim).to('cuda') 
   mlp.double();

optimizer = torch.optim.SGD(mlp.parameters(), lr = learning_rate, momentum=momentum)
criterion = nn.BCELoss()
</code></pre>
<p>All matrixes are transposed when are loaded.<br>
The input data shape has 252 elements in each row.<br>
The target data shape has 88 elements in each row.<br>
The num rows in trainSet is 656445 and less in the others Datasets.</p>
<p>I was checking the accuracy in test, but it is not enough, is very bad, less than 50%. I can’t understand why.</p>
<p>This is the code to train, eval and test the model:</p>
<pre><code class="lang-auto">def train(model, train_loader, optimizer, criterion):
    model.train()
    for idx_batch, (input,target) in enumerate(train_loader):
        optimizer.zero_grad()
        input = input.cuda()
        target = target.cuda()
        outputs = model(input) 
        loss = criterion(outputs, target)
        loss.backward()   
        optimizer.step()
        train_loss += loss.item()
    return train_loss

def val(model, val_loader, optimizer, criterion):
    model.eval()
    val_loss = 0
    with torch.no_grad():
        for idx_batch, (input, target) in enumerate(val_loader):
             input = input.cuda()
             target = target.cuda()
            outputs = model(input)
            loss = criterion(outputs, target)
            val_loss += loss.item()
    return val_loss

for epoch in range(1, N_EPOCHS+1):
    print("------------EPOCHS ",epoch,"------------")
    train_loss = 0
    val_loss = 0
    train_loss = train(mlp, train_loader, optimizer, criterion)
    val_loss = val(mlp, val_loader, optimizer, criterion)
    
    #Mean of losses
    train_loss = train_loss/len(train_loader.dataset)
    val_loss = val_loss/len(val_loader.dataset)
    
    #Stats training and validation.
    print('Epoch: {} \tTraining loss: {:.6f} \tValidation loss:{:.6f}'.format(epoch, train_loss, val_loss))
    
#Training
mlp.eval()
for data, target in test_loader:

    data = data.to('cuda')
    target = target.to('cuda')
    outputs = mlp(data)
    loss = criterion(outputs, target)
    test_loss += loss.item() #*data.size(0)????
    pred = torch.max(outputs, 1)[1]
    t = torch.max(target, 1)[1]
    correct += pred.eq(t).sum()

</code></pre>
<p>Thanks so much.</p>

username: ptrblck
cooked: <p>The code looks generally alright.<br>
You might need to play around with some hyperparameters (e.g. learning rate, hidden size, etc.).<br>
It might also be a good idea to scale down your use case a bit and try to overfit your model on a small data sample (e.g. use just 10 samples from the train dataset).<br>
If your model is not able to overfit these samples perfectly, there might be e.g. some code bugs we missed.</p>

username: Leon_Lopez
cooked: <p>Yes, it really works. Thanks so much, I did a lot of changes to my code and I had a lot of problems. Thank you.</p>

username: god_sp33d
cooked: <p>Hey,</p>
<p>I tried helping you in other thread which you started for the same problem ?All the suggestions seemed to be not much helpful from what you replied back. Could you please <strong>summarize</strong> a few which helped ?</p>
<p>What did you end up using CrossEntropyLoss() ? Did you tweak hyper parameters more ?</p>

username: Leon_Lopez
cooked: <p>I had to use torch.argmax before calculate the loss to have a correct loss and could advance in the training.</p>
<p>I am using now CrossEntropyLoss() because I was thinking about my problem and I have a multiclass problem. I have 88 possible class represented with 0 and 1, so I use torch.argmax to convert indexes in classes.</p>
<p>I think I am in the correct way.</p>
<p>Thanks.</p>