akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Improving precision score by applying weight matrix to cross_entropy loss function?

ratatouille: <p>Hello fellow Pytorchers!</p>
<p>I am working on a CNN to make directional predictions of a market based on a visual representation of the recent market state. The model can make three classifications, down, zero and up. I use the torch.nn.functional.cross_entropy function to determine the loss, which does its job.</p>
<p>However, for practical purposes I want to specifically optimize the precision score of the up and down predictions. To do this, I need to penalize up-predictions that are actually down harder than misclassified zero-predictions for example. I have an evenly distributed set of labels and I know I can apply loss_weight to force more zero-predictions, which slightly improves the precision scores for the up and down predictions.</p>
<p>However, I am looking to apply a weight matrix instead of a vector to more specifically dictate loss, looking to further improve precision. Example of such a matrix would be the following:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/8c36b5d9309ad658f138a107e1abd1e9ddba5826.png" alt="image" data-base62-sha1="k0o3ReQ5CwGd8y9Ipf5ISfmWnj0" width="59" height="61">,<br>
where 0&lt;a&lt;b&lt;1</p>
<p>Is there any built-in functionality for this in Pytorch, or does this require me to do tinkering ‘under the hood’?</p>
<p>Cheers!</p>

ptrblck: <p>I’m not sure how your actual prediction look like, but assuming they would have the same shape as your weight matrix, you could simply multiply both before calling <code>loss.backward()</code>.</p>
<p>Let me know, if that would work!</p>

ratatouille: <p>Thanks for the reply!</p>
<p>I will clarify a little. The <code>output_logits</code> in the code below are  <code>torch.Size([128, 3])</code>, a vector of 3 logit probabilities for every observation. My adjusted loss weights <code>loss_mod</code> is a <code>3*3</code> matrix.</p>
<p>I think I need the loss function to use this matrix along with the input and target. If I correctly understand how optimizing works, the direction in which the parameters are changed depends on the derivative of the loss function, right?</p>
<p>Looking at it more closely, I am wondering how <code>loss.backward()</code> and <code>optimizer.step()</code> are actually connected…</p>
<pre><code class="lang-auto">        loss_function = functional.cross_entropy
        optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,)
#
#
        for images_batch, labels_batch in train_data_loader:
            images_batch = images_batch.to(device)
            labels_batch = labels_batch.to(device)
            optimizer.zero_grad()
            output_logits = model(images_batch)
            loss_mod = np.array([[0, 1/3, 1], [2/3, 0, 2/3], [1, 1/3, 0]])
            loss = loss_function(input=output_logits, target=labels_batch)

            loss.backward()
            optimizer.step()

</code></pre>
<p>Greatly appreciated looking at this!</p>
<p>Cheers!</p>

dyzhou2015: <p>Hi, <a class="mention" href="/u/ratatouille">@ratatouille</a> I meet the same problem as you, have you solved this ?</p>
Copy from GPU to CPU (C++)

adaber: <p>Hi everyone,</p>
<p>I did some testing on copying a tensor from GPU to CPU (.cpu() method) and it seems that Pytorch allocates a new chunk of “CPU” memory (RAM) every time the .cpu() method is executed.</p>
<p>Is there a way to copy a “GPU” tensor to a preallocated buffer in RAM (CPU side) using C++ ? I would like to avoid allocating new memory space for each “GPU to CPU” copy operation.</p>
<p>Thanks!</p>

klory: <p>I don’t think it’s possible with current Python API</p>

adaber: <p>Thanks for replying, Klory <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> My question is if it can be done in C++, though.</p>

tom: <p>You could use <code>cputensor.copy_(gputensor)</code> (both in Python and C++).</p>
<p>Best regards</p>
<p>Thomas</p>

adaber: <p>Excellent. Just what I needed.</p>
<p>Thanks, Thomas <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Delete a dataset sample in the runtime

subhashnerella: <p>I want to delete a sample if does not hold a certain condition. I cannot check for this condition beforehand.  I wrote the sample code on how I wanted it to work.<br>
Will this work?<br>
If not how should i do it.</p>
<pre><code class="lang-auto">class Mydataset(Dataset):

    def __init__(self, data_list):

        self.data = data_list

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
         if some_condition:
               sample = self.data[idx]
        else:
               del self.data[idx]
               self.__getitem__(idx)

        return sample
</code></pre>

ptrblck: <p>I think this won’t work, as you are not changing the length of the dataset and this might use an invalid index.</p>
<p>Would it be possible to iterate the dataset once, store all invalid indices, and remove them before creating your new “clean” dataset?</p>

subhashnerella: <p>I cannot really do that because the behaviour of sample might sometime pass or fail the condition.</p>
<p>How about this code. I will make the shuffle to be false and manually shuffle the dataset beforehand. I think this would make the batch pick samples in a sequence.</p>
<pre><code class="lang-auto">class Mydataset(Dataset):

    def __init__(self, data_list):

        self.data = data_list
        self.len = len(self.data)

    def __len__(self):
        return self.len

    def __getitem__(self, idx):
         if idx&gt; self.len:
               idx = np.random.randint(0,self.len)
         if some_condition:
               sample = self.data[idx]
         else:
               del self.data[idx]
               self.len = len(self.data)
               self.__getitem__(idx)

        return sample
</code></pre>
<p>This would result is some duplicate samples passed to the network. I am okay with that. Do you think this will work?</p>

TinfoilHat0: <p>I’m not sure about deleting the sample from the dataset but did you try checking for the condition as you fetch the batches?</p>
<p>I think it is possible to remove/modify the batches fetched from dataloader in general.</p>