akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How torch::from_blob return a tensor from a batch images?

username: yueyihua
cooked: <p>How torch::from_blob return a tensor from a batch images? I just put image tensor to a vector one by one, but call forward, an exception is thrown. My code is as follows:<br>
std::vector<a>torch::jit::IValue</a> input_tensor;<br>
for (auto file : files)<br>
{<br>
Mat img = imread(file);<br>
if (img.empty())<br>
{<br>
GNLOG(GN_WARNING) &lt;&lt; “Can’t read image data from file”;<br>
continue;<br>
}</p>
<pre><code>cv::cvtColor(img, img, COLOR_BGR2RGB);
cv::Size scale(kIMAGE_SIZE, kIMAGE_SIZE);
cv::resize(img, img, scale);

// convert [unsigned int] to [float]
img.convertTo(img, CV_32FC3, 1.0f / 255.0f);

auto img_tensor = torch::from_blob(
		img.data, {1, kIMAGE_SIZE, kIMAGE_SIZE, kCHANNELS});
img_tensor = img_tensor.permute({0, 3, 1, 2});
img_tensor[0][0] = img_tensor[0][0].sub_(0.4914).div_(0.2023);
img_tensor[0][1] = img_tensor[0][1].sub_(0.4822).div_(0.1994);
img_tensor[0][2] = img_tensor[0][2].sub_(0.4465).div_(0.2010);

img_tensor = img_tensor.to(device_type_);
input_tensor.push_back(img_tensor);
</code></pre>
<p>}</p>
<p>torch::Tensor out_tensor = model_-&gt;forward(input_tensor).toTensor();</p>

username: Jesse_Stone
cooked: <p>example code here:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="http://note4lin.top/post/pytorch%E9%83%A8%E7%BD%B2/">
  <header class="source">
      <img src="http://note4lin.top/favicon.ico" class="site-icon" width="128" height="128">

      <a href="http://note4lin.top/post/pytorch%E9%83%A8%E7%BD%B2/" target="_blank" rel="noopener nofollow ugc" title="12:37PM - 25 December 2019">note4lin.top – 25 Dec 19</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="http://note4lin.top/post/pytorch%E9%83%A8%E7%BD%B2/" target="_blank" rel="noopener nofollow ugc">libtorch c++ 使用例子</a></h3>

  <p>pytorch可使用flask作为服务器部署，但是由于Python的可移植性和速度比不上c++, pytorch还提供将模型转化到c++端运行</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Focal loss gradient vanishing

username: dendi
cooked: <p>I’ve been working on an unbalanced binary classification problem, where true to false ratio is 9:1, and my input is 20 dim tabular data. In order to handle this imbalanced dataset, I decided to use  <a href="https://arxiv.org/abs/1708.02002" rel="noopener nofollow ugc">Focal loss</a>. My implementation of this metric is a PyTorch adaptation of the <a href="https://github.com/tensorflow/addons/blob/ad132da23a8162eb97c435676dd7426e622a0074/tensorflow_addons/losses/focal_loss.py#L104" rel="noopener nofollow ugc">Tensorflow</a> one.</p>
<p>I made a couple of tests and the outcome of my focal loss implementation is the same as the one produced by the TensorFlow one.</p>
<pre><code class="lang-auto">class FocalLoss(nn.Module):
    """
    Weighs the contribution of each sample to the loss based in the classification error.
    :gamma: Focusing parameter. gamma=0 is equivalent to BCE_loss
    """
    def __init__(self, gamma, eps=1e-6):
        super(FocalLoss, self).__init__()
        self.gamma = gamma

    def forward(self, y_pred, y_true):
        y_true = y_true.float()
        pred_prob = torch.sigmoid(y_pred)
        ce = nn.BCELoss(reduce=False)(pred_prob,y_true)


        p_t = (y_true*pred_prob)+((1-y_true)*(1-pred_prob))
        modulator = 1.0
        if self.gamma:
            modulator = torch.pow((1.0-p_t),torch.tensor(self.gamma).to(device) )
        
        return torch.mean(modulator*ce)
</code></pre>
<p>My model is a simple n hidden layers, in this test n=4, fully connected NN with Relu activation. This architecture works reasonably fine when cross-entropy is used as a loss function.<br>
The figure below presents a gradient flow after one epoch. The value of validation loss approached 7e-12 and the validation accuracy is 50%.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6b9f5958edb6d19f7bb86d7d781ee1b7d402b0d.png" alt="image" data-base62-sha1="uDyARTTBOYDjD7QwCcQSAQloMWN" width="432" height="288"></p>
<p>I am using adam optimizer with lr=1e-4.<br>
What do you think about my implementation of the Focal loss? Is it legit?<br>
I am also using a class balanced sampler, therefore each batch contains an equal number of true and false examples.</p>

username: Mona_Jalal
cooked: <p>So, is your implementation correct? How did you conclude it?</p>
DDP with ComplexFloat

username: churchillmic
cooked: <p>I have this same issue with trying to run DDP with a model with ComplexFloat parameters: <a href="https://github.com/pytorch/pytorch/issues/71613" class="inline-onebox" rel="noopener nofollow ugc">NCCL Backend does not support ComplexFloat data type · Issue #71613 · pytorch/pytorch · GitHub</a></p>
<p>In passing in the GH issue, a workaround was suggested to view the complex as real before passing to NCCL. But I don’t quite follow. I have tried adding register_backward_hook, and converting the gradients there using torch.view_as_real, but that doesn’t seem to do anything. I also tried adding register_comm_hook’s, which seemed promising, but it appears they can only be registered on the DDP object, which in this issue doesn’t exist because it throws the error in the DDP call.</p>
<p>It appears this will be fixed, but can someone point to how to modify the example in the issue to successfully use DDP with ComplexFloat models?</p>

username: Yanli_Zhao
cooked: <p>I guess we need to fix this inside DDP packages, will work on this fix</p>
Model.weight ordering question

username: jxmmy7777
cooked: <p>Hello, I would like to ask about how model.weight is ordered. The reason why I’m asking is because when I print out the weight and gradient, it seems like one neuron does not get updated. Is the last row corresponds to the last neuron? Or is the order reversed, which the last row corresponds to the first neuron of the decoder.<br>
Note that my decoder layer is in_dim * 30</p>
<pre><code class="lang-auto">print("grad",self.model.decoder.weight.grad)
print("grad",self.model.decoder.weight)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7aec58fc70990d15be7e165e0341b1a77784f943.png" data-download-href="https://discuss.pytorch.org/uploads/default/7aec58fc70990d15be7e165e0341b1a77784f943" title="Screenshot 2022-02-27 002747"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7aec58fc70990d15be7e165e0341b1a77784f943.png" alt="Screenshot 2022-02-27 002747" data-base62-sha1="hxqCFSbn529PlVoe35qUXXCaudR" width="690" height="375" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-02-27 002747</span><span class="informations">1208×658 26.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>I’m not sure how you are defining “first”, but here is an example showing how a linear layer’s <code>weight</code> is used to create the output:</p>
<pre><code class="lang-python">lin = nn.Linear(2, 3, bias=False)
with torch.no_grad():
    weight = torch.arange(2*3).float().view(3, 2)
    lin.weight.copy_(weight)

print(lin.weight)
# &gt; Parameter containing:
#   tensor([[0., 1.],
#           [2., 3.],
#           [4., 5.]], requires_grad=True)

x = torch.arange(1, 5).float().view(2, 2)
print(x)
# &gt; tensor([[1., 2.],
#           [3., 4.]], requires_grad=True)    

out = lin(x)
print(out)
# &gt; tensor([[ 2.,  8., 14.],
#           [ 4., 18., 32.]], grad_fn=&lt;MmBackward0&gt;)

# corresponds to
# [[(1*0 + 2*1)=2, (1*2 + 2*3)=8, (1*4 + 2*5)=14],
#  [(3*0 + 4*1)=4, (3*2 + 4*3)=18, (3*4 + 4*5)=32]
</code></pre>

username: jxmmy7777
cooked: <p>Thank u for ur reply!</p>
Accuracy and f1 score being returned are identical

username: HenryPowell
cooked: <p>I’m trying to evaluate some models based on their accuracy (% of correctly guessed input examples) and their f1 scores. I’m fairly sure that I’m computing the two in the correct way but I was surprised to see that they are return the exact same score. The code I’m using is below:</p>
<pre><code class="lang-auto">epoch_loss = 0
epoch_accuracy = 0
epoch_f_one = 0

    for i_batch, batch in enumerate(data_loader):
        visual_samples = batch['visual_sample'].float().to(device)
        audio_samples = batch['audio_sample'].float().to(device)
        labels = batch['label'].long().to(device) - 1

        optimizer.zero_grad()
        outputs = net(visual_samples, audio_samples).float()
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        epoch_loss += loss.item()

        batch_accuracy = np.count_nonzero(torch.argmax(outputs, axis=1) == labels) / visual_samples.size(0)
        epoch_accuracy += batch_accuracy

        y_true = labels
        y_pred = torch.argmax(outputs, axis=1)

        f1 = f1_score(y_true=y_true, y_pred=y_pred, average='micro')
        epoch_f_one += f1

    epoch_loss = epoch_loss / (i_batch + 1)
    epoch_accuracy = epoch_accuracy / (i_batch + 1)
    epoch_f_one = epoch_f_one / (i_batch + 1)

    max_batch = i_batch

    return epoch_loss, epoch_accuracy, epoch_f_one, max_batch
</code></pre>
<p>Am I calculating something wrong here or is it just that computing the per epoch accuracy turns out to the be the same as the F1 since they are both means of correct guesses over the epochs?</p>
<p>Many thanks.</p>
Conv2d Vs ConvTranspose2d usage

username: Adam007
cooked: <p>Hi I have a question about ConvTranspose2d, in which situation would you use ConvTranspose2d over conv2d ? Or in which situation I SHOULD use ConvTranspose2d?</p>

username: KFrank
cooked: <p>Hi Adam!</p>
<aside class="quote no-group" data-username="Adam007" data-post="1" data-topic="145166" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/a698b9/48.png" class="avatar"> Adam007:</div>
<blockquote>
<p>in which situation would you use ConvTranspose2d over conv2d ?</p>
</blockquote>
</aside>
<p>In general, if you are building a “conventional” convolutional network<br>
(whatever that means), you should use <code>Conv2d</code>.</p>
<p>The noteworthy use case for <code>ConvTranspose2d</code> that I am familiar with<br>
(I’m sure that there are others.) is in the U-Net architecture, where it is<br>
used as an “upsampling” or “dilation” operation.</p>
<p>I believe that the "up-conv 2x2"operations in the <a href="https://arxiv.org/abs/1505.04597" rel="noopener nofollow ugc">original U-Net paper</a><br>
can be exactly implemented as:</p>
<pre data-code-wrap="python"><code class="lang-python">ConvTranspose2d (in_channels, out_channels, kernel_size = 2, stride = 2)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Upsampling EfficientNet (in Encoder-decoder ⌛ arch)

username: tornikeo
cooked: <p>I’m trying to re-implement <a href="https://github.com/GlebSBrykin/SANET/tree/master/style" rel="noopener nofollow ugc">SANET</a>, an encoder-decoder style network for arbitrary image stylization.</p>
<p>The problem is, these guys use 2 freakin’ VGG19s for inference. I’m trying to replace these with a much more efficient nets (pun intended) to run on mobile devices. I replaced the encoder VGG, no problems there, but what about the decoder?</p>
<p>Is there an equivalent, high-performance, upsampling CNN architecture that can perform just as well as inverted VGG19, but be a lot faster and smaller? (Sadly the U-nets are out question, for this particular project)</p>
<p>I just need some pointers, maybe related papers or something that shows me something equivalent to an inverted efficientNet architecture. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> Also the framework of the paper doesn’t matter.</p>
<p>Thank you so much. <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"></p>