akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pos_weight and reduce='none' in BCEWithLogitsLoss

Jarartur: <p>Hi,<br>
I’ve read <a href="https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/10">here</a> that when doing BCE loss without reduction I need to take care about normalization on my own. Is it also true with <code>pos_weights</code>?<br>
I’ve read about it a bit (i.e. <a href="https://discuss.pytorch.org/t/weight-vs-pos-weight-in-nn-bcewithlogitsloss/114859/4">here</a>) but still I am not sure how I would do it.</p>
How to complete the input data?

qyc1720083040: <p>HI,<br>
I’m producing my own dataset with FEM.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/189bdfdcfb609ef640a9f09697f5a46e1d19bc3c.png" data-download-href="https://discuss.pytorch.org/uploads/default/189bdfdcfb609ef640a9f09697f5a46e1d19bc3c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/189bdfdcfb609ef640a9f09697f5a46e1d19bc3c_2_690x172.png" alt="image" data-base62-sha1="3vHpXQ9YNsCEMuFu3ReTxRjYwq0" width="690" height="172" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/189bdfdcfb609ef640a9f09697f5a46e1d19bc3c_2_690x172.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/189bdfdcfb609ef640a9f09697f5a46e1d19bc3c_2_1035x258.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/189bdfdcfb609ef640a9f09697f5a46e1d19bc3c_2_1380x344.png 2x" data-dominant-color="F8F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1652×414 10.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The figure shows the combination of three features (PCB, chip, SJV), the blue part is the part I can generate, and the red part I can’t generate by software due to technical reasons.</p>
<p>I tried to use the ML model with only the blue part to predict the value of the red part, but the error is very large.</p>
<p>How can I complete the values of the red part?</p>
<p>Thanks!</p>
How to skip quantization of certain layers in static PTQ?

yannbane: <ul>
<li>PyTorch version: 1.8.1</li>
</ul>
<p>I’ve applied static PTQ by changing the source code of <a href="https://github.com/open-mmlab/mmclassification/blob/master/mmcls/models/backbones/swin_transformer.py" rel="noopener nofollow ugc">Swin Transformer from mmclassification</a> (inserting de/quant stubs). I’ve had to change the source code of a lot of other files too, because the model is not exactly standard or simple.</p>
<p>I used this function to perform the quantization of a loaded model:</p>
<pre><code class="lang-python">def static_quantize(m, data_loader):
    backend = 'qnnpack'
    torch.backends.quantized.engine = backend
    m.eval()

    m.qconfig = torch.quantization.get_default_qconfig(backend)
    torch.quantization.prepare(m, inplace=True)

    with torch.no_grad():
        for i, data in enumerate(data_loader):
            result = m(return_loss=False, **data)
            if i &gt; 100:
                break
        
    torch.quantization.convert(m, inplace=True)

    return m # I realize this is unnecessary
</code></pre>
<p>However I’ve noticed a significant drop in accuracy (around 30%). To combat this I would like to selectively quantize layers, i.e. skip the quantization process for certain layers that are problematic. I’ve noticed that <code>prepare</code> and <code>convert</code> effectively quantize everything they can in the model recursively.</p>
<p>For me the simplest way of doing this would be to comment out the de/quant ops in the model source code. Of course this doesn’t actually work because these stubs aren’t used to detect which layers should be quantized.</p>
<p>So how can I tell <code>prepare</code> which layers to skip? Furthermore how can I tell it to skip one <code>Linear</code> layer, but quantize some other <code>Linear</code> layer (if type-level granularity is not enough)?</p>

yannbane: <p>It turns out that the accuracy drop was not due to excessive quantization, but due to reusing the same QuantStubs in different place of the same class. These objects seem to have a dual purpose of collecting quantization statistics and flagging quantization boundaries (which does make sense). They’re stateful.</p>
How to merge multiple autograd::Function or nest them?

anubane: <p>Let us say we write something like:</p>
<pre><code class="lang-auto">struct my_func : torch::autograd::Function&lt;my_func&gt; {}
</code></pre>
<p>I can write my own <strong>forward()</strong> and <strong>backward()</strong> and in my main module I can use <strong>my_func::apply()</strong> to use this function.</p>
<p>My question is, how can create multiple <code>autograd::Function</code> and combine them (pass gradients between and invoke them from my main <code>nn::Module</code> inheriting <strong>struct</strong>)?</p>

ptrblck: <p>You should be able to directly use the function in your custom modules and Autograd will be able to compute the gradients as your custom <code>backward</code> function is also defined.<br>
Let me know, if I misunderstand your use case.</p>

anubane: <p>Well, in my use case, one <code>autograd::Function</code> has to run a <code>parallel_for</code> loop and call a second <code>autograd::Function</code> in each iteration. How to invoke the second Function? Still using <code>apply()</code>? How will the <strong>forward()</strong> and <strong>backward()</strong> handled in this case? Is it possible to provide an example?</p>
<p>I need to understand the syntax, since, the documentation is extremely scant for the cpp frontend.</p>
Back-propagate through torch.stft and torch.istft

pepper8362: <p>Hi,</p>
<p>Do torch.stft and torch.istft support autograd? Is there a way to back-propagate through them?</p>
<p>Thanks</p>

ptrblck: <p>Yes, Autograd is able to backpropagate through these methods as seen in this small example:</p>
<pre><code class="lang-python">input = torch.randn(16, requires_grad=True)
out = torch.stft(input, 10) 
out.mean().backward()
print(input.grad)
# tensor([-3.5224e-02,  1.5741e-01, -6.3721e-02,  1.5741e-01, -2.8497e-02,
#          1.4815e-01, -1.8626e-09,  9.2593e-02, -1.8626e-09,  9.2593e-02,
#          2.8497e-02,  1.0185e-01,  3.5224e-02,  5.5556e-02,  6.3721e-02,
#          2.7778e-02])
</code></pre>

pepper8362: <p>Hi,</p>
<p>Thanks for your reply. I tried your example but got</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>My pytorch version is 1.10.2+cu113. Does it mean I need to update to a newer version?</p>
<p>Thanks.</p>

ptrblck: <p>No, I don’t think you need to update as my code snippet also works in <code>1.10.2</code>:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2+cu113'
&gt;&gt;&gt; input = torch.randn(16, requires_grad=True)
&gt;&gt;&gt; out = torch.stft(input, 10) 
&gt;&gt;&gt; out.mean().backward()
&gt;&gt;&gt; print(input.grad)
tensor([-3.5224e-02,  1.5741e-01, -6.3721e-02,  1.5741e-01, -2.8497e-02,
         1.4815e-01, -1.8626e-09,  9.2593e-02, -1.8626e-09,  9.2593e-02,
         2.8497e-02,  1.0185e-01,  3.5224e-02,  5.5556e-02,  6.3721e-02,
         2.7778e-02])
</code></pre>
<p>so I guess you might not have used my code snippet directly or you might have disabled the gradient calculation with a context manager or globally.</p>

pepper8362: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="161694">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">input = torch.randn(16, requires_grad=True)
out = torch.stft(input, 10) 
out.mean().backward()
print(input.grad)
</code></pre>
</blockquote>
</aside>
<p>Yes it works. I mistakenly disabled the gradient calculation. Thank you very much!</p>
Will dataset be shuffled for each epoch?

Ardeal: <p>Hi,</p>
<p>I define the dataloader as the following code.<br>
will the dataset be shuffled for each epoch?</p>
<pre><code class="lang-python">    dataloader = torch.utils.data.DataLoader(dataset,
                        batch_size=batch_size,
                        num_workers=nw,
                        sampler=sampler,
                        shuffle=True,
                        pin_memory=False, )
</code></pre>

srishti-git1110: <p>Hi,</p>
<p>You are required to specify either of <code>sampler</code> or <code>shuffle</code> as arguments to the dataloader.<br>
You cannot specify both.</p>
<p>In case, you want the data to be shuffled at every epoch and get sampled according to a randomsampler, specify <code>shuffle=True</code> and remove the sampler argument.</p>
<p>If you specify a <code>sampler</code>, the sampling scheme implied by your custom sampler shall be used.</p>

Ardeal: <p>Hi <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a><br>
Thank you!</p>
<p>I should set sampler=None, or just don’t set anything?</p>
<pre><code class="lang-python">    dataloader = torch.utils.data.DataLoader(dataset,
                        batch_size=batch_size,
                        num_workers=nw,
                        sampler=None,
                        shuffle=True,
                        pin_memory=False, )
</code></pre>

srishti-git1110: <p>Do not specify it. It’ll work fine then.</p>
<pre><code class="lang-auto">dataloader = torch.utils.data.DataLoader(dataset,
                        batch_size=batch_size,
                        num_workers=nw,
                        shuffle=True,
                        pin_memory=False)
</code></pre>

Ardeal: <p><a class="mention" href="/u/srishti-git1110">@srishti-git1110</a><br>
Got it. Thank you!</p>
SpectralOps Usage

alvgaona: <p>Hi there! I’m looking for some help related to <code>SpectralOps.cpp</code>.</p>
<p>I’m essentially trying to create a new CPU kernel—in <code>UnaryKernels.cpp</code>—and in this kernel I want to use the inverse FFT defined in the project, but I’m not being able to use in another <code>.cpp</code> file as there is no header file for these functions.</p>
<p>Let me share an example of a dummy CPU kernel:</p>
<pre><code class="lang-cpp">static void function_test(TensorIteratorBase&amp; iter, int64_t a, double b) {
  AT_DISPATCH_FLOATING_TYPES_AND(kBFloat16, iter.dtype(), "function_test_cpu", [&amp;](){
    const int64_t c = a + b; // Just an example
    cpu_kernel(iter, [=](scalar_t a){
      // Lines could be added here
      return c * a;
    });
    
    // TODO: Use at::fft_ifft from SpectralOps.cpp
  });
}
</code></pre>
<p>Does anybody know how to include these functions? It seems they aren’t used in any part of the codebase, other than creating Python bindings.</p>
<p>Thanks!</p>

alvgaona: <p>I kept digging into the codebase and kernels are generally used to apply operations on Tensors in a point-wise fashion. So, I decided to use <code>at::native::fft_ifft</code> after calling the <code>function_test_stub</code> inside a <code>.cpp</code> file that defines the high-level functions.</p>
<p>For example the following dummy code—which you can assume it’s in a file named <code>Dummy.cpp</code>:</p>
<pre><code class="lang-cpp">Tensor high_level_function() { 
  // options
  auto input = at::empty(100, options);
  auto iter = TensorIterator::unary_op(input, options);
  auto output = function_test_stub(input);
  output = at::native::fft_ifft(output, -1, -1, "backward");
  return output;
}
</code></pre>
<p>But in this dummy file, I had to import the FFT functions, and I decided to create a <code>SpectralOps.h</code> file, which does not exist at the time of this writing.</p>
<p>This is the example of <code>SpectralOps.h</code>:</p>
<pre><code class="lang-cpp">#pragma once

#include &lt;string&gt;
#include &lt;stdexcept&gt;
#include &lt;sstream&gt;

namespace at { namespace native {

Tensor fft_ifft(const Tensor&amp; self, c10::optional&lt;int64_t&gt; n, int64_t dim,
                c10::optional&lt;c10::string_view&gt; norm);

}} // at::native
</code></pre>
<p>If anybody knows another way to do this, that may be the standard of the project or whatever other option, please let me know!</p>
Replace Net with VGG16 without any error

bryan123: <p>class Net(nn.Module):<br>
def <strong>init</strong>(self):</p>
<pre><code class="lang-auto">   super(Net, self).__init__()
   self.conv1 = nn.Conv2d(1, 32, 3, 1)
   self.conv2 = nn.Conv2d(32, 64, 3, 1)
   self.dropout1 = nn.Dropout2d(0.25)
   self.dropout2 = nn.Dropout2d(0.5)
   self.fc1 = nn.Linear(9216, 128)
   self.fc2 = nn.Linear(128, 10)

def get_features(self, x):
    x = self.conv1(x)
    x = F.relu(x)
    x = self.conv2(x)
    x = F.max_pool2d(x, 2)
    x = self.dropout1(x)
    x = torch.flatten(x, 1)
    x = self.fc1(x)
    return x
</code></pre>
<p>My goal is to replace this Net model with a VGG16 pre-trained model without any errors. Nevertheless, I received the following error: ValueError: only one element tensor can be converted to Python scalars. Is there a need to add additional layers like fully connected layers to feature_extractor? (nn.Sequential(*list(model.children())[:-1]))</p>

ptrblck: <p>It’s unclear where the error is raised and how this model should be replaced with a VGG16 model, so could you explain your use case a bit more and post a code snippet which raises the current error?</p>
How does Pytorch initalises the weights and the biases by default?

Jeet: <p>I would like to know that which distribution does PyTorch follows now for the default initalisation of the weights and the biases? For example when I write something like -</p>
<p>self.neural=nn.Sequential(nn.Linear(in_features,4))</p>
<p>I need the mathematical formulation of the initalisation.</p>

ptrblck: <p>The parameters are initialized in the <code>reset_parameters</code> method of the corresponding layer. For <code>nn.Linear</code> you can find the method <a href="https://github.com/pytorch/pytorch/blob/1f4f05e59c4cd72dfff9755629f7cc23f8df7abe/torch/nn/modules/linear.py#L103-L111">here</a>:</p>
<pre><code class="lang-python">    def reset_parameters(self) -&gt; None:
        # Setting a=sqrt(5) in kaiming_uniform is the same as initializing with
        # uniform(-1/sqrt(in_features), 1/sqrt(in_features)). For details, see
        # https://github.com/pytorch/pytorch/issues/57109
        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in) if fan_in &gt; 0 else 0
            init.uniform_(self.bias, -bound, bound)
</code></pre>
Rotation angles for Anchors

gonzrubio: <p>I am looking to modify my RetinaNet implementation to support arbitrary orientation object detection using satellite images. What are some common rotation angles to use for generating the target anchors?</p>
<p>Some papers I have looked at only cover the range -[pi/2, 0), I would have thought to cover the range of -[2*pi, 0) or at least -[pi, 0) to avoid duplicate anchors.</p>
<p>Any pointers would be greatly appreciated, Thank you!</p>
Squeeze / unsqueeze leads to non-deterministic model test

lstrgar: <p>I’ve got a model I’m testing with PyTorch Lightning using different batch sizes and I’m observing different results. I found that making the following modification fixed the issue. Does anyone know why? To be frank I’m not even sure how the original formulation came about – it certainly seems wasteful; however, at first glance it doesn’t appear incorrect.</p>
<p>First the original:</p>
<pre><code class="lang-auto">        x = data
        y = y[:, -1:].squeeze(-1).float() 
        w = w[:, -1:].squeeze(-1)
        y_hat = self(x).squeeze(-1)
        if len(y.shape) == 1:
            y = y.unsqueeze(0)
        if len(w.shape) == 1:
            w = w.unsqueeze(0)
        if len(y_hat.shape) == 1:
            y_hat = y_hat.unsqueeze(0)
        return y.cpu(), y_hat.detach().cpu(), w.cpu(),
</code></pre>
<p>And the fix:</p>
<pre><code class="lang-auto">        x = data
        y = y[:, -1:].float() 
        w = w[:, -1:]
        y_hat = self(x)
        return y.cpu(), y_hat.detach().cpu(), w.cpu(),
</code></pre>
Faster quantile computation with libtorch

iiSeymour: <p>The implementation of <code>torch::quantile</code> does a full sort which makes it significantly slower than say the implementation in numpy which uses partition.</p>
<p>I don’t see a partition equivalent in libtorch (C++)… am I missing something?</p>
<pre><code class="lang-python">import time
import torch
import numpy as np

t = torch.rand(1_000_000)
a = t.numpy()

start = time.time()
t.median()
end = time.time()
print("torch median", end - start)

start = time.time()
t.quantile(torch.tensor([0.25, 0.75]))
end = time.time()
print("torch quant ", end - start)

start = time.time()
np.median(a)
end = time.time()
print("numpy median", end - start)

start = time.time()
np.quantile(a, [0.25, 0.75])
end = time.time()
print("numpy quant ", end - start)
</code></pre>
<p>Results</p>
<pre><code class="lang-auto">torch median 0.013309478759765625
torch quant  0.10049819946289062
numpy median 0.012769222259521484
numpy quant  0.014006376266479492
</code></pre>

iiSeymour: <p>Using <code>std::nth_element</code> I can get a ~3x boost over <code>torch::quantile</code> but still behind the performance of numpy.</p>
<pre><code class="lang-cpp">torch::Tensor quantile(const torch::Tensor t, const torch::Tensor q) {
    assert(t.dtype().name() == "float");
    assert(q.dtype().name() == "float");
    if (!torch::equal(q, std::get&lt;0&gt;(q.sort()))) {
        throw std::runtime_error("quantiles q are not sorted");
    }

    auto tmp = t.clone();
    auto res = torch::empty_like(q);

    auto start = tmp.data_ptr&lt;float&gt;();
    auto end = tmp.data_ptr&lt;float&gt;() + tmp.size(0);

    for (int i = 0; i &lt; q.size(0); i++) {
        auto m = tmp.data_ptr&lt;float&gt;() + static_cast&lt;size_t&gt;((tmp.size(0) - 1) * q[i].item&lt;float&gt;());
        std::nth_element(start, m, end);
        res[i] = *m;
        start = m;
    }

    return res;
}

</code></pre>
Quantize Model is bigger than the non quantized mdoel

Rami_Ismael: <p>I went through PyTorch Documentation for Quantize aware training. I prepare and then convert the model to quint 8 models. I save the model to use state_dict(). However,  getting the file size using os.path.gets(). The model size is bigger than a non quantized model.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1TX1JYdjTHDg36VXvnrIqntkyaKrz8Lzv?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20220914-060049-RC00_474248156" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1TX1JYdjTHDg36VXvnrIqntkyaKrz8Lzv?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1TX1JYdjTHDg36VXvnrIqntkyaKrz8Lzv?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Zafar: <p>Can you try running a bigger model? Currently you have  a weight with a single element. Even if you store the weight in int8, you still need 32 bits for the scale and 32 bits for the zero point. Try making a bigger model with multiple conv layers of more realistic sizes.</p>
LSTM with pad_packed_sequence

Nick95: <p>Hi,<br>
I’m using PyTorch to create an LSTM autoencoder that receives a 1D input time series and outputs the reconstruction of the timeserie. The model takes as input sequences of variable length considering one timestep at time.<br>
This is the model:</p>
<pre><code class="lang-auto">class packetAE(nn.Module):

    def __init__(self, lstm1_h: int, ):
        super().__init__()

        self.encoder = nn.LSTM(1, lstm1_h, 1, batch_first=True)
        self.decoder = nn.LSTM(lstm1_h, 1, 1, batch_first=True)

    def forward(self, input):
        encoding = self.encoder(input)
        return self.decoder(encoding)
</code></pre>
<p>This is how I create the dataloader and fed it to the network:</p>
<pre><code class="lang-auto">def collate_padding(batch):
    (xx, yy) = zip(*batch)
    
    # retrieve original lenght of batch sequences
    x_lens = [len(x) for x in xx]
    y_lens = [len(y) for y in yy]

    # add padding
    xx_pad = torch.nn.utils.rnn.pad_sequence(xx, batch_first=True, padding_value=0)
    yy_pad = torch.nn.utils.rnn.pad_sequence(yy, batch_first=True, padding_value=0)

    return xx_pad, yy_pad, x_lens, y_lens

train_loader = torch.utils.data.DataLoader(dataset, batch_size=16, num_workers=16, pin_memory=True, collate_fn=collate_padding)

model = packetAE(128)

for batch_index, (X, y, x_len, y_len) in enumerate(train_dataloader):
    X_packed = torch.nn.utils.rnn.pack_padded_sequence(X, x_len, batch_first=True, enforce_sorted=False)
    pred_packed = model(X_packed)
</code></pre>
<p>However, when running the training code, I got the following error when passing the input to the encoder stage of the model for training:</p>
<p>RuntimeError: input must have 2 dimensions, got 1</p>
<p>Anyone has idea of the reason?</p>
<p>The dataset contains items like:</p>
<pre><code class="lang-auto">tensor([184, 172, 111,  54,  10, 139,   0, 193, 177,  20, 235,  49,   8,   0,
         69,   0,   0,  40,  27,  52,   0,   0,  55,   6, 154, 114,   8, 254,
        250, 126, 192, 168,  10,   5,   0,  80, 192,  36, 223, 215, 173, 139,
        148, 184,  56, 155,  80,  17,   1,  73, 197,  52,   0,   0,   0,   0,
          0,   0,   0,   0], dtype=torch.uint8)
</code></pre>
Getting correct prediction for only one class

Mrinmay_sen: <p>Hi,<br>
I am trying to build a classification model of MNIST data by using newton method of optimization.<br>
After training my model. I am getting right prediction only for class 0. In test data there is 980 nos of O class samples. My model is giving test accuracy 9.8 % only. I checked pred=model(inputs). Found It is only correctly predicting for class O samples in test data.</p>
<p>My approach:</p>
<ol>
<li>
<p>Resnet18 with random weights is used</p>
</li>
<li>
<p>gradients are calculated with the help of<br>
‘’'env_grads = torch.autograd.grad(loss, cnn.parameters(), retain_graph=True, create_graph=True)“”"</p>
</li>
<li>
<p>Hessian has been calculated with the help of "h_col=torch.autograd.grad(env_grads, cnn.parameters(), retain_graph=True, create_graph=False)</p>
</li>
<li>
<p>Training process has been initialized with following script<br>
for batch, (X, y) in enumerate(train_loader):<br>
with torch.set_grad_enabled(True):<br>
cnn.zero_grad()<br>
cnn.train()<br>
wt1= torch.cat([gi.data.view(-1) for gi in cnn.parameters()]).view(-1, 1)<br>
X = X.to(device)<br>
y = y.to(device)<br>
pred=cnn(X)<br>
loss=loss_fn(pred, y)</p>
</li>
</ol>
<p>Please help me to find out the possible reasons behind this issue.</p>
<h3>
<a name="i-run-the-code-with-sgd-in-the-same-setting-it-is-performing-well-1" class="anchor" href="#i-run-the-code-with-sgd-in-the-same-setting-it-is-performing-well-1"></a>I run the code with SGD in the same setting. It is performing well.</h3>
<h3>
<a name="i-checked-the-data-distribution-there-is-no-imbalance-of-data-2" class="anchor" href="#i-checked-the-data-distribution-there-is-no-imbalance-of-data-2"></a>I checked the data distribution. there is no imbalance of data</h3>

KFrank: <p>Hi Mrinmay!</p>
<aside class="quote no-group quote-modified" data-username="Mrinmay_sen" data-post="1" data-topic="161702" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrinmay_sen/48/53252_2.png" class="avatar"> Mrinmay_sen:</div>
<blockquote>
<p>using newton method of optimization.<br>
…</p>
<ol>
<li>
<p>Resnet18 with random weights is used<br>
…</p>
</li>
<li>
<p>Hessian has been calculated</p>
</li>
</ol>
</blockquote>
</aside>
<p>Something is fishy here.  Resnet18 has about 11 million individual<br>
parameters.  The full Hessian will therefore require something like<br>
500 <em>terabytes</em> of storage – not realistic.</p>
<p>It’s not clear what you are actually doing.</p>
<p>As an aside, instead of focusing on your predictions and accuracy,<br>
when you are using a novel optimization method, you should start<br>
by performing a single optimization step and check whether your<br>
objective function actually goes down.  If your optimization algorithm<br>
isn’t reducing your objective function, something isn’t working correctly,<br>
and you should fix that first before looking at higher-level performance<br>
metrics.</p>
<p>Best.</p>
<p>K. Frank</p>

Mrinmay_sen: <p>Hi Frank,<br>
I am not using full hessian. I approximated it.</p>
Whats wrong in my code?

nsh531: <pre><code class="lang-auto">
import torch.optim as optim
import torch.utils.data as D
train=D.Dataset(f("train/scz"))
train2=D.Dataset(f("train/cont"))
train3=D.ConcatDataset([train,train2])
</code></pre>
<p>tnx<br>
nate</p>

srishti-git1110: <p>Hi,<br>
Please specify certain things like the errors you are facing, what’s f (Is it some function that you’ve defined or did you mean to use f strings) and so on.</p>

nsh531: <p>f is function that reads folder of nii.gz files and returns list of covariance between 64 variables for any of files (n<em>64</em>64)<br>
p.s. this function using nilearn</p>
<p>‫בתאריך יום ב׳, 19 בספט׳ 2022 ב-17:07 מאת ‪Srishti Gureja via PyTorch Forums‬‏ &lt;‪<a href="mailto:noreply@discuss.pytorch.org">noreply@discuss.pytorch.org</a>‬‏&gt;:‬</p>

srishti-git1110: <p>Ok, thanks for answering. But, further you need to specify the errors you are facing.<br>
Also, how are you using <code>torch.utils.data.Dataset</code>?</p>
<p>Generally, a custom Dataset class is implemented as a subclass of <code>torch.utils.data.Dataset</code> and its instances are used to create dataloader compatible objects.</p>
How to stop server socket?

Ziyu_Huang: <p>I meet this:</p>
<pre><code class="lang-auto">[W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:29500 (errno: 98 - Address already in use).
[W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:29500 (errno: 98 - Address already in use).
[E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address
</code></pre>
<p>I tried this:</p>
<pre><code class="lang-auto">ps -fA | grep python
And then kill -9 the pid
</code></pre>
<p>But next time I still meet the same error!<br>
Could anyone kindly suggest me something? Thank you!!!</p>

pbelevich: <p>I think it’s unrelated to pytorch itself and there are a lot of options that people suggest on the SO: <a href="https://stackoverflow.com/questions/15198834/bind-failed-address-already-in-use" class="inline-onebox" rel="noopener nofollow ugc">c - Bind failed: Address already in use - Stack Overflow</a></p>
Tensor multiplication 1D to 3D

planet: <p>I was trying to multiply two tensors <code>A - shape [torch.Size([1100, 4, 3])]</code> with <code>B - shape [torch.Size([1100])]</code>.<br>
I would like to do this:</p>
<pre><code class="lang-auto">A[0]
--&gt; 
tensor([[ 2.0025, -1.8551, -0.5907],
        [ 2.0805,  1.8512, -0.5358],
        [-1.9438, -2.0552, -0.5612],
        [-2.0561,  2.0529, -0.7012]], dtype=torch.float64)

B[0]
--&gt; 
tensor(1.4853, dtype=torch.float64)
</code></pre>
<pre><code class="lang-auto">B[0] * A[0]
--&gt;
tensor([[ 2.9742, -2.7553, -0.8773],
        [ 3.0902,  2.7496, -0.7958],
        [-2.8871, -3.0525, -0.8335],
        [-3.0538,  3.0491, -1.0415]], dtype=torch.float64)
</code></pre>
<p>Works if I choose some index, like above. But will not work if I use whole tensors. <code>B * A</code> ,</p>
<pre><code class="lang-auto">Error:
RuntimeError: The size of tensor a (1100) must match the size of tensor b (3) at non-singleton dimension 2
</code></pre>
<p>Not sure where I am missing?</p>

Lezcano: <p>This may help: <a href="https://pytorch.org/docs/stable/notes/broadcasting.html" class="inline-onebox" rel="noopener nofollow ugc">Broadcasting semantics — PyTorch 1.12 documentation</a><br>
That being said, it’d be best not to cross-post these things on the slack channel right after posting them on the forums <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

planet: <p>Oops figured it out. converted shape of B to [1100,1,1]. Worked!!</p>
Discussion about modern training strategies

Zador: <p>Previously I have put a lot of effort into training networks appropriately. However, talking to colleagues, a lot of the things I did may be redundant due to novel optimizers and the theory of deep learning, who are training networks for beating benchmarks in academia. I am hoping for a brief discussion. Here are the things I used to do which might not be necessary, which were inspired by traditional ML classes not DeepL specific classes:</p>
<ol>
<li>
<p>using train-val-test split instead of just train-test: During training I used an additional val split, which I used to check if the training is converging or overfitting. I heard from colleagues that they simply use the training set running loss to check for convergence (and don’t need to check for overfitting)</p>
</li>
<li>
<p>Using learning rate schedulers. My colleagues do not anymore because optimizers such as Adam have their own. I know this to be true but I still did it because it seemed to be a general procedure.</p>
</li>
<li>
<p>Changing training at validation plateau:<br>
3.1. I used to use ReduceLROnPlateau from pytorch. when validation loss starts to increase, the learning rate is reduced<br>
3.2. I used to save checkpoints until reaching plateau → reset to this checkpoint when val loss increases → decrease learning rate and continue</p>
</li>
</ol>
<p>Notice for 1 and 3 I made these decisions based on a patience period, for example, declare overfit/convergence if after 10 epochs the validation loss hasn’t decreased.</p>
<p>What do you think? What is redundant? is anything potentially harmful?</p>
Slicing tensors with data_ptr and accessor

anubane: <p>The libtorch system provides two apis to access a Tensor via reference pointers, viz., <code>data_ptr</code> and <code>accessor</code>. The problem is, to access single element in the tensor, I am able to understand the syntax, but, if I want to perform slicing, the PyTorch cpp forntend documentation <em>does not mention anything</em>.</p>
<p>What is the <strong>syntax</strong> to perform slicing on the Tensor via the above the pointers?</p>
<p>The problem with using the <code>.index()</code> and <code>.slice()</code> functions directly on the tensor is that they <strong>create copies</strong>, thus consuming extra memory.</p>
<p>Any help is welcome.</p>
Getting TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.Image.Image'>

wi-sh: <p>My transforms fn:</p>
<pre><code class="lang-auto">def data_transforms(phase):
    if phase == TRAIN:
        transform = transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.RandomGrayscale(p=0.1),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
        ])
        
    if phase == VAL:
        transform = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
        ])
    
    if phase == TEST:
        transform = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
        ])        




</code></pre>
<p>Dataset and Dataloaders fn:</p>
<pre><code class="lang-auto">image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms(x)) 
                  for x in [TRAIN, VAL, TEST]}

dataloaders = {TRAIN: torch.utils.data.DataLoader(image_datasets[TRAIN], batch_size = 3, shuffle=True), 
               VAL: torch.utils.data.DataLoader(image_datasets[VAL], batch_size = 1, shuffle=False), 
               TEST: torch.utils.data.DataLoader(image_datasets[TEST], batch_size = 1, shuffle=False)}

</code></pre>
<p>but when I am trying to visualize it’s throwing a TypeError:</p>
<pre><code class="lang-auto">def imshow(inp, title=None):
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    inp = std * inp + mean
    inp = np.clip(inp, 0, 1)
    plt.imshow(inp)
    if title is not None:
        plt.title(title)
    plt.pause(0.001)  


inputs, classes = next(iter(dataloaders[TRAIN])) # Error in this line
out = torchvision.utils.make_grid(inputs)
imshow(out, title=[class_names[x] for x in classes])

</code></pre>
<p>Error</p>
<pre><code class="lang-bash">TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class 'PIL.Image.Image'&gt;
</code></pre>

wi-sh: <p>My mistake , I was not returning anything from my <code>data_transform function</code> …</p>
<p>Sorry, for wasting everyone’s time!!! <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=12" title=":pensive:" class="emoji" alt=":pensive:" loading="lazy" width="20" height="20"></p>

srishti-git1110: <p>Hi,<br>
Please try using an additional transformation <code>torchvision.transforms.PILToTensor</code> in <code>transforms.Compose</code>.</p>
<p>Let me know if you still get the error.</p>

wi-sh: <p>Never knew about <code> transforms.PILToTensor(),</code> thanks!!</p>
Is using nightly build with cu116 possible with local cu114 installed?

nestiank: <p>Hello,<br>
I am using CUDA 11.4 for my NVIDIA T4 GPU from AWS.<br>
(That’s what my nvidia-smi says. I have cudatoolkit==11.1.1.)</p>
<p>I need to use nightly build of PyTorch, but there are binaries supporting cu102, cu116, cu117.</p>
<p>I know that using cu113 binaries with cu114 GPU is possible (as back-supported),<br>
but I am not sure about if using cu116 binary with cu114 GPU is possible.</p>
<p>Is that possible?<br>
If not, can I use cu102 binary for my cu114 GPU instead?<br>
If not, is there any way to use nightly build with my cu114 GPU?</p>
<p>This is possibly related to the possibility of using cu116 for a GPU that nvidia-smi says CUDA 11.4.</p>
<p>Thanks.</p>

ptrblck: <p>Yes, you can use the binaries with a newer CUDA 11.6 runtime and an older driver installed on your system.</p>

nestiank: <p>Thanks. You’ve made my day!</p>
Why kornia.augmentation.RandomGaussianBlur() convert the dtype?

joohyunglee: <p>Somehow, <code>kornia.augmentation.RandomGaussianBlur</code> under <code>with torch.cuda.amp.autocast():</code> converts the datatype from float to half. Is this expected? How can I disable the data type conversion? Thank you ahead.</p>

ptrblck: <p>The transformation will eventually call into <a href="https://github.com/kornia/kornia/blob/da2f1325014a158be36d62bdd20293ef97e574f4/kornia/filters/filter.py#L125">this <code>F.conv2d</code></a> operation and is thus enabling the mixed-precision usage. If you want to disallow it, use a nested <code>autocast</code> context and set <code>enabled=False</code> for these operations.</p>

joohyunglee: <h2>
<a name="error-message-1" class="anchor" href="#error-message-1"></a>Error Message:</h2>
<pre><code class="lang-auto">Input type (torch.cuda.HalfTensor) and weight type (torch.FloatTensor) should be the same
</code></pre>
<h2>
<a name="code-training-2" class="anchor" href="#code-training-2"></a>Code (training):</h2>
<pre><code class="lang-auto">model = GERL().cuda()
images = images.cuda()
with torch.cuda.amp.autocast():
    ouput = model(images)
</code></pre>
<h2>
<a name="code-model-3" class="anchor" href="#code-model-3"></a>Code (model)</h2>
<pre><code class="lang-auto">class GERL(nn.Module):
    def __init__():
        self.aug = ImageSequential(ColorJitter(brightness=0.4, contrast=0.4, saturation=0.2, hue=0.1, p=0.8),
                                   RandomGrayscale(p=0.2),
                                   RandomGaussianBlur(kernel_size=(23, 23), sigma=(0.1, 2.0), p=1.0),
        
        self.encoder = resnet50()
        ...

    def forward(x):
        with torch.no_grad():
            x = self.aug(x)
        x = self.encoder(x)
        ...
        return x
</code></pre>
<p>An error occurs at <code> x = self.encoder(x)</code> during model feedforward due to the data type difference between the model param (torch.FloatTensor) and the input type (torch.cuda.HalfTensor). Thank you !!</p>

ptrblck: <p>You might need to transform the inputs to <code>float32</code> if they are already in <code>float16</code> as described in the <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#autocast-and-custom-autograd-functions">docs</a>.</p>
<p>Also, your current code snippet does not show any nested <code>autocast</code> context which is disabling it as I’ve previously suggested, so unsure why you are not using this approach.</p>
What is the relationship between MKL and OpenMP in Torch?

Zarca: <p>Hi,<br>
As the title suggests, I’m wondering what is the relationship between OpenMP and MKL in torch?<br>
Is the MKL default turned off, and Torch just use OpenMP to parallelize the computation like: matrix multiply and etc? While when MKl is turned on, Torch will use MKL for matrix computation instead of the OpenMP runtime?</p>
Partitioning function in at::parallel_for

anubane: <p>The generic <code>parallel_for</code> has a concept of <em>_Partitioner</em><br>
<a href="https://learn.microsoft.com/en-us/cpp/parallel/concrt/reference/concurrency-namespace-functions?view=msvc-170#parallel_for" rel="noopener nofollow ugc">Microsoft parallel_for docs</a></p>
<p>Although I do not see provision for a <strong>partitioner</strong> in the <code>at::parallel_for</code> of PyTorch: <a href="https://github.com/pytorch/pytorch/blob/df048414e0c8485ed22bb7b28a02ab5189d888f4/aten/src/ATen/Parallel.h#L59" rel="noopener nofollow ugc">Parallel.h</a></p>
<p>Is there any way the range can be partioned unevenly for the threads?</p>
Pytorch on riscv64

kush1: <p>Hi all,</p>
<p><code>I cloned master branch of pytorch</code> from link:- <a href="https://github.com/pytorch/pytorch.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a></p>
<p>I am <code>building pytorch on riscv64</code>, I am facing issue:-</p>
<pre><code class="lang-auto">In file included from /home/root/kush/pytorch/third_party/cpuinfo/src/x86/init.c:5:
/home/root/kush/pytorch/third_party/cpuinfo/src/x86/cpuid.h:5:11: fatal error: cpuid.h: No such file or directory
    5 |  #include &lt;cpuid.h&gt;
      |           ^~~~~~~~~
compilation terminated.
make[2]: *** [confu-deps/cpuinfo/CMakeFiles/cpuinfo_internals.dir/build.make:121: confu-deps/cpuinfo/CMakeFiles/cpuinfo_internals.dir/src/x86/init.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1788: confu-deps/cpuinfo/CMakeFiles/cpuinfo_internals.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
</code></pre>
<p>if anyone know, where i am wrong, please guide me to build pytorch on riscv64.</p>
<p>T&amp;R<br>
Kush</p>
Convert Resnet34 to Resnet32

Angelina_Robert: <p>Hi,<br>
Just wonderingg if anyone could guide how to convert the following piece of code of Resnet34  to Reesnet32.</p>
<p>model_dict = {<br>
‘resnet18’: [resnet18, 512],<br>
‘resnet34’: [resnet34, 512],<br>
‘resnet50’: [resnet50, 2048],<br>
}</p>
<p>def resnet34(**kwargs):<br>
return ResNet(BasicBlock, [3, 4, 6, 3], **kwargs)</p>
Can CTCLoss use multi-gpu?

whansk50: <p>I tried to develop mult-gpu CRNN with CTCLoss, when modified code with DataParallel and execute train code and got this error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “train.py”, line 373, in <br>
main()<br>
File “train.py”, line 85, in main<br>
train(model, train_dataloader, criterion, optimizer, epoch, scaler, writer)<br>
File “train.py”, line 230, in train<br>
loss = criterion(output_log_probs, target, images_lengths, target_length)<br>
File “/home/gpudev/miniconda3/envs/gpu38/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/gpudev/miniconda3/envs/gpu38/lib/python3.8/site-packages/torch/nn/modules/loss.py”, line 1743, in forward<br>
return F.ctc_loss(log_probs, targets, input_lengths, target_lengths, self.blank, self.reduction,<br>
File “/home/gpudev/miniconda3/envs/gpu38/lib/python3.8/site-packages/torch/nn/functional.py”, line 2599, in ctc_loss<br>
return torch.ctc_loss(<br>
RuntimeError: input_lengths must be of size batch_size</p>
</blockquote>
<p>But it was not occured when do with single gpu…I cannot understand about it well</p>
<p>I hope it was just my fault but other projects are getting same issue (and it does not work for me, too) : <a href="https://github.com/Holmeyoung/crnn-pytorch/issues/31" rel="noopener nofollow ugc">https://github.com/Holmeyoung/crnn-pytorch/issues/31</a></p>
<p>So I want to know that CTCLoss can use with DataParallel or other way with 2 or more gpus and how to set data with</p>
Adding Two Models of different layers, attention heads, hidden and intermediate size

Sunny_Sanyal: <p>I am trying to add two model checkpoints [transformer] of different no. of layers, hidden size, attention heads and intermediate size. Here is a sample code [screenshot attached] where I am adding two model ckpts trained on different data. Could somebody please help me with this issue I am unable to figure out this by myself.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/6718c91f19b4c79fd023639d45b6f04e8cc008e1.png" data-download-href="https://discuss.pytorch.org/uploads/default/6718c91f19b4c79fd023639d45b6f04e8cc008e1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/6718c91f19b4c79fd023639d45b6f04e8cc008e1.png" alt="image" data-base62-sha1="eI2hvDRLtzciovjsHj2h1UJHTMJ" width="690" height="499" data-dominant-color="EFF1F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">856×620 41.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Get Features of VGG16 Pretrained Model

bryan123: <p>Could anyone show me how to get features of any pre-trained models (say VGG16)?</p>

azhanmohammed: <p>You will first need to load the model, this can be done using:</p>
<pre><code class="lang-python">pretrained_model = torchvision.models.vgg16()
feature_extractor = nn.Sequential(*list(pretrained_model.children())[:-1])
</code></pre>
<p>Then you can use the feature extractor model to extract features from the network.</p>
All_gather does not synchronize training processes at same time?

Vedant_Roy: <p>I have the following lines in my forward method:</p>
<pre><code class="lang-auto">        all_image_features1 = all_gather(image_features1)
        all_image_features2 = all_gather(image_features2)
        print(f"FORWARD (AFTER SYNC): {time.time()}, rank={self.global_rank}")
</code></pre>
<p>where <code>all_gather</code> is a small wrapper around torch’s <code>all_gather</code>.<br>
I have noticed that the timestamp printed after the sync is not the same in all processes (I am using DistributedDataParallel). It can diverge as much as 8 seconds!</p>
<p>Presumably, <code>all_gather</code> should force all the training processes to synchronize. Why is this not happening?</p>
How to share state between dataloader workers?

Vedant_Roy: <p>I’m using torchdata along with Pytorch dataloaders.</p>
<p>I want to read files from a single LMDB database. I suspect I will need to share a single database handle amongst all the dataloader workers, since I’m not sure if LMDB supports multi-process reading.</p>
<p>How can I share a single Python object amongst multiple DataLoader workers. Is this even possible?</p>

srishti-git1110: <p>From what I know, this how DataLoader actually works - an integer valued sampler is created to generate indexes that are sent across the worker processes along with the dataset instance that’s passed to the DataLoader. [This is the case when we do not explicitly specify a sampler via the <code>sampler</code> argument.]</p>
<p>With torchdata (datapipes), an additional step is to chain the <code>sharding_filter</code> datapipe to avoid data duplication.</p>
<p>I am not sure if I answered/understood your question correctly; could you please elaborate more in case what I answered isn’t what you were looking for?</p>

Vedant_Roy: <p>I’m mostly wondering if there is way to share state between the worker processes. E.g, a common database handle or something like that. In hindsight, the answer is “probably not” b/c each worker process is … its own process.</p>
PyTorch memory consumption

sapiosexual: <p>I am training vision CNN model (MoViNetA2 to be specific <a href="https://github.com/Atze00/MoViNet-pytorch" rel="noopener nofollow ugc">this</a> particular implementation). This model has 4.1M parameters which ends up into 16.5 MB of model size. For training I use Tensors size of (4, 3, 50, 224, 224) which consume around 122MB. However <code>torch.cuda.memory_summary()</code> reports over 16GB GPU memory allocated between <code>pred = model(x)</code> and <code>loss.backward()</code></p>
<p>Can somebody clarify why pytorch uses that many GPU memory? Is there a way to decrease memory consumption?</p>

ptrblck: <p>You might be forgetting the intermediate activations, which need to be stored for the gradient computation. <a href="https://discuss.pytorch.org/t/resnet-50-takes-10-13gb-to-run-with-batch-size-of-96/117402/2">This post</a> describes a similar use case for a ResNet.</p>

sapiosexual: <p>Ok, that makes sense. Thank you very much.</p>
Unsupervised Loss functions

Najeh_Nafti: <p>How can I use BCEWithLogitsLoss in the unsupervised learning? or there is any similar loss function to be used?</p>

ptrblck: <p>You can’t use this loss function without targets. Unless your “unsupervised learning” approach creates target tensors somehow, you won’t be able to use it.</p>

Najeh_Nafti: <p>Thank you for your response, and this Loss how can be used for a supervised GAN?</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html#loss-functions-and-optimizers">DCGAN tutorial</a> explains the usage of <code>BCELoss</code> and how the model is trained, so it could be a good starter.</p>