akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to apply different kernels for each example in a batch

ChrisLiu2: <p>I’m implementing a dynamic convolution where the kernel is calculated from another network instead of directly learned, but the F.Conv2d function doesn’t seem to support a kernel of size [batch, in, out, H, W]. Is there a workaround for this? Thank you.</p>

1chimaruGin: <pre><code class="lang-auto">torch.nn.functional.conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1) → Tensor
</code></pre>
<pre><code class="lang-auto">filters = torch.randn(8,4,3,3)
inputs = torch.randn(1,4,5,5)
F.conv2d(inputs, filters, padding=1)
</code></pre>

ChrisLiu2: <p>Sorry my question was confusing, I edited the question a bit. My question is how to do convolution using different filters for each batch.</p>

1chimaruGin: <aside class="quote quote-modified" data-post="2" data-topic="56901">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phan_phan/48/51960_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/apply-different-convolutions-to-a-batch-of-tensors/56901/2">Apply different convolutions to a batch of tensors</a> 
  </div>
  <blockquote>
    So, you have filters in a tensor w of shape: 
[batch_size, out_channels, in_channels, kernel_height, kernel_width], 
and images in a tensor x of shape: 
[batch_size, in_channels, in_height, in_width], 
and you want an output of shape: 
[batch_size, out_channels, out_height, out_width], 
where the i_th output is the convolution of the i_th image by the i_th filter ? 
You can do it by merging the batch and channel dimensions together, and use a grouped convolution, with groups=batch_size : 
o = to…
  </blockquote>
</aside>
nn.TransformerDecoder() - Outputs are empty

Nihir: <p>Hey all,</p>
<p>I’ve been struggling to get nn.TransformerDecoder() to work as intended - 1 of 2 things will happen - and both involve my output being empty.</p>
<p>My target data is as follows:</p>
<pre><code class="lang-auto">[['this', 'is', 'an', 'example', '[EOS]', '[PAD]'],
['another', 'target', 'sentence', '[EOS]', '[PAD]', '[PAD]']]
</code></pre>
<p>My encoders work fine, but the decoder seems to be playing up slightly. During/inside the forward pass, after encoding my source, I actually add an [SOS] token to the target (I’m doing Variational Inference and this is a strategy that one of the papers uses):</p>
<pre><code class="lang-python">target_shifted = torch.cat((sos_token, target[:, :-1]), 1)
</code></pre>
<p>This is followed by generating some target masks:</p>
<pre><code class="lang-python">trg_key_padding_mask = self.generate_pad_mask(target_shifted)
trg_mask = self.generate_square_subsequent_mask(target_shifted.size(-1))
</code></pre>
<p>I permute the embeddings so it’s <code>[S, N, E]</code>, add my latent information to the SOS token and then run the data through the decoder:</p>
<pre><code class="lang-python">target_embedding = self.get_embedding(target_shifted).permute(1, 0, 2) # [S, N, E]
target_embedding[0] = target_embedding[0] + z # z = latent information
decoder_outputs = self.transformer_decoder(target_embedding, encoder_outputs, tgt_mask=trg_mask, tgt_key_padding_mask=trg_key_padding_mask, memory_key_padding_mask=src_mask)
</code></pre>
<p>I then pass these outputs to a linear layer to calculate loss etc:</p>
<pre><code class="lang-python">decoder_outputs = decoder_outputs.permute(1, 0, 2) # [N, S, E]
output = self.output(decoder_outputs)
</code></pre>
<p>Loss is as follows:</p>
<pre><code class="lang-python">output = model(batch)
target = batch["target"]
loss = self.criterion(output.reshape(-1, output.size(-1)), target.reshape(-1)) # CrossEntropyLoss
</code></pre>
<ul>
<li>With target shifting (i.e. <code>target_shifted</code>), my loss stays almost fixed/hovers around a non-zero value and I get no output (not even a repeated token) returned to me.</li>
<li>If I DON’T perform the target shifting, my loss goes to 0 and I still get no output returned to me.</li>
</ul>
<p>Annoyingly, the code works as expected with my own implementation of MHA and a Transformer decoder - the issue occurs when using PyTorch’s nn.TransformerDecoder()</p>
Is QAT Inference not support GPU?

everlastin14: <p>I’m using pytorch 1.7.1 now.<br>
Trying to use QAT for some project.<br>
Everything works well on training.<br>
But when I tried to inference on GPU,<br>
Some errors appeared.</p>
<p>Is QAT inference not support GPU at this moment?<br>
Will it be supported at later patch?</p>

Vasiliy_Kuznetsov: <p>Currently int8 kernels are not supported on GPU.  The workaround is to either eval on GPU without converting to int8 (with fake_quant emulating int8), or to eval int8 on CPU.  Adding int8 support on CUDA is being considered but there is no timeline at the moment.</p>
Implementing log1mexp in Pytorch

Shikhar_Murty: <p>Hi,</p>
<p>I’m trying to implement the log1mexp function (basically log(1 - exp(-x) ) ) which computes the value accurately. This is basically eq(7) in the following note: <a href="https://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf" rel="nofollow noopener">https://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf</a>.</p>
<p>For my application, exp(-x) can be very close to 1 at times, so a numerically stable implementation is necessary. I’m thinking of how to implement this in pytorch and am confused between 3 alternatives: The first is to just compose this with existing pytorch functions, and wrap that in a python def. The second is to extend autograd.Function, and define my own forward and backward procedures. The third would be to write some C code, and interface that with pytorch.</p>
<p>Which one is more suitable? Thanks for the help.</p>

smth: <p>the second procedure seems simple and easy to do, will suit your purpose.</p>

mathDR: <p>Did you ever implement this?  Is there a plan of having torch have a log1mexp function generically?</p>
Numpy.convolve and conv1d in pytorch

masoudpz: <p>Hi all, there is an option in numpy.convolve , ‘same’ mode, that result of convolution will be in the same size of input, is there something like this in pytorch?<br>
size (output of convolution) = size (input)</p>
<p>for numpy.convolve can see <a href="https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.convolve.html" rel="nofollow noopener">Here</a></p>

richard: <p>I don’t think there’s a built in <code>same</code> option to conv1d but you could slice the output of conv1d with your desired size.</p>

masoudpz: <p>thanks <a class="mention" href="/u/richard">@richard</a><br>
I added some zeros at the end of my tensors but my result is differ from numpy.convolve, what do you mean by ‘Slice the output to desired size?’</p>

ptrblck: <p>Note that numpy applies a convolution, i.e. the filter gets flipped before the sliding operation, while Pytorch and other frameworks use a cross-correlation, i.e. without flipping the kernel.</p>

jpeg729: <aside class="quote no-group" data-username="masoudpz" data-post="3" data-topic="12172">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/masoudpz/48/3032_2.png" class="avatar"> masoudpz:</div>
<blockquote>
<p>I added some zeros at the end of my tensors but my result is differ from numpy.convolve</p>
</blockquote>
</aside>
<p>If you want the output to have the same size as the input in a similar way to numpy, then you need to add zero padding to <em>both</em> ends of the input.</p>

Jian_Wang: <p>Are there any functions to achieve accurate convolve operation in pytorch exactly like numpy’s version (<a href="https://numpy.org/doc/stable/reference/generated/numpy.convolve.html" class="inline-onebox" rel="noopener nofollow ugc">numpy.convolve — NumPy v1.19 Manual</a>)?  I am computing the convolution with two given vectors, the result is still different even I flipped the kernel for pytorch  compare with “numpy convolve”.  Thanks!</p>

ptrblck: <p>How large is the difference between both outputs?<br>
Relative errors in the range <code>~1e-6</code> are expected due to the limited floating point precision in <code>float32</code>.</p>
Style transfer, brushstroke size of effect

ars: <p>Hello! I’m using official pytorch examples repo to train my model. I want to use high resolution images (1500x2000) and spent many hours with trying different values of style-weight and content-weight, but I found that the size of grid of effect is always same<br>
Here is my current result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d59e08b42978c436e25aaf73f0a4b1b7381cd46.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5d59e08b42978c436e25aaf73f0a4b1b7381cd46" title="bbb"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d59e08b42978c436e25aaf73f0a4b1b7381cd46_2_374x500.jpeg" alt="bbb" data-base62-sha1="djOXRMraUsExGIyfjyaGkxOwjgq" width="374" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d59e08b42978c436e25aaf73f0a4b1b7381cd46_2_374x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d59e08b42978c436e25aaf73f0a4b1b7381cd46_2_561x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d59e08b42978c436e25aaf73f0a4b1b7381cd46_2_748x1000.jpeg 2x" data-dominant-color="929596"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">bbb</span><span class="informations">1500×2003 1.11 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
You can see, that the grid of effect is very small, so it is trying to reproduce style in small squares. I want to copy bigger parts of these fractals in my image.<br>
How can I control it?<br>
As I understand I can increase dimensions of layers here: <a href="https://github" rel="noopener nofollow ugc">https://github.com/pytorch/examples/blob/master/fast_neural_style/neural_style/transformer_net.py</a><br>
But then my mode has much more bigger size.<br>
Is it good idea or not? Or I can change it somewhere in this code?<br>
Many thanks!</p>
Why nn.Parameter variables seem unchanged during training?

kirk86: <p>I have a simple 2-layer MLP and I’m trying to optimise some hyperparams that govern the network weights but they seem to remain unchanged during training.</p>
<p>E.g.</p>
<pre><code class="lang-auto">model = MLP(fan_in=2, hidden=50, fan_out=3)
# hyper-params to optimise
a = torch.nn.Parameter(torch.tensor([1.3]))
b = torch.nn.Parameter(torch.tensor([2.4]))
opt = torch.optim.SGD([a, b], lr, momentum, weight_decay)

for epochs
  for minibatch
      for p in model.parameters():
          p.data = p.data * a + b
      logits = model(X)
      loss = cross_entropy(logits, y)
   
      opt.zero_grad()
      loss.backward()
      opt.step()
</code></pre>
<p>In every iteration I print <code>a</code> and <code>b</code> and they seem to still contain their initial values, i.e. (1.3, 2.4) equivalently.</p>
<p>Any idea what I’m missing?</p>

ChickenTarm: <p>why do you only perform 1 optimizer step every epoch and not every minibatch?</p>

kirk86: <p>That is just wrong typo in the space formatting from copy pasting. I’ll fix it in the original question but still the point remains that the params <code>(a, b)</code> defined through <code>nn.Parameter</code> are not changing. Why?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d419814c431606ea4da3ddb305e7ea00254ed5d9.png" data-download-href="https://discuss.pytorch.org/uploads/default/d419814c431606ea4da3ddb305e7ea00254ed5d9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d419814c431606ea4da3ddb305e7ea00254ed5d9_2_345x225.png" alt="image" data-base62-sha1="ugjROXst7Esa1KHKbaFC2BlUdlv" width="345" height="225" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d419814c431606ea4da3ddb305e7ea00254ed5d9_2_345x225.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d419814c431606ea4da3ddb305e7ea00254ed5d9_2_517x337.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d419814c431606ea4da3ddb305e7ea00254ed5d9_2_690x450.png 2x" data-dominant-color="EDDBDC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1614×1056 185 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

kirk86: <p>It seems that the values of params <code>a, b</code> do change if they are incorporated into the input to the loss since <code>loss.backward()</code> computes gradients and changes the values of <code>a, b</code> accordingly.<br>
E.g.:</p>
<pre><code class="lang-auto">logits = model(X)
logits += a * b
loss = cross_entropy(logits, y)
loss.backward()
</code></pre>
<p>But what if we want to optimise over params which implicitly affect the loss and are not a direct input to the loss function?</p>
Train Single LSTM Using Multiple Training Sets Collected on Separate Instances

Christopher_Prasanna: <p>I’m a little confused on how to deal with training a single network using data from separate occasions. In my case, I collected experimental data in 30-second intervals for 23 tests. This means that I want to train, validate, and test my network using these 23 separate time series.</p>
<p>Is there a good way to organize my data so that the torch library recognizes separate sequences during training? Concatenating everything would not be a good idea since this would result in discontinuities in the data.</p>
Convolutional Autoencoder weights

sikhar: <p>I am getting this error while running a downloaded program which i am using for my understanding. Input values are in the form of complex number but i am not be able to initialize the weights for convolutional autoencoder in complex number form. so i am getting this error anyone please help me how to initialize weights in complex number form.<br>
Error message:<br>
‘’‘Input type (CUDAComplexDoubleType) and weight type (torch.cuda.FloatTensor) should be the same’’’</p>

ptrblck: <p>You could try to convert all parameters to <code>complex128</code> via <code>model.to(torch.complex128)</code>.<br>
Note however, that this feature is experimental and not all operations might be supported at this moment.</p>

sikhar: <p>Thankyou for the reply <a class="mention" href="/u/ptrblck">@ptrblck</a> but it’s not working. Applying model.to(torch.complex128) i am getting an error  " nn.Module.to only accepts floating point dtypes, but got desired dtype=torch.complex128".   Is there any other way for converting weights float to complex128.?</p>
Gloo backend with CPU

Yasesf: <p>I’m using gloo along with CPU. it gives me the following warning when I submit a job and it wouldn’t continue after the warning:<br>
UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from <a href="http://www.nvidia.com/Download/index.aspx" rel="noopener nofollow ugc">http://www.nvidia.com/Download/index.aspx</a> (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370172916/work/c10/cuda/CUDAFunctions.cpp:100.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0</p>
<p>However when I run in an “interact” mode without submitting jobs it runs fine. Any clue how to get the jobs running?</p>

rvarm1: <p>Hi,</p>
<p>Would it be possible to get a minimal script that reproduces the issue so we can investigate? Also, information about your environment (PyTorch version, etc) would be great as well. Also, this seems like it may be an unexpected bug, so the issues <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a> might be a better place for this.</p>

Yasesf: <p>Sure, in the meantime, is there a way to suppress the use of GPU for gloo? like there is --mca mpi_cuda_support 0 for mpi?</p>
How to add noise

Abhi: <p>I have a mat file dataset(128 x 128). how can I add noise? please explain.</p>
<ol>
<li>add noise in any particular place.</li>
<li>add noise in the file where values are non zero.</li>
</ol>

Thabang_Lukhetho: <p>You have have to convert that into a torch tensor (but please add more detail), if you have torch tensor 128 x 128, then  you can use:</p>
<pre><code class="lang-auto">image = image + torch.randn(image.size()) *1 + 0
</code></pre>
<p>where 1  is the mean and 0  is the standard deviation</p>
Valid loss dependent LR scheduling and DDP

tmain: <p>Hi,</p>
<p>I am training a model where the learning loss is automatically updated (decay on plateau).<br>
This works fine on single GPU.</p>
<p>I am porting my code to work with DDP.<br>
After every epoch, I run a validation loop (only if local_rank == 0) and then decide whether to update or not the learning rate.</p>
<p>Pseudocode:</p>
<pre><code class="lang-python">if local_rank == 0:
    new_lr = run_validation_and_get_lr()
    for param_group in optim.param_groups:
                param_group["lr"] = new_lr
</code></pre>
<p>However, this seems to only update the learning rate in process 0, and not in the other processes.</p>
<p>How can I ensure the LR is updated in all processes ?</p>

rvarm1: <p>Hi,</p>
<p>I’m guessing this is because your update is conditioned on <code>if local_rank == 0</code>. If you only want to run this validation on rank 0, you can <code>broadcast</code> the result to the rest of the ranks as follows:</p>
<pre><code class="lang-auto">if local_rank == 0:
    new_lr = compute_new_lr()
    dist.broadcast(torch.tensor(new_lr), src=0)
else:
    new_lr = torch.zeros()
    dist.broadcast(new_lr, src=0)
# all ranks set param_group[lr] to new_lr.item()</code></pre>
License of pretrained weights for research

Chaithanya_Kumar: <p>Hello,<br>
I have a question regarding the license of pretrained weights available under torchvision.models and their usage for research purposes. The git repository includes the license that regulate the usage of scripts. However, there is no license associated with the pretrained weights that are available for download. I would like to inquire whether the same license apply to these pretrained weights and am I permitted to use those weights in my research. It would be a great help to hear your response on this regard.</p>
Low performance when iterating over ModuleDict in forward()

svenb: <p>I’m currently trying to find a faster way to process many (~100) categorical embeddings in the forward pass.</p>
<p>As the features have distinct vocabulary sizes and embedding dimensions, each requires its own embedding layer. Currently, I initialize them as nn.ModuleDict (<code>emb_model</code> in the snippet below), where the keys correspond to the names of the categorical features and the values correspond to the embedding layers (each with a weight tensor of shape (size_vocab, d_emb)).</p>
<p>In the forward pass, I loop over the ~100 .items() of the ModuleDict and provide the corresponding features as inputs (X is a dictionary with the same keys as the ModuleDict) to each layer:</p>
<pre><code class="lang-auto">def forward(self, X):

    concat_list = []
    for key, layer in self.emb_model.items():
        x = layer(X[key])
        concat_list.append(x)

    x = torch.cat(concat_list, dim=1)
...
</code></pre>
<p>However, this solution is about 3x slower than a equivalent model using the Keras/TF functional API, which I presume is due to the loop being executed in every forward pass.</p>
<p>Is there any better way to handle large numbers of categorical features in PyTorch?</p>

ChickenTarm: <p>This is kinda hacky, but as long as your GPU memory and utilization can handle it, couldnt you make each iteration into its own multiprocessing worker? That would send concurrent forward passes to the GPU allowing you to execute everything at once. Since this doesn’t appear to rely on the previous iteration output, it could be ran concurrently</p>

svenb: <p>Thanks for the tip, I will try that! Although ideally I’d prefer to solve it without relying on multiprocessing workers.</p>
<p>What I have tried is defining an <code>MultiEmbedding</code> class as:</p>
<pre><code class="lang-auto">class MultiEmbedding(nn.Module):
    def __init__(self, modules):
        super().__init__()
        self.emb = modules
    
    def forward(self, X):
        x = torch.cat([m(X[:,i].flatten()) for i, m in enumerate(self.emb)], dim=1)
        return x
</code></pre>
<p>which is initialized with a <code>nn.ModuleList</code>. In the initialization of the main module, I then convert this submodule to TorchScript</p>
<pre><code class="lang-auto">self.emb = torch.jit.script(MultiEmbedding(modules))
</code></pre>
<p>and I then have no more loop in the <code>forward()</code> of the main module. Instead of passing multiply integer arrays to the different embedding layers, all integer-encoded categorical variables are combined in one array and passed directly to the TorchScript <code>MultiEmbedding</code>.</p>
<p>This yields a small performance increase, but the TF/Keras equivalent is still more than 2x faster.</p>
<p>The only other idea I had would be to (in the future) combine <code>vmap</code> (the integer indices) and <code>NestedTensor</code> (the various combined embedding matrices) in a new nn.Module …</p>
Different approach of pretrained model

Adri: <p>Is there any difference between these approaches or maybe there is no difference:</p>
<p>Approach 1 - I use pretrained BERT model inside my model, BERT training == False</p>
<pre><code class="lang-auto"> class Model(nn.Module):
  def __init__(self, input_size, hidden_size):
    super(Model, self).__init__()
    self.model = BertModel.from_pretrained('bert-base-uncased')
    self.rnn = nn.LSTM(input_size, hidden_size,
                       batch_first=True, bidirectional=True)
    self.fc = nn.Linear(hidden_size*2, 10)
 
  def forward(self, token, attention):
    out = model(token, attention)[0]
    out, (hidden, cell) = self.lstm(out)
    out = torch.cat([hidden[-2, :, :], hidden[-1, :, :]], dim=1)
    out = self.fc(out)

    return out
</code></pre>
<p>Approach 2 - I just feed my model with last_hidden_states of BERT as an embedding features</p>
<pre><code class="lang-auto">class Model(nn.Module):
  def __init__(self, input_size, hidden_size):
    super(Model, self).__init__()
    self.rnn = nn.LSTM(input_size, hidden_size,
                       batch_first=True, bidirectional=True)
    self.fc = nn.Linear(hidden_size*2, 10)
 
  def forward(self, last_hidden_states):
    out, (hidden, cell) = self.lstm(last_hidden_states)
    out = torch.cat([hidden[-2, :, :], hidden[-1, :, :]], dim=1)
    out = self.fc(out)

    return out
</code></pre>
<p>Thanks</p>
Converting a column from a Pandas dataset to Tensor; get a KeyError Exception

Geoffrey_Payne: <p>I read data into a Pandas dataset, which I split into 3 via a utility function I wrote.<br>
I want to extract a column (name of Close) from the dataset and convert it into a Tensor.</p>
<pre><code class="lang-auto">path = os.path.join(self.config.data_dir, self.config.company_file)
        data = pd.read_csv(path)
        train_data, val_data, test_data = utilz.split_into_train_val_test(data, 0.7, 0.15, 0.15)
        train_data = T.tensor(train_data["Close"])
</code></pre>
<p>When I run the code I get a “Exception has occurred: KeyError 1”<br>
The column  with the name “Close” does exist.<br>
How do I fix this?</p>

ptrblck: <p>The error is most likely raised by the indexing with <code>"Close"</code> so I don’t think it exists.<br>
Could you print the <code>type</code> of it without using any PyTorch methods?</p>
<pre><code class="lang-python">data = pd.read_csv(path)
train_data, val_data, test_data = utilz.split_into_train_val_test(data, 0.7, 0.15, 0.15)
print(type(train_data["Close"]))
</code></pre>
Loss underfitting, F1 score overfitting

Adel: <p>I used BCEWithLogitsLoss for the loss function (the output of the model is logits) for a multilabel classification (most classes are negative while only a few are positive). During training, I measured the F1 score for train and validation dataset (converted logits to sigmoid outputs). I have ~10 linear layer. The following is the loss and F1 scores for 100 epochs:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4e38c6fb2dca0dd42d4e8e14ce562069c78625a3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4e38c6fb2dca0dd42d4e8e14ce562069c78625a3" title="Screen Shot 2021-01-25 at 10.55.38 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e38c6fb2dca0dd42d4e8e14ce562069c78625a3_2_601x500.jpeg" alt="Screen Shot 2021-01-25 at 10.55.38 AM" data-base62-sha1="b9YT1iHkDputZ2AXziM5XtTs1Ld" width="601" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e38c6fb2dca0dd42d4e8e14ce562069c78625a3_2_601x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e38c6fb2dca0dd42d4e8e14ce562069c78625a3_2_901x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e38c6fb2dca0dd42d4e8e14ce562069c78625a3_2_1202x1000.jpeg 2x" data-dominant-color="FBFCFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-01-25 at 10.55.38 AM</span><span class="informations">1500×1246 145 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As it is shown, the train loss jumps up and down after ~30 epochs. I added L2 penalty like bellow:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model1.parameters(), lr=0.0001, weight_decay=0.00001)
</code></pre>
<p>The ‘weight decay’ didn’t help. Also, adding dropout at the final layer for classification didn’t help (should I add dropout to linear layers?!).</p>
<p>I really appreciate it if you can help me with a possible solution to improve the F1 score of my model. Also I wonder how to interpret the results as the train loss is higher than validation loss but the F1 score of train dataset is better than validation set.</p>
How to know if tensor fits in gpu ram?

omer_sahban: <p>Is there a way to know whether a tensor will fit into the remaining GPU ram, before creating it?</p>

Kabir_Nagrecha: <p>I’ve been trying to figure out the same thing. Recent paper “DNNMem” apparently can analyze a model file to figure out its size BEFORE loading into memory, but they haven’t released any code. Instead, what I’ve been doing at least is adding a lot of exception handling to respond to CUDA OOMs.</p>

omer_sahban: <p>I’ve tried to calculate free memory by <code>total_memory - memory_allocated</code> or <code> total_memory - memory_reserved</code>,  I’ve also tried to do it with pynvml, here is the code snippet:</p>
<pre><code class="lang-python">def remaining_memory():
  pynvml.nvmlInit()
  gpu_handle = pynvml.nvmlDeviceGetHandleByIndex(0)
  info = pynvml.nvmlDeviceGetMemoryInfo(gpu_handle)
  return info.free
</code></pre>
<p>and none of them are accurate.<br>
I guess it has something to do with memory fragmentation, if the remaining space is not contiguous, you can’t fit a large tensor into it even if it shows there are enough free memory. I just want to know how pytorch checks whether a tensor will fit in memory.</p>
How to set the --ntasks or --ntasks-per-node argument in slurm for running multi-node distributed training?

111429: <p>Hi,<br>
I want to run the official video classification script <a href="https://github.com/pytorch/vision/tree/master/references/video_classification" rel="noopener nofollow ugc">here</a>.</p>
<p>I am not sure how to properly set the SBATCH argument --ntasks or --ntasks-per-node  if I want to run this script on 2 nodes with 8 V100 GPUs each on slurm.<br>
like --ntasks 16<br>
or --ntasks-per-node 8 ?<br>
Any advice? thanks</p>
ZeroDivisionError: float division by zero when applying "O2"

sdfadsfa: <p>I’m trying to apply gradient compression with mixed-precision training (amp, mode = “O2” with dynamic loss scale), however I encounter with an error in the middle of training:</p>
<pre><code class="lang-auto">  File "/home/anaconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 248, in run
    self._entrypoint()
  File "/home/anaconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 316, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/home/anaconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 575, in _trainable_func
    output = fn()
  File "/home/resnetTraceDDP.py", line 634, in train
    scaled_loss.backward() # calculate the gradients
  File "/home/anaconda3/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/home/anaconda3/lib/python3.7/site-packages/apex/amp/handle.py", line 123, in scale_loss
    optimizer._post_amp_backward(loss_scaler)
  File "/home/anaconda3/lib/python3.7/site-packages/apex/amp/_process_optimizer.py", line 190, in post_backward_with_master_weights
    models_are_masters=False)
  File "/home/anaconda3/lib/python3.7/site-packages/apex/amp/scaler.py", line 119, in unscale
    self.unscale_python(model_grads, master_grads, scale)
  File "/home/anaconda3/lib/python3.7/site-packages/apex/amp/scaler.py", line 88, in unscale_python
    1./scale,
ZeroDivisionError: float division by zero
</code></pre>
<p>I’m applying this to ResNet model (ResNet 50, for example), and all the hyperparameter values are reasonable. I’m not entirely sure how to fix this, will changing to static loss scale works? I was also wondering what’s the difference between dynamic loss scaling and static loss scaling, and whether that would lead to difference in run time or accuracy</p>

ptrblck: <p>We recommend to use the native mixed-precision training utility via <code>torch.cuda.amp</code> instead of <code>apex/amp</code>.<br>
You can find some examples <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples">here</a>.</p>
Pytorch install question

tianle-BigRice: <p>when i used conda install torch :</p>
<pre><code class="lang-auto"> conda install pytorch torchvision torchaudio cudatoolkit=10.1
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - cudatoolkit=10.1
  - torchvision
  - pytorch
  - torchaudio

Current channels:

  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/win-32
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r/win-32
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r/noarch
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2/win-32
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.
</code></pre>
<p>when i used pip install pytorch :</p>
<pre><code class="lang-auto"> pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.7.1+cu101
ERROR: No matching distribution found for torch==1.7.1+cu101
</code></pre>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/scream.png?v=9" title=":scream:" class="emoji" alt=":scream:"> <img src="https://discuss.pytorch.org/images/emoji/apple/scream.png?v=9" title=":scream:" class="emoji" alt=":scream:"> <img src="https://discuss.pytorch.org/images/emoji/apple/scream.png?v=9" title=":scream:" class="emoji" alt=":scream:"> <img src="https://discuss.pytorch.org/images/emoji/apple/scream.png?v=9" title=":scream:" class="emoji" alt=":scream:"></p>

ChickenTarm: <p>try this</p>
<pre><code class="lang-auto">pip install torch==1.7.1 -f https://download.pytorch.org/whl/cu101
</code></pre>

tianle-BigRice: <p>dont work</p>
<pre><code class="lang-auto">ERROR: Could not find a version that satisfies the requirement torch==1.7.1
ERROR: No matching distribution found for torch==1.7.1
</code></pre>

ptrblck: <p>Which OS and Python version are you using?<br>
If you are using a 32bit Python runtime, install the 64bit version.</p>
Rtx 3070 slower with mixed precision autocast in pytorch 1.7

dmdmello: <p>Hi. I’m using pytorch 1.7 and getting a very slow training speed with my new rtx 3070 whenever I enable torch.cuda.amp.autocast (2x slower).</p>
<p>From what I’ve read, this problem should be solved in 1.7.1, because it incorporates the newer cudnn versions, right? Are there any hopes for it’s stable version being available until the end of this month?</p>
<p>Thank you for your attention.</p>

ptrblck: <aside class="quote no-group" data-username="dmdmello" data-post="1" data-topic="105343">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/f19dbf/48.png" class="avatar"> dmdmello:</div>
<blockquote>
<p>From what I’ve read, this problem should be solved in 1.7.1, because it incorporates the newer cudnn versions, right?</p>
</blockquote>
</aside>
<p>cudnn8.0.5 ships with the updated heuristics for the 3090 and cudnn8.1.x will cover the complete 30xx series. You could try out the nightly PyTorch build, which already uses cudnn8.0.5 and check, if the performance is improved.</p>

dmdmello: <p>Well, I’ve tried the nightly build with cudnn 8.0.5 and now it’s a lot faster without amp, but still something like 10% slower with amp.</p>
<p>I suppose amp will only get full support when cudnn 8.1.x is available, right? Is there any expected month for it’s release?</p>
<p>I thank you very much for your time and help.</p>

ptrblck: <p>Yes, for the 3070 cudnn8.1.x should contain the trained heuristics. I cannot give you a specific data, unfortunately. Could you share your model and input shapes so that I could check the perf. with an internal version?</p>

dmdmello: <p>My code involves a GAN with multiple generators plus a generator classifier, and it’s pretty large, with lots of different classes and lots of argument options. I could still make a simplified version just to reproduce something like the experiment I described here, but I don’t know how condensed you are expecting it to be. I’m knew to this forum, am I supposed to just post it here in the comments?</p>

rafale77: <p>I am observing the same phenomenon running a Resnet50 network inference on a RTX3070:<br>
without autocast: 11ms<br>
with autocast: 16ms.<br>
Interestingly the GPU power consumption and CPU utilization are both lower with autocast but the inference is just significantly slower on Pytorch 1.7.1.</p>
Customize Function with Variable-length Input

wschin: <p>I hit some problems when writing my C++ version of, for example, Concat, which takes some inputs and produce an output. Because Concat consumes varialble-length inputs, I use <code>torch::autograd::variable_list</code> as its input type. Here is a simplified version of my operator.</p>
<pre><code class="lang-auto">struct MyOperator : public torch::autograd::Function&lt;MyOperator&gt;
{
  static torch::autograd::variable_list forward(torch::autograd::AutogradContext *ctx, torch::autograd::variable_list input)
  {
    auto&amp; x = input[0];
    auto&amp; y = input[1];

    ctx-&gt;save_for_backward({x, y});
    // This output's "requires_grad()" returns true.
    torch::autograd::variable_list output = {x * y};
    return output;
  }

  static torch::autograd::variable_list backward(torch::autograd::AutogradContext *ctx, torch::autograd::variable_list grad_output)
  {
    auto saved = ctx-&gt;get_saved_variables();
    auto&amp; x = saved[0];
    auto&amp; y = saved[1];

    auto&amp; dz = grad_output[0];
    torch::autograd::variable_list output = {y * dz, x * dz};
    return output;
  }
}
</code></pre>
<p>The following code may throw because loss.requires_grad() returns false.</p>
<pre><code class="lang-auto">  torch::autograd::Variable x_ = torch::randn({5,5}, torch::requires_grad());
  torch::autograd::Variable y_ = torch::randn({5,5}, torch::requires_grad());
  torch::autograd::variable_list args_;
  args_.push_back(x_);
  args_.push_back(y_);
  auto res = MyEngine::apply(args_);
  auto go = torch::ones({}, torch::requires_grad());
  auto loss = res.at(0).sum();
  // This line throws!
  loss.backward();
</code></pre>
<p>Did I do something wrong?</p>
Overlapping patch extraction from images

aditya_raj: <p>I want to extract overlapping patches from RGB images. The patches should not be randomly extracted. please share some logic or piece of code i. I tried it in MATLAB using the %mat2cell function but unable to extract overlapping patches.</p>
<p>thank you in advance</p>

Mendel123: <p>Maybe <code>Unfold</code>(<a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html?highlight=unfold#torch.nn.Unfold" class="inline-onebox" rel="noopener nofollow ugc">Unfold — PyTorch 1.7.0 documentation</a>) is what you need.</p>
Distributed: other ranks not waiting rank_0's evaluation

George_Fedoseev: <p>Hello!<br>
I am using <strong>DistributedDataParallel</strong> on one node with 4 GPUs. Also using <strong>DistributedSampler</strong> for training.</p>
<pre><code class="lang-python">self.model = torch.nn.parallel.DistributedDataParallel(
    self.model,
    device_ids=[self.local_rank],
    output_device=self.local_rank,
    find_unused_parameters=True
)
</code></pre>
<p>Doing <strong>evaluation after every train epoch only on rank_0</strong>.<br>
During evaluation I observed (through <code>nvidia-smi</code>) that other (1, 2, 3) ranks/gpus continue to be processing something with 100% load.<br>
My questions:</p>
<ol>
<li>Is it possible that other ranks continuing training next epoch without waiting for rank_0 to finish evaluation?</li>
<li>In case (1) is true, is it ok to leave it like this (will <strong>rank_0</strong> process it’s part of the next epoch after it finishes evaluation)? Or is it better to set a <strong>barrier</strong> so that other ranks will wait for rank_0 to do evaluation?</li>
</ol>
<p>Thanks!</p>

mrshenli: <blockquote>
<ol>
<li>Is it possible that other ranks continuing training next epoch without waiting for rank_0 to finish evaluation?</li>
</ol>
</blockquote>
<p>Yes, it is possible. Because all communication/synchronization happen in the backward pass. So other ranks will proceed with their next forward pass and local backward pass, and then block on the AllReduce operation in the backward pass.</p>
<blockquote>
<ol start="2">
<li>In case (1) is true, is it ok to leave it like this (will  <strong>rank_0</strong>  process it’s part of the next epoch after it finishes evaluation)?</li>
</ol>
</blockquote>
<p>Yes, it should be OK to leave it this way. Other ranks is just block waiting on AllReduce until rank_0 finishes evaluations and then runs the subsequent backward pass. It shouldn’t affect the correctness.</p>
<blockquote>
<p>During evaluation I observed (through  <code>nvidia-smi</code> ) that other (1, 2, 3) ranks/gpus continue to be processing something with 100% load.</p>
</blockquote>
<p>Yes, when block waiting for AllReduce, CUDA would show busy, although there might be no real computation running.</p>

George_Fedoseev: <p>great answer, thanks!</p>

ParthTrehan: <p>Thank you for such a great explanation. Really help me figure out how to do evaluation.</p>

tshmak: <p>So is it possible to evaluate on all GPUs and combine the results, or will that be very complicated?</p>
Block matrix multiplication (memory error)

sayan_ghosal: <p>Hi,</p>
<p>I have a block diagonal matrix <strong>A = [ A_1, 0, 0; 0, A_2, 0; 0, 0, A_3]</strong> I am multiplying it with my input vector <strong>X = [ X_1; X_2; X_3], and the output is Y = [Y_1; Y_2; Y_3]</strong>. While training my neural net it seems like during backward pass pytorch is trying to<br>
allocate a huge amount of memory and throwing the error:</p>
<p><strong>"RuntimeError: CUDA out of memory. Tried to allocate 2875.53 GiB (GPU 0; 3.94 GB total capacity; 1.31 GiB already allocated; 1.68 GiB free; 1.44 GiB reserved in total by PyTorch)"</strong></p>
<p>However, if I use a for loop I am not facing any problem but the operation is very slow. I think this problem is happening because during back ward pass PyTorch is trying to  find <strong>d(Y_1)/d(A_1), d(Y_1)/d(A_2), d(Y_1)/d(A_3), though Y_1 depends only on A_1.</strong> Is there any way to solve this problem.</p>
<p>Any help is really appreciated!</p>
<p>Thank you!</p>
Multiple models on a single GPU

Rahul_R: <p>Does running multiple copies of a model (like resnet18) on the same GPU have any benefits like parallel execution?  I would like to find the loss/accuracy on two different datasets and I was wondering if it can done more efficiently on a single GPU. I am currently evaluating the datasets sequentially.<br>
Thanks!</p>

ptrblck: <p>Most likely you won’t see a performance benefit, as a single ResNet might already use all GPU resources, so that an overlapping execution wouldn’t be possible.</p>
<p>If you want to try it nevertheless, you could have a look at the <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-streams">CUDA streams docs</a>. Note however, that you would be responsible for proper synchronizations to prevent race conditions etc.</p>

MasterSkepticista: <p>If I intend to train multiple small models like LeNet in parallel, do I have to create <code>n</code> CUDA streams with each model training function in its own <code>with</code> block?<br>
Can you please give pseudocode on the structure? Say I have 4 models, each with their own dataloaders and optimizers as a list.</p>
Number of dims don't match in permute in birads.py

Pracheta_Sahoo: <pre><code class="lang-auto">x = {
        "L-CC": torch.Tensor(datum_l_cc).permute(3, 1).to(device),
        "L-MLO": torch.Tensor(datum_l_mlo).permute(3, 1).to(device),
        "R-CC": torch.Tensor(datum_r_cc).permute(3, 1).to(device),
        "R-MLO": torch.Tensor(datum_r_mlo).permute(3, 1).to(device),
    }
    #print(x)
    transform = transforms.Compose([transforms.Resize(32),
                                      transforms.ToTensor(),
                                      transforms.Normalize(
                                          (.485, .456, .406), (.229, .224, .225))
                                      ])
    # run prediction:
    for (key,value) in x.items():
        print(key)
        torch_img=(x[key]).to(device)
        print(torch_img.size)
        features, prediction_birads = model(torch_img)
        print(prediction_birads)
        #prediction_birads = model(x).cpu().numpy()

</code></pre>
<p>File “birads.py”, line 96, in <br>
inference(parameters_)<br>
File “birads.py”, line 57, in inference<br>
“L-CC”: torch.Tensor(datum_l_cc).permute(3, 1).to(device),<br>
RuntimeError: number of dims don’t match in permute</p>

ptrblck: <p>Check the number of dimensions via:</p>
<pre><code class="lang-python">datum_I_cc = torch.tensor(datum_l_cc)
print(datum_I_cc.shape)
</code></pre>
<p>and make sure <code>permute</code> uses all dimensions.</p>
<p>Based on the values you are passing I guess the tensor might have 4 dimensions.<br>
If you want to swap <code>dim3</code> with <code>dim1</code>, you could use <code>.permute(0, 3, 2, 1)</code>.</p>

Alpha: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="95756">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>make sure <code>permute</code> uses all dimensions.</p>
</blockquote>
</aside>
<p>Using <strong>All dimensions</strong> solves this error.</p>
Custom transformation on Subset of CIFAR10 - PyTorch

Wanderer3: <p>I am trying to create a custom transformation to part of the CIFAR10 data set which superimposing of an image over the dataset. I was able to download the data and divide it into subsets. Using the following code:</p>
<pre><code>transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

traindata = datasets.CIFAR10('./data', train=True, download=True,
                       transform= transform_train)

partitions = 5
traindata_split = torch.utils.data.random_split(traindata, [int(traindata.data.shape[0] / partitions) for _ in range(partitions)])
</code></pre>
<p>then I wanted to modify part of the splits so I created the following class and functions to use as as follows:</p>
<pre><code>class MyDataset(Dataset): # https://discuss.pytorch.org/t/torch-utils-data-dataset-random-split/32209/3 
    def __init__(self, subset, transform=None):
        self.subset = subset
        self.transform = transform
        
    def __getitem__(self, index):
        x, y = self.subset[index]
        if self.transform:
            x = self.transform(x)
        return x, y
        
    def __len__(self):
        return len(self.subset)
</code></pre>
<p>and</p>
<pre><code>class ImageSuperImpose(object):
    """  Image input as PIL and output as PIL
        To be used as part of  torchvision.transforms
       Args: p, a threshold value to control image thinning        
    """
    def __init__(self, p=0):
        self.p = p                  
        
    def __call__(self, image):
        img = cv2.imread('img.jpg') 
        img = img('float32')/255
        imgSm = cv2.resize(img,(32,32))
        np_arr = image.cpu().detach().numpy().T
        sample = cv2.addWeighted(np_arr, 1, imgSm, 1, 0)
        sample = sample.T
        t = torch.from_numpy(sample)
        return sample

transform_train2 = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    ImagePoisoning(), 
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

datasetA = MyDataset(
    traindata_split[0], transform= transform_train2
)

test_loader = torch.utils.data.DataLoader(datasetA, batch_size=128, shuffle=True)
</code></pre>
<p>But when I tried to train the model on the subset I got the following error:</p>
<pre><code>RuntimeError: The size of tensor a (32) must match the size of tensor b (3) at non-singleton dimension 0</code></pre>

ptrblck: <p>I guess the shape mismatch is raised by the different memory formats used in OpenCV and PIL.<br>
While OpenCV reads images as channels-last (<code>[height, width, channels]</code>), PIL uses channels-first (<code>[channels, height, width]</code>).<br>
Adding these two images would try to add the spatial size of 32 to the channels of 3, which would raise this error. You would thus have to make sure both images are in the same format before adding them.</p>