akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Label and image are the same when augument data in problem reconstruct image

Toan_Nguyen_Duc: <p>I am doing noise restoration, I have 1 set of images for input and one set of images for labels, but when I augment the data with pytorch, when I visualize a batch, the images and labels are different, although it is still the same image but if i do the crop it will be two different areas which is not what i really want.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f347f8f34a2be39c71e1ad08938f87c2d98cef2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1f347f8f34a2be39c71e1ad08938f87c2d98cef2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f347f8f34a2be39c71e1ad08938f87c2d98cef2_2_690x268.jpeg" alt="image" data-base62-sha1="4s3hh2MHDmyM43ymBd9c3uiGuQi" width="690" height="268" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f347f8f34a2be39c71e1ad08938f87c2d98cef2_2_690x268.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f347f8f34a2be39c71e1ad08938f87c2d98cef2_2_1035x402.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f347f8f34a2be39c71e1ad08938f87c2d98cef2.jpeg 2x" data-dominant-color="90938F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1214×472 208 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">class GAN_DATASET(Dataset):
  def __init__(self, ImageFolder, LabelFolder, data_transforms):
    self.transforms = data_transforms
    self.list_image = []
    self.list_label = []
    for i in os.listdir(ImageFolder):
      self.list_image.append(i)

    for i in os.listdir(LabelFolder):
      self.list_label.append(i)

    self.root_hazy_image = os.path.join(ImageFolder)
    self.root_hazy_label = os.path.join(LabelFolder)
    self.file_len = len(self.list_image)

  def __getitem__(self, index, is_train=True):
      hazy = Image.open(self.root_hazy_image + self.list_image[index])
      hazy = self.transforms(hazy)
      
      label = Image.open(self.root_hazy_label + self.list_label[index])
      label = self.transforms(label)
      
      name = self.list_image[index]
      return hazy,label,name
  def __len__(self):
      return self.file_len 
</code></pre>

ptrblck: <p>If I understand your use case correctly, you would like to apply the same random transformations to the image and target tensors.<br>
In this case you could use the functional API as described in <a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">this example</a>.</p>
Torchscript for model with GRU

Sfitsos: <p>I have a model that has a GRU in it. I have traced the model but when I load it and call the forward method it works only if the length of the sequence I use is the same as the dummy input used when tracing. I get this warning when tracing:</p>
<pre><code>TracerWarning: Iterating over a tensor might cause the trace to be incorrect.
Passing a tensor of different shape won't change the number of iterations 
executed (and might lead to errors or silently ​give incorrect results).
</code></pre>
<p>My dummy sequence has a length of 25 with features of size 256 (6400). I chose this at random since the model should work for variable sequence lengths. However, when I load the model and pass a sequence with a different length to the dummy input I get this error:</p>
<p>RuntimeError: shape ‘[-1, 30, 256]’ is invalid for input of size 6400</p>
<p>Is this normal (i.e. what the warning was about) and if so are there any workarounds?</p>
<p>I suppose one workaround is to have a max size and pad the tensor when I use it but this is not optimal.</p>

dfalbel: <p>AFAICT this is exactly tthe case where tracing doesn’t work well. See the <code>Warning</code> section in the <code>torch.jit.trace</code> documentation page:</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.jit.trace.html#torch.jit.trace" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.jit.trace.html#torch.jit.trace</a></p>
<blockquote>
<p>…<br>
Tracing will not record any control-flow like if-statements or loops.<br>
…</p>
</blockquote>
<p>It also offers a possible solution:</p>
<blockquote>
<p>In cases like these, tracing would not be appropriate and <a href="https://pytorch.org/docs/stable/generated/torch.jit.script.html#torch.jit.script" rel="noopener nofollow ugc"><code>scripting</code></a> is a better choice. If you trace such models, you may silently get incorrect results on subsequent invocations of the model. The tracer will try to emit warnings when doing something that may cause an incorrect trace to be produced.</p>
</blockquote>

Sfitsos: <p>The problem in my case seems to be that the handling of variable sized tensors (due to variable sequence lengths). I assume then that this is internally implemented with a loop. I will look into scripting but if the loop is in the pytorch code then how would I do it? Another alternative that I just considered is inputting the hidden state for the GRU and passing it to the network which will output the hidden state for use in future steps. That way I suppose I could use use a dummy with length 1 and perform the loop outside. Would something like this work?</p>

dfalbel: <p>I think scripting works fine in that case and is probably the best option:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class MyGRU (nn.Module):
  def __init__ (self):
    super(MyGRU, self).__init__()
    self.gru = nn.GRU(10, 32, 5)
  
  def forward (self, x):
    return self.gru(x)

module = torch.jit.script(MyGRU())
print(module(torch.randn(100, 15, 10))[0].shape)
print(module(torch.randn(100, 20, 10))[0].shape)
</code></pre>
Implement a large scale Linear layer or use parameter server instead?

ecolss: <p>I want to implement a model which has a very large <code>Linear</code> layer, because the amount of features is very big, say <code>2^32</code> features, actual input will be a sparse tensor.</p>
<p>Typically if the amount of features is not big, I can create an ordinary <code>Linear</code> layer, for example</p>
<pre><code class="lang-auto">self.lin = Linear(in_features, out_features)
</code></pre>
<p>But since <code>in_features=2^32</code>, the above <code>Linear</code> layer won’t work properly.<br>
So I’m thinking about ideas like,</p>
<ol>
<li>Split the huge Linear into multiple small ones, e.g. each has <code>2^20</code> features. And I looked at <code>torch.distributed.rpc</code>, but doesn’t seem to be able to do it.</li>
<li>Or use parameter server, but no idea how to turn the Linear layer into a parameter server.<br>
I didn’t find how to do the above 2 ideas, please give me some advice.<br>
Thanks</li>
</ol>

rvarm1: <p>Regarding the parameter server idea, I guess parameter server and linear are quite different concepts as a PS is a training paradigm while a linear layer is a component of a model.</p>
<p>It looks like you have a use case to shard a linear layer across multiple processes/potentially multiple nodes. We are planning to build out sharding primitives within PyTorch, please see this RFC: <a href="https://github.com/pytorch/pytorch/issues/55207" class="inline-onebox" rel="noopener nofollow ugc">[RFC] Model Sharding for distributed training · Issue #55207 · pytorch/pytorch · GitHub</a> which would suit this use case once it is built out.</p>
<p>Curious, if the input is a sparse feature, would an nn.EmbeddingBag work better than a linear layer?</p>

ecolss: <p>Thanks for your answer Rohan!<br>
Yes, surely EmbeddingBag also looks great, but meanwhile I want to see how to shard Linear layer into multiple processes / machines.<br>
Will definitely check out the RFC.</p>

ecolss: <p>BTW if it’s <code>nn.EmbeddingBag</code> (actually I don’t need the aggregation operation, so <code>nn.Embedding</code> would be better I think), I think we still have to think out sharding.<br>
Because the EmbeddingBag size would still be huge (i.e. 2^32), might not fit into a single machine.</p>
What's the difference between nn.Embedding and nn.Linear

Xin_Perry: <p>what’s the difference between nn.Embedding and nn.Linear ? Does embedding do the same thing as  fc layer ?</p>

stas: <p>There is an excellent answer here: <a href="https://stackoverflow.com/a/65448744/9201239" class="inline-onebox" rel="noopener nofollow ugc">python - What is the difference between an Embedding Layer with a bias immediately afterwards and a Linear Layer in PyTorch - Stack Overflow</a></p>
Deploying a Reinforcement Learning Model (DDPG

uBoscuBo: <p>Are there any kinds of literature that talked about deploying a trained agent? I been looking, but I couldn’t find any. I want to know how to deploy a trained agent. In a particular case using the DDPG algorithm, do I just call the Actor with the same parameters as my trained Actor and load the information? Do I need to include the learning algorithm from DDPG? Is the reward still needed, or can I neglect that?</p>
<blockquote>
<p>import torch<br>
from matplotlib.animation import FuncAnimation<br>
from matplotlib.widgets import TextBox<br>
from tclab import clock<br>
import numpy as np</p>
<p>actor = ActorNetwork(alpha=0.0001, input_dims=(2,), fc1_dims=400,<br>
fc2_dims=300, n_actions=2, name=‘actor’)<br>
actor.load_checkpoint()<br>
env = PIEnv()</p>
<p>facecolor = ‘lightgoldenrodyellow’<br>
fig, axs = plt.subplots(2, figsize=(12, 8), tight_layout=True)</p>
<p>SP_input = env.states[1]</p>
<p>SP_mem = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
PV_mem = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
OP_mem = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
time_mem = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>def animate(i, PV_mem, SP_mem, OP_mem, time_mem):<br>
states = env.reset()<br>
action = None<br>
with torch.no_grad():<br>
state = torch.tensor([states], dtype=torch.float)<br>
if i % 180 == 0 or action is None:<br>
action = actor.forward(state)<br>
states_, PV, SP, OP, Ks = env.step(action.detach().numpy()[0], 50)<br>
states = states_<br>
print(f’Time: {int(i)}, PV: {PV:.2f}, SP: {SP:.0f}, OP: {OP:.2f}')</p>
<pre><code>    SP_mem.append(SP)
    PV_mem.append(PV)
    OP_mem.append(OP)
    time_mem.append(i)

    plt.cla()
    axs[0].plot(time_mem, SP_mem, 'r--', label='Setpoint')
    axs[0].plot(time_mem, PV_mem, 'b-', label='Process Variable')
    axs[0].set_ylabel('Temperature °C')

    axs[1].plot(time_mem, OP_mem, 'g-', label='Heater Output')
    axs[1].set_ylabel('Heater Output')
    axs[1].set_xlabel('Time (seconds)')

    axs[0].set_title(f'IAE: {states_[0]:.0f}', loc='left')
    axs[0].set_title(f'Kp: {Ks[0]:.2f}, Ki: {Ks[1]:.4f}', loc='right')
</code></pre>
<p>try:<br>
ani = FuncAnimation(fig, animate, fargs=(PV_mem, SP_mem, OP_mem, time_mem), interval=1000)<br>
plt.show()<br>
except KeyboardInterrupt:<br>
env.close()</p>
</blockquote>
<p>Thanks is what I have and I don’t know if it captures all the information I need to deploy my RL model. Any advice would be much appreciated. Thanks!</p>
Multi-label classification target padding

safakkbilici: <p>I have a problem that contains target classes from 0 to 6. But the number of targets of samples is not static. For example, while sample <code>x_k</code> has targets <code>[0,1,5,3]</code>, it is possible that sample <code>x_j</code> can have <code>[0,2]</code>.</p>
<p>The first thing that came to my mind is padding all labels with <code>-100</code> to maximum label length (35), then use it with <code>nn.BCEWithLogitsLoss</code>. Then I read that, <code>BCEWithLogitsLoss</code> evaluates targets like they have 35 different classes.</p>
<p>What is the best way I should follow in this case? Thanks.</p>

ptrblck: <p>Assuming each sample can have zero, one, or multiple active classes, you could let the model return logits in the shape <code>[batch_size, nb_classes]</code> where each value in <code>nb_classes</code> would correspond to the logit for the corresponding class index.<br>
The target could then be multi-hot encoded, where a zero would indicate an inactive and a one an active class. For the posted examples the target would thus be:</p>
<pre><code class="lang-python">[0,1,5,3]
target = torch.tensor([[1, 1, 0, 1, 0, 1, 0]]).float()

[0, 2]
target = torch.tensor([[1, 0, 1, 0, 0, 0, 0]]).float()
</code></pre>

safakkbilici: <p>My problem is more likely to sequential decoding. Each target can be repeated, like <code>[5,2,1,0,1]</code>. I padded the targets and, used <code>nn.CrossEntropyLoss()</code> with <code>ignore_index=-100</code> parameter. But still no improvement…</p>
<p>Thanks for your answer.</p>

ptrblck: <p>Thanks for the update.<br>
If I understand the use case correctly you might then be working on a multi-class sequence classification, i.e. each time step has only a single label?<br>
In this case, you could use <code>nn.CrossEntropyLoss</code> with a model output in the shape <code>[batch_size, nb_classes, seq_len]</code> and a target in the shape <code>[batch_size, seq_len]</code> containing the class indices in the range <code>[0, nb_classes-1]</code>.</p>
Choice of activation function

Rajat_Sharma1: <p>I am trying to optimize a model which has output range <strong>[1, -1]</strong> so, I know for the last layer I can not use <code>ReLU</code> as it always gives positive output. But, I was wondering that , can using <code>ReLU</code> in intermediate layers affect the model output as it will make each activation output greater than 1. Thus choosing other activation function such as <code>TanH</code> will be more suitable case here. Or, may be Leaky ReLU. I guess, In case of <code>Leaky ReLU</code>, setting the slope is a crucial hyperparameter. I tried all three activation functions, TanH optimizes over time, <code>ReLU</code> model give <code>nan</code> output after few epoch, <code>Leaky ReLU</code> some time work some time gives output as <code>nan</code>.  Can someone have their thoughts on this topic.</p>

KFrank: <p>Hi Rajat!</p>
<aside class="quote no-group" data-username="Rajat_Sharma1" data-post="1" data-topic="126123" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rajat_sharma1/48/39351_2.png" class="avatar"> Rajat_Sharma1:</div>
<blockquote>
<p>I was wondering that , can using <code>ReLU</code> in intermediate layers affect the model output as it will make each activation output greater than 1.</p>
</blockquote>
</aside>
<p>It is perfectly fine to use <code>ReLU</code> as the activation for intermediate layers.<br>
Even though the output will be greater than zero (I assume that “make<br>
each activation output greater than 1” is a typo.), a subsequent <code>Linear</code><br>
layer can have negative weights that flip the sign of the positive activation<br>
output and negative biases that shift the positive activation output down<br>
below zero.</p>
<aside class="quote no-group">
<blockquote>
<p>I tried all three activation functions, TanH optimizes over time, <code>ReLU</code> model give <code>nan</code> output after few epoch, <code>Leaky ReLU</code> some time work some time gives output as <code>nan</code>.</p>
</blockquote>
</aside>
<p>As a practical modeller, you might then want to use <code>Tanh</code> – because it<br>
works!</p>
<p>As for the <code>nan</code>, <code>ReLU</code> might still work, but you might start with a lower<br>
learning rate as doing so can sometimes avoid the <code>nan</code>s.  One approach<br>
is to start with a small learning rate so that your training can avoid <code>nan</code>s<br>
while the model’s training “settles down,” increase the learning rate for<br>
a while so you make faster progress, and then lower the learning rate<br>
again so you can “fine tune” the model’s parameters.</p>
<p>Best.</p>
<p>K. Frank</p>

Rajat_Sharma1: <p>Like you said in subsequent layers the sign will flip out. But, I case of my model I am using ReLU at all layers except the last one which is TanH. So, I think model will have hard time to learn. Also, for Leaky ReLU the slope is crucial parameter.</p>
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 256 and 130 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71

aniket14: <p><strong>Error :</strong></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 118, in &lt;module&gt;
    main()
  File "main.py", line 48, in main
    model.train()
  File "/workspace/aniket/edge-informed-sisr/src/edge_match.py", line 120, in train
    hr_images_pred, gen_loss, dis_loss, logs = self.sr_model.process(lr_images, hr_images, lr_edges, hr_edges_pred)
  File "/workspace/aniket/edge-informed-sisr/src/models.py", line 210, in process
    outputs = self(lr_images, hr_edges)
  File "/workspace/aniket/edge-informed-sisr/eisr-env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/workspace/aniket/edge-informed-sisr/src/models.py", line 263, in forward
    inputs = torch.cat((hr_images, hr_edges), dim=1)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 256 and 130 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71
</code></pre>
<p><strong>Code :</strong></p>
<pre><code class="lang-auto">    def forward(self, lr_images, hr_edges):
        hr_images = F.conv_transpose2d(lr_images, self.scale_kernel, padding=0, stride=2, groups=3)
        inputs = torch.cat((hr_images, hr_edges), dim=1)
        outputs = self.generator(inputs)
        return outputs
</code></pre>
<p>I am getting dimension mismatch error.</p>

ptrblck: <p>Based on the error message <code>hr_images</code> and <code>hr_edges</code> have different shapes in <code>dim2</code>, while only a size mismatch would be allowed in <code>dim1</code>, which is used to concatenate the tensors.<br>
You can print the shape of both tensors before applying the <code>torch.cat</code> operation and make sure the spatial shapes match.</p>

aniket14: <p>I am training the EISR model <a href="https://github.com/knazeri/edge-informed-sisr.git" rel="noopener nofollow ugc">https://github.com/knazeri/edge-informed-sisr.git</a>, I can train the model properly for the x2 scale factor, but while training for the x4 scale factor, I’m facing the dimension mismatch issue. Can you guide me on how to fix the above issue?</p>

ptrblck: <p>I’m not familiar with the repository, but it seems that the scale factors are used <a href="https://github.com/knazeri/edge-informed-sisr/blob/1b4a6e897a84545fdecdd7acdb5273147b9321cb/src/models.py#L143-L144">here</a> so you could check why the shapes are wrong using a larger scale factor of 4.</p>
Accelerate torch.cholesky() and torch.inverse() for big matrices

Amine08: <p>Hi,</p>
<p>is there any way to accelerate torch.cholesky() and torch.inverse() for big matrices, e.g. by using multiple CPUs or GPUs?</p>
<p>Here an example:</p>
<pre><code class="lang-auto">import torch
import time
start_time=time.time()
matrix=torch.eye(8192)
cholesky_decomp=torch.cholesky(matrix)
matrix_inverted=torch.inverse(cholesky_decomp)
print(time.time()-start_time)
</code></pre>
<p>PS: The matrices to invert are in my case upper triangular matrices (result of cholesky decomposition)</p>
<p>Thanks!</p>

Amine08: <p>Update:</p>
<p>torch.triangular_solve has a better execution time than torch.inverse in this case (triangular matrix)</p>
<pre><code class="lang-auto">import torch
import time
start_time=time.time()
matrix=torch.eye(8192)
cholesky_decomp=torch.cholesky(matrix)
matrix_inverted,_=torch.triangular_solve(torch.eye(cholesky_decomp.shape[0]), cholesky_decomp)
print(time.time()-start_time)
</code></pre>
When DataParallel is used for traning how to know the ground truths of data batches on different GPUs?

sarvani: <p>In my code, every hidden layer output and the ground truths of  that mini-batches are used for estimating a quantity in every iteration. This works fine when single GPU is used (say batch_size =32). As every hidden layer output and ground truths would be of same number  i.e, 32</p>
<p>But when I use datatparallel with batch_size=64 on 2 GPUs, I have the grounds of size 64, but outputs from hidden layers are of size 32 from each of the 2 gpus.</p>
<p>How to know how the 64 images are divided onto 2 GPUs, so that I can get the ground truths of size 32  to use it with outputs from every hidden layer.</p>
<p>(Previously i was passing the entire ground truth values to forward hooks created on each layer)</p>

J_Johnson: <p>What do you mean by “ground truths”?</p>
<p>Also, have you tried using the Model Parallel method? That will divide the model layers between GPUs instead of batches.</p>
<p><a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html</a></p>

sarvani: <p>Ground truths mean the original labels for a given mini-batch of images.<br>
I did not try model parallel , i will try it.</p>

J_Johnson: <p>Ah. Gotcha. Can you post your model here? It’s tough to identify the exact issue with your hidden layer size output without seeing the structure.</p>
More intense Gaussian Noise

KanZa: <p>I want to apply more intense noise to the input data. Is it possible?<br>
The current Noise in the model</p>
<p>class GaussianNoise(nn.Module):<br>
def <strong>init</strong>(self, sigma):<br>
super(GaussianNoise, self).<strong>init</strong>()<br>
self.sigma = sigma</p>
<pre><code>def forward(self, input):
    if self.training:
        noise = Variable(input.data.new(input.size()).normal_(std=self.sigma))
        return input + noise
    else:
        return input</code></pre>

KFrank: <p>Hi KanZa!</p>
<aside class="quote no-group" data-username="KanZa" data-post="1" data-topic="126188" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kanza/48/1529_2.png" class="avatar"> KanZa:</div>
<blockquote>
<p>I want to apply more intense noise to the input data.</p>
</blockquote>
</aside>
<p>Can’t you just use a larger value of <code>sigma</code> when you instantiate your<br>
<code>GaussianNoise</code> object?</p>
<aside class="quote no-group">
<blockquote>
<pre><code>        noise = Variable(input.data.new(input.size()).normal_(std=self.sigma))
</code></pre>
</blockquote>
</aside>
<p>As an aside, this code looks rather out of date.  <code>torch.autograd.Variable</code><br>
has been deprecated for some time now.</p>
<p>Best.</p>
<p>K. Frank</p>

KanZa: <p>Thank you for your suggestion.<br>
Can you please elaborate on your suggestion, I tried by taking larger Sigma values.<br>
Any other current Gaussian Noise method you would like to suggest?</p>

KFrank: <p>Hi KanZa!</p>
<aside class="quote no-group" data-username="KanZa" data-post="4" data-topic="126188" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kanza/48/1529_2.png" class="avatar"> KanZa:</div>
<blockquote>
<p>I tried by taking larger Sigma values.</p>
</blockquote>
</aside>
<p>I assume that you are saying that using larger values for <code>sigma</code> did not<br>
successfully “apply more intense noise.”</p>
<p>Could you post a short, self-contained, runnable script that shows what<br>
you tried, together with its output, and explain what you were looking for<br>
that you didn’t get?</p>
<p>Best.</p>
<p>K. Frank</p>
Explanation about output of torch.cuda.memory_summary()

yncxcw: <p>Hello, I am using <code>torch.cuda.memory_summary()</code> to monitor the memory usage of training job. I am trying to understand what <code>Allocated memory</code>, <code>Active memory</code>, <code>GPU reserved memory</code> and <code>Non-releasable meomry</code> mean.</p>
<p>Thanks,</p>
<p>Wei</p>
How to sync module weights, but not gradients?

hadaev8: <p>So i have encoder-decoder model and also gan discriminator.<br>
Like this out = Generator.decoder(Generator.encoder(inp))<br>
And discriminator_out = D_encoder(out)</p>
<p>I want to share weights between Generator.encoder and D_encoder because it have same layers and do kind of same job, but this modules should not make gan cheat (like make encoder worse so it would give wrong score to fake samples).</p>
<p>How should i approach it?</p>
Batchsize with DDP v.s. without DDP

hhxx: <p>In distributeddataparallel, when local batch-size is 64 (i.e. <code>torch.utils.data.DataLoader(batch_size=64) and torch.utils.data.distributed.DistributedSampler() is used</code>), assume there are N processes totally in ddp (N processes distirbute in one node or more than one node). Is the forward-backward process in ddp similar to the forward-backward process in a single gpu using 64×N batch-size inputs?</p>

Yanli_Zhao: <p>yes, distributed training using DDP is mathematically equivalent to local training</p>

AMellinger: <p>Can you clarify this? The OP is asking if batch_size of 64 per DDP process in a world size of N is the same as a single gpu with a total batch size of 64*N.  There is a note in the DDP docs which state:</p>
<p>“When a model is trained on <code>M</code> nodes with <code>batch=N</code> , the gradient will be <code>M</code> times smaller when compared to the same model trained on a single node with <code>batch=M*N</code> if the loss is summed (NOT averaged as usual) across instances in a batch (because the gradients between different nodes are averaged). You should take this into consideration when you want to obtain a mathematically equivalent training process compared to the local training counterpart. But in most cases, you can just treat a DistributedDataParallel wrapped model, a DataParallel wrapped model and an ordinary model on a single GPU as the same (E.g. using the same learning rate for equivalent batch size).”</p>
<p>It looks to me that they say that they should be the same if the single GPU case if the batch size is 64 and not 64*N (as the OP asked).  Can you clarify?</p>
<p>Thanks for any help!</p>
Torch.nn.functional.cosine_similarity got unexpected behavier with torch.int8 dtype

Jonson: <p>PyTorch version: 1.9.0</p>
<p><code>torch.nn.functional.cosine_similarity</code> got unexpected behavier with torch.int8 dtype.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.randn(10)*100
&gt;&gt;&gt; a
tensor([ 47.8993, -27.2694,  20.9548, -13.5573, -54.0388, -76.8524,  -7.4037,
        -95.7477, 108.9276, -44.5625])
&gt;&gt;&gt; b = a
&gt;&gt;&gt; F.cosine_similarity(a, b, dim=-1)
tensor(1.)
&gt;&gt;&gt; F.cosine_similarity(a.to(torch.int8), b, dim=-1)
tensor(3.5215e+12)
&gt;&gt;&gt; F.cosine_similarity(a.to(torch.int16), b, dim=-1)
tensor(1.0000)
</code></pre>
<p>As we can see, <code>F.cosine_similarity(a.to(torch.int8), b, dim=-1)</code> got result of <code>tensor(3.5215e+12)</code>, while the expected result is <code>tensor(1.)</code>.</p>

ptrblck: <p>This looks like an overflow using <code>int8</code> internally. Would you mind creating an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we could track and fix it?</p>

Jonson: <p>ok, I have created an issue: <a href="https://github.com/pytorch/pytorch/issues/61454" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.cosine_similarity got unexpected behavier with torch.int8 dtype · Issue #61454 · pytorch/pytorch · GitHub</a></p>
What is the 0 and 1 in the grad_func represents?

Saran-nns: <p>Referred from <a href="https://pytorch.org/tutorials/beginner/former_torchies/autograd_tutorial.html" rel="noopener nofollow ugc">Pytorch Tutorial</a></p>
<pre><code class="lang-auto">z = y * y * 3
out = z.mean()

print(z, out)
</code></pre>
<p>Out:</p>
<pre><code class="lang-auto">tensor([[27., 27.],
        [27., 27.]], grad_fn=&lt;MulBackward0&gt;) tensor(27., grad_fn=&lt;MeanBackward1&gt;)</code></pre>
How to know if specific function is differential or not?

Tomer_Weiss: <p>Hi all.<br>
I work a lot with PyTorch and found it great and useful.<br>
I wanted to know if there exist any way to understand from the doc if specific function as supported backward. For most of the function it is clear by definition, but for few it’s not so clear and sometime depend on the implementation.</p>
<p>Thanks<br>
Tomer</p>

ptrblck: <p>I would assume that the majority of mathematically differentiable functions are also differentiable in PyTorch, but to make sure that’s the case, you could check the <code>.grad_fn</code> attribute of the output using inputs, which require gradients e.g. as seen here:</p>
<pre><code class="lang-python">x = torch.randn(10, 10, requires_grad=True)

out = torch.sum(x, dim=1)
print(out.grad_fn)
&gt; &lt;SumBackward1 object at 0x7f79da0b9520&gt; # differentiable

out = torch.argmax(x, dim=1)
print(out.grad_fn)
&gt; None # not differentiable
</code></pre>

Tomer_Weiss: <p>Thanks for your replay.<br>
This is exactly what I am doing.<br>
I think it worth adding it to the doc of each function (or just specify when it’s not differentiable).<br>
What is the way to suggest this idea? Should I open I issue in git?</p>
<p>Thanks again.</p>

ptrblck: <p>Sure, you can create a feature request on GitHub. Would you be interested in working on it as well?</p>

Tomer_Weiss: <p>I don’t have any experience to contribute to such a big library, but I will be happy to help.</p>

tom: <p>Note that there are several subtleties here.<br>
In the first layer we have the following:</p>
<ul>
<li>There are directly differentiable functions (per <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml" rel="noopener nofollow ugc">tools/autograd/derivatives.yaml</a>), these are the easy ones. For those, there is a backward (somewhere).</li>
<li>Then there are functions that reduce to directly differentiable functions (e.g. <code>einsum</code>). This means that they are calling these directly differentiable functions and autograd will do its autograd thing to do the differentiation.</li>
<li>Some of these compositions will go through unexposed/internal functions (e.g. convolutions, ctc loss etc.).</li>
<li>The non-differentiable functions are the remainder…</li>
</ul>
<p>But then there are second derivatives, where the same scheme applies, and there is the question of “differentiable w.r.t. <em>which</em> arguments?”.</p>
<p>Best regards</p>
<p>Thomas</p>
Load pre-trained model w/o AMP and test with AMP

oasjd7: <p>I have pre-trained model which is trained without AMP. (Acc:73%)<br>
I want to load this model and train with AMP.</p>
<p>So I add ‘<span class="mention">@torch.cuda.amp.autocast</span>()’ in model.</p>
<pre><code class="lang-auto">   @torch.cuda.amp.autocast()
    def forward(self, x):
</code></pre>
<p>And also I added <code>with torch.cuda.amp.autocast():</code> in evaluation code.<br>
But the accuracy dropped from 73% to 0.47%.</p>
<p>What should I do?</p>

ptrblck: <p>Could you post the model definition, please, so that we could have a look?</p>
Implementation of Mixed Precision

oasjd7: <p>Hi, 'm new to mixed precision training.<br>
In my case, I have two optimizers and update twice in a epoch.<br>
Only one scaler is enough to use mixed precision training ?<br>
(What about scale(), step() and update() ? )</p>
<p>Here is my pseudo-code.<br>
(Training process is quite complex, so the codes are summarized in a abbreviated version)</p>
<pre><code class="lang-auto">for epoch in epochs:
        ...
        optimizer.zero_grad()
        optimizer2.zero_grad()
        scaler.scale(loss2).backward()
        scaler.step(optimizer2)
        scaler.update()
        ...
        optimizer.zero_grad()
        optimizer2.zero_grad()
        scaler.scale(loss1).backward()
        scaler.step(optimizer)
        scaler.update()
</code></pre>

ptrblck: <p>Yes, one <code>GradScaler</code> would work as described in the <code>amp</code> example using <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#working-with-multiple-models-losses-and-optimizers">multiple models, losses, optimizers</a>.</p>
toTuple() occurs error

Haewon_Lee: <p>When I convert module output to tuple, I got an error like this.<br>
I can’t debug anything. How to debug libtorch? I’m using libtorch1.5.0(debug)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bbe2e3022f9161f1d04edeeb6f5f8b789ba04312.png" data-download-href="https://discuss.pytorch.org/uploads/default/bbe2e3022f9161f1d04edeeb6f5f8b789ba04312" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bbe2e3022f9161f1d04edeeb6f5f8b789ba04312.png" alt="image" data-base62-sha1="qO7o9CdVu3mC85vrxesfXD9Jn4S" width="690" height="257" data-dominant-color="282A2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">781×292 16.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you translate the error message, please?</p>

Haewon_Lee: <p>Oh, I’ve solved the problem.<br>
The cause of the problem was the difference between Torchscript and Libtorch versions.<br>
Thank you ptrblck ! : )</p>
I am getting this error "TypeError: img should be PIL Image. Got <class 'NoneType'>"

Aryan_Philip: <p>I am running a U-net model, I am getting a Type Error  when training the model, ``</p>
<p>‘’</p>
<pre><code class="lang-auto">'num_epochs = 30

#criterion = nn.BCEWithLogitsLoss()

criterion = nn.MSELoss()

optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

scaler = torch.cuda.amp.GradScaler()

# Decay LR by a factor of 0.1 every several epochs

sch = optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma= 0.5)

train(model, criterion, optimizer, num_epochs, scaler,sch)
</code></pre>
<pre><code class="lang-auto">train_transform = transforms.Compose([

       transforms.Resize((256, 256)),

       #transforms.RandomResizedCrop(256),

       transforms.ToTensor(),

       #transforms.ColorJitter(),

       transforms.RandomHorizontalFlip(),

       transforms.Normalize([0.5,0.5,0.5], [0.5,0.5,0.5])

])

test_transform = transforms.Compose([

       transforms.Resize((256,256)),

       #transforms.CenterCrop(256),

       transforms.ToTensor(),

       transforms.Normalize([0.5], [0.5])

   ])
</code></pre>
<p>I am guessing that this part of the code is throwing an error.</p>
<pre><code class="lang-auto">class Flare(Dataset):

    def __init__(self, flare_dir, wf_dir, flare_img_, wf_img_, transform = None):

        self.flare_dir = flare_dir

        self.wf_dir = wf_dir

        self.transform = transform

        self.flare_l = flare_img_

        self.wf_l = wf_img_

        

    def __len__(self):

        return len(self.flare_l)

    def __getitem__(self, idx):

        wf_img = None

        f_img = Image.open(os.path.join(self.flare_dir, self.flare_l[idx])).convert("RGB")

        for i in self.wf_l:

            if (self.flare_l[idx].split('.')[0][4:] == i.split('.')[0]):

                wf_img = Image.open(os.path.join(self.wf_dir, i)).convert("RGB")

                break

        

        f_img = self.transform(f_img)

        wf_img = self.transform(wf_img)

        

        return f_img, wf_img
</code></pre>
<pre><code class="lang-auto">flare_dir = '/content/gdrive/MyDrive/data/Flare_img'

wf_dir = '/content/gdrive/MyDrive/data/Without_Flare_'

flare_img = os.listdir(flare_dir)

wf_img = os.listdir(wf_dir)

wf_img.sort()

flare_img.sort()

print(wf_img[0])

train_ds = Flare(flare_dir, wf_dir, flare_img[0:5000], wf_img[0:5000], train_transform)

train_loader = torch.utils.data.DataLoader(dataset=train_ds,

                                           batch_size=BATCH_SIZE, 

                                           shuffle=True)

print(len(train_ds))
</code></pre>
<p>This is the part of the code where i am suspecting the error is from, not sure how to tackle it.</p>

ptrblck: <p>Check <code>f_img</code> and <code>wf_img</code> inside the <code>__getitem__</code> method for valid objects, as I assume one of the transformations are raising this error.<br>
Given the code, I guess <code>self_wf_l</code> is empty, the loop is never executed, and <code>wf_img</code> is still initialized as <code>None</code>.</p>
Conv2D padding in TensorFlow and PyTorch

mustansarsaeed: <p>I am trying to convert <code>TensorFlow</code> model to <code>PyTorch</code> but having trouble with <code>padding</code>. My code for for relevant platforms are as follow:</p>
<p><strong>TensorFlow</strong></p>
<pre><code>conv1 = tf.layers.conv2d(
                inputs=input_layer,
                filters=32,
                kernel_size=[5, 5],
                padding="same",
                activation=tf.nn.relu,
                name = "conv1")
</code></pre>
<p><strong>PyTorch</strong></p>
<pre><code>conv1 = nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2)
</code></pre>
<p>I have few questions:</p>
<ol>
<li>Are the above codes equivalent?</li>
<li>How many <code>padding</code>s are added on left/right/top/bottom if we use <code>same</code> padding in <code>tensorflow</code>?</li>
<li>How many <code>padding</code>s are added on left/right/top/bottom if we use <code>padding=2</code> in <code>pytorch</code>?</li>
<li>If the above two code snippets are not equivalent then how can we make the same <code>conv</code> layer?</li>
</ol>
<p>Thanks in advance.</p>

ptrblck: <ol>
<li>Not entirely, since the TF implementation seems to add a relu to the conv layer while this is not the case in PyTorch.</li>
<li>I guess 2 on each side, but you would have to confirm it in TF.</li>
<li>2 on each side.</li>
</ol>
Mode write file

Achrouf: <p>Hello, I have the following error that I cannot resolve:<br>
“Could not find a format to write the specified file in %s mode” % modename<br>
ValueError: Could not find a format to write the specified file in single-image mode</p>
<p>the code is :<br>
for f in  jpgFiles:<br>
image=misc.imread(f)<br>
img=misc.imresize(image,(256,256,3))<br>
print(img.shape)<br>
pred=pipeline.predict(img)<br>
output=visualizer(img,pred)<br>
imageio.imwrite(args.outputPath,output)</p>
<p>Help please</p>

ptrblck: <p>This error seems to be raised by <code>imageio</code>, which cannot save the <code>output</code> in a floating point format, so you might need to transform it to an “image format” in <code>uint8</code> before calling <code>imageio.imwrite</code>.</p>
Device memory allocation in pytorch

muart47: <p>Can you, please, explain how device memory allocation is working in pytorch? In this example <a href="https://github.com/sniklaus/pytorch-extension/blob/master/src/HadamardProduct_kernel.cu" rel="nofollow noopener">https://github.com/sniklaus/pytorch-extension/blob/master/src/HadamardProduct_kernel.cu</a> he runs only kernel, but sending memory of input1 and input2 to the GPU is absent, just like loading data of output back to host. Does pytorch use UVA(unified virtual addressing) or something like it?</p>

smth: <p>The memory of input1 and input2 is already on GPU.</p>
<p>If you see <a href="https://github.com/sniklaus/pytorch-extension/blob/master/src/HadamardProduct_kernel.cu#L25-L26">https://github.com/sniklaus/pytorch-extension/blob/master/src/HadamardProduct_kernel.cu#L25-L26</a></p>
<p>The input is a <code>THCudaTensor</code>, which is a GPU tensor.</p>
<p>PyTorch does not use UVA addressing.</p>

Zinuo: <aside class="quote no-group" data-username="smth" data-post="2" data-topic="7755">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar"> smth:</div>
<blockquote>
<p>PyTorch does not use UVA addressing.</p>
</blockquote>
</aside>
<p>Any updates from 2017?</p>
Determine the size of feature maps in generator and discriminator?

ljeonjko: <p>Hi all!</p>
<p>I’m following this tutorial. <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">DCGAN PyTorch</a>, but I’m working with “image-like” tensors containing geographical data. The dimensions of the tensors are<br>
<code>[batch_size, 1 (as the channel is just a single float value, latitudes, longitudes]</code>.</p>
<p>How do I determine the size of feature maps once I get to the generator and discriminator classes?</p>
<p>Thank you!</p>

ptrblck: <p>Assuming you would like to calculate the shape of some intermediate output activations you could either calculate the output shape using the shape formula for e.g. <code>nn.Conv2d</code>, which is given in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html?highlight=conv2d#torch.nn.Conv2d">docs</a> or alternatively you could add <code>print</code> statements into the <code>forward</code> method and print out the shapes.</p>
Inputting dataloader and model to multiprocessing.spawn

marioo: <p>Hi,</p>
<p>I am exploring the use of DistributedDataParallel to train on two GPUs. In all the examples I have found the DataLoader and Model are instanciated separately at each rank. Can I create the model and dataloader outside of the multiprocessing.spaw function and pass them as input arguments to multiprocessing.spawn? I mean something like this:</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp

loader = DataLoader(dataset, batch_size=128, shuffle=True)
model = MyNet()

if __name__ == '__main__':
    mp.spawn(fit, args=(model, devices, loader), nprocs=len(devices), join=True)
</code></pre>
<p>In this case, will be a new model and dataloader created at each rank without shared memory?<br>
I would like to use the same iterable for the dataloader at each rank so each GPU works on the same epoch. Is this possible?</p>
<p>I know that it would be better to use a DistributtedSampler, but I am working with graphs and I cannot do so due to the irregular structure of my data. If you know other similar option I would be nice to hear.</p>

rvarm1: <p>I think that the model’s parameter tensors will have their data moved to shared memory as per <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing best practices — PyTorch 1.9.0 documentation</a>, so you’d essentially be doing Hogwild training and this could cause issues with DistributedDataParallel as usually the model is instantiated individually on each rank. Is there a reason you can’t simply create the model within each subprocess?</p>
<p>Regarding the question about the data loader, I think if the dataloader is serializable this should work, but it would result in each worker training on the same data, and thus DDP would not really help as you’re using the same data and different model. Is it possible for you to do something like pass in a file handle/file name to each worker, and possibly an index/offset so you can shard your data across multiple workers?</p>
Specify parameter sharing pattern in nn.Parameter

Chen-Cai-OSU: <p>Hello,</p>
<p>For some reason, I woud like to use the upper triangular tensor as model parameters. I used the <code>self.L = nn.Parameter(torch.rand((d, d)).triu())</code> to specify the parameters during initialization but this is not right as the total parameter number is still d*d. What I want is to fix the lower part to be 0 all the time. I think I have to unregister those numbers but I didn’t find a clean way to do so. I appreciate if anyone can help me.</p>
<p>Thank you very much!</p>

KFrank: <p>Hi Cai!</p>
<aside class="quote no-group" data-username="Chen-Cai-OSU" data-post="1" data-topic="126031" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chen-cai-osu/48/31025_2.png" class="avatar"> Chen-Cai-OSU:</div>
<blockquote>
<p>What I want is to fix the lower part to be 0 all the time. I think I have to unregister those numbers but I didn’t find a clean way to do so.</p>
</blockquote>
</aside>
<p>An entire tensor is a single <code>Parameter</code>.  You can’t unregister just some<br>
elements of a tensor.</p>
<p>Your best bet will be to “freeze” the lower triangle at zero by zeroing out<br>
that part of the tensor’s gradient.  Here’s a post from <a class="mention" href="/u/ptrblck">@ptrblck</a> that shows<br>
how to do this:</p>
<aside class="quote quote-modified" data-post="2" data-topic="71427">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/creating-a-triangular-weight-matrix-with-a-hidden-layer-instead-of-a-fully-connected-one/71427/2">Creating a Triangular weight matrix with a hidden Layer instead of a fully connected one</a> 
  </div>
  <blockquote>
    You could initialize the weight matrices in your desired shape, e.g. by using torch.triu, and then zeroing out the gradients at the same locations. 
Here is a small code snippet for this use case, which could be a good starter: 
# Init
def triu_init(m):
    if isinstance(m, nn.Linear):
        with torch.no_grad():
            m.weight.copy_(torch.triu(m.weight))

# Zero out gradients
def get_zero_grad_hook(mask):
    def hook(grad):
        return grad * mask
    return hook

# Setup
model = nn…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

Chen-Cai-OSU: <p>This is exactly what I need. Thank you so much!!</p>
Troubleshooting Cuda Memory Overload

J_Johnson: <p>I’m training a model that splits the layers to separate GPUs via model parallel. “cuda:0” starts at around 600MB and slowly keeps etching up until it causes a cuda memory error(12GB per GPU). The other GPU stays under 600MB.</p>
<p>I’ve tried sys.getsizeof(object), which doesn’t do much for tensors. I also now am running empty_cache and gc.collect. Still getting this issue.</p>
<p>Is there a method I can use to troubleshoot where the memory slippage is?</p>

J_Johnson: <p>I found some tools here: <a href="http://pytorch.org/docs/stable/cuda.html" rel="noopener nofollow ugc">pytorch.org/docs/stable/cuda.html</a></p>
<p>under “Memory management”.</p>
<p>Figured out my problem was that I was using the outputs for some statistical calculations and forgot to put them back to cpu before performing those. Somehow that was causing the issue of tensors being duplicated on the device they resided. Running .to(‘cpu’) on all before sending those to stats fixed it pronto.</p>
<p><em>Edit: This ended up just putting the memory overflow on the cpu. I changed it to <code>.detach()</code> with no need to remove from the GPU and that seems to resolve the issue.</em></p>