akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Oddly getting 32 dimensions when I resize a 3D tensor?

talhak: <p>I’m trying to resize a tensor through the <code>Resize</code> function of <code>torchvision.transforms</code> module; but - as the tile is self-descriptive - I’m getting a 32D output when I apply the resize function of a 3D tensor. I’ve added the code snippet below. The shape of the input and the output are <code>(32, 32, 3)</code> and <code>(32, 75, 75)</code>, respectively. Could you please guide me through the thing I miss?</p>
<pre><code class="lang-auto">from torchvision import datasets, models, transforms as T
tensor_x = torch.Tensor(X)  # converting the NumPy array to Tensor
resize_fn = T.Resize((75, 75))
out = resize_fn(tensor_x)
</code></pre>
<p>Also, I’d like to ask why the transforms that I’ve provided did not apply to the dataset below. My aim is to resize the samples of <code>CIFAR-10</code> from <code>(32, 32, 3)</code> to <code>(75, 75, 3)</code>. But I still get the original shape for the samples, <code>(32, 32, 3)</code>.</p>
<pre><code class="lang-auto">from torchvision import datasets, models, transforms as T

m_transform = T.Compose([
    T.Resize((75, 75)),
    T.ToTensor()
])

train_set = datasets.cifar.CIFAR10('/PyTorch-Datasets', train=True, download=False), transform=m_transform)
test_set = datasets.cifar.CIFAR10('/PyTorch-Datasets', train=False, download=False), transform=m_transform)
</code></pre>

ptrblck: <p><code>Resize</code> expects a channels-first tensor, so it should have the shape <code>[3, 32, 32]</code> in your case as seen here:</p>
<pre><code class="lang-python">tensor_x = torch.randn(3, 32, 32)
resize_fn = T.Resize((75, 75))
out = resize_fn(tensor_x)
</code></pre>
BERT Inference fails to run on GPU

Dark_Knight: <p>I have a setup something like this:</p>
<p>model = BERT()<br>
model.to(device)<br>
model.eval()<br>
result = []<br>
for data in batch:<br>
input_ids = data.to(device)<br>
token_ids = data.to(device)<br>
attention_mask = data.to(device)<br>
x = model(input_ids, token_ids, attention_mask)<br>
y = x.cpu().detach().numpy()<br>
result+= covert_to_string(y)</p>
<p>I don’t see GPU being used, I checked using nvidia-smi also. The GPU memory is about 75% being used but not the GPU cores for computation. What am I doing wrong?</p>

ptrblck: <p>How are you checking if the GPU is used for computation? To validate its usage you could check the <code>util.</code> in <code>nvidia-smi</code> or check the used CUDA kernels in a profiler. In general, if the model, input and output data are all on the GPU, the computation will also be performed on the GPU as PyTorch won’t move data to the CPU without explicit <code>to()</code> calls.</p>
Custom resnet50 weights on pytorch faster rcnn backbone

Shantanu_Ghosh: <p>Hi,</p>
<p>I want to detect heart using stanford chestxray dataset with the help of torchvision.models.detection.fasterrcnn_resnet50_fpn. Now when i set torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True) it will have the pretrained weights which i dont want. For my problem, i have already trained a resnet 50 model using stanford chestxray dataset and i want those weights of the checkpoints as the weights of the backbone for the faster rcnn object detector. Please help me how to use them in the torchvision model. Thanks</p>

ptrblck: <p>You could either try to load your <code>state_dict</code> into the backbone via accessing <code>model.backbone</code> and changing some keys in the <code>state_dict</code> if needed or alternatively create a new model as seen <a href="https://github.com/pytorch/vision/blob/1c63096033d8e26b4a4a10180ebb267c9f784d59/torchvision/models/detection/faster_rcnn.py#L394-L401">here</a> with your custom backbone.</p>
Comparison of torch stft with librosa

Nabarun_Goswami: <p>Hi,</p>
<p>I was evaluating the stft function in pytorch vs librosa. I found that the precision is only upto 2 decimal places. I am wondering if that can be improved to atleast 4-5 decimal places.</p>
<p>Also, regarding the speed of execution,</p>
<pre><code class="lang-auto">librosa is ~2x faster than pytorch on CPU (Didn't expect this, thought CPU times should be similar.)
pytorch is ~4x faster than librosa on GPU (Expected this.)

Rig: TitanX pascal, 4-core CPU, pytorch 0.5.0a0+8fbab83
</code></pre>
<p>Do these numbers look ok? Or is there a possibility I am doing something wrong?</p>
<p>Regards<br>
Nabarun</p>

NoahDrisort: <p>I found an answer from stackoverflow said that “The difference is from the difference between their default bit. NumPy’s float is 64bit by default. PyTorch’s float is 32bit by default.”</p>
Implement custom loss functions

Najeh_Nafti: <p>I would like to implemnt the below loss function based on WGAN-gp loss function. How can I do that?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8c1189176d1106ac1cd471fcccaf77f2e88ac9d.png" alt="encoder" data-base62-sha1="sDXcpWjkQik4NN0bdBUzzRx2NWJ" width="673" height="78"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07f1b949e11839d6668fc4fa649d2b6b46945c05.png" alt="Decoder" data-base62-sha1="18hes4RqDvqdJtZvTJujOjrfN5P" width="633" height="168"></p>
Wrong link for AlexNet Paper

LoveU3tHousand2: <p><strong><a href="https://pytorch.org/vision/stable/models.html" class="inline-onebox" rel="noopener nofollow ugc">torchvision.models — Torchvision 0.11.0 documentation</a></strong><br>
<a href="https://arxiv.org/abs/1404.5997" class="inline-onebox" rel="noopener nofollow ugc">[1404.5997] One weird trick for parallelizing convolutional neural networks</a><br>
it’s about parallism.</p>
cuDNN issue when weight.requires_grad=False

Ardeal: <p>Hi,<br>
I wrote the following code to customize a new conv2d layer.</p>
<p>If I set <code>weight.requires_grad=True</code>, there is no issue.<br>
If I set <code>weight.requires_grad=False</code>, The new layer works for forwarding, but I experience error when run loss.backward:</p>
<p>if <code>weight.requires_grad=True</code>, there is no cuDNN issue. why is there cuDNN issue when <code>weight.requires_grad=False</code>?</p>
<pre><code class="lang-python">File "train.py", line 229, in train
    loss.backward()
  File "/root/anaconda3/lib/python3.8/site-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/root/anaconda3/lib/python3.8/site-packages/torch/autograd/__init__.py", line 147, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution
</code></pre>
<pre><code class="lang-python">    def forward(self, *inputs):
        if len(inputs) == 4:
            x, weight, bias, add_tensor = inputs
        else:
            if self.bias_term:
                x, weight, bias = inputs
            else:
                x, weight = inputs
                bias = None
        x = F.conv2d(
            x,
            weight,
            bias,
            self.strides,
            self.padding,
            self.dilation,
            self.groups,
        )
</code></pre>

ptrblck: <p>Could you post an executable code snippet to reproduce the issue as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!<br>
The issue was oberved, but it disappeared automatically. Thank you anyway!</p>
Override AMP casting during bfloat16 training

gahdritz: <p>I’m trying to train a PyTorch model on an A100 with bfloat16 mixed precision.</p>
<p>torch AMP maintains a list of operations deemed numerically unstable in float16 (<a href="https://pytorch.org/docs/stable/amp.html#ops-that-can-autocast-to-float16" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision package - torch.cuda.amp — PyTorch 1.10.1 documentation</a>) and automatically casts the inputs to those operations to float32, even if you’re running AMP with bfloat16. Is there a way to globally tweak that list to prevent inputs to certain operations from being cast to full precision without manually enabling/disabling AMP every time they’re called? I’d like e.g. all softmax calls throughout my model to run using bfloat16.</p>

ptrblck: <p>I’m unsure if there is an easy way to “tune” the autocast list (<a class="mention" href="/u/mcarilli">@mcarilli</a> might know), but for now you could disable <code>autocast</code> for the desired operations, transform your data to <code>bfloat16()</code> and call the operations directly.</p>
Pruning inference time and memory usage

fnak: <p>I understand from this <a href="https://github.com/pytorch/tutorials/pull/605#issuecomment-585994076" rel="noopener nofollow ugc">discussion</a> that pruning API does not provide any memory benefits, nor improve inference time.</p>
<p>I am using the following code to prune my network:</p>
<pre><code class="lang-auto">for module in model.modules():
      if isinstance(module, nn.Conv2d):
          prune.l1_unstructured(module, 'weight', amount=0.5)
          prune.remove(module, 'weight')
      elif isinstance(module, torch.nn.Linear):
          prune.l1_unstructured(module, name='weight', amount=0.5)
          prune.remove(module, 'weight')
</code></pre>
<p>Is there anything that can be done to improve the memory footprint and inference time and not just prune for the sake of pruning?<br>
PS: I can see that a tar version of my model did indeed has a smaller disk size. and the number of parameters are smaller. But this is useless if the computation is the same. no?</p>
Compute CrossEntropyLoss per sentence in MLM task

ThomasGk: <p>Hello everyone. I am using a HuggingFace model, to which I pass a couple of sentences. Then I am getting the logits, and using PyTorch’s <a href="https://pytorch.org/docs/1.9.1/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">CrossEntropyLoss</a>, to get the loss. The problem is as follows:</p>
<p>I want the loss of each <em>sentence</em>. If I have 3 sentences, each with 10 tokens, the logits have size <code>[3, 10, V]</code>, where <code>V</code> is my vocab size. The labels have size <code>[3, 10]</code>, basically the correct labels for each of the 10 tokens in each sentence.</p>
<p>How can I get the cross entropy of each sentence then? If I do <code>reduction='mean'</code>, I am going to get the overall mean loss (1 number). If I use <code>reduction='none'</code>, then I get one number <em>for each</em> token, so basically the loss of each single token. The code I am using is</p>
<blockquote>
<p>loss_fct = nn.CrossEntropyLoss(reduction=‘mean’)  # or ‘none’<br>
masked_lm_loss = loss_fct(outputs.logits.cpu().detach().view(-1, V), target_ids.view(-1))</p>
</blockquote>
<p>Perhaps I need to define the views differently in the second line?<br>
Thanks in advance.</p>

mMagmer: <p>hi<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/0/007dfe9dfbaa60bd44d28102d61c0d467ae78580.png" alt="image" data-base62-sha1="4lWpKeVcTxUjSzfvzqqvw1ius0" width="685" height="277"></p>
<p>you can use <strong>or</strong> form,</p>
<pre><code class="lang-auto">loss_fct = nn.CrossEntropyLoss(reduction=‘none’)
masked_lm_loss = loss_fct(torch.transpose(outputs.logits.cpu().detach(), 1, 2), target_ids)
</code></pre>
<p>and then mean over last dim</p>
<pre><code class="lang-auto">masked_lm_loss.mean(-1)
</code></pre>
<p>you should have 3 positive losses, one for each <em>sentence</em>.</p>

ThomasGk: <p>Thanks. Can you please explain the idea behind this? Why does swapping the 1st and 2nd dimension get us what we want? We end up with the logits having shape <code>[3, V, 10]</code>.</p>

mMagmer: <p>you have to optional shape for <code>CrossEntropyLoss</code> input and target.<br>
commonly use form is to have <code>NxC</code> logit and <code>N</code> “int” vector as target.<br>
other optoin is to have <code>NxCx···</code> as logit (sum over second dim is equal to 1) and have tensor <code>Nx···</code> as target.</p>

ThomasGk: <p>Alright, I see, thank you!</p>
Pretrained Weights for RESNET34

chipakins: <p>I am trying to validate some neural net hardware, and need the weights for RESNET34 which have been pretrained on ImageNet. (Actually I need all the trainable or adjustable parameters from a pretrained RESNET34)<br>
Does anyone know how I can get these weights?<br>
Are they available in a file from some source, or if there is a way to extract the weights from a pretrained model.</p>
RuntimeError second time running loaded module

guillefix: <p>Hello,</p>
<p>I’ve saved a model usint torch.jit.trace, and then loaded it using torch.jit.load.</p>
<p>The first time I run the model it outputs correctly, but the second and subsequent times, it throws this error</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-25-946d12987c4d&gt; in &lt;module&gt;
----&gt; 1 out = model(inputs)

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

RuntimeError: _Map_base::at
</code></pre>
<p>Any idea what could be causing it?</p>
<p>Checked with torch 1.8.0 and torch 1.10.0. Happens both on CPU and GPU</p>

guillefix: <p>As the second pass apparently does the optimization, is there a way to disable this optimization, as that’s where it seems to be failing?</p>
Implementation of Dropout for sparse input

mahsa: <p>Hi,<br>
I want to implement dropout for sparse input. I know that the implementation in tensorflow is as follow, but I don’t know if there is anyway for implementation in pytorch (the source of the following code is <a href="https://github.com/tkipf/gcn/blob/master/gcn/layers.py" rel="nofollow noopener">here</a>)</p>
<pre><code class="lang-auto">def sparse_dropout(x, keep_prob, noise_shape):
    """Dropout for sparse tensors."""
    random_tensor = keep_prob
    random_tensor += tf.random_uniform(noise_shape)
    dropout_mask = tf.cast(tf.floor(random_tensor), dtype=tf.bool)
    pre_out = tf.sparse_retain(x, dropout_mask)
    return pre_out * (1./keep_prob)
</code></pre>

KuanS: <p>Hi<br>
I am doing the same thing currently<br>
here is my solution</p>
<pre><code>class SparseDropout(torch.nn.Module):
    def __init__(self, dprob=0.5):
        super(SparseDropout, self).__init__()
        # dprob is ratio of dropout
        # convert to keep probability
        self.kprob=1-dprob

    def forward(self, x):
        mask=((torch.rand(x._values().size())+(self.kprob)).floor()).type(torch.bool)
        rc=x._indices()[:,mask]
        val=x._values()[mask]*(1.0/self.kprob)
        return torch.sparse.FloatTensor(rc, val)</code></pre>

Juan_Espinosa_Reinos: <p>Hi KuanS, thanks for the solution. Currently I am working on solving the same function. Sadly, the proposal is not working as a simple function. Do you have any idea on how to access the <code>rc=x.indices()[:,mask]</code> when the x features is not a sparse matrix? In the case of transforming it to a sparse matrix then the indices are in 2nd dimension. Moreover, both functionalities <code>x._values()</code> and <code>x._indices()</code> must be deprecated because an error appears.Thank you in advance</p>
RuntimeError: "upsample_nearest2d" not implemented for 'Int'

111220: <p>I wanted to learn about the use of the up sampling API, so I built a matrix, and I sampled it up:</p>
<p>This is my code:</p>
<pre><code class="lang-auto">
model=torch.nn.Sequential()
model.add_module('upsamling2d-1',torch.nn.Upsample(size=(4,4)))

Input=np.asarray([[1,2],[3,4]])
Input=Input.reshape(1,1,2,2)
Input=torch.from_numpy(Input)
print(Input)

yhat=model(Input)
yhat=yhat.reshape((4,4))
print(yhat)
</code></pre>
<p>And this is the error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/8298e890d8bc45dab072e9fdef851d3226844954.png" data-download-href="https://discuss.pytorch.org/uploads/default/8298e890d8bc45dab072e9fdef851d3226844954" title="捕获"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/8298e890d8bc45dab072e9fdef851d3226844954.png" alt="捕获" data-base62-sha1="iDjFUDjfKtu3OCajTBpVbmsGtAU" width="690" height="287" data-dominant-color="333135"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">捕获</span><span class="informations">1238×515 29.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>how should i handle it</p>

111220: <p>OK，I found the solution:</p>
<pre><code class="lang-auto">Input=Input.float()
</code></pre>

adizhol: <p>This is a solution, but why does pytorch force us to increase the memory foot print of an int\byte tensor when doing a simple nearest neighbour interpolation?</p>

ptrblck: <p>I guess the main reason is that this function is not (yet) implemented for int/byte types, since these types cannot be used for gradient calculation.<br>
If you think this feature is not an edge case and would be used by a lot of users, we would be more than happy to accept contributions. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

zzzf: <p>I find it now works for <code>torch.uint8</code> but not for <code>torch.long</code> . Is there a reason why torch only allows for short int but not the long one?</p>

ptrblck: <p><code>unit8</code> is the standard image format, which is why interpolation is supported on this integer format. My guess about the lack of <code>long</code> support would be that it might not have been requested yet (so wasn’t implemented) and would increase the binary size as a new <code>dtype</code> is added to the upsample methods (apparently for an edge case).<br>
If your data isn’t overflowing in <code>float</code>, you could transform if before applying the interpolation.</p>
Control batch size in collate function

talhaanwarch: <p>I am reading video and split them into batched. I cant fit a whole video in my batch. So i split them using collate function. Shape of video is <code>(9*32,3,224,224)</code>.</p>
<p>Output shape from data loader after using collate function is<br>
<code>torch.Size([9, 32, 3, 224, 224])</code> where index 0 is batch. I if pass this bath data , if will get memory error. Is there a way to send a batch of 3 .?</p>
<p>Here is code</p>
<pre><code class="lang-auto">dl=DataLoader(DataReader(df,train_aug), batch_size =1,collate_fn=collate_fn)
</code></pre>
<p>here is custom dataset</p>
<pre><code class="lang-auto">class DataReader(Dataset):
    def __init__(self, dataset, transform=None):
        self.df = dataset
        self.transform = transform

    def __getitem__(self, index):
        x=self.df['path'][index]
        y=self.df['pulse'][index]
        x=readframes(x)
        x1=[]
        for i in x:
            img=cv2.cvtColor(src=i, code=cv2.COLOR_BGR2RGB)
            img=train_aug(image=img)['image']
            x1.append(img)
        return torch.stack(x1),np.float32(y)
        
    def __len__(self):
        return len(self.df)
</code></pre>
<p><code>readframes</code> is a function to extracrt frames of a video.</p>
<pre><code class="lang-auto">def reshaped(i,bs=32):
    i=i[len(i)%bs::].reshape(len(i)//bs,-1,3,224,224)
    return i
def collate_fn(batch):#batch is a tuple. where first value is batch.,2nd is ether array or label
    bs=32
    data=np.concatenate([reshaped(i[0]) for i in batch])  
    label=np.concatenate([[i[1]]*(len(i[0])//bs) for i in batch])
    data,label=data[0:4],label[0:4]
    return torch.tensor(data,dtype=torch.float),torch.tensor(label)
</code></pre>

ptrblck: <p>The <code>collate_fn</code> will get all loaded samples and will create the batch. Removing already loaded and processed samples that late sounds a bit wasteful. Wouldn’t it work to not load the undesired frames in the first place?</p>

talhaanwarch: <p>my gpu memory (8GB) does not fit even a single batch, so i want to split via collate function, and convert them to sub batches and pass the sub batches.</p>

ptrblck: <p>I guess your GPU is running out of memory during the model training, not while creating the batch.<br>
If so, you could split the batch into chunks in the <code>DataLoader</code> loop and perform multiple training steps.<br>
Alternatively, you could only load the desired number of samples in the <code>Dataset</code> and create a batch with the smaller number of samples. Removing samples in the <code>collate_fn</code> could work, but would be wasteful as previously described.</p>
Incomplete Beta Function

Tom_Ginsberg: <p>Hi I’m trying to find a good way to implement the incomplete Beta function B_z(a,b) or better yet the incomplete Beta regularized function I_z(a,b) to work with autograd. The need for this is essential to compute the CDF of the StudentT distribution (which is not currently supported in torch.distributions).</p>
<p>I see that there is a <a href="https://jax.readthedocs.io/en/latest/_autosummary/jax.scipy.special.betainc.html" rel="noopener nofollow ugc">JAX implementation</a> and I’m wondering how trivial/not it would be to directly use this within PyTorch and ideally still get GPU performance. Also, this call is only required for computing loss so it’s not a huge bottleneck and could potentially just be done on CPU.</p>
<p>If anyone has some thoughts/suggestions on solving this problem that would be appreciated.</p>
<p>Thanks</p>

thomassutter: <p>Hi Tom (<a class="mention" href="/u/tom_ginsberg">@Tom_Ginsberg</a>),</p>
<p>Could you find a way to implement the incomplete Beta or incomplete Beta regularized function?<br>
I am also looking for an implementation that works with autograd.</p>
<p>Thanks for your help</p>
<p>Best,<br>
Thomas</p>
Pass dataset to spawn multiprocessing - CUDA

kyriakosschwarz: <p>This minimal example:</p>
<pre><code class="lang-auto">dataset = TensorDataset(torch.randn(20,15, 100), torch.randn(20,15, 1))

def test_mp(dataset):
    print("hello")
    
import torch.multiprocessing as mp
mp.set_start_method("spawn", force=True)

queue = mp.Queue()
q_processes = []

q_processes.append(mp.Process(target=test_mp, args=(dataset)))

q_processes[0].start()
q_processes[0].join()
</code></pre>
<p>Gives me the error:<br>
<code>ValueError: bad value(s) in fds_to_keep</code></p>
<p>How can I correctly pass this dataset to <code>mp.Process</code> ? I use <code>spawn</code> because of <code>CUDA</code>.</p>
<pre><code class="lang-auto">torch: 1.9.1
CUDA: 11.1 - GeForce GTX 1080 Ti
python 3.9.9
</code></pre>
<p>Thanks a lot in advance!</p>
CGAN with images as labels?

ljeonjko: <p>Hello!</p>
<p>I’m following a CGAN tutorail by Aladdin Persson. However, since I’m working with satellite data, I have no labels <em>per se</em>. In lieu of labels, I have “constrained images”: they corresponding images (dupliactes) with some of the pixels removed (let’s say 5% of pixels removed, so that the CGAN doesn’t alter the 95% of the remaining pixels, just the missing ones).  All images are 2 channel (grayscale and alpha).</p>
<p>I’ve managed to edit the dataset to accept images as labels. However, upon executing the code, it returns</p>
<blockquote>
<p>RuntimeError: Expected tensor for argument <span class="hashtag-raw">#1</span> ‘indices’ to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead (while checking arguments for embedding)</p>
</blockquote>
<p>Is there a remedy for this? Could you recommend me any other solution or perhaps a book/tutorial that expands on the embedding part of the code?</p>
<p>Thanks!</p>
Weights stop updating after manual update?

rmorgan: <p>I have a parallelized training setup where I have instances of the same model on different machines. After a set of batches goes through the model, I want to manually set each parameter’s weights to be the average of the all instances of that parameter’s weights.</p>
<p>Following other posts about manually updating weights, my training loop looks like this:</p>
<pre><code class="lang-python">for i_batch, sample_batched in enumerate(train_dataloader):
    optimizer.zero_grad()
    output = model(sample_batched['data'])
    loss = loss_function(output, sample_batched['label']) 
    loss.backward()
    optimizer.step()


nets = [...]  # List of instances of the same model loaded from saved state_dicts.
net_params = [x.named_parameters() for x in nets]

with torch.no_grad():
    for ((name, param), *q) in zip(model.named_parameters(), *net_params):
        param_vals = np.array([p[1].data.numpy() for p in q])
        average_param_val = np.mean(param_vals, axis=0)
        values = torch.ones(param.shape) * average_param_val
        param.copy_(values)
</code></pre>
<p>After this point my training loop continues with more input data, but I notice that the training and validation accuracy no longer change, which to me indicates that the weights are no longer being updated by the optimizer after I do the first manual update.</p>
<p>Does anything seem off with how I am updating the values of the parameter weights that would cause them to stop being updated by the optimizer?</p>

ptrblck: <p>I think your loop is called only once as iterating a generator would exhaust it and you would need to refresh it or are you re-creating <code>net_params</code> before each update?<br>
Add a <code>print</code> statement to the custom update loop and make sure the parameters are indeed being updated.</p>

rmorgan: <p>Thanks for taking a look, this loop happens within a larger loop that instantiates a dataloader for each file containing my data.</p>
<p>The problem was actually that I needed all instances of my network to start with the same weight values for averaging the parameters to work as part of a distributed training setup. So in my case, setting the random seed at instantiation solved this.</p>
Requires_grad directly after .detach()

moka: <pre><code class="lang-auto">foo = foo.detach()
foo.requires_grad = True
</code></pre>
<p>I came across this and am trying to understand what it is doing. Does this actually make sense? Does it mean that foo will be disconnected from the current graph and connected again “at a different location” once used again?</p>
<p>Is there a cleaner way to go about this?</p>

moka: <p>Answering my own question: The snippet is within a loop, at the end of the iteration. So I assume it’s purpose is to only back-propagate through one iteration of the loop and not all of them at once.</p>
Find "bounding box" around ones in batch of masks

LeanderK: <p>So I have a batch of masks (values 0.0/1.0) with dimension (b,h,w) where b is the batch-dim.</p>
<p>For each mask, i need the coordinate of the the bounding box containing all the ones. I thought about using various operators (torch.where/torch.nonzero) but I can’t get it to work in a batched setting.</p>

LeanderK: <p>Ok, I think I’ve solved it:</p>
<pre><code class="lang-auto">#get batched bounds for pytorch
def get_bounds_t(img):
    b,h,w = img.shape[:3]
    rows = torch.any(img, axis=2)
    cols = torch.any(img, axis=1)
    rmins = torch.argmax(rows.float(), dim=1)
    rmaxs = h - torch.argmax(rows.float().flip(dims=[1]), dim=1)
    cmins = torch.argmax(cols.float(), dim=1)
    cmaxs = w - torch.argmax(cols.float().flip(dims=[1]), dim=1)

    return rmins, rmaxs, cmins, cmaxs
</code></pre>
In C++/CUDA extension, how do we include the py wrapper file into setup.py as well?

zfzhang: <p>Hi,<br>
In the official C++/CUDA extension tutorial (<a href="https://github.com/pytorch/extension-cpp" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/extension-cpp: C++ extensions in PyTorch</a>),  setup.py only installs the c++ part of the extension into python’s site-packages directory, leaving the lltm.py wrapper file in local directory.<br>
For usage, we will have to refer to this lltm.py file locally (e.g. <code>import cuda.lltm as LLTM</code>, where cuda is the local folder name), which is not ideal for package distribution.<br>
Is there a way to include lltm.py into the extension installation as well?</p>
<p>The setup.py for cuda version looks like this:</p>
<pre><code class="lang-auto">from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension

setup(
    name='lltm_cuda',
    ext_modules=[
        CUDAExtension('lltm_cuda', [
            'lltm_cuda.cpp',
            'lltm_cuda_kernel.cu',
        ]),
    ],
    cmdclass={
        'build_ext': BuildExtension
    })
</code></pre>
<p>Thanks!</p>

zfzhang: <p>After reading the document of setup script (<a href="https://docs.python.org/3/distutils/setupscript.html" class="inline-onebox" rel="noopener nofollow ugc">2. Writing the Setup Script — Python 3.10.1 documentation</a>), i modify the setup.py file (for cuda folder) as follows, and we can import lltm as installed module :</p>
<pre><code class="lang-auto">from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension

setup(
    name='lltm_cuda',
    py_modules = ['lltm'],
    ext_modules=[
        CUDAExtension('lltm_cuda', [
            'lltm_cuda.cpp',
            'lltm_cuda_kernel.cu',
        ]),
    ],
    cmdclass={
        'build_ext': BuildExtension
    })
</code></pre>
Why backward ops are in another CPU thread?

BruceDai003: <p>I’m currently debugging a OCR model, it seems that there might be some issue related to CPU threads synchronization. I used pytorch profiler to check that in my CPU process, there are 2 threads, one running the forward ops, optimizer.zero_grad(), and later the optimizer.step() stuff in each iteration; another running the backward ops( when you call loss.backward()).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/3656da65c3ee69f618f51b54664e0c132f0b5d1e.png" data-download-href="https://discuss.pytorch.org/uploads/default/3656da65c3ee69f618f51b54664e0c132f0b5d1e" title="pytorch_profiler"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/3656da65c3ee69f618f51b54664e0c132f0b5d1e_2_690x215.png" alt="pytorch_profiler" data-base62-sha1="7KHREo5x1nlBX0Wedr53RWDKbmu" width="690" height="215" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/3656da65c3ee69f618f51b54664e0c132f0b5d1e_2_690x215.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/3656da65c3ee69f618f51b54664e0c132f0b5d1e_2_1035x322.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/3656da65c3ee69f618f51b54664e0c132f0b5d1e_2_1380x430.png 2x" data-dominant-color="E7E9E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_profiler</span><span class="informations">2324×727 117 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As my debug goes, I’m thinking if it is possible to only use a single CPU thread to do this since there is no parallelism in my case between these 2 threads, they seem to be running consecutively.<br>
I checked the source code, seems like this is the starting point for backward function:<a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/python_engine.cpp#L152" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/csrc/autograd/python_engine.cpp at main · pytorch/pytorch · GitHub</a>, and later called into this function: <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/engine.cpp#L998" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/csrc/autograd/engine.cpp at main · pytorch/pytorch · GitHub</a>.<br>
It seems to me that this <code>init_local_ready_queue()</code> function as also documented will create another thread? Should I pass some <code>ready_queue</code> to it? And how?</p>
Different torchvision.resnet output depending on batch size in eval mode

michelemoretti: <p>Hi all,<br>
It seems like models in eval mode return different outputs depending on the batch size of the input provided.<br>
It is my understanding this is not expected.<br>
I already checked <a href="https://discuss.pytorch.org/t/restoring-models-when-batch-size-is-different/71340/4">here</a>, <a href="https://discuss.pytorch.org/t/resnet50-gives-different-outputs-with-respect-to-batch-size/111796">here</a> (and in other 4 posts but the new user limit limits me to 2 links) but the answer is not obvious to me.<br>
I created some dummy code to demonstrated the problem I’m having.<br>
Any Idea if this is expected or why it is happening?</p>
<p>torch version: 1.10.1<br>
python version: 3.9.7</p>
<pre><code class="lang-auto">import torch
import torchvision

model = torchvision.models.resnet18(pretrained=True, progress=True)
model.eval()

a = torch.ones((1, 3, 96, 96))
b = torch.ones((64, 3, 96, 96))

with torch.no_grad():
    a_out = model(a)
    b_out = model(b)

diff = torch.absolute(b_out[0] - a_out[0])
total_diff = torch.sum(diff)
average_diff = torch.mean(diff)
print(f"equal: {torch.equal(a_out[0], b_out[0])}")
print(f"rtol=0.00001: {torch.allclose(a_out[0], b_out[0])}")
print(f"rtol=0.0001: {torch.allclose(a_out[0], b_out[0], rtol=0.0001)}")
print(f"rtol=0.001: {torch.allclose(a_out[0], b_out[0], rtol=0.001)}")
print(f"rtol=0.01: {torch.allclose(a_out[0], b_out[0], rtol=0.01)}")
print(f"rtol=0.1: {torch.allclose(a_out[0], b_out[0], rtol=0.1)}")
print(f"total diff {total_diff}")
print(f"avg diff {average_diff}")
</code></pre>
<p>And this is the output:</p>
<pre><code class="lang-auto">equal: False
rtol=0.00001: False
rtol=0.0001: False
rtol=0.001: True
rtol=0.01: True
rtol=0.1: True
total diff 0.04874414950609207
avg diff 7.616273478561197e-07
</code></pre>
Generate text from tensor

Oscar_Garcia: <p>Hi. Is there any proper approach to generate text from a given tensor/embedding? Let’s say I have a tensor and I want to generate a sequence of N words from it.</p>
<p>Thanks.</p>

huahuanZ: <p>Can you explain more details?<br>
What’s the relation between the tensor and the text. Is there a LUT or neural network for the mapping?</p>

Oscar_Garcia: <p>I have a dataset of paris of word embeddings and that word definition in natural language. During training I use a NN to map word embeddings to definition embeddings. My question is, after that, how could I generate the sequence of words from the definition given those word embeddings. A trasnformer decoder would be enough?</p>
<p>Thanks.</p>

huahuanZ: <p>Correct me if I’m wrong. You want to <strong>generate</strong> sequences with a trained model. Is your model an auto-regressive one?<br>
Usually, we use language model to generate that. A dummy code would be like</p>
<pre><code class="lang-python">model = ...
# tokenizer, a.k.a., the vocabulary mapping text to integer
tokenizer = ...
# start flag of sequence
bos = '&lt;s&gt;'
seq = [bos]
# convert '&lt;s&gt;' to integer, usually 0
last_token_id = tokenizer.encode(bos)
hidden_state = None
for i in range(max_len):
    predict_token_id, hidden_state = model(last_token_id, hidden_state)
    last_token_id = predict_token_id
    seq.append(tokenizer.decode(predict_token_id.item()))
seq = ' '.join(seq)
</code></pre>
<p>Here is a GPT-2 example. <a href="https://huggingface.co/gpt2" rel="noopener nofollow ugc">gpt2 · Hugging Face</a></p>

Oscar_Garcia: <p>Thank you much.</p>
<p>Sorry but: here “hidden_state” is the embedding tesnor? Else, I think my doubt is missing (how to generate from a given tensor).</p>

huahuanZ: <p>No, hidden_states stores the historical states of model predictions. For example</p>
<p>if given ‘h e l l’ , the model would likely to predict the next one is ‘o’;<br>
if given ‘h e a v e’, it would likely to predict the next one is ‘n’.</p>
<p>For your case, if there isn’t any historical information, you can set hidden_state as None</p>
PyTorch (bug?): How to pass gradient through index_add?

Ryan_Burgert1: <p>I’m trying to backpropogate through a Tensor.index_add operation, but it breaks when index_add broadcasts vectors. Is this intended, or is this a bug in PyTorch?</p>
<pre><code class="lang-auto">#The following code does NOT throw an error:

m=torch.zeros(4)
m.requires_grad=True

i=[0,1,2,3]
v=[6,7,8,9]

i=torch.tensor(i).long()
v=torch.tensor(v).float()
v.requires_grad=True

print(m)
m=m.index_add(0,i,v)
print(m)
m.sum().backward()
print(v.grad)
print("DONE")

#The following code throws an error:

m=torch.zeros(4,4)
m.requires_grad=True

i=[0,1,2,3]
v=[6,7,8,9]

i=torch.tensor(i).long()
v=torch.tensor(v).float()
v.requires_grad=True

print(m)
m=m.index_add(0,i,v)
print(m)
m.sum().backward()
print(v.grad)
print("DONE")
</code></pre>
<p>That code, when run, gives the following output and error:</p>
<pre><code class="lang-auto">tensor([0., 0., 0., 0.], requires_grad=True)
tensor([6., 7., 8., 9.], grad_fn=&lt;IndexAddBackward&gt;)
tensor([1., 1., 1., 1.])
DONE
tensor([[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]], requires_grad=True)
tensor([[6., 6., 6., 6.],
        [7., 7., 7., 7.],
        [8., 8., 8., 8.],
        [9., 9., 9., 9.]], grad_fn=&lt;IndexAddBackward&gt;)
</code></pre>
<p>Strangely, it works fine - until we call the second <code>m.sum().backward()</code>. Before then, nothing appears to be any more wrong than the one above it.</p>
<pre><code class="lang-auto">ERROR: RuntimeError: expand(torch.FloatTensor{[4, 4]}, size=[4]): the number of sizes provided (1) must be greater or equal to the number of dimensions in the tensor (2)
</code></pre>
<p>Note: I also asked this question at <a href="https://stackoverflow.com/questions/70612357/pytorch-how-to-pass-gradient-through-index-add" rel="noopener nofollow ugc">Stack Overflow</a>, but right now it’s unanswered</p>
<p>A bit more context about my use case - I need to map a bunch of pixels from one image onto another image, where some pixels overlap (and in that case I’d like their sum). It’s like refracting light - I want parts of the image that are more squished because of a deformation to be brighter. This index_add function is a critical component, and I need it to broadcast across RGB values.</p>

ptrblck: <p>This sound like a valid issue and the workaround via:</p>
<pre><code class="lang-python">m=m.index_add(0,i,v.expand_as(m).contiguous())
</code></pre>
<p>seems to work. Would you mind creating an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> and link to this thread, please?</p>

Ryan_Burgert1: <p>Sure thing; I just posted the issue: <a href="https://github.com/pytorch/pytorch/issues/71155" class="inline-onebox" rel="noopener nofollow ugc">PyTorch bug: Cannot pass gradient through index_add · Issue #71155 · pytorch/pytorch · GitHub</a><br>
Thank you for your help!</p>
Get batch from collate_fn of Pytorch data loader

talhaanwarch: <p>I am reading video and split them into batched. I cant fit a whole video in my batch. So i split them using collate function. Shape of video is (128,3,224,224).<br>
Where 128 is frames, and 3 is channels (RGB).<br>
Output shape from data loader after using collate function is<br>
torch.Size([4, 32, 3, 224, 224]) where index 0 is batch. I if pass this bath data , if will get memory error. Is there a way to send a batch of 2 .?<br>
Here is the code for reproduce-ability purpose.<br>
<strong>Generate dummy data</strong></p>
<pre><code>features=[]
labels=[]
#here the data is in ram, but actually its in disk and i cant load all data to ram to manipulate. 
for i in range(5):
    features.append(np.random.randint(low=0, high=256, size=(128,3,224,224)))
    labels.append(np.random.randint(low=0, high=2))
</code></pre>
<p><strong>dataset</strong></p>
<pre><code>class DataReader(Dataset):
    def __init__(self,features,labels):
        self.features = features
        self.labels = labels

    def __getitem__(self, index):
        x=self.features[index]
        y=self.labels[index]
        return x,y
        
    def __len__(self):
        return len(self.features)
</code></pre>
<p><strong>dataloader</strong></p>
<pre><code class="lang-auto">def reshaped(i,bs=32):
    i=i[len(i)%bs::].reshape(len(i)//bs,-1,3,224,224)
    return i
def collate_fn(batch):
    data=reshaped(batch[0])
    print(data.shape)
    label=[batch[1]]*len(data)
    return data,label
dl=DataLoader(DataReader(features,labels), batch_size =None, num_workers=0,pin_memory=True,shuffle=False,collate_fn=collate_fn)
batch=next(iter(dl))
</code></pre>
<p>batch[0] shape is <code>(4, 32, 3, 224, 224)</code> even with a single batch the generated data is large, so i want to send <code>(2, 32, 3, 224, 224)</code> then another <code>(2, 32, 3, 224, 224)</code></p>
<p>Dont get confused with <code>batch=None</code>, i am just disabling automatic batching. Practically <code>batch=None</code> load one batch. I convert that one batch to four batches in collate function.<br>
But the issue is that now all four will be sent further and i want only 2 of them to be sent, then next 2 to the model, instead of all four at once.</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/control-batch-size-in-collate-function/140879/2">here</a>.</p>
Reasons for class mismatch high accuracy?

Jeffery_Ott: <p>My image is classified into 4 distinct classes but I wrongly have been outputting 20 possible classes, but the thing is my 20 class output was a much more accurate (94%) model then the 4 class (74%) and the image segmentation was looked better? How in the heck? I even checked pixel values and the my target is unique values 0-3 and my prediction is 0-20. I was wondering if anyone could shed some light</p>
<p>Thank you</p>

ptrblck: <aside class="quote no-group" data-username="Jeffery_Ott" data-post="1" data-topic="140935">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeffery_ott/48/44956_2.png" class="avatar"> Jeffery_Ott:</div>
<blockquote>
<p>I even checked pixel values and the my target is unique values 0-3 and my prediction is 0-20.</p>
</blockquote>
</aside>
<p>This sounds counter-intuitive to the higher overall accuracy or are the “additionally” predicted class indices quite sparse and do not lower the accuracy a lot?<br>
I would have assumed your model would learn to predict only the used target classes, as it would never get any signal from classes 5-20 so unsure why the model is learning to predict them at all.</p>
Tensorflow + Pytorch install Cudatoolkit 11.2

Gabor_Korosi: <p>Hi</p>
<p>I have a Windows 10 machine with an nvidia 3080 card. 11.2 with CudaToolkit<br>
I want to install a Pytorch in addition to TensorFlow, which works 100% fine so far.</p>
<p>If I understand the description correctly, the CudaToolkit installed without the Cuda Python env is “independent” of the Cuda toolkit version installed for Windows.</p>
<p>I tried to install Pytorch with this command, but it does not want to recognize the GPU.</p>
<p>pip install torch==1.9.0+cu102 torchvision==0.10.0+cu102 torchaudio==0.9.0 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>

ptrblck: <p>Your 3080 needs CUDA&gt;=11.0, so you have to install the PyTorch binaries with 11.3 or 11.5.<br>
And yes, you are correct that the PyTorch binaries ship with their own CUDA runtime and do not use the locally installed CUDA toolkit unless you build PyTorch from source or a custom CUDA extension.</p>
How to save prediction result of Dataloader batch

bruceszq: <p>I have fine-tuned a Bert model and want to use the model to make new predictions. If I use Dataloader batch to make new predictions, how could I save the prediction label and the original sentence into a csv? My label is [“business”,“news”,“math”]. Could the original sentence in a line of CSV and the predicted label in another line?</p>
<p>Thank you very much!</p>
<p>My code is here:</p>
<pre><code>    eval_sampler = SequentialSampler(dataset)
    eval_dataloader = DataLoader(dataset, sampler=eval_sampler, batch_size=self.args.batch_size)

    # Eval!
    logger.info("***** Running evaluation on %s dataset *****", mode)
    logger.info("  Num examples = %d", len(dataset))
    logger.info("  Batch size = %d", self.args.batch_size)
    eval_loss = 0.0
    nb_eval_steps = 0
    preds = None
    out_label_ids = None

    self.model.eval()

    for batch in tqdm(eval_dataloader, desc="Evaluating"):
        batch = tuple(t.to(self.device) for t in batch)
        with torch.no_grad():
            inputs = {'input_ids': batch[0],
                      'attention_mask': batch[1],
                      'token_type_ids': batch[2],
                      'labels': batch[3]}
            outputs = self.model(**inputs)
            tmp_eval_loss, logits = outputs[:2]

            eval_loss += tmp_eval_loss.mean().item()
        nb_eval_steps += 1

        if preds is None:
            preds = logits.detach().cpu().numpy()
            out_label_ids = inputs['labels'].detach().cpu().numpy()
        else:
            preds = np.append(preds, logits.detach().cpu().numpy(), axis=0)
            out_label_ids = np.append(
                out_label_ids, inputs['labels'].detach().cpu().numpy(), axis=0)

    eval_loss = eval_loss / nb_eval_steps
    results = {
        "loss": eval_loss
    }
    preds = np.argmax(preds, axis=1)
    result = compute_metrics(preds, out_label_ids)

    results.update(result)</code></pre>