akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Backward(gradient=torch.tensor([1.0,1.0]) equivalent to backwarding sum, equivalent to backwarding twice and retaining graph?

gandolfin69420: <p>is backwarding using these 3 methods, even on less trivial examples, equivalent in result?<br>
what about compute usage?</p>
<pre><code class="lang-auto">import torch




def main():
    x = torch.tensor(69., requires_grad=True)
    y = torch.tensor(420., requires_grad=True)
    (x*y + (x+y)/x).backward()
    print(x.grad, y.grad)
    x = torch.tensor(69., requires_grad=True)
    y = torch.tensor(420., requires_grad=True)
    torch.stack((x*y, (x+y)/x)).backward(torch.tensor([1., 1.]))
    print(x.grad, y.grad)
    x = torch.tensor(69., requires_grad=True)
    y = torch.tensor(420., requires_grad=True)
    _ = torch.stack((x*y, (x+y)/x))
    for _ in _:
        _.backward(retain_graph=1)
    print(x.grad, y.grad)



if __name__ == '__main__':
    main()
</code></pre>

soulitzer: <p>The results should be the same yes.<br>
The first is the preferred method.<br>
Stack is expensive because you need to allocate a new buffer to hold the combined results.<br>
Backward twice obviously is also not great because you’re recomputing everything during backward.</p>
How to remove the same elements of two 1-dim int tensors

Joeyoung: <p>a = torch.tensor( [10,2,5,1,9,7,8] )<br>
b = torch.tensor( [1,9,8] )<br>
I want to remove the same numbers as b from a.<br>
What is the fastest way to do it?<br>
The result tensor can be sorted or unsorted.<br>
i.e. the result c is:<br>
torch.tensor( [10,2,5,7] )<br>
or<br>
torch.tensor( [2,5,7,10] )</p>
<p>I can do this by:</p>
<p>from collections import Counter<br>
ca = Counter(a.numpy())<br>
cb = Counter(b.numpy())<br>
c = torch.tensor( list( (ca-cb).elements() ) )</p>
<p>But, I am wondering if there is a better(faster) way to do it.</p>
<p>Thank you!</p>

KFrank: <p>Hi Joseph!</p>
<aside class="quote no-group quote-modified" data-username="Joeyoung" data-post="1" data-topic="204881" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/a88e4f/48.png" class="avatar"> Joeyoung:</div>
<blockquote>
<p>a = torch.tensor( [10,2,5,1,9,7,8] )<br>
b = torch.tensor( [1,9,8] )<br>
…<br>
i.e. the result c is:<br>
torch.tensor( [10,2,5,7] )</p>
</blockquote>
</aside>
<p>At the cost of materializing a <code>[len (a), len (b)]</code> tensor, you can:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.3.1'
&gt;&gt;&gt; a = torch.tensor ([10, 2, 5, 1, 9, 7, 8])
&gt;&gt;&gt; b = torch.tensor ([1, 9, 8])
&gt;&gt;&gt; a[(a == b.unsqueeze (-1)).sum (dim = 0) == 0]
tensor([10,  2,  5,  7])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Joeyoung: <p>Thank you, Frank! I will do the speed benchmark to choose the faster one.</p>
Unable to reproduce training performance

sinntelligence: <p>I have trained a RegNet model on a custom dataset for an image classification task. That was in August 2023. Now I want to train exactly the same model again, using the same dataset. I would expect this new model to achieve about the same performance as the previous one from August 2023, since nothing has changed:</p>
<ul>
<li>I use exactly the same PyTorch and Torchvision versions (1.13 and 0.14)</li>
<li>I use exactly the same image dataset for training/validation/test</li>
<li>I use exactly the same script to train the model via torch</li>
<li>And I use exactly the same training hyperparams as before</li>
</ul>
<p>However, even though nothing has changed, the newly trained model performs significantly worse the the original model from last year. Where the first model from august 2023 achieves an test accuracy of 0.97, now the new model only achives 0.94 on the very same test dataset. During training the train and validation accuracy is about the same though as before.</p>
<p>I understand that two models will not achieve exactly the same performance, but three % difference seems too much. Whatever I do, I cannot get close to these 0.97 test accuracy from last year, about 0.94 is all I get. Even though everything is exactly the same, as described. Even the machine with its four GPUs and the Ubuntu version running on that machine are exactly the same as before in 2023.</p>
<p>I know there is a random seed involved, but I doubt that could lead to such a large test accuracy difference of 3%. Also I know that maybe Nvidia / CUDA driver has been updated on that machine, and of course some dependencies and packages (e.g. numpy). But can that lead to such a huge difference?</p>

ptrblck: <aside class="quote no-group" data-username="sinntelligence" data-post="1" data-topic="204944">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/5daacb/48.png" class="avatar"> sinntelligence:</div>
<blockquote>
<p>there is a random seed involved, but I doubt that could lead to such a large test accuracy difference of 3%</p>
</blockquote>
</aside>
<p>Did you check it using your setup and different seeds? Also, was the previous run from last year a single run or were you able to reproduce it back then?</p>

sinntelligence: <p>Thanks for your reply!</p>
<p>Ad seeds - I trained over 100 models during the last days, and none of them achieved that performance of last year. Since every such trial would be based on a random seed anyway, I conclude that the seed does not lead to large changes of multiple % acccuracy.</p>
<p>Ad reprodudcability - Unfortunately I don’t know. Last year I trained only once, was quite happy with the results, so there was no need to train another model. Thus I cannot say whether that was a super “lucky punch” back then, or the setup was somehow different.</p>
Why do I install pytorch with the names "~orch" and "~-rch" in the site-packages folder?

hrdom: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7ef2bd39a134043fd33ac3f588df3049877ede82.png" alt="image" data-base62-sha1="i72e6CGNMMtU5Aik3bHooXmIvCO" width="316" height="322"></p>

hrdom: <aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/55565760/anaconda-python-site-packages-subfolders-with-tilde-in-name-what-are-they">
  <header class="source">

      <a href="https://stackoverflow.com/questions/55565760/anaconda-python-site-packages-subfolders-with-tilde-in-name-what-are-they" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1637673/mithril" target="_blank" rel="noopener nofollow ugc">
    <img alt="Mithril" src="https://www.gravatar.com/avatar/102059d423f2265e737117ce0947e7ee?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/55565760/anaconda-python-site-packages-subfolders-with-tilde-in-name-what-are-they" target="_blank" rel="noopener nofollow ugc">Anaconda/Python site-packages subfolders with tilde in name - what are they?</a>
</h4>

<div class="tags">
  <strong>python, pip, anaconda, python-packaging</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1637673/mithril" target="_blank" rel="noopener nofollow ugc">
    Mithril
  </a>
  on <a href="https://stackoverflow.com/questions/55565760/anaconda-python-site-packages-subfolders-with-tilde-in-name-what-are-they" target="_blank" rel="noopener nofollow ugc">03:13AM - 08 Apr 19 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Model saved by torch2.1.0 can't load in 2.0.0

dsada: <p>I save a torch model in torch 2.1.0 and try to load in 2.0.0, and I meet such error:</p>
<pre><code class="lang-auto">File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/app/.conda/envs/kuangxiu_dev/lib/python3.8/site-packages/torch/serialization.py", line 809, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/home/app/.conda/envs/kuangxiu_dev/lib/python3.8/site-packages/torch/serialization.py", line 1172, in _load
    result = unpickler.load()
  File "/home/app/.conda/envs/kuangxiu_dev/lib/python3.8/site-packages/torch/_utils.py", line 172, in _rebuild_tensor_v2
    set_tensor_metadata(tensor, metadata)
  File "/home/app/.conda/envs/kuangxiu_dev/lib/python3.8/site-packages/torch/_utils.py", line 163, in set_tensor_metadata
    torch._C._set_tensor_metadata(tensor, metadata)  # type: ignore[attr-defined]
RuntimeError: Unexpected key `base_strides_/1536/1/` passed to setTensorMetadata.
</code></pre>

ptrblck: <p>These issues are expected since PyTorch is not forwards compatible but should be backwards compatible, i.e. loading an older checkpoint in newer PyTorch versions should work.</p>
Casual 3d conv implementation?

Y_shin: <p>Hi, I want to implement a casual 3d conv to process video sequences, with “replicate” padding in spatial and “zero” padding in temporal. Here is my implementation, and I apply this casual conv in some residual blocks, due to the limit of memory, I use util.checkpoint to wrap the residual module.</p>
<pre><code class="lang-auto">class TemporalCasualConv(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, stride=1, dilation=1, spatial_padding_mode="replicate"):
        super().__init__()
        self.kernel_size = format_tuple_size("kernel_size", kernel_size)   # int to tuple
        self.stride = format_tuple_size("stride", stride)
        self.dilation = dilation
        self.padding_dim = tuple((k - 1) * dilation for k in self.kernel_size)
        self.f_pad_param = (self.padding_dim[2] // 2 + self.padding_dim[2] % 2, self.padding_dim[2] // 2, 
                            self.padding_dim[1] // 2 + self.padding_dim[1] % 2, self.padding_dim[1] // 2,
                            0, 0)
        self.spatial_padding_mode = spatial_padding_mode
        self.conv = nn.Conv3d(in_channels, out_channels, kernel_size=kernel_size, stride=stride, padding=(self.padding_dim[0], 0, 0), dilation=dilation)

    def forward(self, x):
        """
        input:  x[B, C_in, T, H, W]
        output: x[B, C_out, T', H', W']
        """
        _, _, t0, _, _ = x.shape
        x = self.conv(F.pad(x, self.f_pad_param, mode=self.spatial_padding_mode))

        last_slice = (t0 + self.padding_dim[0] - self.dilation * (self.kernel_size[0] - 1) - 1) // self.stride[0] + 1
        if self.padding_dim[0] &gt; 0:
            x = x[:, :, :last_slice, :, :]

        return x
</code></pre>
<p>I’m not sure whether my implementation is correct, especially in backward and under checkpoint mechanism. I notice this module decreases the metrics a lot (3x loss) compared with vanilla 3d convolution with replicate padding.</p>
<p>Can anyone provide some instructions?</p>