akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is jax really 10x faster than pytorch?

username: kirk86
cooked: <p>I was reading the following <a href="https://www.assemblyai.com/blog/why-you-should-or-shouldnt-be-using-jax-in-2022/" rel="noopener nofollow ugc">post</a> when I cam accross the figure below and I was wondering whether that’s true for jax vs pytorch, since I haven’t been following closesly the developments in this space? Any thoughts?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/1/0184099ebf08ce4a3ddb163a9f71a666eb568ab8.png" data-download-href="https://discuss.pytorch.org/uploads/default/0184099ebf08ce4a3ddb163a9f71a666eb568ab8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/0184099ebf08ce4a3ddb163a9f71a666eb568ab8_2_345x232.png" alt="image" data-base62-sha1="dpmJvzUilvi9GGMvbCBtfGNEcU" width="345" height="232" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/0184099ebf08ce4a3ddb163a9f71a666eb568ab8_2_345x232.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/0184099ebf08ce4a3ddb163a9f71a666eb568ab8_2_517x348.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/0184099ebf08ce4a3ddb163a9f71a666eb568ab8_2_690x464.png 2x" data-dominant-color="ECD8FC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1480×998 19 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Transformer architecture and theory

username: svv
cooked: <p>Hi! The question is of a theoretical nature. Maybe someone knows or there is a material where a transformer model is described for dummies, so that one pass could be calculated in Excel. With examples, for example, there are 5 sentences, positional coding takes place here, as a result, the matrix of so-and-so. I am especially interested in how Q, K, V and all this magic appear. Is there a matrix of sentences or individual words at the output of the encoder? And how does the decoder turn the matrix into text?</p>

username: AbdulsalamBande
cooked: <p>This is the channel you are looking for  <a href="https://www.youtube.com/channel/UCK_6ta2hzY9ly4gRRRYEYdw" rel="noopener nofollow ugc">Pytorch Modules explained</a> And especially starting with <a href="https://www.youtube.com/watch?v=oCWFyt2kWLg&amp;ab_channel=PytorchModulesExplained" rel="noopener nofollow ugc">Transformer Encoder Overview</a></p>
Graph representation of PyTorch model during forward and backward passes

username: juliagusak
cooked: <p>Hi! If you have ever played with a graph related to PyTorch model could you, please, share what tools did you use?</p>
<ul>
<li>
<p>To build forward / backward graphs? To build A joint graph for both forward and backward?</p>
</li>
<li>
<p>To access and modify nodes/subgraphs?</p>
</li>
<li>
<p>To generate forward/backward pass given model graph?</p>
</li>
</ul>
<p>Thank you!</p>
Make changes on torch jit script after loading

username: chrisrn
cooked: <p>Hi! I wonder if someone can make changes on jit code string parameter. Actually, I have received a trace file which contains cuda code, but I want to run this on cpu. So I wonder if I can change the code like that:</p>
<pre><code class="lang-auto">import torch
torch_script = torch.jit.load("somefile.ptc")
torch_script.code = torch_script.code.replace("cuda", "cpu")
</code></pre>
<p>This code produces the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;input&gt;", line 1, in &lt;module&gt;
  File "venv/lib/python3.9/site-packages/torch/jit/_script.py", line 590, in __setattr__
    return super(RecursiveScriptModule, self).__setattr__(attr, value)
  File "venv/lib/python3.9/site-packages/torch/jit/_script.py", line 310, in __setattr__
    return super(ScriptModule, self).__setattr__(attr, value)
  File "venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 995, in __setattr__
    object.__setattr__(self, name, value)
AttributeError: can't set attribute
</code></pre>
<p>The purpose is to avoid rewriting the torch script code and export it again. Is that feasible? Thanks!</p>
Can i shange current step in dataloader?

username: hadaev8
cooked: <p>For gan training peoples seems to do it like this:</p>
<pre><code class="lang-auto">    trainset = GanLoader(imgs)
    sampler = torch.utils.data.RandomSampler(trainset, replacement=True, num_samples=config['num_samples'])
    train_loader = torch.utils.data.DataLoader(trainset, num_workers=config['num_workers'],
                                               sampler=sampler, batch_size=config['batch_size'],
                                               pin_memory=True, drop_last=True)
</code></pre>
<p>For example, I saved training on step 10k and exited program, then resuming train loop, can I set 10k step in dataloader so it would give the exact same sample as it would without breaking execution?</p>

username: ptrblck
cooked: <p>No, you most likely wouldn’t get the same <code>DataLoader</code> state if you haven’t used e.g. epoch seeds before.<br>
In your current code snippet you are using a <code>RandomSampler</code> without passing a <code>generator</code>, so a new one will be initialized as seen <a href="https://github.com/pytorch/pytorch/blob/88547396eb82bca90cec4f9237277d55b99dcedc/torch/utils/data/sampler.py#L90-L123">here</a>.</p>

username: hadaev8
cooked: <p>Well, this seed should depend on global seed, can I skip first n steps in generator?<br>
Would <code>set_state</code> help?</p>
<p>Maybe I should save the generator/dataloader at the checkpoint?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="hadaev8" data-post="3" data-topic="143103">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hadaev8/48/16280_2.png" class="avatar"> hadaev8:</div>
<blockquote>
<p>Well, this seed should depend on global seed, can I skip first n steps in generator?</p>
</blockquote>
</aside>
<p>If you’ve seeded your code before the beginning of the training, this might work.<br>
If not, your new process would use a new random seed and I don’t think your approach would work.</p>

username: hadaev8
cooked: <p>So, I tested for a bit</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import DataLoader, Dataset

class DSet(Dataset):
    def __init__(self, num=10):
        self.data = range(10)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return len(self.data)

trainset = DSet()

sampler = torch.utils.data.RandomSampler(trainset, replacement=True, num_samples=20)

train_loader = DataLoader(trainset, num_workers=0, sampler=sampler,
                            batch_size=2, pin_memory=False)
</code></pre>
<pre><code class="lang-auto">torch.manual_seed(0)
for i in train_loader:
    print(i)
</code></pre>
<p>Manual seed before calling dataloader make it reproducible, but how can I resume training from the exact step?<br>
Using continue inside loop would make it read all data from disc.</p>
List all the tensors and their memory allocation

username: M2M
cooked: <p>Hello everyone!</p>
<p>Is there a way to list all the tensors and their memory usage? I run out of GPU memory when I start to infer a trained model (not training at all in this code). So I would like to know where is the bottleneck. What uses up all the memory.</p>
<p>Edit: My saved models are more than 700MB in size on disk</p>

username: ptrblck
cooked: <p>You could use a similar approach as described in <a href="https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741/3">this post</a> to get all Python tensors.<br>
However, this would not return the tensors allocated in the backend, so you might (additionally) want to check the memory usage via e.g. <code>print(torch.cuda.memory_summary())</code> to narrow down where you are running out of memory.</p>

username: M2M
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="144108">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>print(torch.cuda.memory_summary())</p>
</blockquote>
</aside>
<p>Thank you so much! That is very helpful.<br>
I could not spot anything unusual. I am running a simple Unet. I presume these networks use a lot of memory. It would have been handy if we can list tensors by their name and their memory usage (size might not tell the full story because of the underlying data type I guess). But it might not possible I think. Thanks again!</p>
How to load a pytorch model without having to import the class

username: nikitha_kona
cooked: <p>I have a notebooks where I have my model and I saved the model.Is there a way on loading the model without importing the class definition ,because that is taking time .</p>
<p>I tried torch.save(model, path)<br>
and tried to load from another notebook using torch.load().I f import the class definition it works.</p>
<p>Thanks</p>

username: Balamurali_M
cooked: <p>I believe torch.load doesn’t expect class definition, only load state dict class definition ?</p>
<p><a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a></p>
<p>This link clearly explains different possibilities</p>

username: nikitha_kona
cooked: <p>Thanks.But it gives me a unpickle error if am using torch save and load.I am trying to load it from a different ipynb file.</p>

username: dingyongchao
cooked: <pre><code class="lang-auto"># Model class must be defined somewhere
model = torch.load(PATH)
model.eval()
</code></pre>
<p>it is from <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="nofollow noopener">pytorch saving_loading_model</a></p>

username: Balamurali_M
cooked: <p>Can you share the error message ?</p>

username: nikitha_kona
cooked: <h2>Thanks.<br>
torch.load(‘issue.pth’) gives me</h2>
<p>AttributeError                            Traceback (most recent call last)<br>
 in <br>
----&gt; 1 a=torch.load(‘issue.pth’)</p>
<p>/anaconda3/lib/python3.6/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)<br>
385         f = f.open(‘rb’)<br>
386     try:<br>
–&gt; 387         return _load(f, map_location, pickle_module, **pickle_load_args)<br>
388     finally:<br>
389         if new_fd:</p>
<p>/anaconda3/lib/python3.6/site-packages/torch/serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)<br>
572     unpickler = pickle_module.Unpickler(f, **pickle_load_args)<br>
573     unpickler.persistent_load = persistent_load<br>
–&gt; 574     result = unpickler.load()<br>
575<br>
576     deserialized_storage_keys = pickle_module.load(f, **pickle_load_args)</p>
<p>AttributeError: Can’t get attribute ‘CharLoopConcatModell’ on &lt;module ‘<strong>main</strong>’&gt;</p>

username: Jingnan_Jia
cooked: <p>You have to define the class of the model before you  load it from disk.</p>

username: trash_Steiner
cooked: <p>Just to end this discussion: You can do it with</p>
<p>model_scripted = torch.jit.script(model) # Export to TorchScript<br>
model_scripted.save(‘model_scripted.pt’) # Save</p>
<p>And then load it with:</p>
<p>model = torch.jit.load(‘model_scripted.pt’)<br>
model.eval()</p>
<p>See also <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc">Saving and Loading Models</a><br>
Under <strong>Export/Load Model in TorchScript Format</strong></p>
A Unet with very small weigths takes long time for inference on CPU

username: spiovesan
cooked: <p>We have trained a UNet network.<br>
When run on a GPU it takes 6 milliseconds. The same image when run on a CPU it takes 4.5 seconds.<br>
We noticed that the trained network has a lot of very small weights (order of e-42).<br>
After manually setting to 0 all those weights, now the network runs in 290 milliseconds on CPU, with the same performance.<br>
You can see with <a href="https://netron.app/" rel="noopener nofollow ugc">https://netron.app/</a> the network <a href="https://visionlinksrl-my.sharepoint.com/:u:/g/personal/stefano_piovesan_univision_it/ET2Cifk905NDiI4-u1CTLlYBXmJFNm3V8hCxqWxyumVeLw?e=1o8xTA" rel="noopener nofollow ugc">here</a><br>
Any idea how to fix it without the manual setting to 0?</p>

username: ptrblck
cooked: <p>I assume one of the workloads is using the CPU? (You are mentioning GPU in both cases)<br>
Assuming the slower use case is executed on the CPU, you could use <code>torch.set_flush_denormal(True)</code> to disable denormal numbers on your CPU, which should then avoid the (potentially slower) denormal code path.</p>

username: spiovesan
cooked: <p>Yes, the slower case was on CPU (I edited the typo). Using <code>torch.set_flush_denormal(True)</code> solves the issue of the slower inference. Thank you for your support!</p>
Pytorch Geometric Datasets

username: Gianmarco_Luchetti_S
cooked: <p>Hello everyone!<br>
I need your help. I have two set of graph structured data, one from Open Graph Benchmark (OGB) and another created with <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html#torch_geometric.data.Dataset" rel="noopener nofollow ugc"><code>torch_geometric.data.Dataset</code></a> from my own data . The data looks like:</p>
<blockquote>
<p>Data(edge_index=[2, 88], edge_attr=[88, 3], x=[39, 9], y=[1, 1])  <span class="hashtag-raw">#OGB</span><br>
Data(x=[23, 9], edge_index=[2, 48], edge_attr=[48, 2], y=[1]) <span class="hashtag-raw">#PyG</span></p>
</blockquote>
<p>I am trying to use a framework developed using OGB functions, this doesn’t work with data created using PyG even though they are both Pytorch objects, the only difference is that the OGB dataset is an InMemoryDataset and the PyG one is a ‘Larger’ dataset (<a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html" class="inline-onebox" rel="noopener nofollow ugc">Creating Graph Datasets — pytorch_geometric documentation</a>). Is there any way to fix this without having to change the source code?</p>
<p>Thanks!</p>
Multiplying 2 tensors properly

username: KURUVILLA_ABRAHAM
cooked: <p>Hi team,</p>
<p>i need to mutiply to images and pass to a conv layer</p>
<p>shape of image is = torch.Size([2, 256, 16, 16])<br>
style =  torch.Size([2, 1, 1, 256])</p>
<h1>
<a name="how-can-i-mutiply-both-the-shape-and-style-with-proper-output-to-produce-1" class="anchor" href="#how-can-i-mutiply-both-the-shape-and-style-with-proper-output-to-produce-1"></a>How can i mutiply both the shape and style with proper output to produce</h1>
<h1>
<a name="code-i-am-using-to-solve-2" class="anchor" href="#code-i-am-using-to-solve-2"></a>Code i am using to solve -</h1>
<p>self.style_scale_transform = nn.Linear(256,256)<br>
self.style_shift_transform = nn.Linear(256,256)<br>
style_scale = self.style_scale_transform(style).unsqueeze(4).unsqueeze(5)<br>
style_bias = self.style_shift_transform(style).unsqueeze(4).unsqueeze(5)<br>
out =  x * style_scale  + style_bias</p>

username: thecho7
cooked: <p>Please let me know the shape of resulting output.</p>
<p>I guess you can do this,</p>
<pre><code class="lang-auto"># (2, 256, 16, 16) -&gt; (2, 256, 256) -&gt; (2, 256, 256)
image = image.flatten(start_dim=2).transpose(2, 1)
# (2, 1, 1, 256) -&gt; (2, 1, 256) -&gt; (2, 256, 1)
style = style.squeeze(dim=1).transpose(2, 1)

# (2, 256, 256) x (2, 256, 1) = (2, 256, 1)
result = torch.bmm(image, style)
</code></pre>
<p>Hope it helps you</p>

username: KURUVILLA_ABRAHAM
cooked: <aside class="quote no-group" data-username="thecho7" data-post="2" data-topic="144186">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/eada6e/48.png" class="avatar"> thecho7:</div>
<blockquote>
<pre><code class="lang-auto">, 256) -&gt; (2, 256, 256)
</code></pre>
</blockquote>
</aside>
<p>but wont this disturb the image dimnesions and final output</p>
Fastest way upscale a batch of images

username: Vladik
cooked: <p>Suppose I have a batch of images in some shape <code>(B, H, W)</code> on the GPU and I want to resize each image such that the result will be doubled in width <code>(B, H, 2*W)</code>.</p>
<p>I tested a few approaches on a big batch of images with shape <code>(3000, 224, 224)</code> and got somewhat surprising results.</p>
<p>First, I naively tried implementing it myself:</p>
<pre><code class="lang-auto">def upscale(images: torch.Tensor) -&gt; torch.Tensor:
    B, H, W = images.shape
    arr = torch.empty((B, H, W*2), device=images.device, dtype=images.dtype)
    arr[..., ::2] = images
    arr[..., 1::2] = images
    return arr

upscale_jit = torch.jit.script(upscale)
</code></pre>
<p>This is roughly 13ms.</p>
<p>Then I tried <code>torch.repeat_interleave(images, 2, -1)</code> which was slightly quicker at 10ms.</p>
<p>Then I tried <code>F.interpolate(images.unsqueeze(0), (H, W*2))</code> and got 5.5ms(!)</p>
<p>So two questions:</p>
<ol>
<li>How the hell is interpolating faster than just straight up repeating\copying data?</li>
<li>Any ideas on what could do it even faster?</li>
</ol>
<p>Using torch==1.8 or newer, these results are on my RTX3060 but this will eventually end up on a Xavier AGX if this makes any difference <img src="https://discuss.pytorch.org/images/emoji/apple/man_shrugging.png?v=12" title=":man_shrugging:" class="emoji" alt=":man_shrugging:"></p>

username: eqy
cooked: <p>I’m seeing a similar ordering of speed on A6000 where “interpolate” by default is the fastest.</p>
<pre><code class="lang-auto">0.006442546844482422 (JIT)
0.0036177635192871094 (repeat interleave)
0.0027103424072265625 (nearest-neighbor interpolate)
</code></pre>
<p>However, it looks like the default setting uses nearest-neighbor interpolation, which amounts to… copying data. When trying another mode such as “bilinear,” repeat-interleave is faster.</p>
<pre><code class="lang-auto">0.006438493728637695 (JIT)
0.003609895706176758 (repeat interleave)
0.0041081905364990234 (bilinear interpolate)
</code></pre>
<p>As for the initial speed difference between interleave and nearest-neighbor interpolation, while they might be very similar in high-level approach (no computation, just data movement), my guess is that their implementation strategies (how the work is coordinated across GPU blocks/threads, and memory access patterns) are sufficiently different to account for the performance difference.</p>

username: Vladik
cooked: <p>Yeah I was suspecting something of this sort.</p>
<p>Still I wonder if there’s anything better</p>
Clone a model from pytorch existing model, only till certain layers

username: nbansal90
cooked: <p>Hello All,</p>
<p>I am looking to define a model, which is a subset of existing models such as <code>ResNet18</code>, <code>ResNet34</code> or <code>ResNet101</code>. In this context i know that pytorch already provides these (pretrained) models through <code>import torchvision.models</code>.</p>
<p>In my case I was looking for a specifc scenario, where supposing I select <code>ResNet34</code> as my base model. Now I am looking to sub-divide this base model in to two models which is <code>Base</code> and <code>Head</code> respectively,  where <code>Base</code> consist of starting few layers of <code>ResNet34</code> and <code>Head</code> the remaining layers.</p>
<p>I was wondering how could I use the existing <code>ResNet34</code>(for example) model provided by Pytorch, to get what I want.</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>You could write a custom model, derive from <code>Resnet34</code> as its base class, and assign the wanted modules to your new <code>Base</code> and <code>Head</code> submodules.<br>
To do so, check the <code>torchvision</code> implementation and make sure e.g. all original functional API calls are used in your new <code>forward</code> method as well (assuming you don’t want to change the execution of the model’s layers).</p>
How to set certain tensors to zero and save the model?

username: foranotherday
cooked: <p>Hello,</p>
<p>I would like to manually edit some tensors in the model weights and save the new version as .pth file. Following this discussion (<a href="https://discuss.pytorch.org/t/how-to-manually-set-the-weights-in-a-two-layer-linear-model/45902" class="inline-onebox">How to manually set the weights in a two layer linear model?</a>), I was able to manually set one of the parameters to zero but I don’t know how it could be saved as a weights file.</p>
<pre><code class="lang-auto">checkpoint = './log/best_state.pth'
checkpoint = torch.load(checkpoint)

for name, param in model.named_parameters():
      if name in param_name:
            with torch.no_grad():
                # Set parameter vals to zero
                param[param_idx] = torch.zeros_like(param[param_idx])

                torch.save(model.state_dict(), "xyz.pth")
</code></pre>

username: ptrblck
cooked: <p>Using <code>torch.save(model.state_dict(), "xyz.pth")</code> should work, as the manipulated parameter will also be saved to the <code>state_dict</code>.<br>
Do you see any issues with this code? If you want to only store the tensor you could use <code>torch.save(model.layer.param, 'path.pt')</code>.</p>
Imagenet Dataloading Correct?

username: just_started_coding
cooked: <p>I have attached dataloader that I have written for Imagenet. I have saved the paths to images with their respective classname in .npy file and load it from there. Hierarchy of file is same as provided in Kaggle dataset for Imagenet. I am not sure if it is correct. It seems to return image tensor and label fine, but model accuracy is very low - so I was wondering if I did any mistake in this part. Any feedback will be highly appreciated! Thank you!</p>
<pre><code class="lang-auto"># Custom Dataloader
class CustomDataset(Dataset):
    def __init__(self,train=True,transform=None):
        self.data = []
        self.transform = transform 
        # Class to Label Encoding Loading
        class_to_label_loading = np.load("~/model_implementation/AlexNet/class_to_label.npy",allow_pickle=True)
        self.class_to_label = class_to_label_loading.item()
        if not train:
            self.data = np.load("~/model_implementation/AlexNet/val_data.npy")
        else:
            self.data = np.load("~/model_implementation/AlexNet/train_data.npy")

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        img_path, class_name = self.data[idx]
        img = Image.open(img_path)
        rgb_img = img.convert('RGB')
        label = self.class_to_label[class_name]
        if self.transform is not None:
            img_tensor = self.transform(rgb_img)
        label_tensor = torch.tensor([label])
        return img_tensor, label_tensor
</code></pre>
<p>and transform is pretty standard one for Imagenet dataset like this:</p>
<pre><code class="lang-auto">transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
</code></pre>

username: ptrblck
cooked: <p>Based on your code it seems as if you’ve stored the paths in the numpy array only. I cannot see any obvious errors, but since the data structure is equal to the “Kaggle dataset” (assuming this means the samples for each class are moved to their respective class folder) you could compare the model training with the standard <code>ImageFolder</code> dataset.</p>
Alternative Model Loading

username: syed_kamal
cooked: <p>Is there a way to load a model from bytes of a file rather than the default load from path:<br>
Module.load(MainActivity.assetFilePath(getApplicationContext(), “deeplabv3_scripted.pt”));</p>
<p>Something like: Module.loadBytes(byteArray);</p>

username: marksaroufim
cooked: <p>Hy <a class="mention" href="/u/syed_kamal">@syed_kamal</a> couold you elaborate a bit more on why you’d need something like this?</p>

username: syed_kamal
cooked: <p>It was a task in my last job, for encrypting the model and decrypting it in runtime in the form of bytes without having to save it anywhere locally.</p>

username: marksaroufim
cooked: <p>We have support for encrypted model loading in torchserve <a href="https://github.com/pytorch/serve/blob/master/docs/management_api.md#encrypted-model-serving" rel="noopener nofollow ugc">https://github.com/pytorch/serve/blob/master/docs/management_api.md#encrypted-model-serving</a></p>
An issue about random seed

username: redocxz
cooked: <p>I’m having an issue, but I’m not sure whether it’s about pytorch</p>
<p>I have fixed the random seed by the following code (at the entrance of the program, or even at the beginning of each file)</p>
<pre><code class="lang-python">seed = 2022
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
os.environ['PYTHONHASHSEED'] = str(seed)
</code></pre>
<pre><code class="lang-python"> Stochastic(nn.Module):
    """
    Base stochastic layer that uses the
    reparametrization trick [Kingma 2013]
    to draw a sample from a distribution
    parametrised by mu and log_var.
    """
    def reparametrize(self, mu, log_var):
        epsilon = Variable(torch.randn(mu.size()), requires_grad=False)

        if mu.is_cuda:
            epsilon = epsilon.cuda()

        # log_std = 0.5 * log_var
        # std = exp(log_std)
        std = log_var.mul(0.5).exp_()

        # z = std * epsilon + mu
        z = mu.addcmul(std, epsilon)

        return z
</code></pre>
<p>And I use the above code from others in my program</p>
<p>And I can’t reproduce the result stably, because I found in the above code that the random values in this line 8 are random each run. I’m confused about that.</p>
<p>I’m not sure whether <code>Variable</code> <code>randn</code> and <code>requires_grad</code> are being used correctly here</p>
<p>Looking forward to your answers, thanks in advance</p>

username: paganpasta
cooked: <p>Tried running your sample code but I am able to get the same value of <code>z</code> for different runs.</p>
<p>Why do you intend to use nn.Module is none of the parameters are trainable? Wouldn’t a simple function called reparameterize suffice which returns <code>z</code> and you won’t have to state if epsilon is trainable or not.</p>

username: redocxz
cooked: <p>That is someone else’s program, I use the FastAPI framework to encapsulate and provide api interface, so I now suspect that is the problem about the FastAPI Background Task, but I do not know how to debug, confused…</p>
Is there a way to check if my cudnn/cuda is consistent among different machines?

username: seyeeet
cooked: <p>I have been able to fix the seed and make sure that the generate weights and biases and data in the dataloader are the same cross two different machines/hardware.</p>
<pre><code class="lang-auto">random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
</code></pre>
<p>but the final results are still different, and I think it comes from optimization part and cuda and cudnn (please let me know if I am wrong). BAsed on my observation, the weights and dataloader and initializations are all consistent.</p>
<p>I was wondering if there is a way to also fix that part or even if there is way to check that?</p>

username: ptrblck
cooked: <p>You can check the versions via:</p>
<pre><code class="lang-python">print(torch.version.cuda)
print(torch.backends.cudnn.version())
</code></pre>
<p>Note however, that there is no guarantee to get the same bitwise identical results if different hardware is used.</p>
How to use nn.TransformerEncoder

username: MatchaTurtle
cooked: <p>I am trying to use nn.TransformerEncoder to intelligently downsample a (3,N) vector into a (1, N) vector that comprises of the important parts of the (3, N).</p>
<p>As a toy experiment, I want the nn.TransformerEncoder to learn the averaging operation such that (3,N) → transformerencoder - &gt; (3,N)[0] → (1,N) is the average across dim=0. However, I am unable to get it working. The L1 loss remains around 0.4 which is pretty high. I would greatly appreciate any pointers for debugging.</p>
<p>Thank you so much!</p>
<p>Attached is my training code:</p>
<pre><code class="lang-auto">encoder_layers = TransformerEncoderLayer(8, 1, 64, 0.0) 
#d_model, n_head, dim_feedforward, dropout

network = nn.Sequential(nn.TransformerEncoder(encoder_layers, 5),)
criterion = nn.L1Loss()
optimizer = torch.optim.Adam(network.parameters(), lr=1e-5)

for i in tqdm(range(30000)):
        optimizer.zero_grad()
        src = torch.rand((3, 1, 8)).to(device)*2 - 1.0
        # src (3,1,8)

        gt = torch.mean(src, dim=0, keepdim=True).to(device).repeat((3,1,1))
        # gt (3,1,8)

        out = network(src)
        # out (3, 1, 8)

        loss = criterion(out, gt)
        loss.backward()
        optimizer.step()
</code></pre>

username: thecho7
cooked: <p>I’m curious about the goal of your experiment.<br>
The source and target are driven from the random then is it what you are really aiming for?</p>
<p>Meanwhile, your code is simply well-designed.</p>

username: MatchaTurtle
cooked: <p>Thanks for the reply!</p>
<p>I am aiming to see if the transformer encoder can learn “torch.mean” the averaging operation. Perhaps my experiment is ill-posed. I am not sure if given my current setup we should expect the transformer encoder to learn the averaging operation.</p>

username: thecho7
cooked: <p>I don’t think it can learn the averaging operation. But nice challenge <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Why when install, libtorch rebuild for ALL archs?

username: jiapei100
cooked: <p>I use cmake to build <strong>libtorch</strong>, it’s easily to build <strong>libtorch</strong> specifically for my <strong>native arch sm_75</strong>. However, after <strong>building (cmake and make)</strong>, <strong>libtorch</strong> has been built out as of size <strong>166M</strong>, while I tried to install <strong>libtorch</strong>, <strong>libtorch</strong> is actually <strong>rebuilt</strong> for <strong>ALL archs, including sm_35, sm_50, sm_52, …sm_75, sm86 …</strong></p>
<p>Did anybody find this issue? How to finally <strong>install libtorch after building</strong>?</p>
Parameters which did not receive grad for rank

username: John_Loverich
cooked: <p>I get the following error when running in distributed (works fine in serial).</p>
<pre><code class="lang-auto"> Parameters which did not receive grad for rank 1: model.head.fc_classification_layer.bias, model.head.fc_classification_layer.weight
</code></pre>
<p>There error is with the “fc_classification_layer” in this code, which is strange, it’s just a linear layer and the regression layer (also linear) is not giving the same issue.</p>
<pre><code class="lang-auto">class ObjPoseRegressionClassificationHead(nn.Module):
    def __init__(self, cfg: CfgNode, input_shape: ShapeSpec):
        super().__init__()
        self.dropout_rate = cfg.MODEL.OBJECT_POSE.HEAD.DROPOUT_RATE
        self.out_shape = [cfg.MODEL.OBJECT_POSE.HEAD.NUM_OUT_VERTICES, 2]
        # Flatten feature map into a vector
        self.flatten_layer = build_obj_pose_flatten(cfg, input_shape)
        # Dropout
        if self.dropout_rate &gt; 0:
            self.dropout = nn.Dropout(self.dropout_rate)
        else:
            self.dropout = None

        # fully connected layer regression layer
        self.fc_regression_layer = nn.Linear(
            input_shape.channels, np.prod(self.out_shape)
        )
        # out dimensions
        self._out_feature_channels = np.prod(self.out_shape)
        self._out_feature_strides = input_shape.width

        # fully connected classification layer
        self.fc_classification_layer = nn.Linear(
            input_shape.channels, cfg.MODEL.OBJECT_POSE.HEAD.NUM_CLASSES,
        )

    def forward(self, x):
        # (N, C, 7, 7)
        x = self.flatten_layer(x)
        # (N, C)
        if self.dropout is not None:
            x = self.dropout(x)
        # (N, C)
        x_regression = self.fc_regression_layer(x)
        # (N, 18)
        x_regression = x_regression.view(
            x.shape[0], self.out_shape[0], self.out_shape[1]
        )
        # (N, 9, 2)

        # NUM_CLASSES
        x_classification = self.fc_classification_layer(x)
        x_classification = F.log_softmax(x_classification, dim=1)

        return x_regression, x_classification

</code></pre>

username: John_Loverich
cooked: <p>Turns out I was returning a scalar in the loss by calling .item() the loss function. Got rid of .item() and it all works.  Should’ve paid more attention with that copy paste.</p>
How to create a random tensor with a given shape in pytorch lite (Java)?

username: zabir-nabil
cooked: <p>I want to create a random tensor with a shape (BxHxWxC) to test my <code>ptl</code> model. How can I create it?</p>
<p>I have seen the documentation here: <a href="https://pytorch.org/javadoc/1.4.0/org/pytorch/Tensor.html" class="inline-onebox" rel="noopener nofollow ugc">Tensor (pytorch_host 1.4.0 API)</a></p>
<p>I see <code>fromBlob</code> but not any method to create a random tensor with a specific shape which is pretty normal in pytorch to test a model.</p>

username: Linbin
cooked: <p>Java API is mainly for model inference and doesn’t support such tensor operations. You may create a random array in Java and use fromBlob to construct the random tensor.</p>
Perform reshape of tensor in JAVA.pytorch

username: SungmanHong
cooked: <p>Hi, I have python code<br>
and converting the code to android, and I have question of some commands of pythons.</p>
<p>First, my python code is <code>unsqueeze(x.0)</code> where x is pytorch.tensor.<br>
e.g) when unsqueeze(x,0) is performed, shape of x will be [3,2,4] to [1,3,2,4] &lt;-- add just 1 dimension</p>
<ol>
<li>
<p>in JAVA, can I do it how…?? (java command is so unusual for me… please help)</p>
</li>
<li>
<p>Also, I want to reorder the pytorch.tensor in JAVA<br>
e.g) [3,2,4] to [2,4,3]<br>
in python it is performed by <code>torch.from_numpy(x).permute(2, 0, 1)</code><br>
In JAVA, how to can I do it? (JAVA command is fine to me… I found some java function code or package that support reshape in python where the input is array… in pytorch.java… what is the type of tensor? it is array?)</p>
</li>
</ol>
<p>thank you.</p>

username: singh-aymcommerce
cooked: <p>hi <a class="mention" href="/u/sungmanhong">@SungmanHong</a> Bro, did you found any workaround for the above task. I am also stuck on the above?</p>

username: Linbin
cooked: <p>You can find Jave API Doc in this page (bottom): <a href="https://github.com/pytorch/pytorch/tree/master/android" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/tree/master/android</a><br>
I just paste it here: <a href="https://pytorch.org/javadoc/1.9.0/" class="inline-onebox" rel="noopener nofollow ugc">pytorch_android 1.9.0 API</a></p>
<p>Java API is mainly for model inference and there’s no such tensor operation API in Java. You may manipulate the raw data before constructing the tensor, or put these operations into a model.</p>
Multiple training runs on the same dataset at the same time?

username: meshghi
cooked: <p>Hi, I want to run my model with different configurations. I don’t want to wait for training with one configuration to be done and then start another one. I want to run all of them at the same time. I don’t have a problem with computation power. But, I don’t want to create multiple versions of the same dataset, and I want to use only one copy of the dataset for all trainings. Therefore, I don’t know if pytorch dataloader in one training waits if the dataset is locked by another training?</p>

username: tom
cooked: <p>Typically the dataloader will only use read-only access, so it should work to have multiple processes read the same dataset. That said, if the bandwidth of the underlying storage becomes the bottleneck, you would necessarily observe a slowdown. Also, caching and the storage characteristics could impact the performance of how the dataset is read (but then, the most standard dataloader setups read random samples from the dataset, too).</p>
<p>Best regards</p>
<p>Thomas</p>
BatchNorm 4D (or N-dim)

username: OverLordGoldDragon
cooked: <p>Does BN care about input dimensionality? The <code>1d</code> vs <code>2d</code> vs <code>3d</code> just assert input dims, all else is same. Is below valid for the N-dim (or at least 4D) case?</p>
<pre><code class="lang-python">class BatchNormNd(nn.modules.batchnorm._BatchNorm):
    def _check_input_dim(self, input):
        pass
</code></pre>

username: eqy
cooked: <p>It should be valid as the underlying functionality is general for higher dimensions because the underlying data layout is unchanged; e.g.,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.randn(3, 65, 9, 9)
&gt;&gt;&gt; bn2 = torch.nn.modules.BatchNorm2d(65)
&gt;&gt;&gt; bn1 = torch.nn.modules.BatchNorm1d(65)
&gt;&gt;&gt; out2 = bn2(a)
&gt;&gt;&gt; out1 = bn1(a.reshape(3, 65, 81)).reshape(3, 65, 9, 9)
&gt;&gt;&gt; torch.allclose(out2, out1)
True
</code></pre>

username: OverLordGoldDragon
cooked: <p>I verified this much but I wonder then what’s the point of having <code>1d</code> etc instead of just one <code>BatchNormNd</code>? An API thing maybe, but as long as forward and backward passes are same within a network, I’m fine</p>

username: OverLordGoldDragon
cooked: <p>For one, there’s</p>
<pre><code class="lang-python">RuntimeError: Expected 2 to 5 dimensions, but got 6-dimensional tensor
for argument #1 'input' (while checking arguments for cudnn_batch_norm)
</code></pre>
<p>which is fixable by <code>view</code>, but unsure what motivates the exception.</p>
Calling individual CUDA kernels directly in Pytorch possible?

username: gradienttrenchcoat
cooked: <p>Hello!</p>
<p>I am currently working on benchmarking performances of CUDA kernels that are widely used in certain scientific PyTorch models and want to know if its possible to directly invoke CUDA kernels from a PyTorch script.</p>
<p>For example, I have identified a handful of target kernels for benchmarking (say <code>aten::native_batch_norm</code>, <code>aten::conv2d</code>) and now want to benchmark each kernel with different sized inputs, different data sparsities, different GPUs, etc.</p>
<p>Is there a way I can access these functions directly in a Pytorch script? Otherwise, are there any established workflows for benchmarking these models without falling back to pure CUDA C++?</p>
<p>Thanks in advance!</p>

username: tom
cooked: <p>If you want to have a way to call TorchScript ops from Python under their TorchScript name, you can use <code>torch.ops</code>, e.g. <code>torch.ops.aten.native_batch_norm</code>. Now, this will go through the various “accounting” layers between Python → cpp and the PyTorch dispatching. The latter can be reduced by using inference mode. But that is about it. Depending on your input sizes, the overhead in addition to the kernels, might be large or small.</p>
<p>Best regards</p>
<p>Thomas</p>
Inference on Specific Layers of a Model

username: Yarden
cooked: <p>Hey!</p>
<p>I’m building a flexible module that implement Monte Carlo Dropout for Bayesian Inference on any kind of model.<br>
(Implementing MC dropout by adding dropout to fc layers during test time and inference for multiple times to get distribution of outputs)<br>
To save time, I want the to divide the model into two parts:</p>
<ol>
<li>No dropout implemented so run only once</li>
<li>Layers which are after the first dropout (first fc layer) therefore will run multiple times.</li>
</ol>
<p>I can’t use nn.Sequential() because many of the models I work with have functionality in forward (like x.permute()).</p>
<p>Do you have any Idea how to inference on only specific layers without changing the model itself or the forward() code?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Yarden" data-post="1" data-topic="98313">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/f6c823/48.png" class="avatar"> Yarden:</div>
<blockquote>
<p>Do you have any Idea how to inference on only specific layers without changing the model itself or the forward() code?</p>
</blockquote>
</aside>
<p>Unfortunately not and I think the cleanest way would be to split the model into different parts and call them separately.</p>

username: pymon
cooked: <p>And how could I do that?<br>
I am trying inference on the penultimate layer of an Alexnet Model but I couldn’t get it.</p>

username: ptrblck
cooked: <p>The author of this question was working on a custom model so could split the model into e.g. blocks which would fit their use case.<br>
If you are working with a pretrained model created in e.g. <code>torchvision</code>, you could e.g. replace the last linear layer with an <code>nn.Identity()</code>, could write a custom model by deriving from <code>AlexNet</code> as its base, or use some helper methods from <code>torchvision</code>.</p>
NN does not train

username: HCH
cooked: <p>Hi all,</p>
<p>I have a NN,  you can see the picture below. I have small number of data (66). I split them into (44 train, 22 test). I’ve used binary_cross_entropy, After training the AUC is 0.5. I’ve tried other loss functions, but nothing changed.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/3001df17924e177fb4cd64b4f1005793c59928a0.png" data-download-href="https://discuss.pytorch.org/uploads/default/3001df17924e177fb4cd64b4f1005793c59928a0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3001df17924e177fb4cd64b4f1005793c59928a0_2_386x500.png" alt="image" data-base62-sha1="6QGVxOC76cV48dhux8YzP9eWyZy" width="386" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3001df17924e177fb4cd64b4f1005793c59928a0_2_386x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3001df17924e177fb4cd64b4f1005793c59928a0_2_579x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3001df17924e177fb4cd64b4f1005793c59928a0_2_772x1000.png 2x" data-dominant-color="0D1111"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1356×1754 144 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks</p>

username: ptrblck
cooked: <p>The dataset is tiny and I don’t know which layers your model contains.<br>
Could you post the model definition here so that we could check for obvious issues?</p>

username: HCH
cooked: <p>here is the model:<br>
n_feature = 9<br>
n_genes = 691</p>
<p>class Net_simple(torch.nn.Module):<br>
def <strong>init</strong>(self, n_feature, n_genes, n_output):<br>
super(Net_simple, self).<strong>init</strong>()</p>
<pre><code>    self.nn = torch.nn.Sequential(torch.nn.Linear(n_genes, 1))
    self.final = torch.nn.Sequential(torch.nn.Linear(n_feature, 1))
    
    self.predict1 =   torch.nn.Sigmoid()
    self.float()

def forward(self, x):
    x = x
    o1 = self.nn(x)
    o = self.final(o1.squeeze())
    w = self.predict1(o)
    return w
</code></pre>
<p>Thanks</p>

username: HCH
cooked: <p>My case is binary class.  What kind of loss function is appropriate for the task?</p>

username: ptrblck
cooked: <p>Your model won’t work, as you are hitting a shape mismatch between <code>self.nn</code> and <code>self.final</code>.<br>
The <code>out_features</code> of one linear layer are used as the <code>in_features</code> of the next one (in the common use case), so remove the <code>squeeze</code> in the <code>forward</code> and use:</p>
<pre><code class="lang-python">self.nn = torch.nn.Sequential(torch.nn.Linear(n_genes, n_feature))
</code></pre>
Does contiguous tensor affect training result?

username: nicozhou
cooked: <p>Hi, I have read a lot of thread regarding contiguos tensor, I know some may say that it affect the speed when forwarding, but I am curious, does it affect network training result? for e.g accuracy, loss, etc.</p>
<p>It may seem that when using <strong>torch.reshape</strong>, there is no need to use contiguous tensor right?</p>

username: ptrblck
cooked: <p>If an operation expects a contiguous tensor, it’ll make the memory contiguous explicitly since otherwise the indexing might be wrong which would influence the training in a bad way. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="nicozhou" data-post="1" data-topic="143921">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nicozhou/48/37762_2.png" class="avatar"> nicozhou:</div>
<blockquote>
<p>It may seem that when using <strong>torch.reshape</strong>, there is no need to use contiguous tensor right?</p>
</blockquote>
</aside>
<p>Yes, <code>reshape</code> combines a <code>view</code> and <code>contiguous</code> op. As described, PyTorch will either raise an error if a contiguous tensor is expected in an operation or call <code>.contiguous</code> internally.</p>

username: nicozhou
cooked: <p>So, can I conclude that using torch.reshape is better than torch.view?</p>

username: ptrblck
cooked: <p>No, it’s not better, since you often don’t need the copy and could thus avoid it.</p>