akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Moving a batch to neural network

Vaibhav_Chahar: <p>Initially, I have a neural network that take a 1d-tensor of 2500 elements, I run this network 20 times in a for loop to get the 20 output 1d-tensors. Now I want to run this network only one time but with a Batch size of 20. My question is</p>
<ol>
<li>Do I need to change anything in my Neural network to take the account of the changes that I have done?</li>
<li>I tried to get the results without making any changes in the neural network, it is still giving me results but there is one problem, all 20 of them have same weights and biases, but I want them to be independent from each other.</li>
</ol>
<p>Please help me to clarify my doubts.</p>

ptrblck: <ol>
<li>No changes to the model are needed assuming you’ve only changed the batch size.</li>
<li>The output does not contain weights and biases, so I’m unsure what exactly is the same.</li>
</ol>
ARM base image does not exist

nvzt: <p>Why there is no official ARM based image in Docker Registry ? All the released images are based on the AMD.</p>
<blockquote>
<p><a href="https://hub.docker.com/r/pytorch/pytorch/tags" class="inline-onebox" rel="noopener nofollow ugc">Docker</a></p>
</blockquote>

ptrblck: <p>I guess because no feature requests came in yet, so feel free to submit one on GitHub.</p>
Model Time Reduction using torch 2.0 compile

Nirmai: <p>I have been using hidet backend for compiling pytorch model. Iam using the torch 2.0 for reducing the appgen time for a model run.</p>
<p>Specify better ways to reduce model run time by using torch.compile.<br>
hidet.torch.dynamo_config.search_space(2)<br>
with torch.no_grad():<br>
hidet.torch.dynamo_config.print_input_graph(True)<br>
hidet.torch.dynamo_config.correctness_report()<br>
pytorch_model = torch.compile(pytorch_model,backend = ‘hidet’)</p>
<pre><code>    return pytorch_model
</code></pre>
RuntimeError: "normal_kernel_cpu" not implemented for 'Byte'

oneluxyou: <p>Having trouble with torchattacks<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f70841a83985e51f592be074c5d79d681fe0104.png" data-download-href="https://discuss.pytorch.org/uploads/default/6f70841a83985e51f592be074c5d79d681fe0104" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f70841a83985e51f592be074c5d79d681fe0104.png" alt="image" data-base62-sha1="fTQ3WRPFFevFX87jvWBuz10K5Io" width="690" height="258" data-dominant-color="1F1923"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">816×306 34.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>torch.randn_like(images)</code> will try to sample random values from a gaussian distribution using the passed <code>images</code> properties such as its <code>dtype</code>. I don’t think it would make a lot of sense to try to use <code>unit8</code> for this sampling operation, so you might either want to transform the <code>dtype</code> of <code>images</code> to a floating point type or use another sampling (e.g. <code>torch.randint</code>).</p>

ndvbd: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> is there a function like randint_like for for more ‘normal distribution’?</p>

ptrblck: <p><code>torch.randn_like</code> will sample from a Normal/Gaussian distribution as explained above, so it this what you are looking for?</p>

ndvbd: <p>Thanks for the quick response and for the awesome work you guys are doing here!<br>
So when we use quantized models, for example Llama model, and have weights that are uint8, if I want to use randn_like, I cannot since it’s int8. So what I am asking if there’s an equivalent function to randn_like that supports uint8 tensors. If not, I guess I need to case it first to float, do the randn_like, and case back to int?</p>

ptrblck: <p>This would be an approach, but note that you might also want to scale the normal distribution, since you would end up with a lot of <code>0</code>s and <code>255</code>s otherwise:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(1000 * 1000 * 1000)

plt.hist(x.numpy())
plt.hist(x.byte().numpy())
</code></pre>
<p>Output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb263359fce9fc9fc58f006eb3f0fa082d7c0ec9.png" alt="image" data-base62-sha1="qHB88XQWoScKkfPIfhYYPnlriBP" width="362" height="259"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/4/046ce3628bcb70662ea874b8c4afd434ed8a100e.png" alt="image" data-base62-sha1="D9cvNWlDkg0Ye0rgRqJHyfeUjk" width="362" height="259"></p>
Confuse about how torch.argmax() supporting `axis` parameter inplace of `dim`

wi-sh: <p><a href="https://pytorch.org/docs/stable/generated/torch.argmax.html?highlight=argmax" rel="noopener nofollow ugc">torch.argmax</a></p>
<pre><code class="lang-python"> torch.argmax(input, dim, keepdim=False) → LongTensor

Parameters:

        input (Tensor) – the input tensor.

        dim (int) – the dimension to reduce. If None, the argmax of the flattened input is returned.

        keepdim (bool) – whether the output tensor has dim retained or not. Ignored if dim=None.
</code></pre>
<p>There is no mentioned in the official docs neither in the <a href="https://github.com/pytorch/pytorch/blob/678ce61cdb0e652ee4dfbb236319bd6acb98b851/aten/src/ATen/native/ReduceOps.cpp#L1564" rel="noopener nofollow ugc">source code</a> but I can pass in <code>axis</code>  instead of <code>dim</code> and it works the same. I am mostly curious about how pytorch does it ?</p>
<pre><code class="lang-python">import torch
a = torch.randn(2,2)
b = torch.argmax(a, dim=1)
c = torch.argmax(a, axis=1)
print(b) # tensor([0, 1])
print(c) # tensor([0, 1])
</code></pre>

ptrblck: <p>The aliasing should be performed in <code>python_arg_parser.cpp</code> <a href="https://github.com/pytorch/pytorch/blob/223f23292873940d1028730fb381d6cf73aac404/torch/csrc/utils/python_arg_parser.cpp#L69-L75">here</a>:</p>
<pre><code class="lang-auto">// Default arg name translations for compatibility with NumPy.
//
// Example:
// ```python
// t = torch.randn(10,10)
// torch.sum(a=t, axis=0, keepdim=True)
// ```
//
// A vector is necessary, because we might need to try multiple values.
// In particular, NumPy sometimes uses "x" and sometimes "a" for the main input
// tensor. Rather than annotate each function separately with whether it should
// take "x" or "a", just try both.
//
// TODO: Allow individual functions to specify non-default translations:
// For example, `torch.pow` should translate "exponent" to "x2".
static const std::unordered_map&lt;std::string, std::vector&lt;std::string&gt;&gt;
    numpy_compatibility_arg_names = {
        {"dim", {"axis"}},
        {"keepdim", {"keepdims"}},
        {"input", {"x", "a", "x1"}},
        {"other", {"x2"}},
};
</code></pre>

wi-sh: <p>I see, this makes it so much clear ! Also, how can we find for what all functions this is applied for or do I have to check <code>numpy</code> docs for the function that has the same name?</p>

ptrblck: <p>I don’t know which functions support this argument overload, but would guess all arguments might be checked for this mapping.</p>
How can i modify the feature map size parameter in ResNET50?

Lina_Yi: <p>I want to change the output feature map size of layer4 of ResNet50,</p>
<pre><code class="lang-python">    self.layer1 = self._make_layer(block, 64, layers[0])
        self.layer2 = self._make_layer(block, 128, layers[1], stride=2,
                                       dilate=replace_stride_with_dilation[0])
        self.layer3 = self._make_layer(block, 256, layers[2], stride=2,
                                       dilate=replace_stride_with_dilation[1])
        self.layer4 = self._make_layer(block, 512, layers[3], stride=2,
                                       dilate=replace_stride_with_dilation[2])
        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
</code></pre>
<p>here, i want to change</p>
<pre><code class="lang-auto">self.layer4 = self._make_layer(block, 512, layers[3], stride=2,
                                       dilate=replace_stride_with_dilation[2])
</code></pre>
<p>to</p>
<pre><code class="lang-auto">self.layer4 = self._make_layer(block, 256, layers[3], stride=2,
                                       dilate=replace_stride_with_dilation[2])
</code></pre>
<p>but this is built-in Model so I have no idea to change that. How can i change this ?</p>

J_Johnson: <p>You can specify any model inside of the class as the parent. And then modify whichever layer you want in the <strong>init</strong>. See here:</p>
<aside class="quote quote-modified" data-post="7" data-topic="54224">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-i-replace-the-forward-method-of-a-predefined-torchvision-model-with-my-customized-forward-function/54224/7">How can I replace the forward method of a predefined torchvision model with my customized forward function?</a> 
  </div>
  <blockquote>
    You could derive a custom class using the resnet class as its parent: 
import torchvision.models as models
from torchvision.models.resnet import ResNet, BasicBlock

class MyResNet18(ResNet):
    def __init__(self):
        super(MyResNet18, self).__init__(BasicBlock, [2, 2, 2, 2])
        
    def forward(self, x):
        # change forward here
        x = self.conv1(x)
        return x


model = MyResNet18()
# if you need pretrained weights
model.load_state_dict(models.resnet18(pretrained=True)…
  </blockquote>
</aside>
Autograd.grad with FSDP

sangkeuc: <p>Hello,</p>
<p>I notice that the output of</p>
<pre><code class="lang-auto">grad = torch.autograd.grad(loss, fsdp_module.parameters())
</code></pre>
<p>to be different on different nodes. However, I thought PyTorch FSDP performs reduce_scatter during the backward pass, so that <code>grad</code> should be same on every node?</p>
<p>Another thing I noticed is that the length of <code>grad</code> is same with the full parameter number rather than the shared parameter number.</p>
<p>In general, I am confused about how <code>torch.autograd.grad</code> works with FSDP. Any clarification would be much appreciated!</p>
Gradient checkpointing causes some parameters to not receive grad error?

yxchng1: <p>I have code written as follows</p>
<pre><code class="lang-auto">
if self.use_checkpoint:
    out = torch.utils.checkpoint.checkpoint(upscale_layer, x_out.permute(0,3,1,2).contiguous())
else:
    out = upscale_layer(x_out.permute(0,3,1,2).contiguous())
</code></pre>
<p>when all the parameters before this line are frozen, using gradient checkpointing will lead to the following error:</p>
<p><code>RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument </code>find_unused_parameters=True<code>to</code>torch.nn.parallel.DistributedDataParallel<code>, and by ...</code></p>
<p>If use_checkpoint is false, no problem occurs.</p>
<p>Why? Is this expected?</p>
Expecting numpy.array.tolist() as input parameter on inference for custom handler

dhx: <p>From python script, I am doing Torch serve on like:</p>
<pre><code class="lang-auto">arr = np.array(np.random.randn(x,y,z))
data = {'arr': arr.tolist()}
response = requests.post(url, json=data)
</code></pre>
<p>In the inference method of the custom handler I have:</p>
<pre><code class="lang-auto">    def inference(self, ds, *args, **kwargs):
        print("type of ds") ##### &lt;class 'list'&gt;
        print(type(ds)) 
        print("type of ds[0]")
        print(type(ds[0])) ##### &lt;class 'dict'&gt;
        count = 0
        whatineed = None
        for k, v in ds[0].items():
            tk = type(k)
            tv = type(v)
            for kk, vv in v.items():
                print("KK: " + str(kk))  ##### KK: arr
                print("Len of vv: " + str(len(vv))) # 1
                whatineed = vv
            count = count+1

        ... [ other stuff ] 
</code></pre>
<p>How can I avoid searching for whatineed?</p>
<p>Is there an easier way to package the input array on the request side?</p>
Preload model dependencies

lsmith77: <p>it seems like torch-model-archiver does not package the dependencies. so torchserve is apparently downloading the dependencies after start up or when the model is added via the management API. is there a way to fetch those dependencies beforehand?</p>
<p>if I run a server I don’t really want to download dependencies at run time.</p>

marksaroufim: <p>You can pass in your virtual environment when you arxiv the model <a href="https://github.com/pytorch/serve/issues/1935" class="inline-onebox">cache per-model dependency in model archiver · Issue #1935 · pytorch/serve · GitHub</a></p>

lsmith77: <p>ok thx … that looks rather hacky. I guess another approach could be to make the model dependencies global dependencies?</p>

marksaroufim: <p>Yup that works too, for any real prod deployment I cant recommend dependencies per model</p>
InPlace BatchNorm

bluesky314: <p>This batchnorm claims it is 60% faster than pytorch’s. Why is this not the default in Pytorch?<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/mapillary/inplace_abn" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/5332499?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/mapillary/inplace_abn" target="_blank" rel="nofollow noopener">mapillary/inplace_abn</a></h3>

<p>In-Place Activated BatchNorm for Memory-Optimized Training of DNNs - mapillary/inplace_abn</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


albanD: <p>Hi,</p>
<p>The main reason is that this is not a batchnorm. This is a “combo” layer with batchnorm + activation (+other stuff I don’t know). This approach allows you to save memory (and a bit of runtime).</p>
<blockquote>
<p>This batchnorm claims it is 60% faster than pytorch’s.</p>
</blockquote>
<p>Where does it claims this? This sounds quite unlikely. It’s not memory usage?</p>
What does it mean to normalize images for Resnet?

Luke_Kollmorgen: <p>I’m using a pretty simple set of steps designed to prepare images for feature extraction from a pre trained resnet 152 model.</p>
<pre><code class="lang-auto">
img=Image.open("Documents/img.png")

# Load the pretrained model
model = models.resnet152(pretrained=True)

# Use the model object to select the desired layer
layer = model._modules.get('fc')

# Set model to evaluation mode
model.eval()

transformations = torchvision.transforms.Compose([
    torchvision.transforms.Resize(256),
    torchvision.transforms.CenterCrop(224),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
t_img = transformations(img)


What I'm trying to understand is what it means to normalize the images with those means and standard deviations? We obviously aren't setting pixel values right? My understanding is that pixel values should be integers between 0 and 255, so what does it mean to normalize in this context?</code></pre>

ebarsoum: <p>Normalize in the above case, mean subtract the <code>mean</code> from each pixel and divide the result by the <code>standard deviation</code>. The input image is float not integer in the range of [0, 1]. So when you load the image, you need to divide it by 255.</p>

Sunshine352: <p>[quote=“Luke_Kollmorgen, post:1, topic:96160”]<br>
<code>torchvision.transforms.ToTensor()</code><br>
[/quote] can transform the pixel range from [0, 255] to [0, 1]</p>

zia_badar: <p>but why a constant value of <code>mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]</code> shouldn’t each image be normalized to 0 mean and 1 standard deviation, or is this mean and std is calculated from the whole dataset?</p>

TigerYan86: <p>I have the same question. How are these values determined?</p>

ptrblck: <p>These stats are calculated from the ImageNet training dataset.</p>
`x.shape` or `x.size()` return torch.Size with elements of type Tensor?

kuldeepbrd1: <p>Hi PyTorch community,</p>
<p>TL;DR<br>
Sometimes <code>torch.Size</code> contains elements of type torch.Tensor when using <code>shape</code> or <code>size()</code>  . Why?</p>
<pre><code class="lang-auto"># x:torch.Tensor [1, 40, 32, 32]

tsize = x.shape[2:]
print(tsize)
&gt; torch.Size([32, 32])

print(tsize[0])
&gt; tensor(32)

type(tsize[0])
&gt; torch.Tensor
</code></pre>
<hr>
<p>Details:</p>
<p>I am trying to trace graph and quantize a LiteHRNet model to run with Vitis AI hardware. In principle, the problem I have is in the following snippet that uses <code>nn.Functional.adaptive_avg_pool2d</code> as following:</p>
<pre><code class="lang-auto">def forward(self, x):
        mini_size = x[-1].shape[2:]
        out = [F.adaptive_avg_pool2d(s, mini_size) for s in x[:-1]] + [x[-1]]
        out = torch.cat(out, dim=1)
</code></pre>
<p>I noticed that when using <code>x.shape</code> or <code>x.size()</code> on some of the tensors, the returned size is <code>torch.Size(tensor(32), tensor(32))</code>  instead of <code>torch.Size(32,32)</code>.</p>
<p>The behavior gets weirder in debugging.  when I put a breakpoint on the line with shape computation. and then step ahead, it returns the correct  type <code>torch.Size(32,32)</code>.</p>
<p>But if I break at the line with adaptive_avg_pool2d, the size returned is <code>torch.Size(tensor(32), tensor(32))</code> where elements are int tensors rather than int.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/9287f2e29c6c7c42381fc4a0ae21cda5edb13402.png" data-download-href="https://discuss.pytorch.org/uploads/default/9287f2e29c6c7c42381fc4a0ae21cda5edb13402" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/9287f2e29c6c7c42381fc4a0ae21cda5edb13402_2_690x235.png" alt="image" data-base62-sha1="kUgYJvY7IrIkIwLVQLm4DGGZpGW" width="690" height="235" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/9287f2e29c6c7c42381fc4a0ae21cda5edb13402_2_690x235.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/9287f2e29c6c7c42381fc4a0ae21cda5edb13402_2_1035x352.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/2/9287f2e29c6c7c42381fc4a0ae21cda5edb13402.png 2x" data-dominant-color="2D2D2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1136×387 46.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>(Context) This causes downstream problems when you convert pytorch modules and ops to the ones supported by Vitis AI, where <code>adaptive_avg_pool2d</code> expects <code>output_size</code> in <code>int</code></p>

ravi_teja1: <p>I’ve faced this too. weird that is Tensor.size(dim) returns Tensor sometimes. its hard to debug as well since the behaviour does not seem deterministic!</p>
Low CPU Utilization and Slow Inference with PyTorch and KServe

Brooks_Ryba: <p>Hello everyone,</p>
<p>I’m currently facing an issue with low CPU utilization and slow inference while using PyTorch and KServe. Here are the details of my setup:</p>
<ul>
<li>I’m running my application on an c5.4xl Amazon Machine.</li>
<li>KServe is being used to initialize and scale the machines.</li>
<li>I have a minimum of 5 instances allocated for serving.</li>
<li>Each pod is allocated 14 cores and 26GB of RAM.</li>
<li>I’m running a Hugging Face LLM (Language Model) model.</li>
<li>The configuration file I’m using has the following settings:</li>
</ul>
<pre><code class="lang-auto">vmargs=-XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0
inference_address=http://0.0.0.0:8085
management_address=http://0.0.0.0:8085
metrics_address=http://0.0.0.0:8082
enable_envvars_config=true
service_envelope=kservev2
model_store=/mnt/models/model-store
install_py_dep_per_model=true
model_snapshot={"name":"startup.cfg","modelCount":1,"models":{"t5xl-llm":{"1.0":{"defaultVersion":true,"marName":"model.mar","minWorkers":1,"maxWorkers":2,"batchSize":1,"maxBatchDelay":5000,"responseTimeout":300}}}}
</code></pre>
<p>Despite hammering the nodes with a load test, the CPU utilization reported by Grafana remains low, ranging between 15-30%. Additionally, each inference request takes approximately 0.9 seconds to process and return to the queue, during which time I would expect CPU usage to be high.</p>
<p>I have tried adjusting the <code>maxWorkers</code> parameter in the <code>model_snapshot</code> configuration, varying it from 1 to 8. Surprisingly, this did not result in any performance enhancements; in fact, it seemed to negatively impact performance.</p>
<p>I would like to understand why the CPU utilization remains low despite the load, and if calling the <code>torch.set_num_threads</code> function is necessary for proper scaling. Any insights or suggestions on improving the performance and utilization of the CPU would be greatly appreciated. I have read through the documentation in these links, and after reviewing with other teammates here we’re becoming confused on how to take the proper steps to fully utilize our hardware:</p>
<ul>
<li><a href="https://pytorch.org/serve/large_model_inference.html?highlight=cache" class="inline-onebox" rel="noopener nofollow ugc">20. Serving large models with Torchserve — PyTorch/Serve master documentation</a></li>
<li><a href="https://pytorch.org/tutorials/intermediate/torchserve_with_ipex.html" class="inline-onebox" rel="noopener nofollow ugc">Grokking PyTorch Intel CPU performance from first principles — PyTorch Tutorials 2.0.1+cu117 documentation</a></li>
</ul>
<p>Thank you in advance for your assistance!</p>
<p>Best regards,<br>
Brooks</p>
PT 2.0 - Are compiled models savable

Zha0q1: <p>Hi PyTorch community,</p>
<p>We are evaluating distributed training for PT 2.0 with compilation. We noticed that compiling a ~ 1B model will cause the first few steps to be slower and it can take ~10 mins for training to reach stable and full throughput state. I am wondering if a compiled model can be saved as some intermediate format so that re-launching training with the same model will take less time.</p>
<p>Thanks!</p>

smth: <p>we do cache the compiles, so that when you run the script again, you shouldn’t be running into recompiles (unless the cache got full).</p>
<p>But we can do a lot more, including having a saveable cache, as well as a moveable (or distributed) cache. At the moment, we don’t have this.</p>

Zha0q1: <p>Thanks <a class="mention" href="/u/smth">@smth</a> ! I wonder where is the cache saved to? Also a related question is that we saw this warning when compiling fairseq roberta 1.3B,</p>
<pre><code class="lang-auto">config.cache_size_limit (64)
 4:    function: 'gelu' (/fairseq/fairseq/modules/gelu.py:24)
 4:    reasons:  tensor 'x' size mismatch at index 0. expected 492, actual 440
 4: to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html.
 2: [2023-01-07 09:23:10,654] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64)
 2:    function: 'gelu' (/fairseq/fairseq/modules/gelu.py:24)
 2:    reasons:  tensor 'x' size mismatch at index 0. expected 368, actual 506
 2: to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html.
</code></pre>
<p>Would you share some insights on what this means? Is this an error or a warning? thanks!</p>

GuillaumeTong: <p>As a workaround for the currently missing savable cache function, would it be possible to do a manual workaround by pickling that cache? As Zhaoqi asks, where is the cache kept currently? Is it of the class fields?</p>
<p>Also, this is potentially out-of-scope in this question, but I would be interested in a “save cache” feature for TorchScript too. It there anything like that already implemented?</p>

marksaroufim: <p><a class="mention" href="/u/guillaumetong">@GuillaumeTong</a> I also have this requirement so we can continue the discussion there <a href="https://github.com/pytorch/pytorch/issues/96152" class="inline-onebox" rel="noopener nofollow ugc">Inductor codecache not saving compilation time in multiprocess env · Issue #96152 · pytorch/pytorch · GitHub</a></p>

Nirmai: <p><a class="mention" href="/u/guillaumetong">@GuillaumeTong</a> did you get solution for above problem? are you able to save the compiled model?</p>

marksaroufim: <p><a class="mention" href="/u/nirmai">@Nirmai</a> <a class="mention" href="/u/guillaumetong">@GuillaumeTong</a> this is more early days but there are ways of loading the cache in python. I have a POC here <a href="https://github.com/pytorch/pytorch/pull/103281" class="inline-onebox">AOT Inductor load in python by msaroufim · Pull Request #103281 · pytorch/pytorch · GitHub</a></p>
Looking for feedback

Nikhilesh_Tayal: <p>Hi there,</p>
<p>I am Nikhilesh Tayal - a non-technical person turned AI professional.</p>
<p>I started my ML learning journey 6 months back.<br>
And when I was looking to learn AI/ML, most of the content was focused on Tech people.</p>
<p>There was no space for “non-technical” people like me.</p>
<p>And that’s the reason, I thought of initiating the space where non-technical people can learn cutting-edge technologies like Artificial Intelligence in simple “no-jargonish” language.</p>
<p>Here is the website - <a href="https://www.aimletc.com/" rel="noopener nofollow ugc">https://www.aimletc.com/</a></p>
<p>I am new to teaching and hence would love to get feedback and suggestions from AI experts and Indie Hackers here on this. Thanks!</p>
Implement conv between two matrix with integer precision

mdatres: <p>Dear all,</p>
<p>for my research I need to perform convolution operation between two 2d matrix. I tried using <code>torch.nn.quantized.functional.conv2d(data.unsqueeze(0).unsqueeze(0), wq.unsqueeze(0).unsqueeze(0), scale = 1.0,bias=None)</code><br>
unsqueezing the first two dimension because <code>torch.nn.quantized.functional.conv2d</code> requires 4d tensors, The problem is that doing like this this operation is quite slow. Is there a function that implement the convolution between two matrices using integer operation directly?</p>
<p>Thanks in advanced,</p>
<p>Max</p>
Why pytorch 1.10 in python 3.6 is slower than pytoch1.10 in python 3.9

Mr.Crab: <p>why pytorch 1.10 in python 3.6 is slower than pytoch1.10 in python 3.9?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/3117211561fb4a4258a31d278b510f054183b775.png" data-download-href="https://discuss.pytorch.org/uploads/default/3117211561fb4a4258a31d278b510f054183b775" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/1/3117211561fb4a4258a31d278b510f054183b775.png" alt="image" data-base62-sha1="70gWUAfJJuYcFjCvo4slMd5ojIh" width="690" height="82" data-dominant-color="242424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1132×136 6.04 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The left is python3.6 and the right is python3.9, i test on the same machine.</p>

ptrblck: <p>Could you share a minimal and executable code snippet showing such behavior in supported versions? Python 3.6 is not supported anymore and PyTorch 1.10 is quite old, so let me know if you see anything similar in new releases.</p>
Scaled_dot_product_attn not giving exact same results as normal attention

kkadry: <p>Hello, I’m trying to substitute my QKV attention function with<br>
torch.nn.functional.scaled_dot_product_attention to benefit from memory efficient attention. However, the result is very slightly different from my own implementation, which mostly follows normal attention calculations.<br>
-Training wise I notice slower convergence and often unstable training when i use sdp, so I’m trying to understand where such differences are coming from<br>
-I checked which method is being invoked and it seems the C++ implementation is the only one that works without erroring out.<br>
-This occurs with or without mixed precision<br>
-I set torch.backends.cudnn.deterministic = True and it did not change the output</p>
<p>My variables are query, key, and value, all of the shape (B,L,E) (1, 32768, 192)</p>
<p>Here is the function I’m using<br>
x=torch.nn.functional.scaled_dot_product_attention(query,key,value)</p>
<p>Here is my older implementation<br>
attn_weight = torch.softmax((query @ key.transpose(-2, -1) / math.sqrt(query.size(-1))), dim=-1)<br>
x2=attn_weight <span class="mention">@value</span></p>
<p>I expect x to be exactly equal to x2 but there are minor differences in some elements of the output tensor.<br>
Am I understanding the sdp function correctly? I would have no issue if the outputs are very slightly off but it seems that this affects training significantly.<br>
Thank you!</p>

ptrblck: <aside class="quote no-group" data-username="kkadry" data-post="1" data-topic="176040">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kkadry/48/58877_2.png" class="avatar"> kkadry:</div>
<blockquote>
<p>I expect x to be exactly equal to x2 but there are minor differences in some elements of the output tensor.</p>
</blockquote>
</aside>
<p>You cannot expect exactly bitwise-identical results using floating point precision dtypes without forcing deterministic algorithms due to the limited floating point precision. How large is the relative error between your manual output and the <code>SDP</code> one?</p>

kkadry: <p>Thank you for the response, I half suspected that the differences were due to floating point cutoffs but good to confirm.<br>
If I take (x-x2).sum() I get a difference of about 0.03 for the tensor sizes  (1, 32768, 192)<br>
The main thing that confused me was the different training dynamics when i substituted that sdp function, I had assumed that this was due to the attention being very slightly inexact but it seems thats not the case?</p>

ptrblck: <p>I don’t know if the errors are expected or not since you are directly subtracting the tensors.<br>
What does <code>print((x - x2).abs().max())</code> and <code>print(x.abs().max(), x2.abs().max())</code> return?</p>
Question with GDB usage

111357: <p>I want to study the source code and try to make some breakpoints at both Python code and cpp code by following <a href="https://github.com/pytorch/pytorch/blob/main/CONTRIBUTING.md#gdb-integration" rel="noopener nofollow ugc">gdb-integration</a>.</p>
<p>The output is not consistent with the <a href="https://github.com/pytorch/pytorch/blob/main/CONTRIBUTING.md#gdb-integration" rel="noopener nofollow ugc">gdb-integration</a>. It seems not to hit the breakpoint. Am I missing something or the tutorial is out of date?</p>
<p>Output:</p>
<pre><code class="lang-auto">$ gdb python
GNU gdb (GDB) 11.2
...

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...
(gdb) break at::Tensor::neg
Function "at::Tensor::neg" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (at::Tensor::neg) pending.
(gdb) run
Starting program: /xx/xx/.conda/envs/torch_x/bin/python 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Python 3.8.1 (default, ...) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
[Detaching after fork from child process 152651]
&gt;&gt;&gt; a = torch.Tensor([1])
&gt;&gt;&gt; a.neg()
tensor([-1.])
</code></pre>
<p>Should I install torch with some special flags?</p>
<p>I installed it with CPU only from source code.</p>
<pre><code class="lang-auto">USE_CUDA=0 python setup.py develop
</code></pre>

111357: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I noticed you have answered a related <a href="https://discuss.pytorch.org/t/gdb-for-pytorch/157239/2">question</a>. Is it possible for you to take a look? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

111357: <p>Ohh, I found the root cause, I did build the torch with <code>DEBUG</code> mode.<br>
Although I build it with <code>DEBUG=1 python setup.py install</code>, I already built it with <code>Release</code> mode, torch used the cache CMakeLists.txt.</p>
Difference between batch size in data loader and num_samples in Weighted Random Sampler

peony: <p>I would like to ask the difference between batch size in data loader and num_samples in Weighted Random Sampler.</p>
<p>I used to assign num_samples as <code>num_samples=len(sample_weights)</code> and my dataloader batch size is 16. Then after about 50 epochs, I changed my num_samples to  num_samples=16 and the training accuracy went down, though my validation accuracy did not change much.</p>
<p>I am confused, I know num_samples means the number of samples we draw for each iteration, but is it the same as data loader batch size?</p>
<p>Thank you in advance</p>

ptrblck: <aside class="quote no-group" data-username="peony" data-post="1" data-topic="182546">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e5b9ba/48.png" class="avatar"> peony:</div>
<blockquote>
<p>I am confused, I know num_samples means the number of samples we draw for each iteration, but is it the same as data loader batch size?</p>
</blockquote>
</aside>
<p>The <code>num_samples</code> argument in <code>WeightedRandomSampler</code> defines the number of samples drawn in each <em>epoch</em>, while the <code>batch_size</code> in the <code>DataLoader</code> defines the number of samples drawn in each <em>iteration</em>.</p>

peony: <aside class="quote no-group" data-username="peony" data-post="1" data-topic="182546">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e5b9ba/48.png" class="avatar"> peony:</div>
<blockquote>
<p><code>num_samples=len(sample_weights)</code></p>
</blockquote>
</aside>
<p>Thanks for clarifying <a class="mention" href="/u/ptrblck">@ptrblck</a> , I wonder which one is better drawing samples from entire length of dataset or drawing less?</p>

ptrblck: <p>The common approach would be to allow the sampler to use all samples unless you have a valid reason to reduce the number of samples and decrease the number of samples in each epoch.<br>
Note that even then you would still be able to draw all samples, but would need to increase the number of epochs to draw the same number of samples.</p>
Multistream resnet50 /I3D

Maths_Electronics_Tu: <p>i trained two models based on I3D from mmaction2  config , one for RGB dataset and the second for optical flow , i need to fuse the best models but i need flexibility to fuse them at  any layer  or  final stage classifier  , i need design class that take the pretarined model (pth) as base and creat new model ,that  i can make choice in which layer i concatenate outputs to feed than  one branch , final classifier .</p>
<p>my first is to fuse them  before feeding  final head as i3d , the second time  i will freeze up layers  and fuse outputs to train new weights and bias of the same architecture of i3d (inflated resnet50)</p>

Maths_Electronics_Tu: <p>any help  please have time to contirbute</p>
Best practice of implementing datapipe loading with both multiprocessing and Python threading

BarclayII: <p>Hi all,<br>
I’m trying to develop a data loading and training pipeline using DataPipes.  Our pipeline consists of three stages:</p>
<ol>
<li>A CPU-compute-intensive stage that computes the indices to fetch necessary features from given the minibatch.  I think this is best done with multiprocessing where I spawn multiple subprocesses to run this stage.</li>
<li>An IO-bound stage that fetches the features from the given indices and transfers them to GPU.  I would like to put this stage in a separate Python thread in the main process.  It then issues I/O requests and initiates GPU transfers to maximize overlap.</li>
<li>A GPU-compute-intensive stage that updates the model given the features on GPU.</li>
</ol>
<p>The question is: <strong>what is the best practice to implement such a pipeline?</strong>  For now, I can think of two options:</p>
<ul>
<li>Wrap the first stage as an <code>IterDataPipe</code> whose <code>__iter__</code> function yields elements from a PyTorch multiprocessing DataLoader.  Then wrap the second stage as an <code>IterDataPipe</code> that yields elements from a thread running in the background.  I’m not sure if allowing a DataPipe object to spawn processes/threads is a good idea.</li>
<li>Write my own <code>ReadingServiceInterface</code>, but it seems that it is only available in <code>DataLoader2</code>, while I would like to use it together with other third party tools like PyTorch Lightning (which only supports <code>DataLoader</code>).</li>
</ul>
<p>Happy to discuss further.</p>
Pytorch not able to use GPU || AWS Graviton Instance

Anoop_Sharma: <p>I installed pytorch and in my <code>.py</code> file I gave</p>
<pre><code class="lang-auto">os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0,1"
</code></pre>
<p>My NVIDIA-SMI output looks like this</p>
<pre><code class="lang-auto">+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03              Driver Version: 530.41.03    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA T4G                      Off| 00000000:00:1E.0 Off |                    0 |
| N/A   52C    P8               10W /  70W|      2MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA T4G                      Off| 00000000:00:1F.0 Off |                    0 |
| N/A   52C    P8               10W /  70W|      2MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
</code></pre>
<p>When I run my application, The model is not loaded in GPU memory. Is there anything that I’m missing ?</p>
<p>Do I need to lower my CUDA version?</p>
<p>Also when I installed pytorch from<br>
<code>pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121</code></p>
<p>It says</p>
<p><code>ERROR: Could not find a version that satisfies the requirement torch (from versions: none) ERROR: No matching distribution found for torch </code><br>
How can I make sure that pytorch uses GPU here</p>

ptrblck: <p>The binaries for ARM do not support CUDA as these were built for Mac only.<br>
You could either build PyTorch from source with CUDA support or e.g. use an <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch">NGC container</a>.</p>
Pytorch installation from source

Srinivasan_Subramani: <p>I forked Pytorch from GitHub and I followed the following steps</p>
<ol>
<li>conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi</li>
<li>git clone --recursive <a href="https://github.com/pytorch/pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a>
</li>
<li>cd pytorch</li>
<li>python tools/amd_build/build_amd.py</li>
<li>python setup.py install</li>
</ol>
<p>I successfully completed the PyTorch build process. However, when I try to run PyTorch, it is unable to detect my AMD GPU. Additionally, PyTorch is unable to assign streams or execute any functions that involve CUDA. This issue is quite perplexing as the GPU-related functionality seems to be completely non-functional.</p>
Differentiable top k masking

damaggu: <p>I would like to mask an input based on the top k masking values, naively doing something as in the following code. Since this is not differentiable, I wanted to ask if there’s a differentiable workaround to achieve the same thing? Thanks</p>
<pre><code class="lang-auto">import torch
top = 2

inp = torch.rand(5, 5, requires_grad=True)
mask = torch.rand(5, 5, requires_grad=True)

top_mask_indices = torch.topk(mask, top, dim=0).indices
masked_inp = torch.gather(inp, 0, top_mask_indices)

loss = torch.mean((masked_inp - torch.rand(top, 5)) ** 2)
loss.backward()

print(mask.grad)
print(inp.grad)
</code></pre>

KFrank: <p>Hi Damaggu!</p>
<aside class="quote no-group" data-username="damaggu" data-post="1" data-topic="163259" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/damaggu/48/53923_2.png" class="avatar"> damaggu:</div>
<blockquote>
<p>I wanted to ask if there’s a differentiable workaround to achieve the same thing?</p>
</blockquote>
</aside>
<p>When you mask with your top-k indices, you are keeping either 100% of the<br>
terms that are included by the mask or 0% of the terms that are excluded.</p>
<p>The approach that comes to mind is to convert the elements of your <code>mask</code><br>
tensor to weight-like or probability-like values, and use them to weight the<br>
terms in your loss function.  The largest values in <code>mask</code> (that would be in,<br>
say, the top k) would weight terms in your loss function the most heavily,<br>
but values that didn’t make it into the top k would still contribute, and you<br>
would avoid the non-differentiable  discreteness of in-vs.-out of the top k.</p>
<p>Something like:</p>
<pre data-code-wrap="python"><code class="lang-python">inp = torch.rand(5, 5, requires_grad=True)
weighted_mask = torch.rand(5, 5, requires_grad=True)   # probability-like
target = torch.rand(5, 5, requires_grad=True)

weighted_loss = torch.sum (weighted_mask * (inp - target)**2) / weighted_mask.sum()
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Arcyl: <p>Hello <a class="mention" href="/u/damaggu">@damaggu</a> , did you find a workaround?</p>
Weighting loss vs changing learning rate (using SGD)

j-piland: <p>I want to be sure that weighting the loss in pytorch gets the same weights result as changing learning rate would.<br>
The gradients attached to the weights will be different, but that’s fine.<br>
This math works on paper, so I’m curious about the implementation.</p>
<p>In short, is <code>Loss * 1</code> with <code>lr = 2</code> the same as <code>Loss * 2</code> with <code>lr=1</code>. EDIT: and would that apply to any equivalent pair?</p>
<p>Here is two short bits of code that demonstrate the behavior that I want:</p>
<ol>
<li>With lr=2</li>
</ol>
<pre><code class="lang-auto">import torch as t

inp = t.tensor([[.1,.2,.3],[.2,.3,.4],[.4,.5,.6]], requires_grad=True)
w1 = t.tensor([[.1,.2,.3],[.2,.3,.4],[.4,.5,.6]], requires_grad=True)
w2 = t.tensor([[.1,.2,.3],[.2,.3,.4],[.4,.5,.6]], requires_grad=True)

a = (inp*w1)
b = (a*w2)
L = b.sum()

L.backward()
opt = t.optim.SGD([w1,w2],lr=2)
opt.step()
print(w1,w2,sep='\n')
print(w1.grad,w2.grad,sep='\n')
</code></pre>
<p>With L*2</p>
<pre><code class="lang-auto">import torch as t

inp = t.tensor([[.1,.2,.3],[.2,.3,.4],[.4,.5,.6]], requires_grad=True)
w1 = t.tensor([[.1,.2,.3],[.2,.3,.4],[.4,.5,.6]], requires_grad=True)
w2 = t.tensor([[.1,.2,.3],[.2,.3,.4],[.4,.5,.6]], requires_grad=True)

a = (inp*w1)
b = (a*w2)
L = b.sum()

L = 2*L

L.backward()
opt = t.optim.SGD([w1,w2],lr=1)
opt.step()
print(w1,w2,sep='\n')
print(w1.grad,w2.grad,sep='\n')
</code></pre>

ptrblck: <aside class="quote no-group" data-username="j-piland" data-post="1" data-topic="182463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j-piland/48/61447_2.png" class="avatar"> j-piland:</div>
<blockquote>
<p>nd would that apply to any equivalent pair?</p>
</blockquote>
</aside>
<p>I’m not sure if I understand this part correctly, but while your approach would work for a plain <code>SGD</code> optimizer, it will not work for more advanced ones, such as <code>Adam</code>.</p>

j-piland: <p>Thank you, I think that was what I was looking for.</p>
<p>Still, for clarity’s sake, I meant any pair of weight/learning rate combinations that result in the same value.<br>
I.e., w=2 and lr=1 should be the same as w=1 and lr=2; w=3/lr=4 same as w=6/lr=2 and so on.</p>
<p>Also, thanks for the reminder that it only works with optimizers that aren’t changing the learning rate or doing other fancy things.</p>