akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Quantization in Federated learning

username: Taghreed
cooked: <p>Based on these tutorials:</p>
<ul>
<li><a href="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/static_quantization_tutorial.ipynb#scrollTo=UBwjcUDJ7L_1" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></li>
<li><a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) FX Graph Mode Post Training Static Quantization — PyTorch Tutorials 1.11.0+cu102 documentation</a></li>
</ul>
<p>Does Pytorch support quantization for Federated learning?<br>
If yes, how to do quantization during training without any discrepancy between local and global’s weights ?</p>

username: jerryzh168
cooked: <p>we haven’t tested this before, but quantization should work for any pytorch model as long as some constraints are met, e.g. the model needs to be symbolically traceable to use fx graph mode quantization, it feels like a question for federated learning itself, could you describe the question in a bit more detail? preferably with some examples?</p>

username: Taghreed
cooked: <p>when I test torch.quantization before applying federated learning, it works and quantize the models before the training! (when apply get_default_qat_qconfig and prepare_qat_fx) the model weights are quantized from 128 to 78, but when weights are aggregated in the global model after training using convert_fx, errors appears! (TypeError: ‘GraphModule’ object is not subscriptable) check this post for the code and the error please: <a href="https://stackoverflow.com/questions/72512591/typeerror-graphmodule-object-is-not-subscriptable-in-pytorch-quantization" class="inline-onebox" rel="noopener nofollow ugc">python - TypeError: 'GraphModule' object is not subscriptable in PyTorch quantization - Stack Overflow</a><br>
Why model’s wights are changed before applied last API (convert_fx)? Does this API quantizes more and makes the discrepancy between global and local models when aggregated?</p>
<p>Basically, before aggregating I can check the model device (e.g., using this command: next(global_model.parameters()).device), but after quantization I can’t! (I got this error: StopIteration: )</p>
Source not available and xxx.cpp is not found?

username: Fuzhiyuan
cooked: <p>Create a tensor in cpu is OK, but create it in GPU(that`s mean use .to(at::Kcuda)   ) will cause an error.</p>
<p>When i debug this program, i found this error locates at a function:  <em><strong>const KernelFunction&amp; lookup(DispatchKey k) const</strong></em>      (line 176 in OperatorEntry.h)<br>
and its parameter value is NULL?<br>
Detail Value can be obtained from the following :</p>
<pre><code class="lang-auto">k             "CUDA(2'\x')
kernel	      {functor_={target_=0x0000000000000000 &lt;NULL&gt; } boxed_kernel_func_=0x0000000000000000 unboxed_kernel_func_=...}	const c10::KernelFunction &amp;
this	      0x0000000001f372a0 {name_={name="aten::empty_strided" overload_name="" } schema_={...} dispatchTable_=...}	c10::impl::OperatorEntry *

</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/1/31f7678b8f863521fcad420c863d925c9f824272.png" alt="QQ图片20220602170055" data-base62-sha1="781snmFCkSwl9urKsskfQdvzhTA" width="565" height="194"></p>
<p>The function code：</p>
<pre><code class="lang-auto"> const KernelFunction&amp; lookup(DispatchKey k) const {
    const auto idx = getDispatchTableIndexForDispatchKey(k);
    if (C10_UNLIKELY(idx == -1)) {
      reportError(k);
    }
    const auto&amp; kernel = dispatchTable_[idx];
    // A valid kernel *always* has a boxed kernel and *may* have an
    // unboxed kernel. However, we typically do unboxed calls in at::
    // APIs, where the kernel 1) will very likely be valid and 2)
    // should have an unboxed kernel. Checking the unboxed kernel
    // first will allow us to avoid touching the boxed kernel at all
    // in the common case.
    if (C10_UNLIKELY(!kernel.isValidUnboxed())) {
      if (!kernel.isValid()) {
        reportError(k);
      }
    }
    return kernel;
  }
</code></pre>

username: ptrblck
cooked: <p>Are you building the application with debug symbols? If not, the debugger might not be able to see all arguments etc.</p>

username: Fuzhiyuan
cooked: <p>Sorry for replaying you so late,  how to check whether i use debug symbol? I just know i built it in debug mode  <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

username: Fuzhiyuan
cooked: <p>Yes, I do use debug symbol that comes from Microsoft  symbol server.</p>
Torch SDE simulation error

username: Rishabh_Kumar
cooked: <p>I tried simulating the Ornstein Uhlenbeck process from torch sde. The SDE model looks like:<br>
$ dX_t = \theta (\mu- X_t)dt + \sigma dW_t $<br>
Here is my code: <code> </code>class OU(nn.Module):</p>
<pre><code>noise_type = "diagonal"
sde_type = "ito"

def __init__(self, theta, mu, sigma):
    super().__init__()
    self.theta = theta
    self.mu = mu
    self.sigma = sigma
def f(self, t, y):
    return self.theta*(self.mu - y)

def g(self, t, y):
    return self.sigma``
</code></pre>
<p>When an object is created from this class, I get the following error:</p>
<pre><code class="lang-auto">AttributeError                            Traceback (most recent call last)
&lt;ipython-input-25-775824621def&gt; in &lt;module&gt;()
      7 
      8 with torch.no_grad():
----&gt; 9     ys = torchsde.sdeint(sde, y0, ts, method='milstein')
     10 plot(ts, ys, xlabel='$t', ylabel='$Y_t$')

1 frames
/usr/local/lib/python3.7/dist-packages/torchsde/_core/sdeint.py in check_contract(sde, y0, ts, bm, method, adaptive, options, names, logqp)
    205     if hasattr(sde, 'g'):
    206         has_g = True
--&gt; 207         g_diffusion_shape = tuple(sde.g(ts[0], y0).size())
    208         _check_2d_or_3d('Diffusion', g_diffusion_shape)
    209     if hasattr(sde, 'f_and_g'):

**AttributeError: 'float' object has no attribute 'size'**
</code></pre>

username: ptrblck
cooked: <p><code>self.sigma</code> seems to be a <code>float</code> while a tensor is expected as <code>.size()</code> is called on this return value.<br>
I don’t know if transforming the <code>float</code> scalar into a tensor would work or if more changes are needed, but it might be worth a try.</p>

username: Rishabh_Kumar
cooked: <p>Thanks for the tip! Transforming the <code>self.sigma</code> scalar to a tensor by <code>self.sigma*torch.ones(y.size())</code> worked for me <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
The equivalent of nn.Parameter in LibTorch

username: JamesDickens
cooked: <p>I am trying to port a python PyTorch model to LibTorch.</p>
<p>In python the line of code is:<br>
nn.Parameter(A) where A is a torch.tensor with requires_grad=True.</p>
<p>What would be the equivalent of this for a torch::Tensor in C++ ?<br>
The autocomplete in my editor gives options for ParameterDict, ParameterList,<br>
ParameterDictImpl, ParamaterListImpl, but no Parameter.</p>

username: ptrblck
cooked: <p>To register a parameter (or tensor which requires gradients) to a module, you could use:</p>
<pre><code class="lang-auto">m.register_parameter("weight", torch::ones({20, 1, 5, 5}), false);
</code></pre>
<p>in <code>libtorch</code>. I don’t know if there is a proper <code>parameter</code> class and have seen the posted approach used in modules.</p>

username: JamesDickens
cooked: <p>Is that third argument, the bool, related to requires_grad? Thanks for the help!</p>

username: ptrblck
cooked: <p>Yes, I think in the module class it should be the <code>requires_grad</code> attribute as seen <a href="https://github.com/pytorch/pytorch/blob/7b701ce2d40480f6f718daacda6fd190de3f282f/torch/csrc/api/src/nn/module.cpp#L306-L326">here</a>:</p>
<pre><code class="lang-auto">Tensor&amp; Module::register_parameter(
    std::string name,
    Tensor tensor,
    bool requires_grad) {
  TORCH_CHECK(!name.empty(), "Parameter name must not be empty");
  TORCH_CHECK(
      name.find('.') == std::string::npos,
      "Parameter name must not contain a dot (got '",
      name,
      "')");
  if (!tensor.defined()) {
    if (requires_grad) {
      TORCH_WARN(
        "An undefined tensor cannot require grad. ",
        "Ignoring the `requires_grad=true` function parameter.");
    }
  } else {
    tensor.set_requires_grad(requires_grad);
  }
  return parameters_.insert(std::move(name), std::move(tensor));
}
</code></pre>
<p>while the <a href="https://github.com/pytorch/pytorch/blob/fc2cf3d26f9b0232bd664dac8c4299293954b997/torch/csrc/jit/api/module.h#L128-L134">jit API</a> seems to use it as <code>is_buffer</code>:</p>
<pre><code class="lang-auto">  void register_parameter(
      const std::string&amp; name,
      at::Tensor v,
      bool is_buffer) {
    type()-&gt;addOrCheckAttribute(name, TensorType::get(), !is_buffer, is_buffer);
    _ivalue()-&gt;setAttr(name, std::move(v));
  }
</code></pre>
Understanding Trace vs. Serialize when using a TorchScript Model

username: JamesDickens
cooked: <p>I have two functions as part of a torch.Module object</p>
<pre><code class="lang-auto">def trace_model(model, sample_input, save_fp=None):
    model.eval()
    traced_model = torch.jit.trace(model, sample_input)
    if save_fp is not None:
        traced_model.save(save_fp)
    return traced_model


def serialize_model(model, sample_input, save_fp=None):
    model.eval()
    serialized_model = torch.jit.script(model, sample_input)
    if save_fp is not None:
        serialized_model.save(save_fp)
    return serialized_model
</code></pre>
<p>One traces the model and saves the result, while the other serializes it. Functionally what is the difference and benefits/draw backs of tracing versus serializing in the context of using the resulting .pt file in a C++ application ?</p>

username: ptrblck
cooked: <p><code>torch.jit.trace</code> will record the used operations via executing the forward pass with the provided input. Conditional control flow and other data-dependent properties will be captured during this execution and saved into the graph.<br>
<code>torch.jit.script</code> is able to understand loops, conditions, etc. and allows for a more flexible graph creation.</p>
Detected CUDA version (11.7) mismatches the version that was used to compile PyTorch (10.2)

username: sharifinjf
cooked: <p>Hi<br>
I am trying to install packages for training a network and I receive the following error:<br>
RuntimeError:<br>
The detected CUDA version (11.7) mismatches the version that was used to compile<br>
PyTorch (10.2). Please make sure to use the same CUDA versions.</p>
<p>ubuntu 20.04.4  LTS<br>
CUDA Version: 11.7<br>
torch  1.11.0<br>
GPU NVIDIA RTX A4000</p>

username: ptrblck
cooked: <p>This issue is raised if your local CUDA toolkit (11.7) doesn’t match the used CUDA runtime from PyTorch (10.2).<br>
Since you are using an Ampere GPU you would need to use CUDA &gt;=11.0 and the 10.2 runtime won’t even work.<br>
Make sure to use the PyTorch binaries with CUDA 11 and try to rebuild.<br>
If you are hitting this issue during a build of a 3rd party library, check if they are using a requirement of a specific <code>torch</code> version and are downloading the CUDA 10.2 runtime in the background.</p>
Expected scalar type Double but found Float

username: ptinn
cooked: <p>Hi,</p>
<p>Running inference using a state_dict from training on modified mobilenet_v2, I get a <code>RuntimeError: expected scalar type Double but found Float</code>  message even though my input image is of type <code>torch.double</code>.  What might be causing the type mismatch if not the input image itself?  Below is my inference setup:</p>
<pre><code class="lang-auto">model = mobilenet_v2()
num_classes = 16
model.classifier[1] = torch.nn.Linear(1280, num_classes)
model.load_state_dict(torch.load(path.join(...)
...
image = torch.as_tensor(np.transpose(image, (2, 0, 1)), dtype=torch.double, device=None)
image = image[None, : ]
pred = model(image)

...
     
     return F.conv2d(input, weight ...)
&gt;&gt; RuntimeError: expected scalar type Double but found Float

</code></pre>

username: ptinn
cooked: <p>Ok…  when I do <code>model = modelnet_v2().float()</code>  and change the input image tensor to <code>dtype=torch.float</code>, the inference actually runs.</p>
<p>BUT, when both the model and image are set to double, it gives a <code>RuntimeError: expected scalar type Float but found Double</code>.     After reading <a href="https://discuss.pytorch.org/t/runtimeerror-expected-object-of-scalar-type-double-but-got-scalar-type-float-for-argument-2-weight/38961">this thread</a>, I still don’t see the pattern.</p>
<p>How does one interpret such behaviors?</p>

username: ptrblck
cooked: <p>Both formats work for me:</p>
<pre><code class="lang-python"># float32
dtype = torch.float32
model = models.mobilenet_v2()
num_classes = 16
model.classifier[1] = torch.nn.Linear(1280, num_classes)
model.to(dtype)
image = torch.randn(1, 3, 224, 224, dtype=dtype)

pred = model(image) # works
print(pred.dtype)
# torch.float32

# float64
dtype = torch.float64
model = models.mobilenet_v2()
num_classes = 16
model.classifier[1] = torch.nn.Linear(1280, num_classes)
model.to(dtype)
image = torch.randn(1, 3, 224, 224, dtype=dtype)

pred = model(image) # works
print(pred.dtype)
# torch.float64
</code></pre>
<p>so could you post a code snippet reproducing the issue?</p>
Coordinate Heatmap

username: Mickey_Han
cooked: <p>Hi, Thank you very much for reading my question. as stated in title, if I am trying to regress a coordinate in a 512x512 image ;</p>
<pre><code class="lang-auto">oo = np.zeros((512, 512))
oo[212,313] = 1   &lt;-- target
</code></pre>
<p>how can I convert this point to a ‘bigger target’ by implementing gaussian range ? and would it be beneficial ? also, what would the final layer of my model be like and what loss function should I be using ?</p>
<p>Thank you very much for your time in reading and helping my with my question and sorry for asking so many questions at once . Thank you !</p>
`torch.mm()` returns RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasSetStream(handle, stream)`

username: Dylan_Hanna
cooked: <p>Steps to reproduce:</p>
<pre><code class="lang-auto"># Create test tensor
x = torch.tensor([[1.0, 2.0]])

# Ensure tensor is on cpu
x = x.cpu()

# This code works fine, so there is no issue with syntax/shapes
torch.mm(x, x.T)

# Send tensor to cuda device (without errors)
x = x.cuda()

# This returns an error.
torch.mm(x, x.T)
</code></pre>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling <code>cublasSetStream(handle, stream)</code></p>
<p>Some specs:</p>
<ul>
<li>
<code>ubuntu</code> 20.04</li>
<li><code>torch==1.10.1</code></li>
<li>RTX 3090</li>
</ul>
<p>Anyone have any idea what’s going on? Most discussions around this topic talk about this error actually being because of <code>shape</code> mis-matches or CUDA memory errors, but neither of these are the case with this example. Happy to provide more detail about my environment if necessary.</p>

username: ptrblck
cooked: <p>Which CUDA runtime is your PyTorch installation using? If it’s 10.2, please update the binaries with the CUDA 11 runtime as this is needed for your Ampere GPU.</p>

username: Dylan_Hanna
cooked: <p>It turns out that I had built <code>torch</code> with the 10.2 runtime.<br>
I uninstalled it and reinstalled the nightly binaries for 11.6 using this command:</p>
<p><code>pip install torch --pre --extra-index-url https://download.pytorch.org/whl/nightly/cu116</code></p>
<p>But now the code above is giving me a different error:</p>
<p>RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.</p>
<p><code>CUDA_VISIBLE_DEVICES</code> is set to <code>0</code>.</p>

username: Dylan_Hanna
cooked: <p>Somehow I fixed the issue by playing around with a bunch of cuda stuff and restarting my machine. I’ll close this issue now.</p>
Custom DataLoader for passing paired data (no augmentation for creation)

username: asha97
cooked: <p>I want to write a custom dataset, where I want to create pair from two different datasets based on its label.<br>
For this I need to use nested loop in the data loader. But whenever I am trying to write loop inside the get_item(self,index) it is not working.</p>
<p>Or I need to write this part somewhere else.<br>
For a given index, I want to match its label and pair it with data points of other dataset and return all the combinations.</p>
How to calculate the required number of FPGA logic elements || Cost of production for pytorch model

username: szahan
cooked: <p>Hi, I am trying to figure out how to calculate the required number of FPGA logic elements for my deep learning model. I can calculate FLOPS and the number of params but have no idea about FPGA. Could anyone help with any suggestions?<br>
I need to estimate the cost of production in an embedded system.</p>
<p>Thanks for any kind of suggestions or directions on where I might get started.</p>
Unbatched vs Batched size=1

username: matinhabi
cooked: <p>Hello everyone,<br>
I want to use <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">LSTM</a> for gesture classification. I wonder if there is any difference between unbatched input and batched input with size set to 1. I think the resulting neural network should be exactly same when we feed it with unbatched input or batched input and batch size = 1. Is this speculation correct? If the answer is no, what’s the difference between them?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>Both approaches will yield the same output with a different shape (with and without a batch dimension) as seen e.g. in this small test:</p>
<pre><code class="lang-python">lstm = nn.LSTM(2, 2)

x = torch.randn(2, 1, 2)
out = lstm(x)
print(out)

out2 = lstm(x[:, 0])
print(out2)
</code></pre>

username: matinhabi
cooked: <p>Thanks for the answer. So in both cases we have used SGD, right?</p>

username: ptrblck
cooked: <p>Assuming you would calculate a loss, the corresponding gradients, and then update the parameters of the model, then you could call this approach a stochastic gradient descent.<br>
In the current code snippet none of this is done (only the forward pass), so I guess you meant a complete training loop.</p>

username: matinhabi
cooked: <p>Thank you for the clarification. Yes I meant a complete training loop. Based on your answer and comment, it seems that both approaches are the same as SGD. Is my understanding correct?</p>
Problem configuring pytorch on local machine

username: anuzcan
cooked: <p>Greetings and thanks for the help<br>
What will be the reason that when executing a .py file it fails to import my “torch” module but if I call it directly within the python interpreter, I have no problems.<br>
Ubuntu 20.04, and python 3.8.10</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/2/423cca242c17bd2fe4ac6c7c72be4a674445a429.png" data-download-href="https://discuss.pytorch.org/uploads/default/423cca242c17bd2fe4ac6c7c72be4a674445a429" title="Captura de pantalla de 2022-06-04 14-02-08"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/423cca242c17bd2fe4ac6c7c72be4a674445a429_2_690x387.png" alt="Captura de pantalla de 2022-06-04 14-02-08" data-base62-sha1="9rXKL8tRAsAKe86bEXNMmuf5sdb" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/423cca242c17bd2fe4ac6c7c72be4a674445a429_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/423cca242c17bd2fe4ac6c7c72be4a674445a429_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/2/423cca242c17bd2fe4ac6c7c72be4a674445a429.png 2x" data-dominant-color="232527"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Captura de pantalla de 2022-06-04 14-02-08</span><span class="informations">1366×768 84.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Your current file is called <code>torch.py</code> which could create these circular imports.<br>
Rename it and it should work.</p>

username: anuzcan
cooked: <p>Thanks a lot. I feel stupid. I think it’s a very basic error</p>

username: ptrblck
cooked: <p>Ah, I think we’ve all ran into it at one point. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
Split pytorch model from input to special layer

username: Hannan_Komari
cooked: <p>Hi all,<br>
Is there any possibility in pytorch where I can split a model from its input to a special layer? Fos this purpose, model.children is not the case as I didn’t care about the functions in between which are not objects.</p>

username: ptrblck
cooked: <p>I’m not completely understanding the issue.<br>
Would you like to forward the input to an additional “special layer” in a model while the default forward pass will still be executed?<br>
If so, you could create a custom module and adapt the <code>forward</code> method accordingly to your use case.</p>
Classification bug

username: Sylvain_Ard
cooked: <p>Hello my code and dataset is here :</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.dropbox.com/sh/qf90423vue22uru/AACZM9f38NElxxNkwnCLcqDZa?dl=0">
  <header class="source">
      <img src="https://cfl.dropboxstatic.com/static/images/favicon-vfl8lUR9B.ico" class="site-icon" width="32" height="32">

      <a href="https://www.dropbox.com/sh/qf90423vue22uru/AACZM9f38NElxxNkwnCLcqDZa?dl=0" target="_blank" rel="noopener nofollow ugc">Dropbox</a>
  </header>

  <article class="onebox-body">
    <img src="https://www.dropbox.com/static/images/spectrum-icons/generated/content/content-folder_dropbox-large.png" class="thumbnail onebox-avatar" width="160" height="160">

<h3><a href="https://www.dropbox.com/sh/qf90423vue22uru/AACZM9f38NElxxNkwnCLcqDZa?dl=0" target="_blank" rel="noopener nofollow ugc">classification</a></h3>

  <p>Shared with Dropbox</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I have the following error :<br>
Traceback (most recent call last):<br>
File “train.py”, line 209, in <br>
loss = criterion(outputs, labels)<br>
File “C:\Users\Sylvain ARD.conda\envs\detectron2\lib\site-packages\torch\nn\modules\module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\Sylvain ARD.conda\envs\detectron2\lib\site-packages\torch\nn\modules\loss.py”, line 1048, in forward<br>
ignore_index=self.ignore_index, reduction=self.reduction)<br>
File “C:\Users\Sylvain ARD.conda\envs\detectron2\lib\site-packages\torch\nn\functional.py”, line 2690, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “C:\Users\Sylvain ARD.conda\envs\detectron2\lib\site-packages\torch\nn\functional.py”, line 2385, in nll_loss<br>
ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
IndexError: Target 3 is out of bounds.</p>
<p>please help me !</p>

username: Sylvain_Ard
cooked: <p>I have found the bug it was at line :<br>
model.fc = nn.Linear(num_ftrs, 3)<br>
3 is the number of classes, I replaced it by num_classes=14 and it works !</p>
Torch.tensor.mps()

username: David_Laxer
cooked: <p>Why doesn’t tensor.mps() work?<br>
E.g.</p>
<pre><code class="lang-auto">In [22]: import torch

In [23]: print(torch.__version__)
1.13.0a0+gitb637810

In [24]: x = torch.tensor([1., 2.]).cpu()

In [25]: x
Out[25]: tensor([1., 2.])

In [26]: y = torch.tensor([1., 2.]).mps()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [26], in &lt;cell line: 1&gt;()
----&gt; 1 y = torch.tensor([1., 2.]).mps()

AttributeError: 'Tensor' object has no attribute 'mps'
</code></pre>
Creating a CNN with bi-temporal image data

username: jbeek00
cooked: <p>I’m looking into ways to create a convolutional net that can classify whether deforestation has occurred in an image based on a prior image of that same area. My dataset is a large set of pairs of image tiles of a general area in the Amazon at a time t1 and later time t2. Each image pair shows the same location on a map. Anyone have advice on how I could best approach this?</p>
<p>Keeping things simple, I define deforestation as: if a pixel has a lighter shade of green or goes from a shade of green to a different color in the second image compared to the first. If this is the case, the pixel could be assigned -1. +1 could be assigned if the opposite is true and 0 if no change has occurred. Then, I’d aggregate this over all the pixels in the 256x256 pixel image for a classification for the image as a whole.</p>
<p>Conceptually, I feel like this shouldn’t be too hard to implement but I’m pretty new to all this so any help would be awesome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: JuanFMontesinos
cooked: <p>Quick hints:<br>
Classification stands for a label for the whole input. If you want to compute per-pixel.</p>
<p>If you have a labeled dataset you can do segmentation easily with a U-Net.<br>
If you have images classified as def and no def you can use metric learning (google it) for a model to bring no def  vs no def images together and no def vs def appart.</p>
<p>Lasty you can prob pose it as a classification problem passint both images as input.</p>
How to validate wait initializations are working as expected?

username: anlgrbz
cooked: <p>I have a simple GAN architecture with a nn.BCEWithLogitsLoss() as a loss function. My final layer is linear and I am using the default weight initialization.</p>
<p>I want to validate my weight initialisation and expect to get an initial loss of log(2) =0.6931 though when I generate random inputs I get a loss value around 0.66 - 0.67</p>
<p>I believe the difference is significant as repeating the experiments still gives loss values in the same range.</p>
<p>Was looking forward to help to find ways to further debug this?</p>
<p>Thanks!</p>
Accessing a Dataset worker's Dataset class

username: Sam_Lerman
cooked: <p>A <code>DataLoader</code> divides a <code>Dataset</code> across many workers. Is there a way to access (from outside the <code>Dataset</code>) a specific worker’s <code>Dataset</code> class instance?</p>
<p>For example:</p>
<pre><code class="lang-auto">my_dataset = DataLoader(...)

a_dataset_instance = my_dataloader.get_worker_instance(id=worker_id)
</code></pre>
<p>Is this possible?</p>

username: nivek
cooked: <p>From your worker, call <code>torch.utils.data.get_worker_info().dataset</code>.</p>

username: Sam_Lerman
cooked: <p>Sorry, I mean from outside the worker.</p>
About FCOS Model

username: Duc_Nguyen
cooked: <p>I am using FCOS for my task and when I checked the source code, I was confused when I saw the source code to calculate the area below. Please check if it is OK and why? Thanks</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b4edea0a03c4f97627288eacab3187f93053e72f.png" data-download-href="https://discuss.pytorch.org/uploads/default/b4edea0a03c4f97627288eacab3187f93053e72f" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4edea0a03c4f97627288eacab3187f93053e72f_2_690x385.png" alt="Capture" data-base62-sha1="pOzFL4yz2Ht9Gn8jpPvXWZeaFYj" width="690" height="385" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4edea0a03c4f97627288eacab3187f93053e72f_2_690x385.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4edea0a03c4f97627288eacab3187f93053e72f_2_1035x577.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4edea0a03c4f97627288eacab3187f93053e72f_2_1380x770.png 2x" data-dominant-color="292C2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1913×1069 163 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Duc_Nguyen
cooked: <p>Does anybody help me?</p>

username: ptrblck
cooked: <p>Without giving enough information it could be hard to help you properly.<br>
If I understand the confusion correctly, you don’t understand how <code>gt_area</code> is calculated.<br>
Based on the screenshot it seems <code>gt_boxes</code> contains coordinates and the area calculation seems to compute the sides of a rectangle first and multiplies these afterwards.<br>
To better understand it, print the values and verify the calculation locally.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>

username: ppwwyyxx
cooked: <p>The source code was wrong and was fixed in <a href="https://github.com/pytorch/vision/pull/5816" class="inline-onebox" rel="noopener nofollow ugc">fix fcos gt_areas calculation by simonJJJ · Pull Request #5816 · pytorch/vision · GitHub</a></p>
Change first model without change the second but still using the grad of the second

username: heitorrapela
cooked: <p>Hello, I am working on a pipeline with two models, model A and model B.</p>
<p>Model B is a frozen model, and model A is a model that changes the input for model B.</p>
<p>So, the forward pipeline is: input_model_A → model_A → output_model_A (input_model_B) → model_B → output_model_B.</p>
<p>For the backward, I want to use the information of the model_B to help change the parameters of the model A without changing the model B. For that, when I calculate the loss part, I do this to freeze the model_B:</p>
<pre><code class="lang-auto">        for param in model_B.parameters():
            param.requires_grad = False
        model_B.eval()
</code></pre>
<p>So, the model_B does not change its weights and makes the batch norm and dropout deactivate, and also, the optimizer is using only the model_A params.</p>
<p>I checked the model B metrics during the training with the original input, and it is fixed (good signal because the model_B is frozen), and the metrics for the new input are increasing (a signal that something is being optimized). Also, the graph shows that the full pipeline is connected. Does this loss make sense? Or do I should try to make the model_B on .train(), change only the BN and dropout layers to false? (because the optimizer is already on the model_A params)?</p>
No Bias in the pretrianed state dictionary of ResNet18

username: papoo13
cooked: <p>I would like to load my model which is modified with the pre-trained weights of a ResNet18. I noticed that there is no bias in the state_dict of this model and bias=False in the conv2d layer. My questions are:</p>
<p>1 - is it possible to access to a version of these pre-trained models in PyTorch that have bias values too?<br>
2 - Why pre-trained bias values are not available and what is the effect of having informed weight values but randomly initialized bias vectors?</p>
<p>I appreciate your guidance and opinions.</p>

username: InnovArul
cooked: <p>In general,  ResNet’s conv layers are followed by batch norm layers which perform <code>mean</code> subtraction. This mean subtraction step will effectively remove the influence of bias. Hence,  bias is not used in conv layers (even if bias is used,  their gradients would be almost 0).</p>
Script `zip` function

username: remiliani93
cooked: <p>Hi, I wanted to know if there is a way to script a function that uses <code>zip</code>, specifically I have a module that interleaves two lists and it is raising an error. To reproduce:</p>
<pre><code class="lang-auto">from typing import List

import torch

def interleave(a: List[int], b: List[int]):
    return sum(zip(a, b), ())

torch.jit.script(interleave)
</code></pre>
<p>This raises a <code>RuntimeError</code> with the message <code>iterabletree cannot be used as a value</code>. Is there another way to do this?</p>

username: Gamrix
cooked: <p>Do you have an example that is closer to what you want to achieve? The above example can be done by converting the two <code>List[int]</code>s into Tensors, then doing a <code>cat</code>, <code>reshape</code>, then <code>sum</code>.</p>
Contributing to PyTorch official docs

username: srishti-git1110
cooked: <p>Hi,<br>
I have been actively using PyTorch since a few weeks now and I am interested in contributing to one specific place in the official docs (I have figured out some edits/additions that I’d like to make).</p>
<p>However, this will be my first time contributing to any open source project, so I do not know anything - from how to get the project on my system (if it’s required) to how to open a PR and everything in between.</p>
<p>I found <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#writing-documentation" rel="noopener nofollow ugc">this</a> on PyTorch’s github, and according to this I will be contributing to the user-facing docs.</p>
<p>Herein, they first ask to install and build PyTorch - what does this mean? Like installing pytorch in jupyter/colab etc.?<br>
Then they ask to install pre-requisites. From what my understanding is, we install pre-requisite libraries/packages when starting to work on a project that we clone from github. But, for contributing to documentation, what exactly do these pre-requisites mean?</p>
<pre><code class="lang-auto">cd docs
pip install -r requirements.txt
</code></pre>
<p>They ask us to use these commands to install the requirement file. I’m guessing I’ll first need to clone pytorch’s repo on my local before this, is that it?</p>
<p>Please help me refer to any resources that can guide me.<br>
Any answer appreciated.</p>
Pytorch custom loss.backward returning "RuntimeError: Found dtype Double but expected Float"

username: Padmaksha_Roy
cooked: <p>Hi, I am creating a custom loss function which has a kernel alignment loss and a reconstruction loss.</p>
<p>Kernel loss is calculated as a Frobenius distance between prior kernel and latent space code.</p>
<p>k_loss = torch.linalg.matrix_norm(torch.sub(code_K_norm,prior_K_norm), ord=‘fro’, dim=(- 2, - 1))</p>
<p>Reconstruction loss is calculated as:<br>
reconstruct_loss = torch.nn.MSELoss()(encoder_inputs, dec_out)</p>
<p>dec_out is the output of encoder decoder network.</p>
<p>Now, I calculate the total loss as:</p>
<p>tot_loss = reconstruct_loss + args.w_reg * reg_loss + args.a_reg * k_loss</p>
<p>I have printed all the data types and they all seem to be in float_64.</p>
<p>TYPE OF PARAMS: &lt;class ‘torch.Tensor’&gt;<br>
Total parameters: 243<br>
RECONS LOSS: &lt;class ‘torch.Tensor’&gt;<br>
RECONS LOSS: tensor(0.9852, dtype=torch.float64,<br>
grad_fn=)<br>
W_REG: &lt;class ‘float’&gt;<br>
REG LOSS: &lt;class ‘torch.Tensor’&gt;<br>
A_REG: &lt;class ‘float’&gt;<br>
K_LOSS: &lt;class ‘torch.Tensor’&gt;<br>
K_LOSS: tensor(1.0471, dtype=torch.float64, grad_fn=)<br>
TYPE tot_loss: &lt;class ‘torch.Tensor’&gt;<br>
tot_loss: tensor(1.1261, dtype=torch.float64, grad_fn=)</p>
<p>But, when I call the loss.backward() , it is throwing the error.</p>
<p>optimizer = torch.optim.Adam(model.parameters(),args.learning_rate)<br>
optimizer.zero_grad()<br>
tot_loss.backward()</p>
<p>tot_loss.backward()<br>
File “/usr/local/lib/python3.7/dist-packages/torch/_tensor.py”, line 363, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/autograd/<strong>init</strong>.py”, line 175, in backward<br>
allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: Found dtype Double but expected Float</p>
<p>Before passing the encoder inputs, I am converting each of them to float tensors.</p>
<p>Kindly take a look on this.</p>

username: Padmaksha_Roy
cooked: <p>I can see,</p>
<p>RECONS LOSS: tensor(0.9852, dtype=torch.float64, grad_fn=)<br>
K_LOSS: tensor(1.0471, dtype=torch.float64, grad_fn=)<br>
tot_loss: tensor(1.1261, dtype=torch.float64, grad_fn=)</p>
<p>The grd_fn for the recons loss is “MseLossBackward0”, for K_loss it is “CopyBackwards” and for tot_loss it is “AddBackward0”. Also, the backward prop works fine if I do a k_loss.backward() but fails with both recons_loss and tot_loss although the data type is still float for all of them.</p>

username: Padmaksha_Roy
cooked: <p>Issue resolved after changing the loss function from nn.MSELoss to calculating it without using any library just by using torch.mean(target - output) **2</p>
Output intermediate layer from the torch.jit.ScriptModule version of the network

username: Andrjusha
cooked: <p>Hi guys,</p>
<p>I have the following problem. I use Torch Script for saving the trained model. Afterwards I need to load it and<br>
take the gradient of the model output with respect to one of the intermediate layers output. At the same time I don’t want to output this intermediate layer with the forward method. As far as I understand register_forward_hook method is not yet implemented in ScriptModule. Could you please tell, if there is another way to solve this problem?</p>
<p>Best,<br>
Andrei</p>

username: jinfagang
cooked: <p>Do u find a solution?</p>
Use `register_hook` to save gradients/gradient computation artifacts

username: zaccharieramzi
cooked: <p>I am currently analyzing a module where <code>register_hook</code> is used for a custom computation of the gradient of some intermediary variable. If you want to take a look, it’s <a href="https://github.com/locuslab/deq/blob/master/MDEQ-Vision/lib/models/mdeq_core.py#L457" rel="noopener nofollow ugc">here</a>.</p>
<p>I would like to save the obtained gradient, as well as computation artifacts that are not the gradient.<br>
I have tried using things like <code>self.my_grad = new_grad</code> and <code>self.artifacts = artifacts</code>, but it did not work, in the sense that <code>model.my_grad</code> returned <code>'MDEQClsNet' object has no attribute 'my_grad'</code>.</p>
<p>How could I go about saving what’s happening in <code>register_hook</code> ?</p>
<p>I think my question is more involved as what is explained <a href="https://discuss.pytorch.org/t/using-hook-function-to-save-gradients/4334/4">in this discussion</a>, since I really want to use <code>register_hook</code> on a variable and not a module + I want to save some computation artifacts not just one gradient.</p>

username: zaccharieramzi
cooked: <p>Apparently, in <a href="https://colab.research.google.com/drive/1knguP00Xpf6c_WNbD8T1EicuJZJL6QJR?usp=sharing" rel="noopener nofollow ugc">a minimal example</a> it works.</p>
<pre><code class="lang-auto">import torch

class MyModule(torch.nn.Module):
  def forward(self, x):
    def my_hook(grad):
      self.my_grad = grad.detach().clone().cpu()
      self.artifacts = 'artifacts'
      return grad
    x.register_hook(my_hook)
    return x

model = MyModule()
x = torch.tensor(1.0).requires_grad_(True)
output = model(x)
output.backward()

model.artifacts
</code></pre>
<p>So I need to understand where exactly the problem happens in my more complex case.</p>
VGG19 avgpool layer before classifier

username: Martin_Pedersen
cooked: <p>Hi there<br>
I am new here.<br>
I have a quick question. When I load the pretrained VGG19 model in python and print the model I noticed that there is a avgpool layer before the classifier layer and this doesn’t seem to appear in the original VGG structure sheet:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de06d30fc73bf61657dfb6b7a0fa0ca09d28f480.png" data-download-href="https://discuss.pytorch.org/uploads/default/de06d30fc73bf61657dfb6b7a0fa0ca09d28f480" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de06d30fc73bf61657dfb6b7a0fa0ca09d28f480_2_493x500.png" alt="image" data-base62-sha1="vG8Cqq9vYh7bWJlZkuL3omqgGwU" width="493" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de06d30fc73bf61657dfb6b7a0fa0ca09d28f480_2_493x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de06d30fc73bf61657dfb6b7a0fa0ca09d28f480_2_739x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de06d30fc73bf61657dfb6b7a0fa0ca09d28f480_2_986x1000.png 2x" data-dominant-color="E1E1E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">996×1010 244 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can anyone explain why the avgpool layer is either not mentioned in the VGG structure or why it has been added to the pytorch model?</p>
<p>Best regards<br>
Martin</p>

username: Martin_Pedersen
cooked: <p>Found this in the original paper<br>
“We then perform global average pooling on the resulting feature map, which<br>
produces a 4096-D image descriptor. The descriptor is then averaged with the descriptor of a horizontally flipped image.”</p>
<p>But I am still confused why such an impactful layer is not displayed in the structure sheet.</p>
The difference between Train accuracy and Test accuracy

username: cjfghk5697
cooked: <p><a href="https://github.com/cjfghk5697/multi-label-classification-Mnist/blob/main/0518%20%EB%AC%B8%EC%A0%9C%EC%A0%90%20%EA%B0%9C%EC%84%A0.ipynb" rel="noopener nofollow ugc">My Train Code</a><br>
<a href="https://github.com/cjfghk5697/multi-label-classification-Mnist/blob/main/Inference.ipynb" rel="noopener nofollow ugc">My Inference Code</a></p>
<h1><a name="question-1" class="anchor" href="#question-1"></a>Question</h1>
<p>I try to classify the Mnist dataset. The models train accuracy is 90% and valid accuracy approach 80%. But the test accuracy is 50%. I think my model is overfitting. So I made a chart about train acc and valid acc.(batch size 16, epoch 100)</p>
<p>[Train acc]<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d47a3893d462e97f6edab9afb1b83b556d224e7.png" alt="image" data-base62-sha1="1TtGWdvYxc3tNnDxiHPkezUcCkT" width="392" height="278"></p>
<p>[Valid acc]<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dce7f3b82beb1acab52270818fd725d834e59279.png" alt="image" data-base62-sha1="vwdZTXzMWyoCOxvYwD7CUALpDhn" width="392" height="278"></p>
<p>Why there is a difference between Train accuracy and Test accuracy?</p>
<h1><a name="things-i-tried-2" class="anchor" href="#things-i-tried-2"></a>Things I tried</h1>
<ol>
<li>Drop out</li>
<li>Add more Activation functions</li>
</ol>
Einsum but works in numpy

username: spyroot
cooked: <p>Hi Folks,</p>
<p>A shape (80, 513)<br>
B shape (2, 80, 501)</p>
<p>in Numpy it does work but in torch.einsum it doesnt.</p>
<pre><code class="lang-auto">x=np.einsum("fm,...mt-&gt;...ft", A, B, optimize=True)
x = shape (2, 513, 501)
</code></pre>
<p>But in torch I’m getting<br>
einsum(): operands do not broadcast with remapped shapes [original-&gt;remapped]: [80, 513]-&gt;[1, 80, 1, 513] [2, 80, 501]-&gt;[2, 1, 501, 80]</p>
<p>I can’t figure what the is difference ?</p>
UnboundLocalError: local variable 'total_seg_cons_loss' referenced before assignment

username: 35454652
cooked: <pre><code>if args.bv and args.gv:
    total_seg_cons_loss = args.bv_wt * total_seg_cons_loss_1 + args.gv_wt * total_seg_cons_loss_2
elif args.gv:
    total_seg_cons_loss = total_seg_cons_loss_2
elif args.bv:
    total_seg_cons_loss = total_seg_cons_loss_1

#global total_seg_cons_loss
total_cons_loss = total_seg_cons_loss</code></pre>

username: ptrblck
cooked: <p>Based on the error message no condition is met and since no <code>else</code> case is defined <code>total_seg_cons_loss</code> is undefined.<br>
Add an <code>else</code> case and set the value to a default one.</p>
Train multiple models from the same dataloader on a single device?

username: ldorigo
cooked: <p>Hi,</p>
<p>I have a model that in itself takes relatively little GPU memory (around 1Gb, 1/8th of what I have available), however the dataset is comparatively large (around 40GB, which fits in RAM). Since loading the data takes a long time, I load data lazily and keep everything in RAM - meaning the first epoch takes around 30 minutes to run, but every epoch after that takes less than one minute.</p>
<p>I’m tuning hyperparameters and thus need to train many iterations of the model. My GPU could easily train 7/8 models in parallel, however since I only have 96GB of ram, I can only fit the dataset twice in memory, meaning I can only train two models at a time.</p>
<p>Would it be possible/how can I use a single Dataset/Dataloader for multiple models so that I don’t need to have duplicate data in memory?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>It should be possible if you run the different models in different streams.<br>
However, not only the memory limits this use case but you would also need free compute resources to be able to execute workloads in parallel on the same device. I.e. even if a single model is using 1/8 of the available memory it could still use all compute resources (SMs) and thus block the other model so you would need to profile the actual workload.</p>
Torch.jit vectorizing loops

username: dash102
cooked: <p>I wrote up a simple example of matrix addition compiled as TorchScript:</p>
<pre><code class="lang-auto">@torch.jit.script
def mat_add(A: torch.Tensor, B: torch.Tensor) -&gt; torch.Tensor:
    C = torch.empty_like(A)
    for i in range(len(A)):
        for j in range(len(A[0])):
            C[i,j] = A[i,j] + B[i,j]
    return C

print(mat_add.graph)
</code></pre>
<p>I thought that torch.jit.script could vectorize loops when appropriate, but the compiled graph still shows element-wise operations and thus no speedup. Is there a way to recognize and vectorize loops like these with torch.jit or some other way?</p>
Class Imbalnace Approach

username: pantazis
cooked: <p>Hello everyone,</p>
<p>I am rather new to pytorch but i used it anyway for my project. The project I deal with is Driver Anomaly Detection. The particular dataset was constructed as an Anomaly Detection project and used unsupervised learning by comparing higher dimension vectors (cosine similarity) in the end. For my project i have altered it a little by applying binary classification implementation (anormal and normal driving). The problem is that the dataset itself has a lot more normal videos (85% normal, rest are anormal) videos. I have tried using data augmentations to stabilize more my data and i have also used WeightedRandomSampler as I have read in this forum. Also, i tried including the parameter pos_weight in the scope and the weight_decay in the optimizer. My questions are the below:</p>
<ol>
<li>
<p>When we use WeightedRandomSampler or pos_weight(for this instance), should we use only one of them or both?</p>
</li>
<li>
<p>I read in a few websites about the value and importance of weight_decay. What would be a great value for it regaarding the learning rate.</p>
</li>
<li>
<p>Finally, because the test_set has a few unseen scenarios, my test_accuracy is quite bad (which was the reason i introduced augmentations to make the model more diverse towards anomaly driving). What bothers me though is that i can achieve high train_accuracy (80% and above ) from the first epoch too. Should I change my approach from binary classification or not?</p>
</li>
</ol>
<p>The Model Architecture I used is a ConvGRU approach, 3D ConvNet + 2D ConvNet and GRU in those features with a classifier in the end.</p>
<p>Thanks in advance!!</p>