akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Loss is not decreasing for Normal distribution prediction

username: Chris_Freiburg
cooked: <p>Hello,</p>
<p>I am working on a model which would predict a Normal Distribution of a Vector.<br>
Thats the code</p>
<pre><code class="lang-auto">



class skilldynamic(nn.Module):





   def __init__(self, state_size, obs_size, fix_var=False, fc1_units=256,fc2_units=256, seed=0):
        """Initialize parameters and build model.
        Params
        ======
            state_size (int): Dimension of each state
            action_size (int): Dimension of each action
            seed (int): Random seed
            fc1_units (int): Number of nodes in first hidden layer
            fc2_units (int): Number of nodes in second hidden layer
        """
        super(skilldynamic, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(state_size, fc1_units)
        self.fc2 = nn.Linear(fc1_units, fc2_units)
        self.mean = nn.Linear(fc2_units, obs_size)
        self.std = nn.Linear(fc2_units, obs_size)
        self.apply(weights_init_)
        self._fix_variance = fix_var
        # self.multi_normal = torch.distributions.MultivariateNormal(torch.Tensor([4,3,5]),torch.Tensor([[1,0,0],[0,1,0],[0,0,1]]))
        if not self._fix_variance:
            self._std_lower_clip = 0.3
            self._std_upper_clip = 10.0

    def forward(self, state):
        """Build a network that maps state -&gt; action values."""
        x = F.relu(self.fc1(state))
        x = F.relu(self.fc2(x))
        mean = self.mean(x)
        logvar =  self.std(x).clamp(self._std_lower_clip, self._std_upper_clip)
        # state_pre = NormalDistribution(mean, logvar)
        return mean, logvar 
    
    def reparam(self, mean, logvar):
        sigma = (logvar / 2).exp()
        epsilon = torch.randn_like(sigma)
        return mean + torch.mul(epsilon, sigma)
    
    def sample(self, state):
        mean, log_std  = self.forward(state)
        std = log_std.exp()
        normal = Normal(mean, std)
        x_t = normal.rsample()
        y_t = torch.tanh(x_t)
        log_prob = normal.log_prob(x_t)
        log_prob = log_prob.sum(0, keepdim=True)
        return x_t, log_prob, normal


class SkillDynamicsModell():
    def __init__(self, state):
        self.dynamic_model = skilldynamic(state.shape[0], state.shape[0])
        self.dynamic_optimizer = torch.optim.Adam(self.dynamic_model.parameters(), 0.003)
    
    def predict(self, state):
        return self.dynamic_model.sample(state)
    
    def learn(self, state, next_state):
        x_t, log_prob, normal = self.dynamic_model.sample(state)
        delta = x_next - state
        model.dynamic_optimizer.zero_grad()
        loss = -normal.log_prob(delta).sum(0, keepdim=True)
        loss.backward()
        model.dynamic_optimizer.step()
        print("Loss :", loss.cpu().detach().numpy())

state =  torch.Tensor(state)
next_state =  torch.Tensor(next_state)
model =  SkillDynamicsModell(state)
print("state", state)
print("next state", x_next)

for i in range(1000):
    model.learn(state, next_state)
</code></pre>
<p>Does anyone see what I am doing wrong ?</p>
Autograd Why accumulation gradient mechanism?

username: Sanglok
cooked: <p>Pytorch autograd why accumulation gradient mechanism?<br>
for what?</p>
<p>Do you have any idea?<br>
Thank you in advance.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m not sure to understand your question.<br>
If you ask why <code>.backward()</code> accumulates gradients, then this post should give you a good idea: <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903" class="inline-onebox">Why do we need to set the gradients manually to zero in pytorch?</a></p>
The current state of Deepfakes and PyTorch - some guidance please

username: bfeeny
cooked: <p>I work for a large company where we are trying to constantly promote awareness of security threats, things like phishing attacks, ransomeware, deep fakes, etc.</p>
<p>I have been asked to give a demonstration on deep fakes, where I will use some public/free/no copyright material and deep fake it with someones face from our organization.</p>
<p>Does it make sense these days to do build this in PyTorch, or is there now such good off the shelf software I should just use that?  Its a basic short demo, 10-30 seconds.  I don’t want to put more work into than necessary.  My company, and myself have no interest in Deepfake technology other than this demonstration on security awareness, which we would like to make an original deep fake.</p>
<p>If PyTorch is a good approach and anyone has some good links, starter code, ideas, etc to share please post in this thread.  If a commercial app is a good solution please share those.  I am currently researching both.  I am quite comfortable with PyTorch and Deep Neural Networks, but obviously I am trying to figure out the approach with least effort and good result.  Thank you for any help and guidance.</p>
Error when running any model converted for Vulkan on Android C++ libtorch

username: Matt_Fischler
cooked: <p>I built the 1.8.0 prototype Android libtorch from source with Vulkan backend enabled and it runs all torchscript models fine but if I run any model that has been converted to Vulkan and put it on GPU with tensor.vulkan() then I get the following error during inference. I’ve even tried the mobilenet2-vulkan.pt model mentioned in the documentation but still, the same error. Note that the call to at::is_vulkan_available() returns True.</p>
<pre><code>     return F.conv2d(input, weight, self.bias, self.stride,
            ~~~~~~~~ &lt;--- HERE
                     self.padding, self.dilation, self.groups)
 RuntimeError: expected scalar type Float but found UNKNOWN_SCALAR</code></pre>

username: ssjia
cooked: <p>There is a similar issue on Github right now: <a href="https://github.com/pytorch/pytorch/issues/57893" class="inline-onebox" rel="noopener nofollow ugc">Android API to run model on Vulkan backend · Issue #57893 · pytorch/pytorch · GitHub</a></p>
<p>In that thread I linked a MobileNet V2 model that you can try. If that works, I would suggest re-building pytorch with USE_VULKAN=ON as mentioned in the thread.</p>
When and Why "prim::DifferentiableGraph" occurs?

username: alanzhai219
cooked: <p>[PyTorch 1.6]<br>
I try to torch.jit.trace or torch.jit.script alexnet, the graph displays “prim::DifferentiableGraph”. However, sometimes, it only fuses some small network into a “FusionGroup”.<br>
When prim::DifferentiableGraph and FusionGroup happens?<br>
If I want to use cuda fusion(nvrtc), how to <em>close</em> prim::DifferentiableGraph and turn on “FusionGroup”?<br>
Thanks.</p>
Read video with different fps

username: 111429
cooked: <p>Hi,<br>
I use <code>torchvision.io.read_video</code> to read a video, and the number of frames is based on the intrinsic fps of the video.<br>
I wonder if there is easier way to use this api, but one can feed a specific fps value?</p>
<p><a href="https://github.com/pytorch/vision/blob/5bb997c6173e016438c765c8190c0747d472e1f6/torchvision/datasets/video_utils.py#L75" rel="noopener nofollow ugc">VideoClips</a> class has a frame_rate argument, but the size is fixed by <code>clip_length_in_frames</code> and <code>frames_between_clips</code>.<br>
I want videos with variant duration have their corresponding frames based on the specified fps.<br>
Thanks.</p>
Issues installing Pytorch and CUDA

username: lory
cooked: <p>Hello, i’m struggling for 2 days to make this work. So i have Geforce GTX 950M, v 388.00 and Python 3.9( i can also do 3.8 if necessary) and Windows 10 and pip for installation. I need home help choosing the right CUDA and pytorch versions for my driver… i tried many combinations but i couldn’t make it work, everytime torch.cuda.is_available() returns False.I checked in Nvidia control panel and it point to CUDA 9.1 version but then i do not know which Pytorch to install, which one is compatible. Many thanks in advance.</p>

username: ptrblck
cooked: <p>The binaries for the current PyTorch release <code>1.8.1</code> and the nightly ship with CUDA10.2 and CUDA11.1 as given in the <a href="https://pytorch.org/get-started/locally/">install instructions</a>.</p>
<p>Your local CUDA9.1 installation won’t be used, if you are installing the conda binaries or pip wheels. Also note that you would need a newer NVIDIA driver, since even CUDA9.1 needs &gt;=390.46 based on <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility">Table 1</a>.</p>

username: lory
cooked: <p>So i can safely remove the installed CUDA 9.1 from the computer and just install 1.8.1 and CUDA 10.2 / 11.1 (using pip) and work fine?</p>

username: lory
cooked: <p>I tried pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio===0.8.1 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a> but when it reaches at the end of downloading everything freezes and I cannot do anything( move mouse or type). Is that normal? do I have to wait for it to recover?<br>
Edit:I uninstalled the CUDA 9.1 from machine.<br>
I managed to install using : pip --no-cache-dir install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio===0.8.1 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>
<p>But now when i try to run the code i receive RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu. Any idea how to solve this, please?</p>
<p>P.S. I updated the driver and now i have v 398.35 and NVCUDA.dll  9.2. Should I uninstall the above and install something for CUDA 9.2(with pip)?</p>

username: ptrblck
cooked: <p>Your NVIDIA diver seems to be too old, since CUDA11.1 needs driver &gt;= 450.80.02 as described in the linked table.</p>

username: lory
cooked: <p>Many thanks, I managed to fix that problem but I’ve bumped into another issue, if you could take a look, please. <a href="https://discuss.pytorch.org/t/issues-training-bert/121499" class="inline-onebox">Issues training BERT</a></p>
Problem with Einsum

username: Sohrab_Samimi
cooked: <p>Hi everyone,i am having some trouble with torch.einsum.Basically i am trying to mutliply two tensors in a certain way:</p>
<pre><code class="lang-auto">first = torch.rand(12,8192,2)

weights1 = torch.rand(12,8192,2)

torch.einsum('bix,iox-&gt;box',first,weights1)
</code></pre>
<p>But i get the following error:<br>
<code>einsum() operands do not broadcast with remapped shapes [original-&gt;remapped]: [12, 8192, 2]-&gt;[12, 1, 2, 8192] [12, 8192, 2]-&gt;[1, 8192, 2, 12]</code></p>
<p>Can someone explain to me what I am doing wrong?</p>

username: tom
cooked: <p>Something is up with the i dimensions: in the first tensor it is 8192, in the second 12.</p>
<p>I must admit the error message isn’t as clear as it could be, so I took the liberty to file an issue with your example: <a href="https://github.com/pytorch/pytorch/issues/58380" class="inline-onebox" rel="noopener nofollow ugc">Clarity of error message in einsum regressed in performance improvements · Issue #58380 · pytorch/pytorch · GitHub</a></p>
<p>Thank you for asking this and providing a straightforward example, it gives us a chance to improve the error message.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Sohrab_Samimi
cooked: <p>Thank you so much for your answer!</p>
Gaussian Mixture Model maximum likelihood training

username: whoab
cooked: <p>Typically, GMMs are trained with expectation-maximization, because of the need for implementing the unitary constraint over the categorical variables.</p>
<p>However, in Pytorch, it is possible to get a differentiable log probability from a GMM. Why is this possible? How exactly is the constraint implemented in the code?</p>

username: googlebot
cooked: <p>Do you refer to MixtureSameFamily? That one contains a nested “Categorial” distribution object, with non differentiable distribution parameters.</p>
<p>It is possible (though not trivial) to train Categorical with sampling - docs describe REINFORCE / score function. This issue is orthogonal to using gradient descent to train GMMs in general, or “unitary constraint” (that is handled by using Categorical distribution [class])</p>

username: whoab
cooked: <p>All I did was use the logprob function of the MixtureSameFamily and trained it with gradient descent over samples in my dataset, it works.</p>

username: googlebot
cooked: <p>Ah, right, log_prob / local MLE simultaneous estimation works, it is just not too good with random nn initializations and SGD. May be ok for some tasks.</p>
<p>To directly answer your question:</p>
<pre><code class="lang-auto">    def log_prob(self, x):
        x = self._pad(x)
        log_prob_x = self.component_distribution.log_prob(x)  # [S, B, k]
        log_mix_prob = torch.log_softmax(self.mixture_distribution.logits,
                                         dim=-1)  # [B, k]
        return torch.logsumexp(log_prob_x + log_mix_prob, dim=-1)  # [S, B]
</code></pre>
<p>this is just transformed mixture density formula, where everything is differentiable (similarly to a weighted sum).</p>
Transforms.resize() the value of the resized PIL image

username: Xiaoyu_Song
cooked: <p>Hi, I find that after I use the <code>transforms.resize()</code> the value range of the resized image changes.</p>
<pre><code class="lang-auto">a = torch.randint(0,255,(500,500), dtype=torch.uint8)
print(a.size())
print(torch.max(a))
a = torch.unsqueeze(a, dim =0)
print(a.size())
compose = transforms.Compose([transforms.ToPILImage(),transforms.Resize((128,128))])
a_trans = compose(a)
print(a_trans.size)
print(a_trans.getextrema())
</code></pre>
<p>The result:</p>
<pre><code class="lang-auto">torch.Size([500, 500])
tensor(254, dtype=torch.uint8)
torch.Size([1, 500, 500])
(128, 128)
(79, 179)
</code></pre>
<p>The original range is [0,255], after the transforms.resize(), the value range change to [79,179]</p>
<p>I want to do the resize without the change of value range, someone could help? Thank you</p>

username: Xiaoyu_Song
cooked: <p>The problem is solved, the default algorithm for <code>torch.transforms.resize()</code> is BILINEAR<br>
SO just set<br>
<code>transforms.Resize((128,128),interpolation=Image.NEAREST)</code><br>
Then the value range won’t change!</p>

username: tlim
cooked: <p><a class="mention" href="/u/xiaoyu_song">@Xiaoyu_Song</a>,</p>
<p>did you get this error?</p>
<pre><code class="lang-auto">UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
</code></pre>
<p>It still trains, not too sure what does this error mean.</p>

username: ptrblck
cooked: <p>This warning points to <a href="https://github.com/pytorch/vision/pull/2952">this PR</a>, which seems to have introduced the <code>InterpolationMode</code> argument, since <code>torchvision.transforms</code> are supporting <code>PIL.Image</code>s as well as tensors now (at least the majority, if I’m not mistaken).</p>
<p>CC <a class="mention" href="/u/vfdev-5">@vfdev-5</a> to correct me.</p>

username: sriram2000na
cooked: <p>Can you please see this: <a href="https://discuss.pytorch.org/t/regarding-transforms-resize-and-drastic-changes-in-accuracy/121371" class="inline-onebox">Regarding transforms.resize and drastic changes in accuracy</a>, I have the same question regarding which is better or a preferred way to resize an image. Thanks in advance, Sriram Na.</p>
Get error in torch.onnx.export

username: rabbit_hog
cooked: <p>I used the code below to export interpolation operation to onnx file.<br>
repeat = torch.nn.UpsamplingNearest2d(scale_factor=2)<br>
repeat.eval()<br>
torch.onnx.export(repeat, torch.ones((1, 3, 4, 4)), “repeat.onnx”)<br>
And the console output is:<br>
torch.onnx.export(repeat, torch.ones((1, 3, 4, 4)), “repeat.onnx”)</p>
<p>File “D:\ProgramData\Anaconda3\lib\site-packages\torch\onnx_<em>init</em>_.py”, line 25, in export<br>
return utils.export(*args, **kwargs)</p>
<p>File “D:\ProgramData\Anaconda3\lib\site-packages\torch\onnx\utils.py”, line 131, in export<br>
strip_doc_string=strip_doc_string)</p>
<p>File “D:\ProgramData\Anaconda3\lib\site-packages\torch\onnx\utils.py”, line 363, in _export<br>
_retain_param_name, do_constant_folding)</p>
<p>File “D:\ProgramData\Anaconda3\lib\site-packages\torch\onnx\utils.py”, line 278, in _model_to_graph<br>
_disable_torch_constant_prop=_disable_torch_constant_prop)</p>
<p>File “D:\ProgramData\Anaconda3\lib\site-packages\torch\onnx\utils.py”, line 188, in _optimize_graph<br>
graph = torch._C._jit_pass_onnx(graph, operator_export_type)</p>
<p>File “D:\ProgramData\Anaconda3\lib\site-packages\torch\onnx_<em>init</em>_.py”, line 50, in _run_symbolic_function<br>
return utils._run_symbolic_function(*args, **kwargs)</p>
<p>File “D:\ProgramData\Anaconda3\lib\site-packages\torch\onnx\utils.py”, line 589, in _run_symbolic_function<br>
return fn(g, *inputs, **attrs)</p>
<p>File “D:\ProgramData\Anaconda3\lib\site-packages\torch\onnx\symbolic.py”, line 130, in wrapper<br>
args = [_parse_arg(arg, arg_desc) for arg, arg_desc in zip(args, arg_descriptors)]</p>
<p>File “D:\ProgramData\Anaconda3\lib\site-packages\torch\onnx\symbolic.py”, line 130, in <br>
args = [_parse_arg(arg, arg_desc) for arg, arg_desc in zip(args, arg_descriptors)]</p>
<p>File “D:\ProgramData\Anaconda3\lib\site-packages\torch\onnx\symbolic.py”, line 90, in _parse_arg<br>
raise RuntimeError("Failed to export an ONNX attribute, "</p>
<p>RuntimeError: Failed to export an ONNX attribute, since it’s not constant, please try to make things (e.g., kernel size) static if possible</p>
<p>My pytorch version is 1.1.0, py3.6_cuda90_cudnn7_1.</p>
What happens to memory region of tensor object on CPU when tensor (in pytorch) is sent to Cuda device?

username: Hong_Dinh
cooked: <p>My question is: What happens to memory region of tensor object on CPU when tensor (in pytorch) is sent to Cuda device?</p>

username: tom
cooked: <p>That depends. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
In PyTorch, moving a CPU tensor to GPU does create a copy, so they both live independently after.</p>
<ul>
<li>
<p>If the CPU tensor is still referenced eg</p>
<ul>
<li>by your program,</li>
<li>by autograd (if you used it in a computation involving values that required grad which and PyTorch needs it for computing the backward),</li>
<li>by a “view”-Tensor (e.g. a slice)</li>
</ul>
<p>it will be still around.</p>
</li>
<li>
<p>If the CPU tensor (or the storage, to be more precise) is not used after the copy, it will be freed (with the usual caveats about how memory allocation works with system libraries potentially caching etc.).</p>
</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>
Pytorch Cuda out of memory

username: smca
cooked: <p>Hello ! Im trying to run my CNN but it fails due to memory shortage. I guess the problem is that i have some variables which consume all the memory but i do not see the mistake. It is to say that i am quite new to working with neural networks and worring about memory ! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"><br>
I really appreciate any help.<br>
Here is the code which i guess does the trouble :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0aa8b3f12173a44b82580028deab59f08083717d.png" data-download-href="https://discuss.pytorch.org/uploads/default/0aa8b3f12173a44b82580028deab59f08083717d" title="Annotation (1)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0aa8b3f12173a44b82580028deab59f08083717d.png" alt="Annotation (1)" data-base62-sha1="1widzP3iE2wgxyfXFmrxfWFKbG5" width="690" height="390" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Annotation (1)</span><span class="informations">1077×610 37.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>with the following error :<br>
RuntimeError: CUDA out of memory. Tried to allocate 6.60 GiB (GPU 0; 31.75 GiB total capacity; 21.92 GiB already allocated; 1.87 GiB free; 28.56 GiB reserved in total by PyTorch)</p>
While creating a custom dataloader, which is faster, reading from dictionaries or reading from CSVs

username: keshav-b
cooked: <p>If i am having say quite a large dataset with around 50k images, will it be efficient to store the paths in a dictionary and read them in the custom dataloader? Or is CSV a better option?</p>

username: tom
cooked: <p>A dataset, probably?<br>
This depends on lots of factors (not lest how you intend to read CSVs), but to be honest, I would expect the time for either to be very small compared to the rest of the training. So my advice not worry about it until it becomes the bottleneck.</p>

username: keshav-b
cooked: <p>Okay, thanks <a class="mention" href="/u/tom">@tom</a> for the reply, unfortunately its a custom dataset, so i wont be able to sshare it. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Why pytorch pip install include 10.x except 10.0

username: leigaoxiang
cooked: <p>i want to pip install pytorch 1.7+ with cuda 10.0 but  there is no cuda 10.0</p>

username: ptrblck
cooked: <p>The currently built binaries ship with the CUDA10.2 and 11.1 runtime, so you could pick one of these two or alternatively build from source using a locally installed CUDA10.0 toolkit.</p>
FASTER_RCNN image size and weird transform behaviour

username: Faiz_Alqorni
cooked: <p>Hi, I have been trying to train and test faster_rcnn where I want to train and test pytorch image with the following parameter</p>
<p><code>model =torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True, min_size = 200,max_size = 4000)</code></p>
<p>but it seems image is resized upon inference eventhough it shouldn’t be inference image sizes are : (912,912), (1824,1824), (3648,3648)</p>
<p>I know that image is resized upon inference since the time and GPU RAM usage doesn’t make sense (all size are similar even if I set the min_size = 200, and max_size = 4000, but if I change min_size and max_size close to the original size like (3648, 3648) the time and RAM usage will be different which is why I draw conclusion that the image is resized)</p>
<p>problem is, because I felt like the generalized transform is so whimsical I don’t even know if my train is the right size/ model learn the right information.</p>
<p>Although now that I think about it I didn’t use dataloader while inferencing (straight using util_dataset) does it affect the inference image size in any way?</p>

username: Faiz_Alqorni
cooked: <p>or maybe someone can help me avoid this issue by completely avoiding automatic transform altogether? I really feel like automatic transform is out of scope for Faster-RCNN responsibility and this closely coupled structure is really making it hard to use. it’s coupled in both Generalized-RCNN and automatically in Faster-RCNN too, I have tried manually decoupling the transform process but it feels like rabbit hole since other process depends on transformation result</p>
<p>logically when i set min_size, max_size (200,4000) it should accept image between those sizes right? but the thing is when i said min_size, max_size (200,4000) say I detect 2 Positive objects but when i set min_size, max_size (3648,3648) the results changed a lot, but by logic it shouldn’t right?</p>
Scalar parameters not showing up in model.parameters()

username: lm_modeler
cooked: <p>I am getting <code>ValueError: optimizer got an empty parameter list</code> when I run this code. What am I missing? Tried using <code>nn.Parameters</code> and still getting error on model parameters <code>a</code> and <code>b</code>. I just want to track and update these two scalars on this toy problem.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import TensorDataset, DataLoader

class RosenbrockFunction2N(nn.Module):
  def __init__(self):
    super(RosenbrockFunction2N, self).__init__()
    self.a = torch.rand(1, requires_grad=True)
    self.b = torch.rand(1, requires_grad=True)

  def forward(self, x, y):
    return (self.a - x)**2 + self.b(y-x**2)**2

# Dataset
inputs = [[1., 3.], [5., 7.], [2., 4.], [3., 9.]]
targets = []
for i in inputs:
  targets.append([(1-i[0]**2) + (i[1] - i[0]**2)**2])
inputs = torch.tensor(inputs)
targets = torch.tensor(targets)

train_ds = TensorDataset(inputs, targets)
train_dl = DataLoader(train_ds)

# Load model
model = RosenbrockFunction2N()

# Loss and Optimizer
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
    
for i in range(5):
  for inputs, target in train_dl:
    optimizer.zero_grad()
    output = model(inputs[0][0], inputs[0][1])
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()
</code></pre>

username: ptrblck
cooked: <p>Registering the <code>a</code> and <code>b</code> tensors as <code>nn.Parameters</code> should work (alternatively you could also use <code>self.register_parameter</code>). In your current code you are assigning tensors as model attributes, which won’t be registered as parameters internally.</p>

username: lm_modeler
cooked: <p>Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a>. Sorry, I’m not used to working with scalars like this. When I change it to this:</p>
<pre><code class="lang-auto">    self.a = nn.Parameter(torch.tensor(1., requires_grad=True))
    self.b = nn.Parameter(torch.tensor(1., requires_grad=True))
</code></pre>
<p>I get this:<br>
<code>TypeError: 'Parameter' object is not callable</code></p>

username: ptrblck
cooked: <p>This error is raised by trying to call the parameter in:</p>
<pre><code class="lang-python">self.b(y-x**2)**2
</code></pre>
<p>What kind of operation should be performed in this line of code?</p>

username: lm_modeler
cooked: <p>Such a simple problem. Yeah, it’s just a missing multiplication symbol. Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
Normalising parameters of a model

username: nmourdou
cooked: <p>Hey guys,</p>
<p>I have probabilities as parameters in my model and I wish to normalise them (divide by their norm so they sum up to 1) as they are being optimised. Probably I should impose this constraint during training with:</p>
<p><strong>with torch.no_grad():</strong></p>
<p>However  I haven’t found a way to so, as any assignments, like</p>
<p><strong>model.probs = model.probs/model.probs.sum()</strong></p>
<p>don’t work, bc you cannot assign a float to a parameter. Any suggestions?</p>
<p>Thank you in advance,<br>
Nikos</p>

username: ptrblck
cooked: <p>I think you could try to apply a similar approach as is used in <a href="https://pytorch.org/docs/stable/_modules/torch/nn/utils/weight_norm.html#weight_norm"><code>weight_norm</code></a>, which uses hooks to normalize the parameters.</p>
What does it mean if accuracy and recall are same?

username: Sahil_malik
cooked: <p>Hi members,</p>
<p>I am evaluating multi-class classification task using precision, recall, accuracy, and F-Score metrics. In all my experiments accuracy and recall are the same but other two metrics have bit different values. I am calculating average=“weighted” through scikit learn library.</p>
<p>How should I interperate this behavior?</p>
<p>Thanks!</p>
How to prepare tensor correctly for loss computation?

username: julliet
cooked: <p>I am new to the NN and PyTorch, so some things are not obvious to me. I would really appreciate help here. I wanted to use nn.CrossEntropyLoss on a tensor of shape [3, 14, 136], where 3 is a batch_size, 14 is sequence_length and 136 is number of tokens. It is one-hotted by number_of_tokens dimension.<br>
I put this tensor into the nn.CrossEntropyLoss with the reference tensor of shape [3, 14] with correct tokens index in second dimension. However I’ve got an error:<br>
ValueError: Expected target size (3, 136), got torch.Size([3, 14])<br>
As I thought from the documentation, my dimensions were correct, what do I do wrong to receive this error?</p>

username: ptrblck
cooked: <p>Based on the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss">docs</a> the output of the model should have the shape <code>[batch_size, nb_classes, *]</code>, while the target should have the shape <code>[batch_size, *]</code>, so you would need to <code>permute</code> the output to the shape <code>[batch_size=3, nb_classes=nb_tokens=136, seq_len=14]</code> (assuming <code>136</code> equals the number of classes). <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: julliet
cooked: <p>Thank you for your help!<br>
I did that before, but my loss becomes very big (4.9) and model generates rubbish of the same letter over and over again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> could you advise me where to keep attention to beat my problem?</p>

username: ptrblck
cooked: <p>I would recommend to try to overfit a small data sample (e.g. just 10 samples) and make sure your current model is able to do so e.g. by playing around with some hyperparameters. Once this is done, you could try to scale up the use case again.</p>
Sample Batches according to IDs

username: SaadMunir
cooked: <p>Hello everyone !<br>
I have a large dataset about a casting machine and its corresponding process parameters . So basically each row is an observation with about 200 parameters and then the corresponding class label of 0 or 1 where 0 means the part is NOT defective and 1 means the part was badly manufactured. We are trying to map these process parameters to the correct class prediction and right now I am using a feedforward network with 5 layers and the area under ROC curve is 0.76.<br>
What I have been thinking is maybe we lack temporal information in our batches which are now created using stratified train test split and later on balanced from both classes with a WeightedRandomSampler.<br>
So my idea is to first sort the observations in ascending order of time,  do not shuffle and then create the batches so that the temporal order is preserved. However when I take this approach, my mini batches become severely imbalanced. Nevertheless, there might be some mini batches where the minority class has a clear majority , for example 8 out of 10 samples in a mini batch belong to minority class. Lets say the IDs of such mini batches are from 40 to 60. So I would like the first iteration of my epoch to take a normal, highly unbalanced mini batch as the input, then the second iteration to take a minority class dominant mini batch as input i.e. a batch from ID between 40 and 60, then the third iteration again takes a normal , unbalanced batch as input and the fourth iteration again samples a minority class dominant batch, and then this process should repeat for a fixed number of iterations. In this way, temporal information would be preserved. I think to implement this strategy, I would have to create a custom generator or Dataloader in Pytorch but I have little idea about that. Could anybody please guide me over this?</p>

username: ptrblck
cooked: <p>Based on your description I think the proper way to implement this custom sampling strategy would be in a custom <code>sampler</code>. By default the <code>DataLoader</code> would use a <code>SequentialSampler</code> or a <code>RandomSampler</code>, which you could replace with your custom class. This custom sampler could then take the targets and create the batch indices using your sampling logic.</p>

username: SaadMunir
cooked: <p>Thankyou very much for the idea! Could you please give me an idea of how my custom sampler class should look like ?</p>

username: ptrblck
cooked: <p>You could take a look at the <a href="https://github.com/pytorch/pytorch/blob/71f4c5c1f436258adc303b710efb3f41b2d50c4e/torch/utils/data/sampler.py">sampler implementations</a> and adapt one to your use case. In particular I think you might want to pass the target tensors to the sampler and create the desired sample indices in the <code>__iter__</code> method.</p>
Keyerror 0 while zipping the dataloader objects

username: raghava_vinay
cooked: <p>training_patch_loader = zip(training_patch_loader_inp, training_patch_loader_op)<br>
training_patch_loader = list(training_patch_loader)</p>
<p>I want to join two dataloader objects but it throwing keyerror 0 when i am trying to run.</p>
Extracting image features to feed to Transformer layer

username: EvanZ
cooked: <p>I’m trying to understand how to use Transformers for OCR starting with something very basic like MNIST. My thought was to extract image patches from each MNIST image and then feed those into a Transformer with a classifier on the output. For example:</p>
<pre><code class="lang-auto">model = torch.nn.Sequential(
          torch.nn.Conv2d(1,16,3,3),
          torch.nn.ReLU(),
          torch.nn.Conv2d(16,32,3,3),
          torch.nn.ReLU()
        )

model(images).shape
&gt; torch.Size([64, 32, 3, 3])
</code></pre>
<p>So here is a batch of 64 32-layer patches of 3x3. Given that the input to the Transformer is supposed to be a sequence of vectors, my thought is I should “unroll” the 3x3 patches to end up with a sequence of 32 9-dimensional vectors. Does this make sense?</p>

username: Unity05
cooked: <p>Hi,</p>
<p>for me at least unrolling those 3x3 patches as you’ve proposed does not really make sense, as you would just replace the FC - classifier with a Transformer what seems kind of weird to me (as it’s not really a reasonable sequence - please correct me, if it should be so after all).<br>
For your task, I would recommand checking out <a href="https://arxiv.org/abs/2010.11929#:~:text=An%20Image%20is%20Worth%2016x16%20Words%3A%20Transformers%20for%20Image%20Recognition%20at%20Scale,-Alexey%20Dosovitskiy%2C%20Lucas&amp;text=While%20the%20Transformer%20architecture%20has,to%20computer%20vision%20remain%20limited." rel="noopener nofollow ugc">this paper</a>. I am however not sure about how this approach could be adapted for more challenging OCR tasks.</p>
<p>Regards,<br>
Unity05</p>

username: EvanZ
cooked: <p>Thanks <a class="mention" href="/u/unity05">@Unity05</a>. I tried with Transformer and without, and performance did not improve much so perhaps you’re right. Maybe it will make more sense with a string of characters.</p>

username: Unity05
cooked: <p>Indeed. Excited to see if you manage do adapt it for general OCR use case. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Gradientflow through UNets

username: Luesch
cooked: <p>hello,</p>
<p>I’m curious about how you backpropagate through a Unet skip connection. Do you just ignore the gradients of the part that skipped from the downward part or are they retained like in resnets? In other words do you backprop through those skip connections or are they really only there to get features from one end to the other. And if you backprop through them ,how?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/2061533399c18223b0e58acedf808cf081c0298d.png" alt="example" data-base62-sha1="4CrNplWs0CbwRCTzY7zJVlZiw6V" width="603" height="273"></p>

username: JuanFMontesinos
cooked: <p>U-Net typically concatenates the output from encoding blocks to the decoding blocks.<br>
Backprop is a normal backprop depending on the envolded ops (usually upsimpling or upconvolution).</p>

username: Luesch
cooked: <p>@ <a href="https://discuss.pytorch.org/u/JuanFMontesinos">JuanFMontesinos</a> so you just ingnore the skip connections when you backprop through the network</p>

username: tom
cooked: <p>No. The backward of “using the same value twice” is “add the gradients from both uses”.</p>
<p>More formally if you have a function of three parameters f(x, y, z) and now consider<br>
g(u, v) = f(u, u, v) (with u twice), you have that (under suitable conditions) dg/du = df/dx + df/dy .</p>
<p>Best regards</p>
<p>Thomas</p>

username: Luesch
cooked: <p>thanks you so much! Already thought it couldnt be quite right to just ingore those connections.<br>
have a nice day <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Batched bincount

username: jojono
cooked: <p>Hi,<br>
I would l like to apply torch.bincount on batches of vectors supposing that the maximal possible value is the same for every vector of the batch, how can I do it ? Because bincount does not support batch and a for-loop would not be efficient.</p>
<p>Thank you</p>

username: AK47
cooked: <p>You can just use scatter_add_(dim, index, other) for batched bincount, where ‘index’ is your input and ‘other’ is just torch.ones(…).</p>

username: seam
cooked: <p>Hi, were you ever able to achieve a batched bincount? I’m facing the same issue where looping throughout a dim is not an option.</p>

username: GuillaumeLeclerc
cooked: <p>Hi,<br>
This is what I got</p>
<pre><code class="lang-python">def batched_bincount(x, dim, max_value):
    target = ch.zeros(x.shape[0], max_value, dtype=x.dtype, device=x.device)
    values = ch.ones_like(x)
    target.scatter_add_(dim, x, values)
    return targe</code></pre>