akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Improving model's performance

username: gamebo5000
cooked: <p>I am using a pre-trained mobilenetv2 model running experiments for 30 epochs with the learning rate being adjusted by 0.1 every 10 epochs and these hyperparameters:</p>
<p>learning rate = 0.001<br>
weight decay = 4e-5<br>
momentum = 0.9<br>
batch size = 64<br>
optimizer = SGD<br>
dropout = 0.2</p>
<p>I am using class_weights since my training dataset is imbalanced, but my test dataset is perfectly balanced. The formula for determining each weight is minority class size/class size. I am getting good results as my confusion report is showing that each class is generating accuracies in the 80s and 90s. The issue I am facing is that after a while my model will start to fluctuate, for example in my latest experiment after 11 epochs the accuracy keep bouncing between 87% and 91%.</p>
<p><strong>What I have tried so far:</strong></p>
<p>doubling the weight of the minority class, so the model can learn it as well as the others.</p>
<p>changing the learning rate: either 0.01 or 0.0001</p>
<p>changing how much the learning rate is adjusted</p>

username: rwightman
cooked: <p>oversampling w/ meanginful augmentation might work better than weighting the loss</p>

username: gamebo5000
cooked: <p>I tried your method and I am not seeing any difference, aside from the fact that the detection accuracy of my minor class went maxes out in the 70% range, where as when I use of class weights i would high 80% to high 90%. I used the exact same setup as the one I posted in my question. It’s stuck between 87% and 89% validation accuracy.  This is the data augmentation method I am using.</p>
<blockquote>
<p>train_transform = transforms.Compose([<br>
transforms.RandomResizedCrop(224,scale = (0.2,1.0)),<br>
transforms.RandomRotation(15),      # rotate +/- 10 degrees<br>
transforms.RandomHorizontalFlip(),  # reverse 50% of images<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406],<br>
[0.229, 0.224, 0.225])<br>
])</p>
</blockquote>
Pytorch Nesterov Implementation

username: n4tman
cooked: <p>Hi All,</p>
<p>It seems that pytorch’s way of implementing SGD with Nesterov is different to:</p>
<pre><code class="lang-auto">x_ahead = x + mu*v
v = learning_rate * (mu*v - dx_ahead)
x = x + v
</code></pre>
<p>Does pytorch’s implementation give the same answer as the one above?</p>
<p>As well are pytorch’s implementations of SGD+Momentum and SGD+Nesterov non adaptive learning rate algorithms?</p>
<p>Thanks!</p>
Help understanding how to implement a LSTM

username: icarosadero
cooked: <p>Hello,</p>
<p>I don’t have a lot of experience with PyTorch and neural networks in general but I am trying to make a model. In this model, I want to generate the graph o a function with a set of parameters, add some noise and have the neural network find out which parameters were used in the function.</p>
<p>For the input, I created a function which makes the graph of a Gaussian pulse with random parameters. As for the model, I wanted to take the graph as a time sequence, pass that through an LSTM, get its last output and pass it through a linear layer for scaling. However, the model always makes the same prediction for all random sequences.</p>
<p>I don’t know what I’m doing wrong and need help to find out how to properly implement it. Here’s a link to Google Colab that I’m using to test the model: <a href="https://colab.research.google.com/drive/1eawJxIifCjaEQWBraW1CuqTbXVMTRVbr?usp=sharing" rel="nofollow noopener">https://colab.research.google.com/drive/1eawJxIifCjaEQWBraW1CuqTbXVMTRVbr?usp=sharing</a></p>
SGD and no effect of momentum

username: Jacek_Poplawski
cooked: <p>hello, I am experimenting with different optims and I found that momentum in SGD has no effect on results</p>
<p>I am working in fully reproducible environment (all seeds set)<br>
this is how I create SGD optimizer:</p>
<p>optimizer = optim.SGD(model.parameters(), lr = lr / 30)</p>
<p>this is how I can create one with momentum:</p>
<p>optimizer = optim.SGD(model.parameters(), lr = lr / 30, momentum = 0.9)</p>
<p>then I tried this one too:</p>
<p>optimizer = optim.SGD(model.parameters(), lr = lr / 30, momentum = 0.5)</p>
<p>all results (losses and scores) are same in each case</p>
<p>now I tried this one:</p>
<p>optimizer = optim.SGD(model.parameters(), lr = lr / 30, nesterov = True, momentum = 0.9)</p>
<p>and finally I see different results</p>
<p>does it mean by default momentum is unused? I tried to analyse code and looks like it is 0 by default and it should be used, any tips?</p>
<p>torch version ‘1.5.1’</p>

username: Kushaj
cooked: <p>Can you share the training code also, it is difficult to tell without that?</p>
Assign values to a pytorch tensor

username: curious
cooked: <p>I have a 2D tensor and I have the following problem:</p>
<pre><code class="lang-auto">a=tensor([[1296, 1295, 1292,    4, 1311,    4, 1293,    2],
        [1297, 1295, 1292, 1404, 1294,    4, 1293,    2]]
</code></pre>
<p>I need to mask all values greater than 1292, also I want to mask values in sorted order by incrementing values. What I want is:</p>
<pre><code class="lang-auto">tensor([[3, 2, 1292,    4, 5,    4, 1,    2],
        [4, 3, 1292, 5, 2,    4, 1,    2]]
</code></pre>
<p>How can I do this?</p>

username: Kushaj
cooked: <pre><code class="lang-python">def func(x):
    _, indices = torch.topk(x, k=x.size(1), dim=1)
    _, indices = torch.sort(x, dim=1)
    mask = x &lt;= 1292
    num_greater = mask.sum(dim=1, keepdim=True)
    
    _, s_indices = torch.sort(indices, dim=1)
    s_indices -= num_greater - 1
    
    return torch.where(mask, x, s_indices)

x = torch.tensor([[1296, 1295, 1292, 4, 1311, 4, 1293, 2],
                  [1297, 1295, 1292, 1401, 1294, 4, 1293, 2]], dtype=torch.int64)
func(x)

# tensor([[   3,    2, 1292,    4,    4,    4,    1,    2],
#         [   4,    3, 1292,    5,    2,    4,    1,    2]])
</code></pre>

username: curious
cooked: <p>How to do the same thing when values are repeated on a dimension?</p>
<pre><code class="lang-auto">a=tensor([[1296, 1295, 1292,    4, 1311,    4, 1293,    2],
        [1297, 1295, 1292, 1404, 1293,    4, 1293,    2]]
</code></pre>
<p>a should become:</p>
<pre><code class="lang-auto">a = tensor([[3, 2, 1292,    4, 5,    4, 1,    2],
        [3, 2, 1292, 4, 1,    4, 1,    2]]
</code></pre>

username: Kushaj
cooked: <p>I don’t think that can be done with the above code, without introducing more complexity. You can also write a <code>for</code> loop that does the same in brute force manner.</p>
What exactly is happening in this ResNet? Why multiply?

username: Blizzard_boi2020
cooked: <p>nn.Sequential(<br>
block(in_channels , out_channels, *args, **kwargs, downsampling=downsampling),<br>
*[block(out_channels * block.expansion,<br>
out_channels, downsampling=1, *args, **kwargs) for _ in range(n - 1)]<br>
)</p>

username: dejanbatanjac
cooked: <p>I think the implementation of ResNet in PyTorch is complicated. They used this block expansions as factors, and if I recall they used dilation extensively although it was constant 1 all the time.</p>
<p>You may assume they wrote it this way to help their researches experiment.</p>
<p>If you need simpler implementation check it from <a href="https://dejanbatanjac.github.io/resnet18" rel="nofollow noopener">resnet from scratch</a>.</p>

username: Blizzard_boi2020
cooked: <p>I understand it, but why exactly it’s shown block(—)*[block() for _ …]?</p>
PyTorch optimizer __getstate__ implementation

username: ilovescience
cooked: <p>This is a somewhat advanced question, but I am going through <a href="https://github.com/pytorch/pytorch/blob/91c80d122ab271d36ce37d60acf430fdbd54d249/torch/optim/optimizer.py#L54" rel="nofollow noopener">the source code</a> for the optimizer, and I found something interesting about the pickle-specific function <code>__getstate__</code>:</p>
<pre><code class="lang-auto">    def __getstate__(self):
        return {
            'defaults': self.defaults,
            'state': self.state,
            'param_groups': self.param_groups,
        }

    def __setstate__(self, state):
        self.__dict__.update(state)
</code></pre>
<p>My limited understanding of pickling/serialization is that <code>__getstate__</code> is going to just return the state of the class in whatever implemented form, often by just returning a copy of the dictionary of attributes <code>self.__dict__</code>. Here, it is returning a defined dictionary of the attributes <code>self.defaults</code>, <code>self.state</code>, and <code>self.param_groups</code>. But as far as I can tell, these are the only attributes of the optimizer class. So why can’t <code>__getstate__</code> just return <code>self.__dict__.copy()</code>?</p>
Conversion torch model to onnx model ignoring slice operation

username: natan
cooked: <p>I’m converting a pytorch model to onnx model. in this model there an assignment of tensor to a slice of another tensor.<br>
when i’m running the converted model with onnxruntime he crashes when trying to assign the small tensor to the big tensor and ignoring the slice operation.</p>
<p>i isolated the problem to this forward function:</p>
<pre><code class="lang-auto">def forward(self, x: torch.Tensor, y: torch.Tensor) -&gt; torch.Tensor:
    x[:y.size(0), 0, :] = y
    return x
</code></pre>
<p>onnxruntime returns this error where shape of x is (56,6,256) ans shape of y is (22,256):<br>
<strong>The input tensor cannot be reshaped to the requested shape. Input shape:{22,256}, requested shape:{56,1,256}</strong></p>
<p>my env is:</p>
<ul>
<li>PyTorch Version: 1.5.1</li>
<li>OS (e.g., Linux): Linux ubuntu 16.04</li>
<li>Python version: 3.6.8</li>
<li>onnxruntime verion: 1.4.0</li>
</ul>
<p>to reproduce this bug just ran this code:</p>
<p>``import torch<br>
import onnx<br>
import onnxruntime</p>
<p>class Test(torch.nn.Module):<br>
def  <strong>init</strong> (self):<br>
super(). <strong>init</strong> ()</p>
<p>def forward(self, x: torch.Tensor, y: torch.Tensor) -&gt; torch.Tensor:<br>
x[:y.size(0), 0, :] = y<br>
return x</p>
<p>x = torch.zeros((56,6,256))<br>
y = torch.rand((22,256))</p>
<p>m = Test()<br>
torch_outputs = m(x, y)</p>
<p>path = “/tmp/m.onnx”<br>
torch.onnx.export(m, (x, y), path,<br>
do_constant_folding=True,<br>
opset_version=12,<br>
input_names=[“x”, “y”],<br>
output_names=[“z”])</p>
<p>onnx_model = onnx.load_model(path)<br>
onnx.checker.check_model(onnx_model)<br>
inferred_model = onnx.shape_inference.infer_shapes(onnx_model)<br>
onnx.save(inferred_model, path)</p>
<p>ort_model = onnxruntime.InferenceSession(path)</p>
<p>ort_outputs = ort_model.run([“z”], {“x”: x.numpy(), “y”: y.numpy()})``</p>
"Member not initialized" error in PyTorch jit code

username: Victory
cooked: <p>While building a CUDA extension on Windows, I am getting the following errors in the “module.h” header of PyTorch code. Any idea about the cause/fix?<br>
Thanks.</p>
<p>PathToCondaEnv/lib/site-packages/torch/include\torch/csrc/jit/api/module.h(474): error: member “torch::jit::detail::ModulePolicy::all_slots” may not be initialized</p>
<p>PathToCondaEnv/lib/site-packages/torch/include\torch/csrc/jit/api/module.h(487): error: member “torch::jit::detail::ParameterPolicy::all_slots” may not be initialized</p>
<p>PathToCondaEnv/lib/site-packages/torch/include\torch/csrc/jit/api/module.h(501): error: member “torch::jit::detail::BufferPolicy::all_slots” may not be initialized</p>
<p>PathToCondaEnv/lib/site-packages/torch/include\torch/csrc/jit/api/module.h(514): error: member “torch::jit::detail::AttributePolicy::all_slots” may not be initialized</p>
Precomputing part of a loss function

username: Shuaib_Osman
cooked: <p>I’m trying to use pytorch to calculate a complex function but still require the gradient of the output as a function of the inputs. For example:</p>
<pre><code class="lang-auto">a=torch.tensor([1,2,3], dtype=torch.float32, requires_grad=True)
b=torch.tensor([3,2,1], dtype=torch.float32, requires_grad=True)
v=torch.tensor([0.2], dtype=torch.float32, requires_grad=True)

# here precalc represents some (fairly expensive) sequence of operations
precalc = a.dot(b)+a*b+a*a+b*b

def calc(precalc, v):
    z=torch.randn(3,1000)
    batch=v*precalc.matmul(z)
    return torch.relu(batch).mean()
</code></pre>
<p>So the precalc tensor is a fixed function of a and b (but expensive to compute).</p>
<p>When I call calc for the first time as <code>x=calc(precalc, v)</code> and then <code>x.backward()</code>, I get the correct gradients for a,b and v. However, if I call calc a second time, i.e. <code>x2=calc(precalc, v)</code> and then <code>x2.backward()</code>, I get the pytorch error that I’m going backward through the graph a second time and I should retain the graph.</p>
<p>I would ideally like to free the graph as precalc is expensive to compute but small in memory and the calc itself is easier to compute but will use a lot of memory.</p>
<p>Is there a way to achieve this?</p>
<p>Thanks.</p>
Pytorch 1.5.1 + ubuntu 20.04 + cuda 11

username: wolfviking0
cooked: <p>Hello,</p>
<p>I need to test some work on ubuntu 20.04 and cuda 11.<br>
I tried to build pytorch from source and I face just one hiccup easy to fix inside SparseCUDABlas.cu.</p>
<p>But now I am blocking at 89% (so close), and the build failed because of context_gpu.cu.<br>
Does any one have an idea how I can fix this one ?</p>
<p>Thanks</p>
<hr>
<p>[ 89%] Building NVCC (Device) object caffe2/CMakeFiles/torch_cuda.dir/utils/math/torch_cuda_generated_broadcast.cu.o<br>
[ 89%] Building NVCC (Device) object caffe2/CMakeFiles/torch_cuda.dir/utils/math/torch_cuda_generated_elementwise.cu.o<br>
[ 89%] Building NVCC (Device) object caffe2/CMakeFiles/torch_cuda.dir/utils/math/torch_cuda_generated_reduce.cu.o<br>
/home/sluser/Desktop/bitbucket/hugo/libs/a2kpf/external/torch_cuda/linux/src/c10/util/TypeIndex.h(164): error: expression must have a constant value<br>
/home/sluser/Desktop/bitbucket/hugo/libs/a2kpf/external/torch_cuda/linux/src/c10/util/TypeIndex.h(90): note: expression cannot be interpreted<br>
/home/sluser/Desktop/bitbucket/hugo/libs/a2kpf/external/torch_cuda/linux/src/c10/util/TypeIndex.h(110): note: called from:<br>
detected during:<br>
instantiation of “c10::string_view c10::util::get_fully_qualified_type_name() [with T=caffe2::Tensor]”<br>
/home/sluser/Desktop/bitbucket/hugo/libs/a2kpf/external/torch_cuda/linux/src/c10/util/typeid.h(415): here<br>
instantiation of “c10::string_view caffe2::TypeMeta::TypeName() [with T=caffe2::Tensor]”<br>
/home/sluser/Desktop/bitbucket/hugo/libs/a2kpf/external/torch_cuda/linux/src/caffe2/core/blob.h(88): here</p>
<p>1 error detected in the compilation of “/home/sluser/Desktop/bitbucket/hugo/libs/a2kpf/external/torch_cuda/linux/src/caffe2/core/context_gpu.cu”.<br>
CMake Error at torch_cuda_generated_context_gpu.cu.o.Release.cmake:281 (message):<br>
Error generating file<br>
/home/sluser/Desktop/bitbucket/hugo/build/__torchcuda/caffe2/CMakeFiles/torch_cuda.dir/core/./torch_cuda_generated_context_gpu.cu.o</p>
<p>make[2]: *** [caffe2/CMakeFiles/torch_cuda.dir/build.make:1303: caffe2/CMakeFiles/torch_cuda.dir/core/torch_cuda_generated_context_gpu.cu.o] Error 1<br>
make[2]: *** Waiting for unfinished jobs…<br>
make[1]: *** [CMakeFiles/Makefile2:2807: caffe2/CMakeFiles/torch_cuda.dir/all] Error 2<br>
make: *** [Makefile:161: all] Error 2<br>
CMake Error at tools/cmake/include/HugoPrebuild.cmake:288 (MESSAGE):<br>
[torchcuda] Failed to build<br>
Call Stack (most recent call first):</p>

username: Kushaj
cooked: <p>Reclone the repo recursively again (this sometimes helps).</p>
Hard attention in Pytorch

username: Neofytos
cooked: <p>Hi everyone,</p>
<p>I’ve been trying to find an implementation of the stochastic “hard” attention described in the seminal work of Xu et al (Show, Attend, and Tell) but so far I have only come across a Tensorflow implementation (<a href="https://github.com/uzaymacar/attention-mechanisms" rel="nofollow noopener">https://github.com/uzaymacar/attention-mechanisms</a>). I find it strange that no one has open-sourced this widely known attention mechanism. Please let me know if I have missed something <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thank you!</p>

username: Kushaj
cooked: <p>Check this <a href="https://paperswithcode.com/paper/show-attend-and-tell-neural-image-caption" rel="nofollow noopener">link</a>.</p>

username: Neofytos
cooked: <p>Hey! Thanks for the reply. Indeed that is an excellent repository <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> However, unfortunately, they do not implement hard attention.</p>

username: Kushaj
cooked: <p>None of the 37 implementations implemented hard attention.</p>
Stack expects each tensor to be equal size, but got [2, 18432] at entry 0 and [1, 0] at entry 4

username: costa98
cooked: <p>I am trying to train a neural network using the data loader</p>
<pre><code>data_loader = torch.utils.data.DataLoader(dataComplete, batch_size=16, shuffle=True)
</code></pre>
<p>but it gives me the error at the second for cycle</p>
<pre><code>for epoch in range(num_epochs):
  for i, (video, _, label) in enumerate(tqdm(data_loader)):
    ##training...
</code></pre>
<p>The dataset is UCF101 imported as such:</p>
<pre><code>dataComplete = torchvision.datasets.UCF101('ucf_dir','ucf_ann',
                                       frames_per_clip=n_frames, step_between_clips=1, frame_rate=None, fold=1, train=True,
                                       transform=transformF,
                                       _precomputed_metadata=None, num_workers=4, _video_width=0,
                                       _video_height=0, _video_min_dimension=0, _audio_samples=0)
</code></pre>
<p>where transformF = <code>transforms.Compose([lambda img:F.interpolate(img.permute(0,3,2,1).float(), size=(H,W)),])</code></p>
<p>But if I use batch size = 1 (when creating the data_loader) it will work</p>

username: Kushaj
cooked: <p>Check the inputs you are passing to <code>torch.stack</code>, they should be of same size. The error message says they are of different size.</p>
<p>Use this example as reference</p>
<pre><code class="lang-python">a = torch.randn(4,3)
b = torch.randn(4,3)

c = torch.stack([a,b])
c.shape
# torch.Size([2, 4, 3])
</code></pre>
Can you set a dataset attribute using python formatting?

username: bfeeny
cooked: <p>You can set the transforms of a dataset programatically like so:</p>
<pre><code class="lang-auto">for a_idx in range(128):
    train_loader.dataset.transforms=albumentations_transform_0
</code></pre>
<p>Is it possible to do this with Python formats?  I tried this:</p>
<pre><code class="lang-auto">for a_idx in range(128):
    train_loader.dataset.transforms=f"albumentations_transform_{a_idx}"
</code></pre>
<pre><code class="lang-auto">TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "&lt;ipython-input-18-4a491769a098&gt;", line 30, in __getitem__
    augmented = self.transforms(image=image)
TypeError: 'str' object is not callable
</code></pre>

username: Kushaj
cooked: <p>Python stores all the local objects in <code>locals()</code>, so you can do something as</p>
<pre><code class="lang-python">def func():
    pass

locals().get('func')() # To run the function by string name
</code></pre>
<p>There is also <code>globals()</code>, which you can use instead as</p>
<pre><code class="lang-python">my_globals = globals().copy()
my_globals.update(locals())

# then same as above
</code></pre>

username: bfeeny
cooked: <p>I may try that.  I ended up doing this for now, which appears to work, but is crude:</p>
<pre><code class="lang-auto">for a_idx in range(128):
    exec(f"train_loader.dataset.transforms=albumentations_transform_{a_idx}")
</code></pre>
Does number of torchvision.transforms used affect data loading speed?

username: BadTimeManagement
cooked: <p>As the title says, I’m trying to do some speed test on my data loading code. I am wondering if adding all kinds of transform to transform.Compose() will affect the speed of data loading?</p>
<p>Thank you.</p>

username: Kushaj
cooked: <p>All compose does is apply the transform one by one</p>
<pre><code class="lang-python">for t in self.transforms:
    img = t(img)
return img
</code></pre>
<p>So adding all the transforms to compose is same as applying them one-by-one.</p>

username: BadTimeManagement
cooked: <p>So if i want to make it as slow, i should add as much transformation as possible? Or creating own augmentation function would be better?</p>

username: Kushaj
cooked: <aside class="quote no-group" data-username="BadTimeManagement" data-post="3" data-topic="91336">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/e19b73/48.png" class="avatar"> BadTimeManagement:</div>
<blockquote>
<p>So if i want to make it as slow, i should add as much transformation as possible?</p>
</blockquote>
</aside>
<p>This is true.</p>
<aside class="quote no-group" data-username="BadTimeManagement" data-post="3" data-topic="91336">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/e19b73/48.png" class="avatar"> BadTimeManagement:</div>
<blockquote>
<p>Or creating own augmentation function would be better?</p>
</blockquote>
</aside>
<p>Torchvision transforms are written in C++ internally, so they are already fast.</p>
The difference between OpenBLAS and MKL

username: khalil
cooked: <p>I want to compile the source code in ARMv8 machine, and MKL does not support ARM.Then, I choose OpenBLAS as the BLAS.However, when I compile it successfully, I found the CPU utilization is low.My machine has 128 cores but when I run the following benchmark:</p>
<pre><code class="lang-auto">import timeit
runtimes = []
threads = [1] + [t for t in range(2, 49, 2)]
for t in threads:
    torch.set_num_threads(t)
    r = timeit.timeit(setup = "import torch; x = torch.randn(1024, 1024); y = torch.randn(1024, 1024)", stmt="torch.mm(x, y)", number=100)
    runtimes.append(r)
# ... plotting (threads, runtimes) ...
</code></pre>
<p>The CPU untilzation just like this and  time has hardly changed.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/6/16ff0f4520535737b2e01babdd92836ef113d40f.png" alt="image" data-base62-sha1="3hqXXDlD1XB0YHauxguqxv5ouQf" width="637" height="168"></p>
<p>The compile step just like this:<br>
export USE_OPENMP=1<br>
export BLAS=OpenBLAS<br>
export USE_CUDA=0<br>
python setup.py install</p>
<p>machine:aarch64</p>
<p>Can OpenBLAS do parallel computing on Pytorch?If not, how can I do to improve the performance?</p>
Numpy-like tensor processing on Android

username: ryanaleksander
cooked: <p>Hi everyone. I’m deploying a deeplab model onto an android application. The output of the model is a [1,21, 400,400] tensor. In order to get my final result mask, I have to run an argmax operation on the 2nd dimension. In python I figure the operation would be something like this out = np.argmax(out, axis=1). Then I’ll have to create a mask based on the result (out[out==classId] = colorValue).</p>
<p>But there is nothing quite like numpy on android. Looping through the whole thing to find all the maximum values for each pixel, and then set the color is going to be very slow. Is there any way I can do this efficiently. Any help would be greatly appreciated</p>

username: tom
cooked: <p>The easiest probably is to code this in TorchScript and call that.<br>
You can also run libtorch proper on PyTorch and do your own JNI wrapper for whatever you need.</p>

username: ryanaleksander
cooked: <p>Thanks for the reply. I’m not all that familiar with TorchScript and Android. What would the process be like? Would you mind recommending me a starting place?</p>

username: tom
cooked: <ol>
<li>You write your numpy-like processing in PyTorch as a plain python function or better yet as a nn.Module,</li>
<li>run it through torch.jit.script similar to the TorchScript tutorials,</li>
<li>now you can use it similar to the Android tutorials.</li>
</ol>
<p>Using a nn.Module in step 1 isn’t better in the sense that it would be preferable per se, but it will reduce the amount of adaptation you need to do in steps 2 and 3.</p>
<p>Best regards</p>
<p>Thomas</p>
Constant loss while training

username: Angry_potato
cooked: <p>Hello all,<br>
I am getting constant loss while iteration.<br>
Here is how i define my optimizer.</p>
<pre><code class="lang-auto">input_img=torch.randn(3,64,64,requires_grad=True, device='cuda')
optimizer=optim.Adam([input_img], lr=1e-05)
</code></pre>
<p>And after that i simply calculate loss and use <code>optimizer.step()</code> to update <code>input_img</code>.<br>
But the loss is constant.I couldn’t find the mistake that i made.</p>
<p>Here is the iteration loop.</p>
<pre><code class="lang-auto">while True:
    input_img.data.clamp_(0,1)
    optimizer.zero_grad()
  
    input_img=input_img.view(3,64,64)
    input_img=norm(input_img)
    input_img=input_img.view(1,3,64,64)
    score=net(input_img)
  

    style_score = 0
    content_score = 0
      
    for ip in range(4):
          if ip==1:
              content_score=content_score+content_loss(content[1],score[1])
          else:
              style_score=style_score+style_loss(gram_matrix(style[ip]),gram_matrix(score[ip]))*10
    print('style_loss = ',style_score)
    print('content_loss = ',content_score)
    loss_total=style_score+content_score
    print('total_loss = ',loss_total)
    loss_total.backward(retain_graph=True)
    optimizer.step()
    count=count+1
    print('#################################',count,'#######################################')
</code></pre>

username: ptrblck
cooked: <p>Don’t use the <code>.data</code> attribute, as it might yield unexpected side effects.<br>
Also, you are currently overriding <code>input_img</code> in:</p>
<pre><code class="lang-python">input_img=input_img.view(3,64,64)
input_img=norm(input_img)
input_img=input_img.view(1,3,64,64)
</code></pre>
<p>so you should use another variable name to keep the original <code>input_img</code> tensor alive.</p>
Unfreeze BatchNorm only in a deep net

username: WeiQin_Chuah
cooked: <p>Hi,</p>
<p>How can I unfreeze only BatchNorm2D and freeze all the other layers is a deep net like ResNet50?</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>You could first freeze all parameters and later unfreeze the parameters of all batchnorm layers:</p>
<pre><code class="lang-python"># Freeze all parameters
model = models.resnet50()
for param in model.parameters():
    param.requires_grad_(False)

# Unfreeze bn params
for module in model.modules():
    if isinstance(module, nn.BatchNorm2d):
        if hasattr(module, 'weight'):
            module.weight.requires_grad_(True)
        if hasattr(module, 'bias'):
            module.bias.requires_grad_(True)

# Check
out = model(torch.randn(1, 3, 224, 224))
out.mean().backward()

for name, param in model.named_parameters():
    if param.grad is not None:
        print(name, param.grad.abs().sum())
</code></pre>
CUDAerror: device-side assert triggered

username: chetan06
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/8/987f2ecba96217e7457f33bcf86c5621704efc57.png" data-download-href="https://discuss.pytorch.org/uploads/default/987f2ecba96217e7457f33bcf86c5621704efc57" title="cud-error"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/8/987f2ecba96217e7457f33bcf86c5621704efc57.png" alt="cud-error" data-base62-sha1="lL33VFDJS18ObEwXJOtrzdXYPWf" width="690" height="185" data-dominant-color="3A3B3C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cud-error</span><span class="informations">741×199 23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I got this error while running a simple line of code. Most of the time when I get this error I just switch off my gpu and run it on cpu , and always I get what is wrong with my code. I did the same with this but It ran fine with cpu.</p>

username: ptrblck
cooked: <p>Could you rerun the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the complete stack trace here, please?</p>

username: chetan06
cooked: <p>Actually I was running this on google colab. So How can I run this command there?</p>
<p>Anyway I switched off my gpu and restarted it after 1 day and was not getting any error…but then too I want to know the reason</p>

username: ptrblck
cooked: <p>You could try to set this environment argument via <code>os.environ['CUDA_LAUNCH_BLOCKING'] = "1"</code>.<br>
This would have to be added before any other imports in your script to work properly.</p>
When to save the hidden state during training in LSTMs

username: niata
cooked: <p>When training an LSTM when should i save the hidden state?<br>
During the run or after running the optimizer?<br>
Here is an example:</p>
<pre><code class="lang-auto">import torch
class neuralnet(torch.nn.Module):
    def __init__(self):
        super(neuralnet, self).__init__()
        self.lstm = torch.nn.LSTM(
            input_size=64,
            hidden_size=64,
            num_layers=1,batch_first=False
        )
        self.front = torch.nn.Linear(128,64)
        self.behind=torch.nn.Linear(64,2)
        self.reset_hidden()

    def reset_hidden(self):
        self.hidden=(torch.rand(1, 1,64),torch.rand(1, 1,64))

    def forward(self, inp,storehidden=False):
        # frontlayers
        x=self.front(inp)
        x, prx= self.lstm(x.view(-1,1,64), self.hidden)
        if storehidden:
            self.hidden=prx
        return torch.sigmoid(self.behind(x))

net=neuralnet()
optimizer=torch.optim.Adam(net.parameters(),lr=0.3)
loss_function = torch.nn.BCELoss()
input=torch.rand(128)
output=net(input)
loss = loss_function(output,torch.tensor([0,1], dtype=torch.float).view(1,1,2))
loss.backward(retain_graph=True)
optimizer.step()
with torch.no_grad():
    print("direct:",net(input))
    net(input,True)
    print("after the optimizer:",net(input))
</code></pre>
<p>currently i store it after the optimizer and there is little difference to before it.</p>
<pre><code class="lang-auto">direct: tensor([[[0.0197, 0.9791]]])
after the optimizer: tensor([[[0.0092, 0.9885]]])
</code></pre>
Xtensor and libtorch conflict results in "Error C3861 'lib': identifier not found include\c10\util\variant.h 958"

username: Shisho_Sama
cooked: <p>Hello everyone,<br>
I’m trying to convert some Python modules into their C++ counter part and for this I need to use <code>Xtensor</code>. However, I notcied upon including  both <code>torch</code> and <code>xtensor</code>, they have a conflic and thus I face the following error from libtorch side! :</p>
<pre><code class="lang-auto">Severity	Code	Description	Project	File	Line	Suppression State
Error	C3861	'lib': identifier not found	test	G:\cpp\...\libtorch\include\c10\util\variant.h	958	
Error	C2653	'lib': is not a class or namespace name	test	G:\cpp\...\libtorch\include\c10\util\variant.h	958	
Error	C2143	syntax error: missing ',' before '&lt;'	test	G:\cpp\...\libtorch\include\c10\util\variant.h	958	
Error	C2653	'lib': is not a class or namespace name	test	G:\cpp\...\libtorch\include\c10\util\variant.h	982	
Error	C2061	syntax error: identifier 'type_pack_element_t'	test	G:\cpp\...\libtorch\include\c10\util\variant.h	982	
Error	C2059	syntax error: '&gt;'	test	G:\cpp\...\libtorch\include\c10\util\variant.h	982	
Error	C7510	'array': use of dependent template name must be prefixed with 'template'	test	G:\cpp\...\libtorch\include\c10\util\variant.h	995	
Error	C2653	'lib': is not a class or namespace name	test	G:\cpp\...\libtorch\include\c10\util\variant.h	1033	
Error	C2061	syntax error: identifier 'enable_if_t'	test	G:\cpp\...\libtorch\include\c10\util\variant.h	1033	
Error	C2653	'lib': is not a class or namespace name	test	G:\cpp\...\libtorch\include\c10\util\variant.h	1034	
Error	C2065	'T': undeclared identifier	test	G:\cpp\...\libtorch\include\c10\util\variant.h	1037	
Error	C2065	'Ts': undeclared identifier	test	G:\cpp\...\libtorch\include\c10\util\variant.h	1037	
Error	C2631	'find_index_checked_impl': a class or enum cannot be defined in an alias template	test	G:\cpp\...\libtorch\include\c10\util\variant.h	1040	
Error	C2988	unrecognizable template declaration/definition	test	G:\cpp\port\LibtorchPort\LibtorchPort\LibtorchPort.cpp	1	
Error	C2059	syntax error: 'template'	test	G:\cpp\port\LibtorchPort\LibtorchPort\LibtorchPort.cpp	1	
Error	C1903	unable to recover from previous error(s); stopping compilation	test	G:\cpp\port\LibtorchPort\LibtorchPort\LibtorchPort.cpp	1	

</code></pre>
<p>Whats the cause here and how can I fix this ?<br>
Any help in this reagard is greatly apprecaited</p>
Update billions of Bernoulli parameters

username: jchaykow
cooked: <p>Does anyone have an example of how to set up a model of <em>n</em> bernoulli random variables? I want to sample billions of bernoulli random variables and then update them with something like NLL loss against an array of billions of 1’s and 0’s.</p>
<p>Something like this, but scales better. If <em>n</em> is so large (1e6 x 1e6 matrix) how might I structure this problem?</p>
<pre><code class="lang-auto">sample = np.array([[ 1.,  1.,  0.,  1.],
                   [ 0.,  1.,  1.,  1.],
                   [ 0.,  0.,  1.,  1.],
                   [ 1.,  0.,  0.,  1.]])

x = Variable(torch.from_numpy(sample)).type(torch.FloatTensor)

n = 4
ps = torch.rand((n,n), requires_grad=True)

learning_rate = 2e-4
for _ in range(num_epochs):
    NLL = -torch.sum(torch.log(x @ p + (1-x) @ (1-p)) )
    NLL.backward()

    p.data -= learning_rate * p.grad.data
    p.grad.data.zero_()
</code></pre>
Random Transforms not consistent with the same seed

username: balbok0
cooked: <p>I was recently trying to train a resnet on ImageNet with consistent images inputs across runs, yet still with data augmentation, such as cropping, flipping rotating, etc.<br>
I run into a problem with the fact, that there is no way of consistently getting the same random crops.<br>
Here is a minimal example I created:</p>
<pre><code class="lang-auto">import torch
from torchvision import transforms


torch.random.manual_seed(1)

x = torch.rand((3, 10, 10))

tf_crop = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomResizedCrop(4),
    transforms.ToTensor(),
])
tf_flip = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
])
tf_rot = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomRotation(45),
    transforms.ToTensor(),
])

# Consistent x among the calls
print(x[:2, :2, :2])

# RandomRotation, RandomResizedCrop and Random HorizontalFlip changes stuff
# even if seed stays the same
for idx in range(2):
    torch.random.manual_seed(1)
    print(f'Crop {idx + 1}')
    print(tf_crop(x)[:2, :2, :2].numpy())
for idx in range(2):
    torch.random.manual_seed(1)
    print(f'Flip {idx + 1}')
    print(tf_flip(x)[:2, :2, :2].numpy())
for idx in range(2):
    torch.random.manual_seed(1)
    print(f'Rotation {idx + 1}')
    print(tf_rot(x)[:2, :2, :2].numpy())
</code></pre>
<p>Each iteration of each of the for loops produces different results. However, at least in my understanding, across iterations, the results should be the same since the seed is set to the same number.</p>
<p>I would appreciate it if someone could point in direction on how to achieve consistency across different iterations of for loops (NOT across different for loops).</p>

username: ptrblck
cooked: <p>Some <code>torchvision</code> transformation use the Python <code>random</code> library to sample the numbers.<br>
Add <code>random.seed(1)</code> to the loop and you should get the same results.</p>

username: balbok0
cooked: <p>Thank you, that worked flawlessly!</p>

username: weikunhan
cooked: <p>In PyTorch 1.6, you need to use torch.manual_seed(5) and random.seed(5) same time. Please see the issue in: <a href="https://github.com/pytorch/pytorch/issues/42331" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/42331</a></p>
Multiple gathering

username: Nick_8229
cooked: <p>torch.gather could help in the following case,</p>
<pre><code class="lang-auto">out[i][j][k] = input[index[i][j][k]][j][k]  # if dim == 0
out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1
out[i][j][k] = input[i][j][index[i][j][k]]  # if dim == 2
</code></pre>
<p>However, what we do when we have multiple index tensors?</p>
<pre><code class="lang-auto">source = torch.ones(dim1, dim2, dim3, dim4)
index1 = torch.ones(dim3, dim4)
index2 = torch.ones(dim3, dim4)

for i in range(dim3):
    for j in range(dim4):
        ret[i, j] = source[index1[i, j], index2[i, j],i, j]
</code></pre>
Query about weight decay

username: DLopezG
cooked: <p>I understand this is the formula for L2 regularization:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/9102a3197ea5cf961009290854d0d53f0c92135e.png" alt="image" data-base62-sha1="kGOSKZunM8jzintqMTq15uC80y2" width="453" height="137"></p>
<p>Does weight decay equal lambda in this equation?</p>

username: Kushaj
cooked: <p>Here lambda is l2-regularization factor. In case of SGD, this value is proportional to weight decay but for other optimizers like Adam this is not the case.</p>
<p>In short, weight decay is something that you subtract from the weight update equation directly.</p>

username: DLopezG
cooked: <p>Then how to add L2 regularization when using Adam?</p>

username: Kushaj
cooked: <p>You don’t. L2 regularization does not work well with the modern optimizers like Adam, weight decay is the option to go.</p>
<p>But if you still want to add l2 regularization just use optim.Adam and provide weight_decay argument in it (pytorch will use that argument for l2 regularization, AdamW solves this). I may be wrong on this as I have not followed the complete AdamW discussion on pytorch.</p>
Custom concatenation of 2d torch.Tensor rows

username: plhngts
cooked: <p>Hi,</p>
<p>let’s say I have a tensor like this:</p>
<pre><code class="lang-auto">a = torch.randint(1, 9, (12, 3))
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">tensor([[2, 1, 5],
        [2, 4, 7],
        [8, 4, 6],
        [3, 8, 5],
        [7, 3, 5],
        [6, 6, 8],
        [6, 8, 3],
        [5, 1, 4],
        [2, 4, 6],
        [4, 8, 7],
        [3, 3, 6],
        [1, 7, 8]])
</code></pre>
<p>and now I want to concatenate specific rows according to the following pattern:</p>
<pre><code class="lang-auto">triplets_list = []
for i in range(0, a.shape[0], 4):
    # Get quadruplet
    x1, x2, x3, x4 = a[i:i+4]
            
    # Create concatenated pairs from quadruplet
    pair_x1x2 = torch.cat([x1, x2], dim=0)
    pair_x1x3 = torch.cat([x1, x3], dim=0)
    pair_x3x4 = torch.cat([x3, x4], dim=0)
           
    # Stack concatenated tensors
    triplets_list.append(torch.stack([pair_x1x2, pair_x1x3, pair_x3x4], dim=0))

# Concatenate list of triplets
wanted_result = torch.cat(triplets_list, dim=0)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">tensor([[2, 1, 5, 2, 4, 7],
        [2, 1, 5, 8, 4, 6],
        [8, 4, 6, 3, 8, 5],
        [7, 3, 5, 6, 6, 8],
        [7, 3, 5, 6, 8, 3],
        [6, 8, 3, 5, 1, 4],
        [2, 4, 6, 4, 8, 7],
        [2, 4, 6, 3, 3, 6],
        [3, 3, 6, 1, 7, 8]])
</code></pre>
<p>So for every 4 rows (indexed from 0 to 3) I need to concatenate 0 with 1, 0 with 2 and 2 with 3. Is there a way to vectorize that operation? - for now I come up with non-vectorized presented above.</p>
Autograd supported operations

username: n4tman
cooked: <p>Hi All,</p>
<p>I have been reading through autograd recently.</p>
<p>Which operations I should be using in order for gradients to be correctly calculated by autograd?</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You should be using torch implementation of any function you use.<br>
All pytorch function will work. Or raise an error if you’re asking for things that are not supported/non-differentiable.</p>

username: n4tman
cooked: <p>Thanks, so if I use an operation or function that is not support by autograd, an error is raised?</p>

username: albanD
cooked: <p>Yes.<br>
All differentiable functions should be implemented though. Only a few linear algebra functions that could be differentiated are not.<br>
Of course non-differentiable function won’t work.<br>
And the functions like <code>.detach()</code> and <code>torch.no_grad()</code> are built to do things outside of the autograd so they are special cases<br>
.</p>

username: n4tman
cooked: <p>Thanks, just to clarify. Using autograd supported operations and functions allows gradients to be correctly calculated?</p>

username: albanD
cooked: <p>Yes it will compute the correct gradients if you don’t get any error <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: n4tman
cooked: <p>And if gradients cannot be correctly computed, I get an error?</p>

username: albanD
cooked: <p>If the function is not differentiable, you’ll get an error.<br>
If gradients cannot be computed (usually due to inplace ops), yes you will get an error.<br>
In general, if a gradient is computed, it is correct.</p>
Why (or when) to use unsqueeze ( ) in pytorch?

username: Eric_Cartman
cooked: <p>Why (or when) to use Unsqueeze in pytorch ?</p>

username: Kushaj
cooked: <p>If you want to run your model on a single image. You can load the image and then use unsqueeze to add the batch dimension in the starting. You can also use unsqueeze during broadcasting to make sure the things are broadcasted as you want.</p>
Memory conscious resized matrix multiplication

username: Pietro_Cicalese
cooked: <p>Hello, first time posting here.</p>
<p>I have become interested in transformers and their application in vision. This being said, torch.matmul of two flattened vectors representing all of the pixels in an image is virtually unusable with images much larger than a thumbnail.</p>
<p>For this reason I have been looking at scaled down versions of transformers, but something doesn’t seem right to me.</p>
<p>Generally, If I want to scale attention down (to say size [Batch, ((0.25 * Height) * (0.25 * Width))^2], lets say [1, 256 * 256]) and want to multiply it by my values at full scale (of size [Batch, Height * Width], so [1, 64 * 64] in our example) I would need to scale the attention values up and them perform matrix multiplication with torch.matmul:</p>
<pre><code class="lang-auto"># weights.shape = [1, 256, 256] in our case, else [1, 4096, 4096] at full scale
# flat_v.shape = [1, 4096]
attn = torch.matmul(weights, flat_v.transpose(2, 3))
# attn.shape = [1, 4096, 1] is what we want in both cases
</code></pre>
<p>This seems very poorly optimized for memory and I am sure there must be a smarter way to approach this problem in pytorch. Is there some concerted operation that will allow me to multiply the vectors together without having to overwhelm my memory? I dont need to interpolate values when scaling for this implementation (all values in the region of interest can be the same).</p>
Loss divided by the item() of itself

username: hongyuanluke
cooked: <p>Hi there, I am just wondering that in Pytorch, if I do:</p>
<p>loss=loss/loss.item(), I know that the resulting loss will be having a scalar value of 1. Do the gradients of the loss still preserve? What is the difference between loss/loss and loss/loss.item() ?</p>

username: tom
cooked: <p><code>loss_norm = loss/loss.item()</code> will have gradient in the same direction as <code>loss</code>, but scaled by `1/loss.item() because the gradient flows through loss but not loss.item() (which is a constant number to autograd).</p>
Has anyone ever been in this situation，about a bug of torch.jit.trace

username: li_jiachen
cooked: <p>Traceback (most recent call last):<br>
File “/home/lijiachen/Projects/binlang/scripts/show_img.py”, line 29, in <br>
traced_script_module = torch.jit.trace(model, img)<br>
File “/home/lijiachen/anaconda3/lib/python3.7/site-packages/torch/jit/<strong>init</strong>.py”, line 875, in trace<br>
check_tolerance, _force_outplace, _module_class)<br>
File “/home/lijiachen/anaconda3/lib/python3.7/site-packages/torch/jit/<strong>init</strong>.py”, line 1027, in trace_module<br>
module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, _force_outplace)<br>
RuntimeError: 0 INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1591914880026/work/torch/csrc/jit/ir/alias_analysis.cpp:318, please report a bug to PyTorch. We don’t have an op for aten::to but it isn’t a special case.  Argument types: Tensor, None, int, Device, bool, bool, bool, int,</p>

username: tom
cooked: <p>It would help if you could provide a reproducing example and say something about which PyTorch version you’re using. This would make it much faster to look at locating the problem.</p>
Will the cuda memory cache be helpful for loading the highly similar batch-data?

username: lzypku
cooked: <p>When we use the mini-batch for training, if the next batch-data is highly similar to the previous batch, will the cuda memory cache be helpful for loading it? (compared with the next batch-data which is totally different from the previous batch)</p>

username: ptrblck
cooked: <p>Pasting the answer from the PM:</p>
<blockquote>
<p>The PyTorch caching allocator is only concerned with the re-usage of the device memory.<br>
I.e. it it trying to avoid expensive cudaMalloc calls, which would be synchronizing, and thus keeps the memory allocated and tries to reuse it. If that’s not possible, new memory is allocated and added to the reserved pool.<br>
You might be referring to the L1 or L2 memory cache, which is used as a faster memory pool on the device to avoid loading from the global GPU memory.</p>
</blockquote>

username: lzypku
cooked: <p>Thanks again for your reply!</p>