akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch DataLoader multiprocessing, could not unlink shared memory file

jambo6: <p>Hello,</p>
<p>I’m getting the following error with a multi-worker DataLoader</p>
<pre><code class="lang-auto">2022-11-25 23:18:06 [info] RuntimeError: falseINTERNAL ASSERT FAILED at "/opt/pytorch/pytorch/aten/src/ATen/MapAllocator.cpp":319, please report a bug to PyTorch. could not unlink the shared memory file /torch_2999188_550744154
</code></pre>
<p>I’m using a PyTorch DataLoader to load data on-the-fly since my full dataset does not fit into RAM. With multiple workers the data loading time is hidden by the model forward, this is not the case when <code>num_workers = 0</code> so this isn’t what I’m hoping for as a solution.</p>
<p>It’s a bit strange since everything is ok for a couple of epochs (which is on the order of a few hours), but then at some point one of the splits crashes with this error.</p>
<p>Does anyone have any idea what this might be caused by, and how to go about fixing it?</p>
<p>My dataloader is wrapping a dataset that I have ensured only contains elements with a fixed refcount to ensure we dont hit the memory issues from <a href="https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662" rel="noopener nofollow ugc">this thread</a>, so I dont think its related to that.</p>
<p>Thanks!</p>
100% at mini-batch gradient decent testing

IoannisTselekoglou: <p>Hey, let’s say you have a small dataset with 200 data points. When testing the model, I get 100% accuracy, which seems false. However, considering I test in very small batches with batch size = 8-10, it makes sense to me that occasionally every prediction matches the label. So should you even test in such small batch sizes?</p>

ptrblck: <p>The batch size during testing shouldn’t matter, as you would have to use a separate test dataset and only test the final model once on it (the validation dataset is usually used once per training epoch).<br>
In case you are using the training data to “test” the model, note that you are leaking data and the “test” accuracy is indeed the training accuracy of a single batch.</p>
TorchData Augmentations

Samuel_Lewis: <p>Hi,</p>
<p>I am new to datapipes and I am unsure how to implement augmentation on images, similar to how I would in the normal dataset class.</p>
<p>I have a datapipe which currently returns an image, and a mask, but I am not sure how to map my transforms.</p>
<p>when i run something like:</p>
<pre><code class="lang-auto">  ds_stack = ds_stack.map(
        A.Compose(
  [
  A.HorizontalFlip(p=0.5),
  #A.Rotate(limit=35, p=1.0),
  A.VerticalFlip(p=0.5),
  A.RandomRotate90(p=1),
  A.Transpose(p=0.5),
  A. Normalize(
      mean=[0.5]*15,  ## adjust to number of bands
      std=[0.5]*15,   ## adjust to number of bands
      max_pixel_value=10000,  ### adjust to the max pixel value
  ),
  ToTensorV2()
  ])
</code></pre>
<p>i get the error: KeyError: ‘You have to pass data to augmentations as named arguments, for example: aug(image=image)\nThis exception is thrown by <strong>iter</strong> of MapperIterDataPipe()’</p>

nivek: <p>The exception is raised by the function that you pass in. Based on the message, I believe you can wrap a function around it to provide named arguments:</p>
<pre><code class="lang-python">transform = A.Compose(...)

def apply_transform(image):
    return transform(image=image)

dp = dp.map(apply_transform)
</code></pre>

Samuel_Lewis: <aside class="quote no-group quote-modified" data-username="nivek" data-post="2" data-topic="167049">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nivek/48/39672_2.png" class="avatar"> nivek:</div>
<blockquote>
<pre><code class="lang-auto">def apply_transform(image):
    transform(image=image)
dp = dp.map(apply_transform)
</code></pre>
</blockquote>
</aside>
<p>fantastic, thank you very much Kevin</p>
What’s the best practice to make a list of lists that collect the scatter result?

SJoJoK: <p>Here I got two tensor, A and it’s corresponding index tensor, for example</p>
<pre><code class="lang-auto">A = torch.tensor([2,3,41,5,12,45,4576,2,1])
index = torch.tensor([0,0,1,1,2,2,2,2,3])
</code></pre>
<p>What I want is a list of lists, for example</p>
<pre><code class="lang-auto">Res = [ 
       [2,3],
       [41,5],
       [12,45,4567,2],
       [1]
      ]
</code></pre>
<p>that Res[i] is a list contains all the elements in A which corresponding index is i.<br>
Actually I can use a loop to get this, but I think there may be a better way to do this in torch (loop seems slow without vectorization)<br>
Is there any better way to do it? I can get the max length of list, so some solutions with fix-length is also good.<br>
(Briefly speaking,  what I want is torch_scatter.segment_coo without reduce)<br>
Thanks in advance for any help or suggetions.</p>
Torchaudio.load not working as expected

ADONAI_TZEVAOT: <p>The dataloader script is working fine with all audio files but for a few audio files, the torchaudio.load is saying that it cannot load file. But, when I try using torchaudio.load separately and try to load the same file, then its working. Please help</p>

ptrblck: <p>Could you post the error message which indicates the loading fails?</p>
Specifying stride in Conv2D and ConvTranspose2d

sroyyors: <p>hi,<br>
I am trying to implement a Convolutional AutoEncoder following the python example here <a href="https://medium.com/dataseries/convolutional-autoencoder-in-pytorch-on-mnist-dataset-d65145c132ac" rel="noopener nofollow ugc">https://medium.com/dataseries/convolutional-autoencoder-in-pytorch-on-mnist-dataset-d65145c132ac</a>. However, I  am not sure how to specify the stride or padding parameters for the Convolution or Transpose convolution.</p>
<p>I looked at the MNIST example code here <a href="https://github.com/pytorch/examples/blob/main/cpp/mnist/mnist.cpp" rel="noopener nofollow ugc">https://github.com/pytorch/examples/blob/main/cpp/mnist/mnist.cpp</a><br>
but no stride is mentioned.</p>
<p>How can we specify the stride parameter?<br>
thanks,<br>
Sushmita</p>

sroyyors: <p>I resolved this. You can pass these parameters using the torch::nn::Conv2dOptions and torch::nn::TransposeConv2dOptions types.</p>
Cache/store transformed input to speed up data loading

JXuan: <p>Hey there,</p>
<p>The inputs to my dataloader are audio files (wav.), which during the preprocessing go through stft and other transformations. The spectrograms are then fed into the network.</p>
<p>Instead of computing stft and performing the same transformations at each epoch, I would like to cache the transformed inputs either in RAM or on disk. What’s a more efficient way of doing it in pytorch?</p>
<p>After searching online, I found that people cache those transformed as numpy, which I don’t know if is slower than saving as tensors. Others use tfrecord and custom pytorch dataloader.</p>
<p>Many thanks in advance!</p>

ptrblck: <p>Assuming you don’t use ay random transformations you could try to save the tensors directly via <code>torch.save</code> and check how much memory it would take of your SSD.<br>
During training you could write a custom <code>Dataset</code> to load each tensor and pass it to the model.</p>

nivek: <p>Aside from ptrblck’s suggestion, you can also try <a href="https://pytorch.org/data/main/" rel="noopener nofollow ugc"><code>TorchData</code></a> which provides a few build-in caching functionalities (in-memory or disk).</p>
Why intra_op parallelism threads still consume CPU resource even when the computation is done?

Zeyu-ZEYU: <p>I do optimizer.step() in python using CPU. By default, PyTorch set_num_threads to 20 on my server. After optimizer.step() I let the program time.sleep(3). When optimizer.step() is in process, the CPU usage increases to ~1900% and optimizer.step() lasts for ~50ms. But after optimizer.step() the CPU usage still keeps high as ~1900% for ~450ms. Then CPU usage decreases to 0.</p>
<p>If I set_num_threads to 5, the CPU usage increases to ~500% when optimizer.step() starts. But when the computation is done, the CPU usage still keeps ~400-500% for ~8x computation time.</p>
<p>I’m wondering why intra_op parallelism threads still consume CPU resource even when the computation is done? Clean-up? or just some async operations?</p>
How to Upsample a Featuremap with ConvTranspose2d?

abolfazl_saheban: <p>Hello, I want to upsample a feature map by a scale factor=2 but with Transposed Convolution.<br>
Here is a code for this purpose but by using torch.nn.Upsamle.<br>
nn.Upsample(scale_factor=2, mode=‘nearest’)<br>
Is there any possibility to do this with Transposed Convolution and if it is possible, what parameter should I give to<br>
torch.nn.ConvTranspose2d(<em>in_channels</em> , <em>out_channels</em> , <em>kernel_size</em> , <em>stride=1</em> , <em>padding=0</em> , <em>output_padding=0</em> , <em>groups=1</em> , <em>bias=True</em> , <em>dilation=1</em> , <em>padding_mode=‘zeros’</em> , <em>device=None</em> , <em>dtype=None</em> )?</p>

KFrank: <p>Hi Abolfazl!</p>
<aside class="quote no-group" data-username="abolfazl_saheban" data-post="1" data-topic="166623" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abolfazl_saheban/48/44817_2.png" class="avatar"> abolfazl_saheban:</div>
<blockquote>
<p>I want to upsample a feature map by a scale factor=2 but with Transposed Convolution.</p>
</blockquote>
</aside>
<p>Use <code>kernel_size = stride = 2</code> (and for <code>in_channels</code> and <code>out_channels</code>,<br>
use whatever is appropriate for your use case):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.0'
&gt;&gt;&gt; t = torch.ones (1, 1, 8, 8)
&gt;&gt;&gt; t.shape
torch.Size([1, 1, 8, 8])
&gt;&gt;&gt; convtran = torch.nn.ConvTranspose2d (in_channels = 1, out_channels = 1, kernel_size = 2, stride = 2)
&gt;&gt;&gt; convtran (t).shape
torch.Size([1, 1, 16, 16])
&gt;&gt;&gt; convtran.weight
Parameter containing:
tensor([[[[ 0.0681, -0.1747],
          [-0.3762,  0.0049]]]], requires_grad=True)
&gt;&gt;&gt; convtran.bias
Parameter containing:
tensor([0.3983], requires_grad=True)
</code></pre>
<p>Note, that <code>ConvTranspose2d</code> has trainable <code>weight</code> and <code>bias</code> parameters,<br>
so it is a <em>trainable</em> up-sampling (which is probably a good thing).</p>
<p>Best.</p>
<p>K. Frank</p>

abolfazl_saheban: <p>Thank you for your attention</p>
What state_dict to save for model fine-tuning

nicozhou: <p>Hi, I am training a model with synthetic data and wish to fine-tune it with my real-dataset. So, what state_dict should I save for continue training with my real-dataset?</p>
<p>Does saving model.state_dict is enough for fine-tuning? Or should I save optimizer.state_dict too?</p>

ptrblck: <p>For a general fine-tuning use case, storing the <code>model.state_dict()</code> might be sufficient, as it’s also done when you are trying to fine-tune e.g. the <code>torchvision</code> models (you can’t load the <code>optimizer.state_dict()</code> as they are not available).<br>
However, if you would like to “continue” the training, then you should store the <code>optimizer.state_dict()</code> as well (additionally, you might also want to store the learning rate scheduler’s <code>state_dict</code> if used).</p>

nicozhou: <p>Thanks for the answer, I am curious does loading optimizer.state_dict() will make huge difference? Because I found no different between loading and not loading optimizer.state_dict in both fine-tuning and ‘continue’ training.</p>

ptrblck: <p>If the optimizer has internal states (e.g. such as <code>Adam</code>) I would claim you should see a difference.</p>

nicozhou: <p>I see, maybe in my case there is no significant difference. But I still have one question in mind, is it a good practice to load <em><strong>model.state_dict</strong></em> and <em><strong>optimizer.state_dict</strong></em> to do <strong>fine-tuning</strong> on other dataset? Which is like the “continue” training do.</p>

banton: <p>Were you able to find an answer to this question anywhere?</p>
Gram matrix in mixed precision

Andrej: <p>Hi all,<br>
I was trying to train a model using a Style loss with VGG19 in float16 precision however, when I calculated the Gram matrix it contains Nan and Inf. This does not happen in float32.<br>
Here is the code of Calculating Gram matrix:</p>
<pre><code class="lang-auto"> def getGramMatrix(self, x):
        b, ch, h, w = x.size()
        f = x.view(b, ch, w * h)
        G = f.bmm(f.transpose(1, 2))
        return G
</code></pre>
<p>It happens when doing the multiplication.</p>
<p>I can think only of two options. First one is to remove NaNs and Infs with<br>
<code>        G = torch.nan_to_num(G, nan=0.0, posinf=0.0, neginf=0.0)</code></p>
<p>or cast the inputs to float32 before estimation the loss.</p>
<p>Is there any other solution to fix this issue?</p>
<p>Here is the full code of the loss function</p>
<pre><code class="lang-auto">class StyleLoss(_LossBase):
    def __init__(self, weight = 1.0):
        super(StyleLoss, self).__init__(weight)
        self.add_module('vgg', VGG19())
        self.criterion = torch.nn.L1Loss()

    def getGramMatrix(self, x):
        b, ch, h, w = x.size()
        f = x.view(b, ch, w * h)
        G = f.bmm(f.transpose(1, 2))
        return G

    def __call__(self, x, y):
        x_vgg, y_vgg = self.vgg(x), self.vgg(y)

        style_loss = 0.0
        style_loss += self.criterion(self.getGramMatrix(x_vgg['relu2_2']), self.getGramMatrix(y_vgg['relu2_2']))
        style_loss += self.criterion(self.getGramMatrix(x_vgg['relu3_4']), self.getGramMatrix(y_vgg['relu3_4']))
        style_loss += self.criterion(self.getGramMatrix(x_vgg['relu4_4']), self.getGramMatrix(y_vgg['relu4_4']))
        style_loss += self.criterion(self.getGramMatrix(x_vgg['relu5_2']), self.getGramMatrix(y_vgg['relu5_2']))

        return style_loss * self.weight
</code></pre>
<p>Thank you.</p>

J_Johnson: <p>Try this:</p>
<pre><code class="lang-auto">def remove_inf_nan(x):
    x[x!=x]=0
    x[~torch.isfinite(x)]=0
    return x
</code></pre>
<p>Another option is to clamp your tensors before multiplication. This would probably be a better solution than setting the values to zero.</p>
<pre><code class="lang-auto">
x = torch.clamp(x, max=1.0e2, min=-1.0e2)

</code></pre>
<p>Or if you plan to use for gradient descent:</p>
<pre><code class="lang-auto">
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0e2)

</code></pre>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html</a></p>
Framework for storing/ loading mobile models

mmisiur: <p>Hi!<br>
Do you know if there is any framework for storing and loading models to mobile device? I’m thinking about functionality similar to TensorFlow Lite + Flutter + Firebase:</p>
<ul>
<li>you can store your ML models on the server side</li>
<li>you can load it and update it to mobile device</li>
</ul>
<p>Have you heard about same thing for PyTorch?</p>
Torch::cuda::is_available() return 0

Larry_Li: <p>I compiled pytorch from source code. And I got libc10.so, libc10_cuda.so, libtorch.so, libtorch_gpu.so, libtorch_cuda.so.</p>
<p>When using torch::cuda::is_available() api to check cuda, it returned false. I confirmed that 4 GPUs are avalaible on the machine  and torch.cuda.is_available() return False in python.</p>
<p>Please help to tell me how to fix it. Thx!</p>

khushi-411: <p>Hi, I think the CUDA path is not set properly. You can fix this using these commands:</p>
<pre><code class="lang-python">export PATH=/usr/local/cuda-11.3/bin/:$PATH
</code></pre>
<p>and,</p>
<pre><code class="lang-python">export LD_LIBRARY_PATH=/usr/local/cuda-11.3/lib64:$LD_LIBRARY_PATH
</code></pre>
<p>PS: check your CUDA path, update the commands and then use it. Thanks!</p>

Larry_Li: <p>These Env Vars already added.</p>
<p>(base) root@b92d948aa67e:/workspace/code/example# echo $PATH<br>
/root/.vscode-server/bin/6261075646f055b99068d3688932416f2346dd3b/bin/remote-cli:/root/anaconda3/bin:/root/anaconda3/condabin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/anaconda3/bin/conda:/root/anaconda3/bin/conda<br>
(base) root@b92d948aa67e:/workspace/code/example# echo $LD_LIBRARY_PATH<br>
/usr/local/nvidia/lib:/usr/local/nvidia/lib64</p>
<p>source code:<br>
<span class="hashtag">#include</span> <br>
<span class="hashtag">#include</span> &lt;torch/torch.h&gt;<br>
<span class="hashtag">#include</span> &lt;torch/script.h&gt;<br>
<span class="hashtag">#include</span> </p>
<p>int main() {<br>
std::cout &lt;&lt; "CUDA: " &lt;&lt; torch::cuda::is_available() &lt;&lt; std::endl;<br>
std::cout &lt;&lt; "CUDNN: " &lt;&lt; torch::cuda::cudnn_is_available() &lt;&lt; std::endl;<br>
std::cout &lt;&lt; "GPU(s): " &lt;&lt; torch::cuda::device_count() &lt;&lt; std::endl;</p>
<pre><code>torch::cuda::is_available();

auto device = torch::kCPU;
if(torch::cuda::cudnn_is_available()) {
    device = torch::kCUDA;
}
int size = 4;

auto t = torch::tensor({1});

auto a = torch::randn({size, 9}).to(device);
auto b = torch::randn({size, 9}).to(device);
auto c = a + b;

std::cout &lt;&lt; c &lt;&lt; std::endl;

return 0;
</code></pre>
<p>}</p>
<p>CmakeLists.txt<br>
cmake_minimum_required(VERSION 3.0 FATAL_ERROR)</p>
<p>project(main)</p>
<p>set(CMAKE_INSTALL_RPATH “/workspace/code/mytorch/pytorch/build/lib/”)</p>
<p>set(CMAKE_CXX_FLAGS “${CMAKE_CXX_FLAGS} -std=c++14 -g”)</p>
<p>add_executable(main main.cpp)</p>
<p>target_include_directories(main PUBLIC</p>
<p>“/workspace/code/pytorch/torch/csrc/api/include”</p>
<p>“/workspace/code/pytorch/torch/include”</p>
<p>)</p>
<p>target_link_libraries(main</p>
<p>“/workspace/code/mytorch/pytorch/build/lib/libc10.so”</p>
<p>“/workspace/code/mytorch/pytorch/build/lib/libc10_cuda.so”</p>
<p>“/workspace/code/mytorch/pytorch/build/lib/libtorch.so”</p>
<p>“/workspace/code/mytorch/pytorch/build/lib/libtorch_cpu.so”</p>
<p>#“/workspace/code/mytorch/pytorch/build/lib/libtorch_cuda.so”</p>
<p>#“/lib/x86_64-linux-gnu/libm.so”</p>
<p>#“/usr/lib/x86_64-linux-gnu/libdl.so”</p>
<p>)</p>
<p>set_property(TARGET main PROPERTY CXX_STANDARD 14)</p>
How can I compute L2 norm of weight difference?

BobKim: <p>Hello everyone!</p>
<p>I want to check the weight of a specific layer while training ResNet.<br>
And i’ll compute the L2 norm of weight difference.<br>
The formulation is as follows.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/2/2201ad0ec50251b337b3574cba689d1edb5a8b34.png" alt="image" data-base62-sha1="4QPP8Co24kiyrYhJrEGLKEVYoxC" width="252" height="39"><br>
(where w is the t th weight in ResNet)</p>
<p>How can I calculate this formulation?</p>

azhanmohammed: <aside class="quote no-group" data-username="BobKim" data-post="1" data-topic="167058">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bobkim/48/52081_2.png" class="avatar"> BobKim:</div>
<blockquote>
<p>And i’ll compute the L2 norm of weight difference.</p>
</blockquote>
</aside>
<p>Here by weight difference you mean the difference in weights of two layers, or difference in weights of the same layer between two different training steps.</p>

BobKim: <aside class="quote no-group" data-username="azhanmohammed" data-post="2" data-topic="167058">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/azhanmohammed/48/49537_2.png" class="avatar"> azhanmohammed:</div>
<blockquote>
<p>difference in weights of the same layer between two different training steps.</p>
</blockquote>
</aside>
<p>I mean the difference of the same layer between two different training steps!<br>
difference between weight of t th step and weight of t - 1 th step.</p>

azhanmohammed: <p>So for this you first need to access the weights of a certain layer, this can be done using:</p>
<pre><code class="lang-python">import torch
from torchvision import models
import torch.nn as nn

model = models.resnet18()

for name, param in model.named_parameters():
  print(name)
  print(param)
</code></pre>
<p>The above script prints all the layer names and the weights. To store a certain weight layer you can use:</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
  if name=="layer1.0.conv1.weight": #replace the layer name with layer name of your choice
    weight1 = param.detach().cpu().numpy()
</code></pre>
<p>This stores the weight parameters as a numpy array in the variable weight1. Between two training steps, you can use the code snippet to get weights for the same layers, and then compute the weight difference, and L2 norm easily.</p>

BobKim: <p>Thank you very much. I’ll try this!</p>
PiNNs solving PDEs fail to converge to right solutions

robinho: <p>1.why would PiNNs solving PDEs converge to wrong solutions? That is when the loss seems to converge to 0, the solution is far from the analytic.<br>
2.why would PiNNs solving PDEs jump out a good solution towards a wrong one? That is the loss first seems to converge to 0 while the solution looks good but then increases accompanied by the solution becoming worse.</p>

robinho: <p>is there any answer?</p>
Pytorch and custom compilers

Andre_Chang: <p>I wanted to share a tutorial on compilers and optimizations with pytorch. Hope it helps</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://medium.com/@achang67/bring-your-own-compiler-optimization-in-pytorch-5ba8485ca459">
  <header class="source">
      <img src="https://miro.medium.com/1*m-R_BkNf1Qjr1YbyOIJY2w.png" class="site-icon" width="32" height="32">

      <a href="https://medium.com/@achang67/bring-your-own-compiler-optimization-in-pytorch-5ba8485ca459" target="_blank" rel="noopener nofollow ugc" title="06:51AM - 26 November 2022">Medium – 26 Nov 22</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:372/500;"><img src="https://miro.medium.com/max/474/1*fTDjWV4y_EImGd8sDnIOpQ.png" class="thumbnail" width="372" height="500"></div>

<h3><a href="https://medium.com/@achang67/bring-your-own-compiler-optimization-in-pytorch-5ba8485ca459" target="_blank" rel="noopener nofollow ugc">Bring Your Own Compiler/Optimization in Pytorch</a></h3>

  <p>TLDR: code example</p>

  <p>
    <span class="label1">Reading time: 6 min read</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Manually assign half precision on a specific linear layer

malioboro: <p>What is the best way to assign bias/weights of a specific layer to be half precision? This layer is a simple linear layer (no dropout or non-linear function).</p>
<p>I code this and I don’t know why it works even without gradient scaling. But if I move the <code>dtype=torch.float16</code> to the weight (<code>self.fc1w</code>), it yields an error:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    
    def __init__(self):
        super(Net, self).__init__()
        self.fc1w = nn.Parameter(torch.rand((100,100), requires_grad=True)/100)
        self.fc1b = nn.Parameter(torch.zeros(100, requires_grad=True, dtype=torch.float16))
        self.fc2 = nn.Linear(100,100)
        self.fc3 = nn.Linear(100,10)

    def forward(self, x):
        x = F.linear(x, self.fc1w, self.fc1b)
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x
    
model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
x = torch.rand(100)
y = torch.tensor([0]*9+[1], dtype=torch.float32)
for _ in range(20):
    out = model(x)
    loss = criterion(out, y)
    print(loss)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>In general, I want to know, is it possible to assign half-precision manually to a specific simple linear layer? If possible, what is the best way?</p>

ptrblck: <p>Your current code might work if e.g. type promotion is used internally.<br>
For a manual approach (assuming you don’t want to use the mixed-precisition training util. via <code>torch.amp</code>) you could either manually cast the tensors and parameters to the desired <code>dtype</code> in your <code>forward</code> method or you could also use the <code>autocast</code> context manager and specify the types in addition to casting as explained <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#autocast-and-custom-autograd-functions">here</a>.</p>

malioboro: <p>Thank you! I will check the reference.</p>
<p>Also I have tried to cast the tensors in the forward method, like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    
    def __init__(self):
        super(Net, self).__init__()
        self.fc1w = nn.Parameter(torch.rand((100,100), requires_grad=True)/100)
        self.fc1b = nn.Parameter(torch.zeros(100, requires_grad=True))
        self.fc2 = nn.Linear(100,100)
        self.fc3 = nn.Linear(100,10)

    def forward(self, x):
        self.fc1w.half()
        self.fc1b.half()
        x = F.linear(x, self.fc1w, self.fc1b)
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<p>the training works, but when I print the type of <code>self.fc1w</code> and <code>self.fc1b</code> it returns <code>torch.float32</code></p>

ptrblck: <p><code>half()</code> is not an inplace operation on tensors, so either assign the <code>flaot16</code> parameters to new tensors in the <code>forward</code> method or initialize them in the desired <code>dtype</code> in the <code>__init__</code> before wrapping them into <code>nn.Parameter</code>s.</p>
Vectorized sum different from looped sum

Ivano_Donadi: <p>Hi, I am using torch 1.7.1 and I noticed that vectorized sums are different from sums in a loop if the indices are repeated. For example:</p>
<pre><code class="lang-auto">import torch

indices = torch.LongTensor([0,1,2,1])
values = torch.FloatTensor([1,1,2,2])
result = torch.FloatTensor([0,0,0])

looped_result = torch.zeros_like(result)

for i in range(indices.shape[0]):
    looped_result[indices[i]] += values[i]

result[indices] += values

print('result',result)
print('looped result', looped_result)
</code></pre>
<p>results in:</p>
<pre><code class="lang-auto">&gt;&gt; result tensor([1., 2., 2.])
&gt;&gt; looped result tensor([1., 3., 2.])
</code></pre>
<p>As you can see the looped variable has the correct sums while the vectorized one doesn’t. Is it possible to avoid the loop and still get the correct result?</p>

ptrblck: <p>Since you are using duplicated indices, the behavior is UB and you should use <code>index_put_</code> instead with <code>accumulate=True</code>:</p>
<pre><code class="lang-python">result.index_put_((indices,),  values, accumulate=True)
</code></pre>

Ivano_Donadi: <p>Thank you very much! it works. I did not know of this method</p>
IndexError: index out of range in self for AlbertForTokenClassification

cantbelieveimshook: <p>When converting columns of csv files into torch token and label tensors, I got this error when attempting to run the AlbertForTokenClassification model on the token tensor.</p>
<p>Code:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/3/03129148fcef0cacd0fe27702530fdd2df7ed4b1.png" data-download-href="https://discuss.pytorch.org/uploads/default/03129148fcef0cacd0fe27702530fdd2df7ed4b1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/3/03129148fcef0cacd0fe27702530fdd2df7ed4b1.png" alt="image" data-base62-sha1="rbddj5h0HLqBVBwEY3MUbgp8nD" width="684" height="500" data-dominant-color="F1F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1085×793 47.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried rotating the input so it is [256, 34546] instead of [34536, 256], but that did not work, either. I ended up getting the error “RuntimeError: The expanded size of the tensor (34546) must match the existing size (256) at non-singleton dimension 1.  Target sizes: [256, 34546].  Tensor sizes: [1, 256].”</p>
<p>Any help would be appreciated, thanks</p>

cantbelieveimshook: <p>Update: I have figured out the issue. I had used a model with a different pretrained dataset to the default one, and I think that resulted in a model with a smaller vocabulary, which resulted in some of my train_token_ids tensor max values being larger than the max allowed value. If you’re experiencing this issue and scouring other threads hasn’t helped, maybe check your model and determine its vocab size if possible.</p>
Libtorch community

sroyyors: <p>hi,<br>
I am wondering if there is dedicated user community to develop and troubleshoot code for Libtorch (C++) code. Any pointers would be helpful.<br>
Thanks,<br>
Sushmita</p>

ptrblck: <p>This discussion board is used for all parts/utils. of PyTorch, so you can post your question here, too.</p>
How to take the value of center frame?

tanvir_14: <p>I have a tensor of shape (256, 7, 16, 2) where the 2nd dimension is the total number of frames (e.g: 7). How can I take the value of the center frame so that it becomes of shape (256, 1, 16, 2)?</p>

ptrblck: <p>I assume you are referring to the frame at index position 3 as the “center frame”. In this case, you could directly index and <code>unsqueeze</code> it or slice the tensor:</p>
<pre><code class="lang-python">x = torch.randn(256, 7, 16, 2)

y = x[:, 3].unsqueeze(1)
print(y.shape)
# torch.Size([256, 1, 16, 2])

z = x[:, 3:4]
print(z.shape)
# torch.Size([256, 1, 16, 2])
</code></pre>
KeyError when accessing activation from pytorch register_forward_hook

Debojyoti_Biswas: <p>I am trying to save the output from the <strong>fc1 layer</strong> of the model. Below is my model formulation:</p>
<pre><code class="lang-auto">model_ft = models.resnet50(pretrained=True)
num_ftrs = model_ft.fc.out_features
model_ft.fc1 = nn.ReLU(nn.Linear(num_ftrs, 512))
model_ft.fc2 = nn.Linear(num_ftrs, num_classes)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()
optimizer_ft = optim.SGD(model_ft.parameters(), lr=learning_rate, momentum=0.9)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
</code></pre>
<p>The below code shows the current activation access method:</p>
<pre><code class="lang-auto">activation = {}
def get_activation(name):
    def hook(model_ft, input, output):
        activation[name] = output.detach()
    return hook

model_ft.fc1.register_forward_hook(get_activation('fc1'))

x = torch.randn(12,3,224,224).to('cuda')
output = model_ft(x)
block4_output = activation['fc1']
</code></pre>
<p>The above method throws <strong>KeyError: ‘fc1’</strong>. Could anyone help me to fix this problem?</p>
<p>Thank You!!</p>

ptrblck: <p><code>model_ft.fc1</code> and <code>model_ft.fc2</code> are new attributes, which are never used in the original <code>forward</code> method.<br>
If you want to replace the original <code>model_ft.fc</code> with these two new layers, assign them to the same attribute and wrap them into an <code>nn.Sequential</code> container:</p>
<pre><code class="lang-python">model_ft = models.resnet50(pretrained=False)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Sequential(
    nn.ReLU(),
    nn.Linear(num_ftrs, 10)
)

activation = {}
def get_activation(name):
    def hook(model_ft, input, output):
        activation[name] = output.detach()
    return hook

model_ft.fc.register_forward_hook(get_activation('fc'))

x = torch.randn(12,3,224,224)
output = model_ft(x)
block4_output = activation['fc']
</code></pre>
<p>Your usage of <code>out_features</code> is also wrong since you are using it as the <code>in_features. Also, in case you want to use different "branches" in your model, you should override the </code>forward` method and create a custom model.</p>
Adding further transformation to preset_transformations

minmax: <p>I have loaded the resnet50’s transformations from the weights</p>
<pre><code class="lang-auto">resnet_50_weights = ResNet50_Weights.IMAGENET1K_V2
resnet_50_model = resnet50(weights=resnet_50_weights)
resnet_50_transformations = resnet_50_weights.transforms()
</code></pre>
<p>However I would like to add another transformation to this, say <code>transforms.RandomRotation(15, expand=True)</code> How would I add this. Using Compose threw an error</p>

ptrblck: <aside class="quote no-group" data-username="minmax" data-post="1" data-topic="166997">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/e5b9ba/48.png" class="avatar"> minmax:</div>
<blockquote>
<p>Using Compose threw an error</p>
</blockquote>
</aside>
<p>It seems to work for me:</p>
<pre data-code-wrap="python"><code class="lang-python">resnet_50_weights = models.resnet.ResNet50_Weights.IMAGENET1K_V2
resnet_50_transformations = resnet_50_weights.transforms()

trans = transforms.Compose([
    resnet_50_transformations,
    transforms.RandomRotation(15, expand=True)
])

x = torch.randn(3, 224, 224)

ref = resnet_50_transformations(x)
out = trans(x)

plt.imshow(ref.permute(1, 2, 0).numpy())
plt.imshow(out.permute(1, 2, 0).numpy())
</code></pre>
<p>Output:<br>
<code>ref</code>:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/504e13388547b1d15d037116ba42e37f5f2032f4.png" alt="image" data-base62-sha1="bsptj9OKQfcScB1eBhSureyrhL6" width="257" height="252"></p>
<p><code>out</code>:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db13596305bc2e5fc0edf4e4fef530bc2e966e13.png" alt="image" data-base62-sha1="vg21kmRUjM1W7xp1xTUOu1PtZ1V" width="262" height="252"></p>