akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Spliting data into imbalanced data

username: user3
cooked: <p>hi<br>
i would like to create a data that is imbalanced by classes. par example, i want 5 classes that are 30%<br>
of the data, and 5 other classes that are 70% of the data. and by using subset, it doesn’t work.<br>
i would like for some help, thanks.<br>
this is what i did -<br>
_,set1 = createDataset(trainset, np.array([0,2,4,6,8]), isTrain=True,batchSize=1, test=False)<br>
set1 = partialDataset(set1, num=1750,class_num=5)<br>
_,set2 = createDataset(trainset, np.array([1,3,5,7,9]), isTrain=True,batchSize=1, test=False)<br>
set2 = partialDataset(set2, num=750,class_num=5)<br>
trainset = set1.dataset + set2.dataset</p>
<p>(createDataset return the trainset with the classes that are in the array.<br>
partial dataset is th e following function</p>
<p>def partialDataset(trainset, num=0,class_num=10):<br>
indices = np.arange(len(trainset))<br>
train_indices, _ = train_test_split(indices, train_size=num * class_num, stratify=trainset.targets)</p>
<pre><code># Warp into Subsets and DataLoaders
train_dataset = torch.utils.data.Subset(trainset, train_indices)
return train_dataset
</code></pre>
<p>thanks a lot!<br>
Eilon</p>

username: ptrblck
cooked: <p>Since you are using <code>stratify=trainset.targets</code>, you should get balanced splits as is described in the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">docs</a> of <code>sklearn.model_selection.train_test_split</code>.<br>
If you want to manually create imbalanced splits, you could sample the desired amount of class indices from <code>trainset.targets</code> and create a new <code>Subset</code> using the new sample distribution.</p>
Does amp autocast cache fp16 copies of model parameters?

username: vadimkantorov
cooked: <p>I’m thinking of using autocast for speeding up processing of frozen backbone. If I use autocast context manager in forward, will it every time spend time on conversion of model weights?</p>
<p>How do I ensure it does conversion of model parameters fp32-&gt;fp16 exactly once?</p>
<p>My alternative is to manually convert frozen backbone’s parameters to fp16, and maybe not use any autocast at all, but I assume autocast is a better bet because it has special cases for ops that still must be run in fp32.</p>

username: ptrblck
cooked: <p>Yes, <code>autocast</code> uses caching internally unless you manually disable it and the checks can be found <a href="https://github.com/pytorch/pytorch/blob/240e8d5cc5a6913375cd8507cf7d22727334b953/aten/src/ATen/autocast_mode.cpp#L111-L133">here</a>.</p>
<aside class="quote no-group" data-username="vadimkantorov" data-post="1" data-topic="136168">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vadimkantorov/48/365_2.png" class="avatar"> vadimkantorov:</div>
<blockquote>
<p>How do I ensure it does conversion of model parameters fp32-&gt;fp16 exactly once?</p>
</blockquote>
</aside>
<p>It’ll be done automatically for you if the parameter is found in the cache (which won’t be the case if e.g. you’ve modified it).</p>
How to load intermediate layers of deep neural network in a different file?

username: Achint_Kumar
cooked: <p>I have saved parameters of a trained neural network(actually a variational autoencoder) using</p>
<blockquote>
<p>state = {<br>
‘epoch’: epoch,<br>
‘state_dict’: model.state_dict(),<br>
‘optimizer’: optimizer.state_dict()}<br>
torch.save(state, PATH)</p>
</blockquote>
<p>I would like to load the parameters and only use certain layers of my neural network(decoder of VAE) for other purposes. I am able to load the state using</p>
<blockquote>
<p>state = torch.load(PATH)</p>
</blockquote>
<p>But I don’t know how to access the decoder layer of my neural network without defining the full neural network in the new file.</p>
Installed CUDA after pytorch

username: pavlidic
cooked: <p>Hello, a few days ago i decided to follow a tutorial that was using pytorch. I installed pytorch with the cuda command even though i didnt previously have CUDA installed. Today i got to the GPU part, and it seemed to go awfully slow, even though in the video it went much faster than me. I have a gtx 1070 and it was a simple MNIST example with 1 hidden layer. I did something similar on octave before and it went MUCH faster. So, i thought that probably not having cuda was the problem, i googled and i came to this: <a href="https://discuss.pytorch.org/t/is-it-required-to-set-up-cuda-on-pc-before-installing-cuda-enabled-pytorch/60181" class="inline-onebox">Is it required to set-up CUDA on PC before installing CUDA enabled pytorch?</a><br>
I then installed cuda 11.5 on my machine. I tried the pytorch code again and it still had the same speed. Am i doing something wrong here?</p>

username: ptrblck
cooked: <p>If you are already pushing the input tensor as well as the model to the GPU via <code>.cuda()</code> or <code>.to('cuda')</code> then your GPU should already be used the the util. in <code>nvidia-smi</code> as well as the memory usage would also verify it. To further isolate your bottleneck, profile your code and try to narrow down which part of your code is the slowest.</p>
Pass Stream as arg to torch.jit.script function

username: fumihwh
cooked: <p>I can’t pass cuda stream as an arg to torch.jit.script function.<br>
Following is code snippet.<br>
Any help?</p>
<pre><code class="lang-auto">  @torch.jit.script
  def f(d, s: torch.Stream, i: int):
    with torch.cuda.stream(s):
        #do sth

  @torch.jit.script
  def r(tensors: List[Tensor], num_gpus: int):
    futures: List[torch.jit.Future[Tensor]] = []
    streams = [
        torch.cuda.Stream(torch.device(f"cuda:{i}"))
        for i in range(num_gpus)
    ]
    for i in range(num_gpus):
      futures.append(torch.jit.fork(f, tensors[i], streams[i], i))
    results = [torch.jit.wait(fut) for fut in futures]
    return results
</code></pre>
<p>Got RuntimeError:</p>
<pre><code class="lang-auto">f(Tensor d, Stream s, int i) -&gt; (Tensor):
Expected a value of type 'Stream' for argument 's' but instead found type '__torch__.torch.classes.cuda.Stream (of Python compilation unit at: 0)'.
</code></pre>
<p>I also tried typing <code>torch.cuda.Stream</code>, and got:</p>
<pre><code class="lang-auto">'__torch__.torch.cuda.streams.Stream (of Python compilation unit at: 0x3e98e70)' object has no attribute or method 'cuda_stream'. Did you forget to initialize an attribute in __init__()?:
  File "/data0/fumi/.pyenv/versions/venv374/lib/python3.7/site-packages/torch/cuda/streams.py", line 98
    @property
    def _as_parameter_(self):
        return ctypes.c_void_p(self.cuda_stream)
                               ~~~~~~~~~~~~~~~~ &lt;--- HERE
'Stream.___as_parameter__getter' is being compiled since it was called from '__torch__.torch.cuda.streams.Stream'
</code></pre>
Strange function type named "boolean_dispatch.<locals>.fn" for torch.nn.functional.max_pool2d

username: shengchunnan
cooked: <h1>
<a name="strange-function-type-named-boolean_dispatchlocalsfn-for-torchnnfunctionalmax_pool2d-1" class="anchor" href="#strange-function-type-named-boolean_dispatchlocalsfn-for-torchnnfunctionalmax_pool2d-1"></a>Strange function type named “boolean_dispatch..fn” for torch.nn.functional.max_pool2d</h1>
<p>When I was checking function references of <strong><code>functional.relu</code></strong>, <strong><code>functional.avg_pool2d</code></strong> or <strong><code>functional.max_pool2d</code></strong>, I found description of <strong>max_pool2d</strong> is a bit different from other function types. This name <strong>‘boolean_dispatch’</strong> may be a bit confusing.</p>
<p>See the python command lines as follows:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; func1 = torch.nn.functional.max_pool2d
&gt;&gt;&gt; func2 = torch.nn.functional.avg_pool2d
&gt;&gt;&gt; func3 = torch.nn.functional.relu
&gt;&gt;&gt; func1
&lt;function boolean_dispatch.&lt;locals&gt;.fn at 0x7f3e99f928c0&gt;
&gt;&gt;&gt; func2
&lt;built-in function avg_pool2d&gt;
&gt;&gt;&gt; func3
&lt;function relu at 0x7f3e99f97ef0&gt;
</code></pre>
<h2>
<a name="python-version-37-2" class="anchor" href="#python-version-37-2"></a>python version: 3.7</h2>
<h2>
<a name="torch-version-110-3" class="anchor" href="#torch-version-110-3"></a>torch version: 1.10</h2>
How can I switch between two batch sizes for train_loader during training?

username: JimW
cooked: <p>I hope to switch between two batch sizes for training data during data, for different epochs.<br>
For example,</p>
<pre><code class="lang-auto">if epoch%2==0:    train with batch_size 4 (i.e. this is a for loop to grab each batch of size 4)
else:  train with batch_size 8 (i.e. this is a for loop to grab each batch of size 8)
</code></pre>
<p>I tried to define two train_loaders at the very beginning like this:</p>
<pre><code class="lang-auto">train_small = torch.util.data.Dataloader(.., batch_size=4 ,,,)
train_large = torch.util.data.Dataloader(.., batch_size=8 ,,,)
if epoch%2==0:
    train_loader =  train_small
else 
    train_loader = train_large
for input, target in tqdm(train_loader ...):
    do some training.
</code></pre>
<p>But got the “out of cuda memory” error.  However,  if I just use “train_large” for all epochs, it is ok.<br>
Is this the correct approach to switch between 2 batch sizes during training?</p>
<p>The reason I am doing this is, different training epoch consumes different size of memory, so I want to use smaller batch for even epochs but a larger batch size for odd epochs. Could someone provide some guidance on how to achieve that?  Thanks.</p>

username: OrielBanne
cooked: <p>Hi</p>
<p>You tell very little about your model architecture and data structure. However, in even epoch cycles, you could copy the part of the batch you want and delete from memory the other parts before you start any computation. This way- when you start computing for the batch, memory is of the size you want. It will not cover the whole dataset but will indicate if a smaller batch helps even epochs</p>

username: JimW
cooked: <p>Thanks for the response. Can you provide some exemplary code to achieve “in even epoch cycles, you could copy the part of the batch you want and delete from memory the other parts before you start any computation”?  I never manually handled memory before.</p>

username: OrielBanne
cooked: <p>so what I think you should do:<br>
if epoch%2 == 0:<br>
data_small = data[0:new_size]<br>
del data</p>
<p>give it a try and let me know what happens</p>
<p>I think that in your previous attempt you created a confusion between the data loaders because they are data loaders from the same dataset. in the method suggested here - you use only the large data loader, but use only a portion of the batch each even epoch. So if the suggestion above works, it will skip part of the data. This can be fixed later - maybe use only the small batch data loader taking 4 batches every ‘odd’ epoch. We can see after you try the above</p>
PyTorch Implementation of CTC Loss

username: ThomasGk
cooked: <p>Hi everyone. I am studying about CTC from this wonderful article <a href="https://distill.pub/2017/ctc/" rel="noopener nofollow ugc">Sequence Modeling with CTC </a>, and I’d like to ask something regarding PyTorch’s way of computing CTC loss.</p>
<p>In the article it says that to compute the loss, a typical way would be to sum the probability of all valid alignments. However, this might become too costly, as the valid alignments may be too many.</p>
<p>It then goes on to explain an alternative, less costly way to compute the loss, which approximates the above mentioned calculation.</p>
<p>Reading PyTorch’s <a href="https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html#torch.nn.CTCLoss" rel="noopener nofollow ugc">documentation </a>, I think that it computes the loss using the first way, that is summing over <em>all</em> possible alignments. Is my understanding correct, and if yes, is there a way to use the second approach described in the article, which uses a dynamic programming algorithm?</p>
<p>Any help will be greatly appreciated, thanks.</p>
Loading External Data with Torchtext

username: jdbeck66
cooked: <p>I have two Arrays - one contains letter sequences and the other corresponding scores. After days looking through the documentation, examples and tutorials, there doesn’t seem to be any straightforward explanation on how to use the most current version of torchtext to load this data. Tons of examples exist on how to load the datasets, but not for external data?</p>
<p>I can save my data as a csv and then use legacy to use the Field and LabelField tools, but I can’t believe that using deprecated functions is the best approach. Sorry for being frustrated, but am I missing some documentation where this is laid out?</p>
<p>I’ll take any hints or suggestions.</p>
Jit: Tried to access nonexistent attribute or method 'forward' of type 'Tensor'

username: eagomez
cooked: <p>I am new to TorchScript and I was trying to use torch.jit.script for one of my models and when I do so, I get the following:</p>
<pre><code class="lang-auto">RuntimeError: 
Tried to access nonexistent attribute or method 'forward' of type 'Tensor'.:
  File "custom_layers.py", line 32
    def forward(self, x):
        result = super().forward(x)
                 ~~~~~~~~~~~~~ &lt;--- HERE
    
        if self._padding != 0:
</code></pre>
<p>The method that is causing issues is the <code>forward()</code> method of the <code>nn.Conv1d</code> instance that is called from the derived class as follows:</p>
<pre><code class="lang-auto">class CausalConv1d(nn.Conv1d):
    def __init__(self,
                 in_channels,
                 out_channels,
                 kernel_size=1,
                 stride=1,
                 dilation=1,
                 groups=1,
                 bias=True):
        self._padding = (kernel_size - 1) * dilation

        super().__init__(
                 in_channels,
                 out_channels,
                 kernel_size=kernel_size,
                 stride=stride,
                 padding=self._padding,
                 dilation=dilation,
                 groups=groups,
                 bias=bias
                )

    def forward(self, x):
        result = super().forward(x)

        if self._padding != 0:
            return result[:, :, : - self._padding]

        return result
</code></pre>
<p>If I hint the input type x as suggested in <a href="https://discuss.pytorch.org/t/error-with-my-custom-concat-class-with-torchscript/112440/2">this topic</a> I still get an error.</p>
<p>Thanks for your help!</p>

username: Prashant
cooked: <p>Hi I am noticing the same problem as you. Any chance you fixed it ?</p>

username: eagomez
cooked: <p>I tried a few changes with no luck and had to move to something else so it is still unsolved</p>

username: Prashant
cooked: <p>Looks like inheritance is not supported <a href="https://github.com/pytorch/pytorch/issues/42885#issuecomment-678385108" class="inline-onebox" rel="noopener nofollow ugc">Unable to call `super` method with TorchScript · Issue #42885 · pytorch/pytorch · GitHub</a></p>

username: eagomez
cooked: <p>Thanks for sharing it! That explains the issue…so apparently only tracing the model is an option in this case since it will only care about operations, but may not work too depending on the model’s logic.</p>

username: Prashant
cooked: <p>My plan is make the subclass own an instance of the parent class and muck around with that instance’s state / call its forward method</p>

username: eagomez
cooked: <p><a class="mention" href="/u/prashant">@Prashant</a> did this workaround work for you? It seems like as of today the issue still remains</p>

username: Prashant
cooked: <p>Yes my workaround worked out for me.</p>

username: eagomez
cooked: <p>Thanks!</p>
<p>I tried it and worked fine, although since my final code could be traced, I ended up using jit.trace for this section and jit.script for the rest.</p>
Pytorch c++ Frontend and torchscript model

username: DSL
cooked: <p>Hello together,</p>
<p>I found this blog for pytorch training in c++ (<a href="https://krshrimali.github.io/categories/pytorch/" class="inline-onebox" rel="noopener nofollow ugc">pytorch :: Kushashwa Ravi Shrimali (Kush) — Hello Friend NG HZ Theme</a>). I see how he trained the neuronal network but i have some understanding problems. I see there will be used “torch::load”. Is there a way to load a torchscript model like this?</p>
<pre><code class="lang-auto">module_ = torch::jit::load(model_file_path);
module_.train();
module_.to(torch::kCUDA);
torch::optim::Adam optimizer(module_.parameters(),torch::optim::AdamOptions(2e-4).betas(std::make_tuple (0.5, 0.5)));
</code></pre>
<p>when i try to compile this i get this error:</p>
<pre><code class="lang-auto">torchscript.cc:22:27: error: no matching function for call to ‘torch::optim::Adam::Adam(torch::jit::parameter_list)’
   22 |       module_.parameters());

</code></pre>
<p>Do i something wrong?</p>
How to estimate size of serialized tensor (in C++)?

username: fulltopic
cooked: <p>Hi.</p>
<p>If I write a tensor into an ostringstream named testStream, I can get the size(in bytes) of the serialized tensor by testStream.str().length(). How can I get this value without actually executing any action such as “serializing into ostringstream”?</p>
<p>Thank you.</p>
Now I just want to train the two W of linear layer, w (w1, w2, w3,...... W100), now just want to train W1, w2, how to choose these two weights to train?

username: xiaojin
cooked: <p>Now I just want to train the two W of linear layer, w (w1, w2, w3,…  W100), now just want to train W1, w2, how to choose these two weights to train?</p>
AdderNet Backprop Understanding

username: Fabfi
cooked: <p>Iam trying to understand the implementation of AdderNet regarding this: <a href="https://github.com/huawei-noah/AdderNet/blob/master/adder.py" rel="noopener nofollow ugc">https://github.com/huawei-noah/AdderNet/blob/master/adder.py</a></p>
<p>especially the backprop, can someone maybe elaborate on the X_col and W_col, and how this is in relation to the AdderNet paper:  <a href="https://arxiv.org/pdf/1912.13200.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1912.13200.pdf</a></p>
<blockquote>
<pre><code>@staticmethod
def backward(ctx,grad_output):
    W_col,X_col = ctx.saved_tensors
    grad_W_col = ((X_col.unsqueeze(0)-W_col.unsqueeze(2))*grad_output.unsqueeze(1)).sum(2)
    grad_W_col = grad_W_col/grad_W_col.norm(p=2).clamp(min=1e- 
    12)*math.sqrt(W_col.size(1)*W_col.size(0))/5
    grad_X_col = (-(X_col.unsqueeze(0)- 
    W_col.unsqueeze(2)).clamp(-1,1)*grad_output.unsqueeze(1)).sum(0)
    
    return grad_W_col, grad_X_col
</code></pre>
</blockquote>
<p>especially how the hardthan function is executed here is something I dont really understand (which is stated in 3.2 of the Paper), and how the backprop has something to do with the X_col, since how Iam understanding it X_col is just the input columns</p>
Using a target size that is different to the input size is deprecated. Please ensure they have the same size

username: Tornike
cooked: <p>Hi, Doing Background Removal With U2net.<br>
The error i’m getting is :</p>
<p><code>Using a target size (torch.Size([5, 3, 320, 320])) that is different to the input size (torch.Size([5, 1, 320, 320])) is deprecated. Please ensure they have the same size.</code></p>
<p>As you may know, This is a Segmentation Type task. so, it has RGB Image as X and Ground Truth Grayscale Image as Y.</p>
<p>The error is when i apply this model output to Loss Function. I’m Using BCE Loss.</p>
<p>What should i do? i don’t want to debug that error by just converting input image to Grayscale.<br>
is there any other way ?</p>
<p>Thanks in advance.</p>

username: Programmer-RD-AI
cooked: <p>your input size is <code>(5,1,320,320)</code> the model is expecting to have <code>(5,3,320,320)</code>.</p>
<p>If you don’t want to pass RGB images</p>
<p>you can change<br>
<code>Conv2d(3,-,(-,-))</code> to <code>Conv2d(1,-,(-,-))</code></p>
<p>if you are using a linear layer you can change</p>
<p><code>Linear(3*320*320,-)</code> to <code>Linear(1*320*320)</code></p>
<p>Hope this helps</p>
<p>With best regards,<br>
Ranuga</p>

username: Programmer-RD-AI
cooked: <p>And also I have seen this more in BCELoss (In my personal Projects)</p>
<p>I am not sure if it has anything to do with BCELoss.</p>

username: Tornike
cooked: <p>Hi,</p>
<p>Thanks for your reply.</p>
<p>What you have said make sense in spite of the model but the problem was in processing the image Data. I could have explained that extensively.</p>
<p>Again, Thanks for the reply.</p>
'NoneType' object has no attribute 'parameters'

username: mkarn3yru
cooked: <p>Below is the training function I am writing, yet, there is this error: ‘NoneType’ object has no attribute ‘parameters’. I don’t really know how to solve it</p>
<pre><code class="lang-auto">def train(net, train_loader, test_loader, logger):    
    # TODO: Define the SGD optimizer here. Use hyper-parameters from cfg
    
    optimizer = optim.SGD(net.parameters(),lr=cfg['lr'],momentum=cfg['momentum'],weight_decay=cfg['weight_decay'], nesterov=cfg['nesterov'])
    # TODO: Define the criterion (Objective Function) that you will be using
    criterion = nn.CrossEntropyLoss()
</code></pre>

username: ptrblck
cooked: <p>It seems you are passing the <code>net</code> argument as a <code>None</code> so check where <code>net</code> is created and that it holds a valid <code>nn.Module</code> object.<br>
My guess would be that you might be using a convenient function to create the model and might have forgotten to return the model itself. E.g. this would break with the same error:</p>
<pre><code class="lang-python">def create_model():
    net = nn.Linear(1, 1)
    # remove the '#' and it'll work
    # return net  

net = create_model()
net.parameters()
&gt; AttributeError: 'NoneType' object has no attribute 'parameters'
</code></pre>
Developing new backend for custom hardware in Glow

username: jr.r
cooked: <p>Hi,<br>
I am trying to develop a backend for custom hardware like Nvdla in glow. I created an external backend and try to consume Glow IR for code generation.<br>
Here, I am not sure about, the way to perform IR instruction processing in codegen, usage of the symbol table, what data is required to keep from compile-time to runtime, and where exactly I can add/update my runtime module and address mapping.</p>
<p>Please provide information or any useful material to go through</p>
Training pretrained model(multi label classification) for SSD Base weight model

username: pjg205
cooked: <p>Dear all,</p>
<p>I have custom datasets, which was made in a synthetic way matching the image format with sensor data.</p>
<p>Then I am trying to change the pre-trained model. So I utilized the applied Multi-label classification using modified vgg16.</p>
<p>However, training loss is iterable when starting. What should I check to solve it?</p>
<p>Any suggestions? Thank you in advance.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbc9b28cdeadab0e43af43e86a1cb5f1dac144a2.png" alt="image" data-base62-sha1="t4N3NIox6V0kU9R90oN39yaoAb8" width="624" height="32"></p>
Dynamic/variable convolution kernel/filter for image smoothing/blurring

username: Bruce_Muller
cooked: <p>I would like to smooth/blur each part of an image to an extent dependent on a separate input tensor specific for that image.<br>
So, depending on the part of the given input tensor I can smooth the corresponding part of the image.</p>
<p>Is there a way to do this using Conv2d/Unfold/Fold functional calls? Or another way?</p>
<p>It seems like I need a conv function with dynamic kernel weights?</p>

username: ptrblck
cooked: <p>Could you explain how the separate input tensor look like and how a “part of the given input tensor” would be used to smooth the image?</p>
<p>Generally you can use the functional API via <code>F.conv2d</code> to apply a kernel on an image tensor, but I’m unsure about the aforementioned aspects.</p>

username: Bruce_Muller
cooked: <p>I think I figured it out but I need to make it less expensive! Thanks!</p>

username: shapovalov
cooked: <p>Sorry for reviving this old topic but I think this can be a relatively common use case that can help implementing bilateral filter or even learnable input-conditioned filter.</p>
<p>More specific proposal may be to extend <code>F.conv2d</code> so that <code>weight</code> parameter can be a 6D tensor with dimensions (out_channels, in_channels, kH, kW, iH, iW). So the current conv2d producing for the pixel (i, j)</p>
<pre><code class="lang-auto">output[i,j] = sum(crop(input, i, j, kH, kW) * weight)
</code></pre>
<p>can be generalised to</p>
<pre><code class="lang-auto">output[i,j] = sum(crop(input, i, j, kH, kW) * weight[..., i, j])
</code></pre>
<p>It is probably difficult to implement it now in a vectorised way without resorting to writing new CUDA kernels.<br>
Another useful API may be generic filter similar to <a href="https://docs.scipy.org/doc/scipy/reference/reference/generated/scipy.ndimage.generic_filter.html#scipy.ndimage.generic_filter" rel="noopener nofollow ugc">the one in scipy</a>. It allows to avoid re-implementing boundary checks at least.</p>
Using cuda ipc memory handles

username: kurt_fisher
cooked: <p>Hi everyone! I try to write custom handle reader in pytorch based on <a href="https://discuss.pytorch.org/t/using-cuda-ipc-memory-handles-in-pytorch/17548/4">this</a> topic.<br>
I read topic with torch.multiprocessing.reductions.rebuild_cuda_tensor and its work Ok, but if I stop reading this handle GPU memory leaked.</p>
<p>Can someone introduce me how its work and how overwrite handle for solve memory leaks?<br>
And second? can I read rebuild_cuda_tensor with multiple processes?<br>
PS. I use independent processes and can not transfer Queue here.</p>
Nan training and testing loss

username: ashcher51
cooked: <p>When trying to use a LSTM model for regression, I find that I am getting NaN values when I print out training and testing loss. The DataFrame I pass into the model has no NaN values, so I believe it is an issue with my model or my training/testing loop functions. Any help in this regard would be greatly appreciated.</p>
<p>Model Class:</p>
<pre><code class="lang-auto">class PKLSTM(nn.Module):
    def __init__(self, input_size, hidden_units, num_layers):
        super(PKLSTM, self).__init__()
        self.input_size = input_size
        self.hidden_units = hidden_units
        self.num_layers = num_layers 

        self.lstm = nn.LSTM(
            input_size=input_size, 
            hidden_size=hidden_units,
            batch_first=True,
            num_layers=num_layers
        )

        self.linear = nn.Linear(hidden_units, 1)
    
    def forward(self, x):
        batch_size = x.shape[0]
        h0 = torch.zeros(self.num_layers, batch_size, self.hidden_units).requires_grad_()
        c0 = torch.zeros(self.num_layers, batch_size, self.hidden_units).requires_grad_()

        _, (hn, _) = self.lstm(x, (h0, c0))
        out = self.linear(hn[0]).flatten()

        return out
</code></pre>
<p>Training and testing loop:</p>
<pre><code class="lang-auto">def train_model(data_loader, model, loss_function, optimizer):
    num_batches = len(data_loader)
    total_loss = 0
    model.train()

    for X, y in data_loader:
        output = model(X)
        loss = loss_function(output, y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        total_loss += loss.item()

    avg_loss = total_loss / num_batches
    print(f"Train loss: {avg_loss}")

def test_model(data_loader, model, loss_function):

    num_batches = len(data_loader)
    # print('Num of batches', num_batches)
    total_loss = 0

    model.eval()
    with torch.no_grad():
        for X, y in data_loader:
            # print('input: ', X)
            output = model(X)
            # print('output: ', output)
            total_loss += loss_function(output, y).item()

    avg_loss = total_loss / num_batches
    print(f"Test loss: {avg_loss}")

</code></pre>

username: JuanFMontesinos
cooked: <p>if your learning rate is too big model will diverge which eventually would lead to an Inf/NaN.<br>
Aaand the second option is there is a NaN in the input data. Which may not appear in the data frame but after preprocessing?</p>

username: ashcher51
cooked: <p>Thank you. Some of my code in my dataset class was generating NaNs. Appreciate all the help!</p>
Equivalent of `register_buffer` with torch.optim.Optimizer

username: AlphaBetaGamma96
cooked: <p>Hi  All,</p>
<p>I was wondering if there’s an equivalent to<code>register_buffer</code> for custom optimizers? I’m currently building a custom optimizer and I want to be able to save some floats (which are used within the optimization process)</p>
<p>Some methods within the class don’t use <code>self.param_groups</code> so is there a way to save attributes within the <code>torch.optim.optimzier</code> such that if I were to use <code>load_state_dict</code> / <code>save_state_dict</code> it would save those float values?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>You should be able to add your custom buffers to the corresponding <code>param_group</code> via:</p>
<pre><code class="lang-python"># setup
lin = nn.Linear(1, 1)
optimizer = torch.optim.Adam(lin.parameters(), lr=1e-3)

# dummy training
out = lin(torch.randn(1, 1)).backward()
optimizer.step()

# check param_group
print(optimizer.param_groups[0])

# register custom tensor
optimizer.param_groups[0]['my_buffer'] = torch.tensor(100)
print(optimizer.state_dict())

# save state_dict and restore
sd = optimizer.state_dict()
optimizer = torch.optim.Adam(lin.parameters(), lr=1e-3)
print(optimizer.state_dict())
optimizer.load_state_dict(sd)
print(optimizer.state_dict())

# dummy training
out = lin(torch.randn(1, 1)).backward()
optimizer.step()
</code></pre>
Setting up PyTorch/XLA on a local TPU

username: Maxwell175
cooked: <p>Hello,</p>
<p>I have a physical, local TPU connected to my computer. I am looking to use it for PyTorch models, but I can’t seem to understand how to install the torch_xla module (<a href="https://pytorch.org/xla/release/1.9/index.html" rel="noopener nofollow ugc">which i seem to need</a>) if I’m not using a VM or cloud TPU.</p>
<p>Thank you.</p>
Warnings.warn(.. prints after epoch

username: Junyong_Lee
cooked: <p>Hi, when I train my network (<code>self.network</code>) with the following scheme, pytorch prints out a warning <code>warning.warn(...The default behaviour for interpolate/upsample with float scale_factor changed in 1.6.0...</code> at the beginning of the training loop, which is totally OK with me.</p>
<p>I have two versions of <code>self.network</code>.<br>
For one version of <code>self.network</code>, the warning prints only at the very begging of the training loop.</p>
<p>But strangely, for the other version of <code>self.network</code>, the warning is additionally printed out for every beginning of the validation loop during the training.</p>
<p>The differences between the two networks are the network architecture and additional input that I give for the second version of <code>self.network</code>.</p>
<p>The main reason I’m asking this question is that the second version undergoes out of GPU memory problem during the training. (If I just do the validation, the two networks take almost the same memory, and there is no out of GPU memory problem.) I strongly believe it is somehow related to the reason why the warning prints out for every validation loop.</p>
<p>It would be very helpful if you guys share any clues on this problem.<br>
Thank you!</p>
<pre><code class="lang-auto">self.state = ['train', 'valid']

for epoch in self.epoch_range:
    for state in self.states:
        if state == 'train':
            self.network.train()
            self.iteration(epoch, state)
        elif state == 'valid':
            self.network.eval()
            with torch.no_grad():
                self.iteration(epoch, state)

def iteration(self, epoch, state):
    is_train = True if state == 'train' else False
    data_loader = self.data_loader_train if is_train else self.data_loader_eval

    for inputs in data_loader:
        result = self.network(inputs, is_train)

</code></pre>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation (Feed Forward RNN)

username: dennis.trujillo
cooked: <p>I’m getting the following error for the code listed below, any suggestions are appreciated!</p>
<p>File “scripts/rnn/rnn.py”, line 132, in <br>
train_load_save_model(rnn,path,device)<br>
File “scripts/rnn/rnn.py”, line 104, in train_load_save_model<br>
train(model_obj,device)<br>
File “scripts/rnn/rnn.py”, line 86, in train<br>
outputs,hidden = model_inp(inputs,hidden)<br>
File “/Users/dennis/Library/Python/3.7/lib/python/site-packages/torch/nn/modules/module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “scripts/rnn/rnn.py”, line 59, in forward<br>
<strong>hidden   = self.lin1(combined.view((batch_size,size_in+size_hidden))).view((batch_size,size_hidden))</strong><br>
File “/Users/dennis/Library/Python/3.7/lib/python/site-packages/torch/nn/modules/module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/Users/dennis/Library/Python/3.7/lib/python/site-packages/torch/nn/modules/linear.py”, line 96, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/Users/dennis/Library/Python/3.7/lib/python/site-packages/torch/nn/functional.py”, line 1847, in linear<br>
return torch._C._nn.linear(input, weight, bias)<br>
(Triggered internally at  …/torch/csrc/autograd/python_anomaly_mode.cpp:104.)<br>
allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag<br>
Traceback (most recent call last):<br>
File “scripts/rnn/rnn.py”, line 132, in <br>
train_load_save_model(rnn,path,device)<br>
File “scripts/rnn/rnn.py”, line 104, in train_load_save_model<br>
train(model_obj,device)<br>
File “scripts/rnn/rnn.py”, line 91, in train<br>
loss.backward()<br>
File “/Users/dennis/Library/Python/3.7/lib/python/site-packages/torch/_tensor.py”, line 255, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/Users/dennis/Library/Python/3.7/lib/python/site-packages/torch/autograd/<strong>init</strong>.py”, line 149, in backward<br>
allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [256, 128]]</p>
<pre><code class="lang-auto">class RNNModel(nn.Module):
    def __init__(self,size_in,size_hidden,size_out):
        super(RNNModel,self).__init__()
        self.lin1 = nn.Linear(size_in+size_hidden, size_in)
        self.lin2 = nn.Linear(size_in+size_hidden, size_out)


    def forward(self, input, hidden):
        combined = torch.cat((input,hidden),1).view((batch_size,size_in+size_hidden)) 
        hidden   = self.lin1(combined).view((batch_size,size_hidden))
        x        = self.lin2(combined)
        return x, hidden 

    def initHidden(self): 
        return torch.zeros(1,size_hidden)
</code></pre>
<p>My thoughts are the line in bold is to blame but I don’t know how to remedy the issue.</p>