akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Managing shared models in pytorch

username: Sam4
cooked: <p>I don’t know what category to put this in, so I hope that people read the uncategorized bucket.</p>
<p>I’m using AllenNLP to do some NLP work. AllenNLP uses pytorch (version 1.9.0 is what I have). I’m using Python 3.7.9 on macOS 11.6.1 (latest version of Big Sur). I’m trying to use multiprocessing (and yes, I’m using <code>torch.multiprocessing</code>) to distribute some decoding work among multiple subprocesses, using <code>torch.multiprocessing.Pool</code>. The problem I’m having is that, no matter what I try, the actual processing time (which I’m measuring in the children, after the processes have started up, etc.) is increasing with the number of subprocesses.</p>
<p>I know that pytorch makes use of shared memory, and the shared memory doesn’t seem to have a notion of read-only vs. read-write, and I’m assuming that what’s happening is that the model is being locked so that only one subprocess at a time is able to read from it. But I can’t exactly tell, because I can’t figure out where the sharing is happening. What’s more bizarre to me is that when I load the model in the subprocesses, the processing time <em>still</em> scales with the number of subprocesses. I’ve convinced myself that I’m truly loading the models independently in the subprocesses.</p>
<p>What occurs to me, now, is that perhaps the shared memory implementation works on process groups, and looks at the model, rather than the model instance, to decide whether to use shared memory. But I’m not nearly good enough a C++ programmer to confirm this by reading the source. Can anyone cast any light on this issue?</p>
<p>Thanks in advance.</p>
Long underflow/overflow?

username: Yash_Jakhotiya
cooked: <p>What could be going wrong here?</p>
<p>converting a scalar to a LongTensor returns an arbitrarily large value.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6353fa9e4fc7482f6d94866276e25878b51eafb.png" alt="image" data-base62-sha1="shqCGoEI6DjaEEHcyI0hPigllq3" width="435" height="57"></p>
<p>This is the source.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4bef46bc1fca284126b0c8b08fd2fb968bfed8ec.png" alt="image" data-base62-sha1="aPKsmhsuC2TWac6ut5aHxQVENre" width="583" height="60"></p>
<p>A bug?</p>
<p>Changing the source to <code>id = torch.LongTensor([bos_id])</code> works fine.</p>

username: ptrblck
cooked: <p>You are creating a <code>LongTensor</code> of the size <code>1</code> using uninitialized memory.<br>
Using the “tensor type” constructors is not recommended, as it’s creating this unexpected behavior (the same applies for e.g. <code>torch.FloatTensor</code>).<br>
E.g. if you are passing a scalar value of <code>10</code> you would get a tensor with 10 uninitialized values:</p>
<pre><code class="lang-python">torch.LongTensor(10)
tensor([ 94443093879952,  94439303731984, 140676049261888,               0,
                      0,               0,               0,               0,
                      0,               1])
</code></pre>
<p>Use <code>torch.tensor(bos_id)</code> instead (which will automatically derive the <code>dtype</code>) or specify the <code>dtype</code> explicitly <code>torch.tensor(bos_id, dtype=torch.long)</code>.</p>

username: Yash_Jakhotiya
cooked: <p>I see. Indeed, the docs don’t use those constructors explicitly. torch.tensor seems the way to go. Thanks!</p>
Gradients of Torch.pow() gives NAN when the base is non-positive values

username: windson
cooked: <p>Hi, am using pytorch lightning to train some model<br>
and i use torch.pow function in my loss function…<br>
then it keeps giving NAN , then i found when computing the gradients of torch.pow() function,<br>
if the base is non-positive, the gradients its nan  which makes sense for base 0, but not sure for negative values.</p>
<p>It can be reproduced using this:</p>
<p>a = torch.tensor([-2.011, -0.000001,0,1,200], requires_grad=True)<br>
q=torch.pow(a,1/2.2)<br>
q.backward(a)</p>
<p>I wonder in this case, how do we deal with it?<br>
Anybody has ideas?  Thanks  <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: albanD
cooked: <p>Hi,</p>
<p>Since your exponent is 1/2.2, then the function is not defined for negative bases. Hence the nan gradient.</p>

username: windson
cooked: <p>hmm yeah just realize the issues. thanks</p>
Online hard negative mining (OHNM) - On Background or on every class?

username: YASJAY
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, reading a lot of iterature the OHNM is being suggested as a method to be used with background class (say for object detection). Does it really work? This is my implementation of ohnm where I sort losses from each image from a batch and create a boolean mask for values &gt;top_k:</p>
<blockquote>
<pre><code>    loss = F.cross_entropy(logits, batch_labels , weight=class_weights, reduction='none')

    batch_size = loss.shape[0]

    for i in range(batch_size):

        i_loss = loss[i]

        k_per = 0.9

        loss_sorted=torch.sort(i_loss.flatten())

        loss_len=len(loss_sorted[0])

        top_k_index = int(k_per*loss_len)

        ohnm_thresh = loss_sorted[0][top_k_index]

        loss_ohnm = i_loss&gt;ohnm_thresh

        loss[i] = loss[i]*loss_ohnm

    loss = loss.sum()/class_weights[batch_labels].sum()
</code></pre>
</blockquote>
<p>My question is, how does OHNM help and should it only be applied on the background?</p>

username: addisonklinke
cooked: <p>Hard mining (whether on or offline) is designed to help prevent class imbalance from hindering the network’s ability to learn. Typically in object detection there are way more background pixels than object class pixels, so that’s why it is applied on background. There’s no reason you couldn’t take the same approach for an object class that was overrepresented though, it’s just not as common</p>
Implementation of torch `pad_sequence` function seems like it will generate errors?

username: 00krishna
cooked: <p>I was looking at the implementation of the torch <code>torch.nn.utils.pad_sequence</code> function, which is designed to pad a sequence with a specified padding value if the sequence is less than the length of the longest example in the batch.</p>
<p>Perhaps I am not understanding something, but won’t this implementation create problems because different batches may have different length sequences? If I have a bunch of text sequences, then batch number 1 might have a longest sequence of 10, while batch 2 might have a sequence of length 30. So the neural network will likely expect fixed sized batches right? Hence I was wondering why the function was written this way, instead of specifying a fixed length for the sequence length, and then padding any sequence that was less than this length?</p>
<p>Is there a different version of this padding function in pytorch that allows the user to specify the sequence length, and hence avoid these errors with different sized batches? Thanks.</p>
Different learning rate within a layer

username: lasun
cooked: <p>Hi there, may I ask if I want to get different learning rate for different neurons within a layer, is there any code example to implement it? Thank you.</p>

username: mMagmer
cooked: <p>I’m not sure whether that is  what you want or not, but if you have modules with named parameters, you can do something like:</p>
<pre><code class="lang-auto">L = nn.Linear(5,3)
optim.SGD([
                {'params': L.weight },
                {'params': L.bias, 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)
</code></pre>
<p>You can read torch.optim <a href="https://pytorch.org/docs/stable/optim.html" rel="noopener nofollow ugc">doc</a> for more information.</p>

username: lasun
cooked: <p>Hi mMagmer, thank you for your reply. Not really my case.<br>
What I want to set different learning rate on weights in the same layer. For example, in your example<br>
{‘params’: L.weight }, will update all the weights of L at the same lr. Is it possible to let different weights have different lr? Thank you.</p>

username: mMagmer
cooked: <p>I don’t know, maybe you can write your own custom optimizer.</p>
Question about torch.histc function

username: scintel
cooked: <p>Why does</p>
<pre><code class="lang-auto">torch.histc(torch.tensor([1,3.,3,3]), bins=1, min=3, max=3)
</code></pre>
<p>output [4] instead of [3]?</p>

username: ptrblck
cooked: <p>You are using <code>bins=1</code>, which will return a single bin containing all values.<br>
Since you are using 4 input values, the returned bin will contain the count of 4.</p>

username: scintel
cooked: <p>Shouldn’t the value 1 be ignored since min=3?</p>
Profile GPU usage of a model during inference

username: andreabonvini
cooked: <p>Hi everyone,</p>
<p>I trained and serialized a segmentation model in PyTorch and now I need to profile the maximum GPU memory occupied by the model in a C++ application. Can somebody tell me what is the correct way to profile GPU usage in this case?</p>
<p>Suppose that my executable looks like this:</p>
<pre><code class="lang-auto">int main() {

   // Setup
   const torch::Device device = torch::Device(torch::kCUDA, 0);
   c10::InferenceMode guard; // See: https://pytorch.org/cppdocs/notes/inference_mode.html

   // Load model
   torch::jit::script::Module model;
   std::string model_path = "path/to/model.pt";
   try {
       model = torch::jit::load(model_path, device);
       model.eval();
   }
   catch (const c10::Error&amp; e) {
       std::cerr &lt;&lt; "error loading the model\n";
   }

   // Build a tensor to feed the model with.
   int batch_dimension = 1;
   auto tensor = torch::zeros({ batch_dimension, 3, 512, 512 }).to(device);
   std::vector&lt;torch::jit::IValue&gt; inputs;
   inputs.emplace_back(tensor);

   // Forward pass
   auto output = model.forward(inputs).toTensor();
   torch::cuda::synchronize();

   return 0;
}</code></pre>

username: a_d
cooked: <p>Seems like <code>nvprof</code> might be useful in your case.<br>
Suppose <code>./a.out</code>  is the executable, you can profile it using <code>nvprof ./a.out</code></p>
<p>more about <code>nvprof</code> here - <a href="https://docs.nvidia.com/cuda/profiler-users-guide/index.html" class="inline-onebox" rel="noopener nofollow ugc">Profiler :: CUDA Toolkit Documentation</a></p>
Triangulated points from motion_from_ essential_choose_solution

username: adizhol
cooked: <p>In what coordinate system does motion_from_essential_choose_solution return the triangulated points?<br>
In Camera1 or Camera2 coordinate system?<br>
I don’t think it can be in world coordinate because we don’t know the world coordinate of either camera</p>

username: edgarriba
cooked: <p>the function <a href="https://kornia.readthedocs.io/en/latest/geometry.epipolar.html#kornia.geometry.epipolar.motion_from_essential_choose_solution" rel="noopener nofollow ugc">motion_from_essential_choose_solution</a> should return the points in the camer2 coordinate system</p>
<p>check also this example:<br>
<a href="https://drive.google.com/file/d/1bKHTXLpA3VSBvJWd2eupbcKuqBAjCEC2/view?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/file/d/1bKHTXLpA3VSBvJWd2eupbcKuqBAjCEC2/view?usp=sharing</a></p>
Bandpass filter of Signal

username: Jaideep_Valani
cooked: <p>Is there bandpass filter api available in pytorch</p>

username: ptrblck
cooked: <p>You could create the bandpass filter either manually (e.g. via a difference of gaussians) or using another Python library (scipy should have methods for it) and apply the filter e.g. using <code>F.conv1d</code>. Alternatively, you could also filter the signal in the frequency domain.</p>

username: edgarriba
cooked: <p>checkout kornia filters API here<br>
<a href="https://kornia.readthedocs.io/en/latest/filters.html#filtering-api" class="onebox" target="_blank" rel="noopener nofollow ugc">https://kornia.readthedocs.io/en/latest/filters.html#filtering-api</a></p>
Kornia library: Does not require grad and does not have a grad_fn

username: pulsar
cooked: <p>Dear all,<br>
I’m using Kornia library in order to use some useful functions for my experiments.<br>
However, I get always the same error when I try to compute the image generated by canny filters with another very similar (a simple MSE by PyTorch function).<br>
I looked the code inside Kornia function (canny filter) and I noticed that the required_gradient attribute is empty.<br>
I’m sure to feed a tensor image with gradient set True.<br>
I’m new on Kornia and I will appreciate so much any useful suggestions…<br>
Thanks a lot!  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p><code>required_gradient</code> is not a tensor attribute, so you might want to replace it with <code>requires_grad</code> and make sure it’s set to <code>True</code>.</p>

username: edgarriba
cooked: <p>in principle the canny operator should work with back propagation<br>
<a href="https://kornia.readthedocs.io/en/latest/filters.html?highlight=canny#kornia.filters.canny" class="onebox" target="_blank" rel="noopener nofollow ugc">https://kornia.readthedocs.io/en/latest/filters.html?highlight=canny#kornia.filters.canny</a></p>
<p>see also: <a href="https://github.com/kornia/kornia/blob/0deb07968053a6f6dea38754b4167498ad59e7b7/test/filters/test_canny.py#L264" class="inline-onebox" rel="noopener nofollow ugc">kornia/test_canny.py at 0deb07968053a6f6dea38754b4167498ad59e7b7 · kornia/kornia · GitHub</a></p>
Listing variables in a computational graph

username: Kamil_Adamczewski
cooked: <p>There are two neural networks. In theory, the loss which is computed at the end for both of them should rely on distinct sets of variables. But for some reason, I get a common error:</p>
<p>RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed).</p>
<p>Is it possible to check all the variables in the computational graph for the backpropagation (backward function)?<br>
And Is it possible to see the variable where the graphs coincide?</p>

username: ZimoNitrome
cooked: <p>What does your code look like? Do you backprop on a single, combined loss variable once or two individual ones each?</p>

username: Kamil_Adamczewski
cooked: <p>Basically, there are two reinforcement learning policy networks (but that shouldn’t matter). The losses are separate  (I tried to feed them with different rewards, etc.). But I must have made a mistake somewhere and somewhere they coincide.</p>
DataLoader slows down exponentially

username: pbjarterot
cooked: <p>I have two different dataloaders for the same dataset, the first one runs through the model agnostically but the second takes the 315 consecutive images from a csv file as each batch.</p>
<p>The iteration basically identical except one layer in the model and measuring only the time for this code snippet it increases seemingly exponentially:</p>
<pre><code class="lang-auto">data_loading_time = time.time()
for batch_idx, (data2, _) in enumerate(train_loader2):
    data_loading_time2 = time.time()
    print("data loading time", data_loading_time2  - data_loading_time)
</code></pre>
<p>The code for the dataloaders is here:<br>
the loop seems to use more of the cpu (~60%) with num_workers=0 compared to num_workers=8 where it only uses 12%.<br>
(the second epoch using num_workers=0 had cpu usage at 30%)<br>
How do i structure my dataloader to not increase time to load data exponentially?</p>
<pre><code class="lang-auto">train_set, test_set = torch.utils.data.random_split(dataset, [int(0.8*len(dataset)), int(0.2*len(dataset))])
train_loader = DataLoader(dataset=train_set, batch_size=2048, shuffle=True, **kwargs)
test_loader = DataLoader(dataset=test_set, batch_size=2048, shuffle=True, **kwargs)

train_set2, test_set2 = torch.utils.data.Subset(dataset, [i for i in range(0, 504000)]), torch.utils.data.Subset(dataset, [i for i in range(504000, 630000)])
train_loader2 = DataLoader(dataset = train_set2, batch_size=315, shuffle=False, num_workers=0, pin_memory=True)
test_loader2 = DataLoader(dataset = test_set2, batch_size=315, shuffle=False, num_workers=0, pin_memory=True)
</code></pre>
Train a convnet with images of different size?

username: maffe
cooked: <p>Hi,<br>
I would like to train a Resnet-18 network on GPU using different sized images, and I do not want to resize the images nor pad. The network allows the task, as it contains an AdaptiveAvgPooling layer.</p>
<p>The problem I’m stuck with is how to create a DataLoader that loads different sized images.<br>
It appears from a number of posts on this that a custom collate_fn function should be used instead of the default one in the DataLoader.</p>
<p>Can anybody direct me in the right direction in how to create such a DataLoader for different sized images that can be used for training ?</p>
<p>Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: my3bikaht
cooked: <p>All resnets start with Conv2d layer, expecting tensor with predefined shape (batch_num, num channels, H, W). You can pass images with arbitrary shape, yes, but it has to be same shape (HxW) for all images.</p>
Consistency between DistributedSampler and DataLoader

username: junb
cooked: <p>I’m trying to ensure consistency when creating a <strong>DataLoader</strong> instance between two cases:</p>
<ul>
<li>I pass a <strong>DistributedSampler</strong> to the <code>sampler</code> argument of the <strong>DataLoader</strong>:<br>
<code>dataloader = DataLoader(..., sampler=DistributedSampler(...))</code></li>
<li>I directly instantiate the <strong>DataLoader</strong>:<br>
<code>dataloader = DataLoader(..., sampler=None)</code></li>
</ul>
<p>The consistency I’m referring to is when my <em>dataset size</em> is not divisible by the <em>batch size</em>. The documentation specify that I can handle that through the <code>drop_last</code> argument. That way, using either:</p>
<ul>
<li><code>dataloader = DataLoader(..., sampler=DistributedSampler(...,shuffle=False, drop_last=True))</code></li>
<li><code>dataloader = DataLoader(..., shuffle=False, drop_last=True, sampler=None)</code></li>
</ul>
<p>should ensure iterating identically (<em>i.e.</em> same ordering) over the batches with an equal size per batch.</p>
<p>I have several questions linked to my case:</p>
<ol>
<li>
<p>Regarding the way I handle the <code>drop_last</code> argument, is my conclusion correct?</p>
</li>
<li>
<p>Both <strong>DataLoader</strong> and <strong>DistributedSampler</strong> possess <code>shuffle</code> and <code>drop_last</code> arguments. When used in conjunction, which arguments have priority?</p>
</li>
<li>
<p>Both the documentations of <strong>DataLoader</strong> and <strong>DistributedSampler</strong> state that:</p>
</li>
</ol>
<blockquote>
<p><strong>drop_last</strong> (<a href="https://docs.python.org/3/library/functions.html#bool" rel="noopener nofollow ugc"><em>bool</em></a><em>,</em> <em>optional</em>) – if <code>True</code>, then the sampler will drop the tail of the data to make it evenly divisible across the number of replicas. If <code>False</code>, the sampler will add extra indices to make the data evenly divisible across the replicas. Default: <code>False</code>.</p>
</blockquote>
<p>I wonder how the extra indices are added (I assume through uniform sample with replacement over the whole set of indices?).</p>
<ol start="4">
<li>Is there any way to ensure consistency between my two cases if I don’t want to drop the last batch with uneven size (<em>i.e.</em> use <code>drop_last=False</code>)? I guess I would need to control how the extra indices are added; is it possible?<br>
&nbsp;</li>
</ol>
<p>If some of these points have already been addressed, I apologize and please redirect me towards the specific topics. Thanks in advance.</p>
Training Neural Network Using Two Datasets

username: Uuurial_Yow
cooked: <p>Hello. Can someone with kind heart explain to me how training with two different datasets work? The first dataset is purely for person detection while the second dataset is for head detection. The process should be, the model should detect a person then after detecting a person it then detect the head. Thank you!</p>

username: Kapil_Rana
cooked: <p>You can train two network one by one.<br>
one will person detection<br>
other with head detection.</p>
<p>After training you can pass the input to first network and based on output you can pass it to next network.</p>
Pickle data corruption with big tensors

username: Giorgio
cooked: <p>Hello, I’ve experienced a serious data corruption while saving tensors to a pickle file (about 1 giga byte of data). Probably I’ve tried to write to it again before the previous writing was complete? It’s a google colab script that saves data to its google drive environment, and each writing operation came a minute or so after the previous writing… no multithreading involved, just a saving operation in a while loop. How can I make sure that this doesn’t happen again?? Is there a way to check if there’s a file lock, or if there’s a process still writing to it?</p>
<p>Thanks</p>
<p>it saves like this</p>
<blockquote>
<p>torch.save(colonne_out, ‘0.pt’)</p>
</blockquote>
<p>after saving the file a lot of times, I tried to load the file but it gave an error</p>
<blockquote>
<p>colonne_out = torch.load(pickle_file, map_location=torch.device(device)</p>
</blockquote>
<hr>
<p>RuntimeError                              Traceback (most recent call last)</p>
<p> in ()<br>
1 # se sono offline, carico il file delle colonne di output<br>
----&gt; 2 colonne_out = torch.load(pickle_file, map_location=torch.device(device))</p>
<p>1 frames</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/serialization.py in <strong>init</strong>(self, name_or_buffer)<br>
240 class _open_zipfile_reader(_opener):<br>
241     def <strong>init</strong>(self, name_or_buffer) → None:<br>
 → 242         super(_open_zipfile_reader, self).<strong>init</strong>(torch._C.PyTorchFileReader(name_or_buffer))<br>
243<br>
244</p>
<p>RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue using this small example:</p>
<pre><code class="lang-python"># 4GB
x = torch.randn(1024**3)

for _ in range(10):
    torch.save(x, "tmp.pt")
    y = torch.load("tmp.pt")
</code></pre>
<p>Could you check, if you are hitting the same error using my code?</p>

username: Giorgio
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="137105">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto"># 4GB
x = torch.randn(1024**3)

for _ in range(10):
    torch.save(x, "tmp.pt")
    y = torch.load("tmp.pt")
</code></pre>
</blockquote>
</aside>
<p>Thanks for your feedback I’ve made it run long enough on google colab and I got no error. Since it happened to me at least once a week, I wonder where the error came from. Anyway, how does the corruption happen? Is there a way to check if the data has been written, or the only solution is trying to load it to check it?</p>

username: ptrblck
cooked: <p>Often file corruptions are caused by writes to the same files from multiple processes. Since you have apparently excluded this issue I don’t know what might be causing this error as I haven’t seen it before.</p>
Recommended way to replace a partcular value in a tensor

username: thyr
cooked: <p>What is the recommend way to replace a particular value in a tensor with something else?<br>
is torch.where() the most efficient solution?</p>
<pre><code class="lang-auto">q =[[0,1,2],[3,0,4],[5,6,0]]
tense_tensor = torch.tensor(q)
print(torch.where(tense_tensor==0,torch.tensor(-1),tense_tensor))
</code></pre>

username: justusschock
cooked: <p>What about</p>
<pre><code class="lang-auto">tensor[tensor==4] = 16
</code></pre>
<p>Assuming you want to replace for example each 4 with a 16.</p>
<p>Edit: you could also do this with a tensor, but you would need to do it like <code>tensor[tensor==4] = sixteens_tensor.clone()</code> if you want to do it in autograd on a leaf variable. Also the <code>sixteens_tensor</code> must contain exactly as many values as there are 4s in <code>tensor</code></p>

username: thyr
cooked: <p>This definitely looks cleaner! But I don’t have a sense of how much more efficient it is. It looks like it though.<br>
Also I dont want to modify the original tensor. So will need to make a clone() first.</p>

username: justusschock
cooked: <p>I don’t think you will see much differences there regarding the performance.</p>
<blockquote>
<p>Note: if you want to replace the 4s by arbitrary values defined in a tensor, I would recommend <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.masked_scatter_" rel="noopener nofollow ugc">masked_scatter_</a> instead of looping over tensors yourself. This approach should also work with a tensor of always the same value (e.g. always 16). However, this function will also modify the original tensor. If you don’t want this behavior, you would also need to clone it before.</p>
</blockquote>

username: dragen
cooked: <p><a class="mention" href="/u/justusschock">@justusschock</a> Thanks.<br>
Is there a batch way to replace several particular values? for example, <code>old_new_value = [[2,22],[3,33]]</code>, which means 2 should be replaced by 22, and 3 should be replaced by 33.<br>
Can I have a efficient way to achieve this ? Thank you.</p>

username: Giorgio
cooked: <p>Currently I’m trying to replace values that fall in a range,</p>
<p>to get the idea</p>
<pre><code class="lang-auto">tensor[tensor&gt;=4 and tensor &lt;= 10] = 0
</code></pre>
<p>but it doesn’t work…</p>
<p>Iterating seems the only choice.</p>
Equivalent of Tensorflow Records for PyTorch

username: AndriPi
cooked: <p>Hi all,</p>
<p>As per object, I’d like to know if there is a PyTorch equivalent of <a href="https://github.com/open-mmlab/mmdetection" rel="noopener nofollow ugc">Tensorflow Records</a>. I’m pretty sure that image loading (my images are high-resolution JPEG) consumes the most time during training, so I’d like to know if there are alternative ways to store my data which can lead to significant speedups. I’m using <a href="https://github.com/open-mmlab/mmdetection" class="inline-onebox" rel="noopener nofollow ugc">GitHub - open-mmlab/mmdetection: OpenMMLab Detection Toolbox and Benchmark</a></p>
<p>Thanks in advance for your support,</p>
<p>Andrea</p>
Use torchaudio cause torch jit load model error,help!

username: c9412600
cooked: <p>we use pytorch version of 1.7.0+cu101 to turn torchscript，and I successfully loaded the torchscript model in torch version of 1.7.0+cu101 to inference, but when I use pytorch version of 1.8.1+cu101 to inference, there are some error</p>
<pre><code class="lang-auto">python3 torch_test_jit_1109.py 
Traceback (most recent call last): File "torch_test_jit_1109.py", line 35, in &lt;module&gt; main(sys.argv[1], sys.argv[2]) File "torch_test_jit_1109.py", line 17, in main e2e_model = torch.jit.load(script_model) File "/data3/users/suziyi/Miniconda3/lib/python3.8/site-packages/torch/jit/_serialization.py", line 161, in load cpp_module = torch._C.import_ir_module(cu, str(f), map_location, _extra_files) RuntimeError: Class Namespace cannot be used as a value: Serialized File "code/__torch__/torchaudio/sox_effects/sox_effects.py", line 5 effects: List[List[str]], channels_first: bool=True) -&gt; Tuple[Tensor, int]: in_signal = __torch__.torch.classes.torchaudio.TensorSignal.__new__(__torch__.torch.classes.torchaudio.TensorSignal) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE _0 = (in_signal).__init__(tensor, sample_rate, channels_first, ) out_signal = ops.torchaudio.sox_effects_apply_effects_tensor(in_signal, effects) 'apply_effects_tensor' is being compiled since it was called from 'SynthesizerTrn.forward' Serialized File "code/__torch__/models_torchaudio_torchscript.py", line 41 sr: Tensor, vol: Tensor) -&gt; Tensor: _0 = __torch__.torchaudio.sox_effects.sox_effects.apply_effects_tensor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE _1 = torch.tensor([(torch.size(x))[-1]], dtype=None, device=None, requires_grad=False) x_lengths = torch.to(_1, 4, False, False, None)
</code></pre>
<p>we use torchaudio to change audio speed,volume and so on, please help me<br>
when we create torchscript the pytorch version is 1.7.0+cu101 , the torchaudio version is 0.7.0</p>

username: nateanl
cooked: <p>Hi <a class="mention" href="/u/c9412600">@c9412600</a>, thanks for sharing the issue. Could you share your script that uses <code>sox_effect</code>?<br>
Also you can post it on torchaudio’s GitHub page. This helps other people who also meet this problem.</p>

username: c9412600
cooked: <p>Thanks for your reply，that’s my script ，please check</p>
<pre><code class="lang-auto">import torch
import torchaudio.sox_effects as sox_effects

 def forward(self, audio: torch.Tensor, speed: torch.Tensor, pitch: torch.Tensor, sr: torch.Tensor, vol: torch.Tensor): 

    speed = str(speed.item())

    pitch = str(pitch.item())

    sr = str(int(sr.item()))

    vol = str(vol.item())

    effects = [['tempo','-s', speed], ['pitch', pitch],['rate', sr], ['vol', vol]]

    waveform, _ = sox_effects.apply_effects_tensor(audio, int(sr), effects)

    return waveform

</code></pre>
PyTorch ops to native function mapper

username: vjayd
cooked: <p>I am looking for a mapping table between PyTorch ops like (sigmoid, narrow, lstm, etc) to its native function. I agree there are some ops which invoke multiple native functions like<br>
empty_strided → empty and stride. Where can I have a detailed table where I can see the mapping. In other words, if I have a log of native functions how can I reverse map to it’s corresponding pytorch operator</p>

username: ptrblck
cooked: <p>I don’t think there exists a mapping as the internal implementations could change and could also dispatch to different operations/methods for different backends. The best way would be to check the implementation in the source code and see which operations are used.</p>
Using DistributedDataParallel on shared GPU

username: Jacfger
cooked: <p>Hi,</p>
<p>I’m now a research assistant and I want to replicated a experiment which had used distributed data parallel method. I have also heard good things about it and therefore I kinda want to try it as well. However, I noticed that in the documentation, it says I have to ensure that the program has exclusive access to the GPU when I’m using NCCL backend (the source code of the project had also used barrier(), which was only supported by nccl without extra steps like compiling). Is the exclusiveness includes other process running on GPU that’s not distributed? Like can I have a small model (probably using non-distributed dataparallel or simple single gpu training) on the same GPU? I just want to make sure and ask before I try because it’s shared and I’m afraid I might crash others’ programs. I have looked online but I can only see people reporting problems when they’re trying to run two distributed instance on a GPU.</p>

username: rvarm1
cooked: <p>Yes, you could run into deadlocks or hard to debug hangs if your program does not have exclusive access to the GPU when using the NCCL backend. This is because synchronization done by the application can result in things such as waiting for all operations in a stream to complete, which could result in unpredictability if it ends up waiting on ops from other applications.</p>

username: Jacfger
cooked: <p>But according to what you say, as long as I’m the only one who’re using nccl backend, the other program shouldn’t be locked? Since those wouldn’t need to wait for anything to end. And as long as they finished my program should continue to run. Is that how it works?</p>
Nn.embedding too big (cuda out of memory)

username: EJHyun
cooked: <p>Hi<br>
Im dealing with memory issue,<br>
which is because I need to use huge size of nn.embedding layer.<br>
Ex) self.layer = nn.embedding(huge_dimension, emb_dim)</p>
<p>Before start,<br>
I have 2 gpus, and both have vram memory of 32GB.<br>
My embedding layer(my model) 's memory usage is  17~18GB.<br>
Im using Adam optimizer.</p>
<p>Okay uhh…<br>
It seemed to be okay for just using one 32GB gpu to handle 17~18GB size model, but<br>
Im facing Cuda out of memory when my Adam optimzer begins loss.backward().<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad9c8c3328dbbd7af7aaa9aa0ad1f3bb80b3e05f.png" alt="image" data-base62-sha1="oLQ0g7jxuiISrkh2xRFm2VafCdp" width="550" height="52"></p>
<p>So… I’ve tried</p>
<ol>
<li>nn.embedding(~~, sparse = True) with SparseAdam</li>
<li>nn.DataParallel</li>
</ol>
<p>The first one,<br>
because the cause of Cuda Out of memory is loss.backward(),<br>
which is the step of backpropagation loading the gradient-graph before optimizer.step(),<br>
I thought using sparse embedding layer with SparseAdam could help because<br>
I heard SparseAdam only need the “used part” of total embedding layer(sparse matrix).<br>
In fact, it worked about the loss.backward(), but now optimizer.step() is showing me Cuda out of memory.</p>
<p>The Second one, nn.DataParallel,<br>
because I have one more GPU with same spec,<br>
I thought I could deal with this problem using nn.DataParallel.</p>
<p>but the problem is,<br>
when nn.DataParallel calls nn.parallel.replicate(), which copies model to each GPUs,<br>
it shows Cuda out of memory.<br>
I’m not sure about how nn.parallel.replicate works but It seems it need extra memory to copy<br>
my 17~18GB model.</p>
<p>So…can anyone please tell me<br>
how to deal with huge size of embedding layer or…<br>
why my optimizer.step()is showing me Cuda out of memory or…<br>
why nn.parallel.replicate needs extra memory or…<br>
nn.parallel.replicate seems to copy model from gpu to gpu, but i think just copying model from cpu to each gpu seems fair enough… but i don’t know the way. or…<br>
how to seperate my nn.embedding layer to 2 gpus or…<br>
Can I just use one gpu for forward and another gpu for backward?<br>
or just any comments for help</p>
<p>thanks</p>

username: my3bikaht
cooked: <p>Usually to deal with memory constraints during training pass, you need to call optimizer more often. First choice to go is to make batches smaller.</p>

username: EJHyun
cooked: <p>Oh I solved this problem with model sharding!<br>
I splitted my embedding layer to 2 pieces with differnet size<br>
and allocated them to each gpus<br>
and ran forward operation on gpu who has smaller piece of embedding layer</p>
<p>and it works!</p>
How to remove SpectralNorm parametrizations in PyTorch?

username: Glory_Chen
cooked: <p>The statement is as the title.</p>

username: ptrblck
cooked: <p><code>torch.nn.utils.remove_spectral_norm</code> should work.</p>

username: Glory_Chen
cooked: <p>Could you give an example on like how to remove <a href="https://pytorch.org/docs/1.10.0/generated/torch.nn.utils.parametrizations.spectral_norm.html?highlight=spectral%20norm#torch.nn.utils.parametrizations.spectral_norm" rel="noopener nofollow ugc">ParametrizedLinear</a> with _SpectralNorm? Thanks</p>

username: ptrblck
cooked: <p>You could use this approach:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 10)
lin = torch.nn.utils.spectral_norm(lin)

# check spectral_norm params
print(lin.state_dict())

lin = torch.nn.utils.remove_spectral_norm(lin)

# check params again
print(lin.state_dict())
</code></pre>
<p>It seems that using the reimplemented <code>torch.nn.utils.parametrizations.spectral_norm</code> might be missing the corresponding <code>remove_spectral_norm</code> call, as using <code>torch.nn.utils.remove_spectral_norm</code> doesn’t work:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 10)
lin = torch.nn.utils.parametrizations.spectral_norm(lin)

# check spectral_norm params
print(lin.state_dict())

lin = torch.nn.utils.remove_spectral_norm(lin)
ValueError: spectral_norm of 'weight' not found in ParametrizedLinear(
  in_features=10, out_features=10, bias=True
  (parametrizations): ModuleDict(
    (weight): ParametrizationList(
      (0): _SpectralNorm()
    )
  )
)
</code></pre>
<p><a class="mention" href="/u/alband">@albanD</a> do you know, if this method is just missing or if the removal of <code>spectral_norm</code> via the new API should be done in another way?</p>

username: Lezcano
cooked: <p>The correct way to remove parametrisations (this or any <code>nn.utils.parametrizations</code>) is through <code>torch.nn.utils.remove_parametrization</code>: <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.spectral_norm.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.utils.parametrizations.spectral_norm — PyTorch 1.10.0 documentation</a></p>
<p>As a side note, all this is documented in the parametrizations tutorial: <a href="https://pytorch.org/tutorials/intermediate/parametrizations.html#removing-parametrizations" class="inline-onebox" rel="noopener nofollow ugc">Parametrizations Tutorial — PyTorch Tutorials 1.10.0+cu102 documentation</a></p>
Epochs and Accuracy

username: KanZa
cooked: <p>Hi, I am working on Active Learning methods. I implemented this model here <a href="https://github.com/razvancaramalau/Sequential-GCN-for-Active-Learning" class="inline-onebox" rel="noopener nofollow ugc">GitHub - razvancaramalau/Sequential-GCN-for-Active-Learning</a><br>
I  tried different models with different numbers of epochs and that increased accuracy, now I am increasing epochs, and accuracy is increasing. I am so confused about how is this possible or what kind of trick they have used for such a model. I have invested a lot of my time in understanding the code.</p>
Tensor operations gives me different outputs

username: thesekyi
cooked: <pre><code class="lang-auto">index_separator = 2
all_probs = tensor([[1.7453e-01, 8.2525e-01, 2.1337e-04],
        [7.9364e-01, 2.0613e-01, 2.2616e-04],
        [4.1631e-02, 9.5827e-01, 1.0366e-04],
        [9.2631e-01, 7.3529e-02, 1.6452e-04],
        [9.9914e-01, 8.1824e-04, 4.1946e-05],
        [8.0214e-01, 1.9750e-01, 3.6684e-04],
        [9.9894e-01, 1.0302e-03, 3.3393e-05],
        [9.5733e-02, 9.0413e-01, 1.3462e-04],
        [6.5114e-01, 3.4861e-01, 2.4305e-04],
        [4.2742e-02, 9.5717e-01, 8.7021e-05],
        [6.7704e-01, 3.2274e-01, 2.1942e-04],
        [2.8399e-02, 9.7148e-01, 1.1770e-04],
        [1.0530e-01, 8.9452e-01, 1.7405e-04],
        [7.6687e-02, 9.2317e-01, 1.3761e-04],
        [9.8485e-01, 1.4972e-02, 1.7385e-04],
        [8.1003e-01, 1.8972e-01, 2.4742e-04],
        [9.5397e-01, 4.5884e-02, 1.4194e-04],
        [6.4258e-02, 9.3563e-01, 1.0830e-04],
        [9.7275e-01, 2.7137e-02, 1.1549e-04],
        [9.9825e-01, 1.6978e-03, 5.4864e-05],
        [8.5200e-02, 9.1468e-01, 1.2489e-04],
        [1.1248e-01, 8.8739e-01, 1.3314e-04],
        [9.9868e-01, 1.2721e-03, 4.9347e-05],
        [9.8094e-01, 1.8952e-02, 1.0916e-04],
        [9.9782e-01, 2.0968e-03, 8.1622e-05],
        [9.2653e-01, 7.3261e-02, 2.1026e-04]])

normal_index_sum = torch.sum(all_probs[:, :index_separator], dim=1, keepdim=True)
anomaly_index_sum = torch.sum(all_probs[:, index_separator:], dim=1, keepdim=True)

difference = abs(normal_index_sum - anomaly_index_sum).reshape(-1).tolist(); difference
[0.9995732307434082,
 0.9995477199554443,
 0.9997926354408264,
 0.9996709227561951,
 0.9999160766601562,
 0.9992663264274597,
 0.9999333024024963,
 0.9997307062149048,
 0.9995139241218567,
 0.9998259544372559,
 0.9995611310005188,
 0.999764621257782,
 0.9996519684791565,
 0.9997246861457825,
 0.9996522665023804,
 0.9995051026344299,
 0.9997161030769348,
 0.9997833371162415,
 0.9997689127922058,
 0.9998903274536133,
 0.9997502565383911,
 0.9997337460517883,
 0.9999013543128967,
 0.9997817873954773,
 0.999836802482605,
 0.9995794296264648]
</code></pre>
<p>However, if I take the item at <strong>index 0</strong> and do the computation <code>1.7453e-01 + 8.2525e-01 - 2.1337e-04</code>, I get an output of <code>0.99956663</code> which is close to <code>difference[0]</code> but not quite. Is there a way to make sure the values returned in both cases are consistent? Is there a way to also track very small changes without the approximation effects seen here?</p>

username: ptrblck
cooked: <p>This small error is expected due to the limited floating point precision. A different order of operation could create these errors and on my system the difference between the manual calculation vs. the <code>difference</code> tensor is ~4e-8.</p>
How to customize load_state_dict?

username: traveller59
cooked: <p>I have a custom convolution layer that support two layouts: KRSC and RSKC, RS is kernel size, K is out channel, C is input channel, used by different algorithms. Now I want to load RSKC state dict to KRSC layer, I can’t find any useful hooks to achieve this. a hook named “_register_load_state_dict_pre_hook” exists but it’s a private method and don’t have guarantee that available in torch 1.5+.</p>

username: ptrblck
cooked: <p>You can manipulate the tensors in the <code>state_dict</code> directly before loading it.<br>
Here is a simple example:</p>
<pre><code class="lang-python">conv1 = nn.Conv2d(3, 6, 3)
conv2 = nn.Conv2d(3, 6, 2)

sd = conv1.state_dict()

# won't work
conv2.load_state_dict(sd)
# &gt; RuntimeError...

# manipulate
sd['weight'] = sd['weight'][:, :, :2, :2]

# works
conv2.load_state_dict(sd)
# &gt; &lt;All keys matched successfully&gt;
</code></pre>
Computing Hardware Resources for deep learning model

username: learner12345
cooked: <p>Hi everyone<br>
Can anyone guide me how can we compute resources required by our model before deploying it. For instance<br>
if we want to apply ‘A’ neural network (with ‘N’ layers) to a ‘D’ dataset. How can we compute how hardware resources we should have to run it perfectly. Instead of deploying on too smaller or too larger hardware that is wastage of time and cost.</p>
<p>Please help if anyone have any clue. I have been googling a lot about it but couldn’t find a perfect solution.</p>
<p>Thanks in Advance</p>
Restrict output to an arbitrary bound

username: codeflux
cooked: <p>Hi,<br>
I am implementing a A2C algorithm for RL. For this purpose, I need to train the actor, which has a Gaussian distribution as a policy. Therefore, the mean and variance are two of my outputs and are continuous numbers. They need to be restricted to both being &gt; 0 and the mean &lt; than another adaptable threshhold.</p>
<p>My third output should have just three values, which is encoded into the decision making of the algorithm. Here, I am not sure if I can keep it in the same neural net or create a seperate one.</p>
<p>Anyway, the main question is, how I could restrict the above mentioned outputs, to fullfilling my conditions.</p>

username: Eta_C
cooked: <p>I think you can add one ReLU layer before output.</p>

username: codeflux
cooked: <p>That would enable me to make it &gt;= 0, but what I want is &gt;0 and &lt; a bound that is dependent on one of the inputs.</p>

username: Eta_C
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.clamp.html?highlight=clamp#torch.clamp" rel="noopener nofollow ugc">torch.clamp</a></p>
So I think I may have built an attention transformer

username: djlindsay
cooked: <p>I pretty faithfully followed the code found at <a href="https://github.com/aladdinpersson/Machine-Learning-Collection/blob/master/ML/Pytorch/more_advanced/transformer_from_scratch/transformer_from_scratch.py" class="inline-onebox" rel="noopener nofollow ugc">Machine-Learning-Collection/transformer_from_scratch.py at master · aladdinpersson/Machine-Learning-Collection · GitHub</a></p>
<p>I even wrote a little training function for it.</p>
<pre><code class="lang-auto">import torch.optim as optim

device= torch.device("cuda" if torch.cuda.is_available() else "cpu")

src= torch.tensor([[1,2,3,4,5,0],[3,4,5,6,7,8]]).to(device)
trg=torch.tensor([[6,7,8,9,10,11],[8,9,10,11,12,13]]).to(device)

model=Transformer(src_vocab_size=14, trg_vocab_size=14, src_pad_idx=0, trg_pad_idx=0).to(device)

opt=torch.optim.SGD(model.parameters(), lr=0.01)
loss_fn=nn.functional.cross_entropy

def train_model(epochs=501):
    total_loss=0
    train_loss_list, validation_loss_list = [], []
    for e in range(epochs):
        model.train()
        pred=model(src, trg)
        if e%50==0:
            print("-"*25, f"Epoch {e + 1}","-"*25)
            print(pred.shape)
            print(torch.argmax(pred,dim=2))
            print(trg)
        loss=loss_fn(pred, torch.nn.functional.one_hot(trg, num_classes=14).type(torch.FloatTensor))
        
        opt.zero_grad()
        loss.backward()
        opt.step()
        
        total_loss += loss.detach().item()/(e+1)
        train_loss_list += [total_loss]
        
        if e%50==0: print(f"Training loss: {total_loss:.4f}")
        print('...')
        
    return train_loss_list
print(model)
train_loss_list = train_model()
</code></pre>
<p>After 400 or so epochs, it actually predicts the right target sequence given the source!</p>
<p>What I am asking you is,</p>
<ol>
<li>
<p>Did I actually implement this training function correctly? Is my model learning or do I have data leakage? I’m pretty sure my loss is being calculated incorrectly, so if you have any tips on that too, that’d be great.</p>
</li>
<li>
<p>How do I test this function on data it hasn’t seen? For example, the sequence [2,3,4,5,6,7] should predict [7,8,9,10,11,12], but I don’t know how to run the model with only inputs (it calls for src and trg).</p>
</li>
</ol>
<p>Your guidance is really appreciated. I’m looking to take this transformer and generate song lyrics with it.</p>
Post load_state_dict hook

username: iiSeymour
cooked: <p>I have a use case for modifying a model’s parameters after its values get loaded but I don’t think PyTorch has a hook for <code>load_state_dict</code>. I currently just have a caching method that runs the transformation once on the first call the <code>forward</code> but this makes the model jit unfriendly. Is there a better way to handle this?</p>

username: ptrblck
cooked: <p>PyTorch uses the internal <code>_register_state_dict_hook</code> and <code>_register_load_state_dict_pre_hook</code>. However, since these methods are internal their interface could change without a deprecation warning, so I wouldn’t depend on them.</p>
<p>Based on your description it seems you are looking for a “post_state_dict_hook”?<br>
If so, wouldn’t applying the desired manipulations to the model parameters after calling <code>model.load_state_dict</code> work? I’m unsure why a hook would be needed for it.</p>
Can tensor values go outside [0,1] range?

username: useasdf_4444
cooked: <p>Hello!</p>
<p>I’m taking my first steps in machine learning and experimenting with cnns and pooling recently.<br>
I’m trying to implement a fuzzy pooling method, described in the paper by Diamantis et. al. (<a href="https://is-innovation.eu/iakovidis/wp-content/uploads/2020/09/IEEE-TFS-Fuzzy-Pooling.pdf" rel="noopener nofollow ugc">https://is-innovation.eu/iakovidis/wp-content/uploads/2020/09/IEEE-TFS-Fuzzy-Pooling.pdf</a>)<br>
Though I have implemented most of it, I’m kinda stuck in the part of the membership functions that usually end up in tensors of zeros or ones, messing up the overall results.</p>
<p>In page 3 of the paper’s PDF there’s a schema with a sample patch with values way outside of the [0,1] range. How is that possible in pytorch, since all tensor values will be in the [0,1] range?<br>
The input patches are categorized using membership functions based on a max value of 6 (rmax), but is it possible for a tensor value to go that high - or even above 1 in general?</p>
<p>Sorry if it’s too “newbie” question but this is driving me nuts for days!</p>

username: ivan-jgr
cooked: <p>Hi <a class="mention" href="/u/useasdf_4444">@useasdf_4444</a>,</p>
<p>Yes, in torch the values of a tensor can go outside [0, 1] but remember that if you’re using <code>ToTensor</code> somewhere it automatically normalize the values between 0 and 1.</p>