akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Glow training problem: the train loss has a sudden rise (inf) when learning rate up 1e-3 from 0

username: yiqiang_wu
cooked: <p>Hi, I met a cuda runtime error as following:<br>
RuntimeError: svd_cuda: the updating process of SBDSDC did not converge (error: 22)<br>
Recently, I am studying normalizing flow, such as Glow, however, a strange svd problem has arisen when I try to train Glow from scratch. In my opinion, due to Glow contains “tensor.slogdet()” operation in affine coupling layer, it may involve SVD decomposition, and thus casue above problem.<br>
Specifically, I first use a small learning rate, such as 1e-6, the training loss begins  to fall slowly. However, when the learning rate reaches 0.0004, the training loss has a sudden rise (inf) and the error information is presented as above.<br>
How can I avoid this error during training process of Glow？<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/56667875d9c995545d18ca138a8a7804c9a6df72.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/56667875d9c995545d18ca138a8a7804c9a6df72" title="捕获"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56667875d9c995545d18ca138a8a7804c9a6df72_2_690x397.jpeg" alt="捕获" data-base62-sha1="ckkBu45TTgPgf3NbpqUqUSNpfrA" width="690" height="397" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56667875d9c995545d18ca138a8a7804c9a6df72_2_690x397.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/6/56667875d9c995545d18ca138a8a7804c9a6df72.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/6/56667875d9c995545d18ca138a8a7804c9a6df72.jpeg 2x" data-dominant-color="0F3640"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">捕获</span><span class="informations">945×544 87.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: zyabo
cooked: <p>I also encountered a similar problem</p>
How can I split the input data into training and validation in the C++ API?

username: tmaric
cooked: <p>I would like to be able to do k-fold validation or use random sampling to split the data into training and validation sets, but I’m having trouble finding the functions in the C++ API, can anyone help?</p>

username: tom
cooked: <p>The low-level sampling functions should be there, I don’t think torch.distributions has an equivalent in C++. However, in terms of process I’d recommend to assign each datapoint to a fold ahead of time.</p>
<p>Best regards</p>
<p>Thomas</p>

username: tmaric
cooked: <p>Thanks! I found <a href="https://pytorch.org/cppdocs/api/classat_1_1_tensor.html?highlight=tensor" rel="noopener nofollow ugc">tensor_split</a> but the problem is that my data have geometric ordering, for example, data stored at points surrounding a sphere. Constructing a fold manually would mean I’ve selected a sub-set of the sphere’s surface for testing, which won’t work. Random sampling would be better… if there aren’t libtorch native functions for that, I can use STL and hack something, but it would be nice to have something like a <code>random_split(const at::tensor &amp;, 0.1)</code> giving <code>std::vector&lt;at::tensor&gt;</code> that splits the data randomly into 90% training, 10% validation sets.</p>
Getting "RuntimeError: CUDA error: out of memory" when memory is free

username: blade
cooked: <p>I’m trying to run a test code on GPU of a remote machine. The code is</p>
<pre><code class="lang-auto">import torch

foo = torch.tensor([1,2,3])
foo = foo.to('cuda')
</code></pre>
<p>I’m getting the following error</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “/remote/blade/test.py”, line 3, in <br>
foo = foo.to(‘cuda’)<br>
RuntimeError: CUDA error: out of memory<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
</blockquote>
<p>From <a href="https://stackoverflow.com/questions/54374935/how-to-fix-this-strange-error-runtimeerror-cuda-error-out-of-memory/67617190#67617190" rel="noopener nofollow ugc">this discussion</a>, the conflict between cuda and pytorch versions may be the cause for the error. I run the following</p>
<pre><code class="lang-auto">print('python v. : ', sys.version)
print('pytorch v. :', torch.__version__)
print('cuda v. :', torch.version.cuda)
</code></pre>
<p>to get the versions:</p>
<blockquote>
<p>python v. : 3.9.7 (default, Sep 16 2021, 13:09:58)<br>
[GCC 7.5.0]<br>
pytorch v. : 1.11.0.dev20211206<br>
cuda v. : 10.2<br>
Does anything here look off?</p>
</blockquote>

username: ptrblck
cooked: <p>The out of memory error might be wrongly returned and it seems your setup is unable to use the GPU.<br>
Were you able to run PyTorch workloads on the GPU on this system before? If so, what did you change?<br>
Also, which GPU, NVIDIA driver, OS etc. are you using?</p>

username: blade
cooked: <p>This is a remote research computing machine and I used to be able to run on GPUs before. I open an interactive session using</p>
<p>qrsh -q gpu -l gpu_card=1 -pe smp 1</p>
<p>I’m using:</p>
<p>GPU: Tesla V100 PCIe 32GB<br>
NVIDIA driver: version 460.80<br>
OS: Red Hat Enterprise Linux Server release 7.9 (Maipo)</p>

username: ptrblck
cooked: <p>It’s hard to tell what might be the root cause, as it was working before and I don’t know what has changed.<br>
I don’t believe there is a PyTorch-related fix, as the error points towards a setup issue.<br>
You could try to test other environments (e.g. create a new conda env, reinstall PyTorch, and check if it’s working), try to run CUDA samples by rebuilding them and executing etc.</p>

username: blade
cooked: <p>For the folks having a similar issue, the issue in my case was caused by conflicting python libraries. I removed the python directory outside of conda and it fixed the issue.</p>
'gelu_backward_out_cuda' is not a member of 'at:native' in PyTorch 1.10

username: hs_matt
cooked: <p>Hi PyTorch team,</p>
<p>After upgrading to PyTorch 1.10, I can not call the C++ API for gelu backward function. I do find that the API for gelu backward has been changed from <code>gelu_backward_cuda </code> into <code>gelu_backward_out_cuda </code>. However, I still met the following error:</p>
<pre><code class="lang-auto">error: ‘gelu_backward_out_cuda’ is not a member of ‘at::native’
</code></pre>
<p>A similar error can also be observed for <code>softmax_backward_cuda_out </code>. Are there any solutions for that? or any examples for using C++ API for calling gelu and softmax backward function in PyTorch 1.10?</p>
<p>Kind regards.</p>

username: tom
cooked: <p>Are you calling into <code>at::native::foo</code>? This has never been supported, but only using <code>at::foo</code>. <code>at::native</code> is an implementation detail. (And even the internal uses have been switched to <code>at::</code> if I remember correctly.)</p>
<p>Best regards</p>
<p>Thomas</p>

username: hs_matt
cooked: <p><a class="mention" href="/u/tom">@tom</a> Hi Thomas, thanks for your reply!</p>
<p>I am using <code>at::native::</code>. For example,</p>
<pre><code class="lang-auto">at::native::gelu_backward_out_cuda(grad_output, input);
</code></pre>
<p>However, it will result in an error with PyTorch 1.10 as mentioned before. But with PyTorch 1.9 and lower versions, I can call <code>at::native::gelu_backward_cuda(grad_output, input);</code> successfully.</p>
<p>Kind regards.</p>

username: tom
cooked: <p>Yes, but that this worked was by accident more than by design. at::native is an internal namespace of PyTorch and the official API is at::… without native. Does using it without native work for you?</p>
<p>Best regards</p>
<p>Thomas</p>

username: hs_matt
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a>, thanks for your quick reply. Unfortunately, it doesn’t work for me to use <code>at:: gelu_backward_out_cuda</code>. However, I just found that <code>at::gelu_backward</code> and <code>at::_softmax_backward_data</code> work for me. Thank you for the very important tips.</p>
<p>Cheers.</p>

username: tom
cooked: <p>Glad you solved it! There should be an out variant, but cuda/cpu is done by the dispatcher, likely.</p>
I want to implement a keras architecture on pytorch . i have uploaded both code on pa

username: Hridoy_Buffy
cooked: <p>. i have uploaded both code on pastebin<br>
<a href="https://pastebin.com/x0wdTMLK?fbclid=IwAR2dz9ZhSy9snnbpKxVg-3fNgwCYoJfBKb25aEDo1h_ciG_vzCLdRqLt5yw" rel="noopener nofollow ugc">pytorch implementation</a></p>
What's the corresponding function to torch.nn.CrossEntropyLoss() in C++ API?

username: EsdeathYZH
cooked: <p>I want to use <code>torch.nn.CrossEntropyLoss</code> to compute loss in C++, but I couldn’t find a corresponding function in C++ API, only binary_cross_entropy is found. Is there a function corresponding to <code>torch.nn.CrossEntropyLoss</code>?</p>

username: 21fa417e3fb06e56040c
cooked: <p>I also could not find the function. Try the function below.<br>
<code>torch::nll_loss(out.log_softmax(1), targets);</code></p>

username: EsdeathYZH
cooked: <p>Thanks for you advice!</p>

username: user31
cooked: <p>have you found the function?</p>

username: user_123454321
cooked: <p>Perhaps it was not available in 2019, but now I believe it is</p>
<p><a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_cross_entropy_loss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_cross_entropy_loss.html</a></p>

username: user31
cooked: <p>I did not find the c++ source code</p>
`num_features` argument in BatchNorm1d?

username: pascalm
cooked: <p>Hello,</p>
<p>the signature of <code>BatchNorm1d()</code> is <code>torch.nn.BatchNorm1d(num_features, ...)</code>.</p>
<p>whereas</p>
<blockquote>
<p><code>num_features</code> – C from an expected input of size (N,C,L) or  from input of size (N,L)</p>
</blockquote>
<p>and</p>
<blockquote>
<p>Input: (N,C) or (N,C,L)</p>
</blockquote>
<p>Currently my input is a timeseries sampled at 2048 Hz and I take 1s of it i.e. I have 2048 numbers. I currently feed the network a tensor of size [1, 2048] and use</p>
<p><code>torch.nn.BatchNorm1d(1)</code></p>
<p>Now while the results are bad (but that’s probably because my NN is random/bad atm) I’m wondering if I use it correctly.</p>
<p>Of course, I have 2048 features and not 1 but then the docs say that the input has to be of the shape (N, C) and num_features is actually C meaning that if I put in a tensor of shape [1, 2048] I actually have num_features=2048.</p>
<p>Which confuses me, because I did set num_features=1. So the question arises: Why the need to set num_features anyway if it is deduced automatically from the input shape?</p>
<p>What’s going on here?</p>
<p>Also what is N, C, L to begin with?</p>

username: user_123454321
cooked: <p>I believe <code>num_features</code> in <code>BatchNorm</code> is the number of channels rather than time/spatial dimensions.</p>
<pre><code class="lang-auto">N - Batch size
C - Features / Channels, 1 in your case
L - Length (number of samples), 2048 in your case
</code></pre>
<p>You can definitely have more than 1 <code>C</code>, think about ECG where there are 12 leads and thus <code>C=12</code>.</p>
How to define a pytorch class Dataset which take dataset with and without label

username: KNEC
cooked: <p>Hello, I ma trying to define a Dataset class which is suppoose to treat dataset with label and without label. But I ma getting and error <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">Item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()} AttributeError: ‘list’ object has no attribute ‘items’ 
</code></pre>
<p>code</p>
<pre><code class="lang-auto"># Create torch dataset
class Dataset(torch.utils.data.Dataset):
	def __init__(self, encodings, labels=None):
		self.encodings = encodings
		self.labels = labels

	def __getitem__(self, idx):
		item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
		if self.labels:
			item["labels"] = torch.tensor(self.labels[idx])
			#print(item)
		return item

	def __len__(self):
		print(len(self.encodings["input_ids"]))
		return len(self.encodings["input_ids"])


# prepare dat for classification

tokenizer = FlaubertTokenizer.from_pretrained(model_name)
	print("Transform xml file to pandas series core...")
	text, file_name = transform_xml_to_pd(file)  # transform xml file to pd
	
	# Xtest_emb, s = get_flaubert_layer(Xtest['sent'], path_to_model_lge)  # index 2 correspond to sentences
	#print(text)
	
	print("Preprocess text with spacy model...")
	clean_text = make_new_traindata(text['sent'])
	#print(clean_text[1])  # clean text ; 0 = raw text ; and etc...
	
	X = list(clean_text)
	X_text_tokenized = []
	
	for x in X:
		#print(type(x))
		x_encoded = tokenizer(str(x), padding="max_length", truncation=True, max_length=512)
		#print(type(x_encoded))
		#print(x_encoded)
		X_text_tokenized.append(x_encoded)
		
	#print(type(X_text_tokenized))
	
	
	X_data = Dataset(X_text_tokenized)
	
	print(type(X_data))
	print(X_data['input_ids'])
Error :slight_smile:


  File "/scriptTraitements/classifying.py", line 153, in __getitem__
    item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
AttributeError: 'list' object has no attribute 'items'


</code></pre>
<p>Any idea  ?</p>
<p>furthermore How can I access element of the corpus pass to the Dataset class. When printing it , I get only this : &lt;<strong>main</strong>.Dataset object at 0x7fc93bec1df0&gt;</p>

username: mMagmer
cooked: <p>it says that you’re not suppose to use .items ( a dict method) on list object.</p>

username: user_123454321
cooked: <p><code>self.encodings</code> is a list (of dictionaries) and thus it does not have a <code>.items()</code> function. Taking a guess that <code>X_text_tokenized</code> contains all data and we need to index it to get one item, I think you need to update</p>
<pre><code class="lang-auto">item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
</code></pre>
<p>to</p>
<pre><code class="lang-auto">item = {key: torch.tensor(val) for key, val in self.encodings[idx].items()}
</code></pre>
Does `torch.nn.Transformer` uses caching for autoregressive decoding?

username: seewoo5
cooked: <p><a href="https://tunz.kr/post/4" rel="noopener nofollow ugc">This blog post</a> says that we can make transformer much faster while decoding by caching the previous inputs, since there’s no need to re-compute all the same things we already computed. When I read the implementation of the <code>torch.nn.Transformer</code> code, it doesn’t seem to have such feature (at least for 1.7.0). If I’m right, is there any plan to implement such feature, or is it already implemented by someone and will be released in 1.8.0? Thanks!</p>

username: yijiang
cooked: <p>As far as I know, <code>nn.Transformer</code> remains unchanged since its release. Nonetheless, I found <a href="https://scale.com/blog/pytorch-improvements" rel="noopener nofollow ugc">a post</a> that might answer your question. Or, another option is to dive into the source code of huggingface transformer to learn how they implement the cache and other optimization tricks (which is tough but highly worth!!). Moreover, after implementing these tricks, we can pass a <code>custom_decoder</code> also <code>custom_encoder</code> as args to instantiate a <code>nn.Transformer</code>.</p>
Question about Convolution Underlying Operations

username: J_Johnson
cooked: <p>I was going through the Python source code for the ConvNd modules and seem to have hit a dead end.</p>
<p>What I am wondering is how the convolutions are being calculated. I.e. if we have the weights and input image, how are these being shaped/multiplied?</p>
<p>Is it done by reshaping the kernel to a classic Toeplitz matrix and then doing matmul?</p>
<p>Or some other method?</p>

username: ptrblck
cooked: <p>Yes, a <code>matmul</code> approach will be used for the native implementations as given <a href="https://discuss.pytorch.org/t/source-code-f-conv2d-definition-location/81569/4">here</a>. Different backends (e.g. cuDNN) could call different algorithms internally, which depend on the workload shape etc.</p>

username: J_Johnson
cooked: <p>I found this article helpful in detailing the underworkings of the Conv2d operation:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://towardsdatascience.com/conv2d-to-finally-understand-what-happens-in-the-forward-pass-1bbaafb0b148">
  <header class="source">
      <img src="https://miro.medium.com/fit/c/256/256/1*ChFMdf--f5jbm-AYv6VdYA@2x.png" class="site-icon" width="256" height="256">

      <a href="https://towardsdatascience.com/conv2d-to-finally-understand-what-happens-in-the-forward-pass-1bbaafb0b148" target="_blank" rel="noopener nofollow ugc" title="05:20PM - 03 May 2021">Medium – 3 May 21</a>
  </header>

  <article class="onebox-body">
    <img src="https://miro.medium.com/freeze/max/660/1*DTTpGlhwkctlv9CYannVsw.gif" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://towardsdatascience.com/conv2d-to-finally-understand-what-happens-in-the-forward-pass-1bbaafb0b148" target="_blank" rel="noopener nofollow ugc">Conv2d: Finally Understand What Happens in the Forward Pass</a></h3>

  <p>Visual and mathematical explanation of the 2D convolution layer and its arguments</p>

  <p>
    <span class="label1">Reading time: 9 min read</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>So the kernel is first applied via matmul with the set stride, dilation, etc to each channel. Then those output channels are summed across channels. That is repeated for each kernel.</p>
<p>In Pytorch, the weights are of size:<br>
out_channels, in_channels, kernel_dim0, kernel_dim1</p>
Idiomatic way to standardize data by blocks (days)

username: efremale
cooked: <p>I have a dataset that consists of samples recorded on different days, i.e. ~100 days, variable number of samples (~10K - 100K) per day.</p>
<p>Given the nonstationary nature of the data, it makes sense for us to standardize the data per day. Standardized-by-day data is used for training, for inference we store mean of all daily means and stds and standardize inference inputs using those aggregates (which also makes sense in our domain, there is no consistent trend in the data, but there are daily fluctuations in mean and std).</p>
<p>The current pipeline is the following:</p>
<p>Training:</p>
<ul>
<li>Load the data by day</li>
<li>On day i subtract mean and divide by std of all samples on that day</li>
<li>Store mean of daily means and mean of daily stds (for inference)</li>
<li>Concatenate all standardized samples over all days into a single tensor</li>
<li>Create torch.utils.data.Dataset and torch.utils.data.DataLoader which will shuffle the data and split into batches</li>
<li>Train the model</li>
</ul>
<p>Online inference:</p>
<ul>
<li>On every input sample subtract stored mean and divide by stored std</li>
<li>Evaluate prediction</li>
</ul>
<p>This works for us but the solution is not the most elegant, now there are two different code paths of getting data into the model (training and inference), training data is pre-processed while inference samples are passed raw.<br>
Ideally we would like to do per-day standardization in the torch.utils.data.Dataset or torch.utils.DataLoader  based on train/inference flag (train will estimate and store mean/std from the data, inference will use this values) or some other idiomatic alternative (maybe nn.Module that would be called inside forward call or torchvision.transformer-like?). We compile our models via TorchScript for online inference outside of Python environment, so it would be great to have the standardization step compiled as well rather than have it outside of the pipeline. I hope there is an idiomatic solution that would let us avoid different code paths in inference and training.</p>
<p>It is not immediately obvious how to do this in torch.utils.DataLoader (or any step after this) since it splits data into random batches (there is no way of getting daily mean or even day corresponding to the sample at this point) and there doesn’t seem to exist any solution for by-group (by-day) data transforms (standardization).</p>
<p>Any ideas on the best practices to do this?</p>
How to train hierarchical models in PyTorch

username: motam79
cooked: <p>I have a PyTorch architecture similar to this example</p>
<pre><code class="lang-auto">class MainModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.sub_model_1 = SubModel1()
        self.sub_model_2 = SubModel2()

    def forward(self, features1, features2):
        target_1 = self.sub_model_1(features1)
        target_2 = self.sub_model_2(target_1, features2)
        return dict(
            target_1=target_1,
            target_2=target_2,
        )


class SubModel1(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = nn.Linear(2, 1)

    def forward(self, features1):
        return self.layer(features1)


class SubModel2(nn.Module):

    def __init__(self):
        super().__init__()
        self.layer = nn.Linear(3, 1)

    def forward(self, target_1, features2):
        concatenated_features = torch.cat([target_1, features2], dim=1)
        target_2 = self.layer(concatenated_features)
        return target_2
</code></pre>
<p>I would like to train the first network (e.g. <code>SubModel1</code>) by providing <code>target_1_hat</code> and once that network os trained, I would like to fix to freeze the parameters of the model and try to optimizer the model <code>SubModel2</code> by providing <code>target_2_hat</code></p>
<p>Does PyTorch or a similar library has the option of training such models?</p>

username: InnovArul
cooked: <p>I do not see any complications with it.  It’s a design choice and you can write code to achieve that.<br>
I.e., you can choose to train <code>submodule1</code> first,  then <code>submodule2</code> or vice versa,  it’s all upto how you write the code, use the optimizer (or) loss functions appropriately etc. Do you face any issues while doing that?</p>
_augmentation_space gives invalid range to Solarize threshold

username: micahr123
cooked: <p>The threshold argument of the Solarize transform can’t accept values above 255 when the image is a uint8.  AutoAugment, RandAugmentcan, and TrivialAugmentWide produce augmentation spaces that go up to 256. This will give an error like:</p>
<p>File “/python3.9/site-packages/torchvision/transforms/autoaugment.py”, line 397, in forward<br>
return _apply_op(img, op_name, magnitude, interpolation=self.interpolation, fill=fill)<br>
File “/python3.9/site-packages/torchvision/transforms/autoaugment.py”, line 40, in _apply_op<br>
img = F.solarize(img, magnitude)<br>
File “/python3.9/site-packages/torchvision/transforms/functional.py”, line 1320, in solarize<br>
return F_t.solarize(img, threshold)<br>
File “/python3.9/site-packages/torchvision/transforms/functional_tensor.py”, line 874, in solarize<br>
return torch.where(img &gt;= threshold, inverted_img, img)<br>
RuntimeError: value cannot be converted to type uint8_t without overflow: 256</p>
<p>Please correct the code in the _augmentation_space function in the autoaugment.py file:<br>
wrong: “Solarize”: (torch.linspace(256.0, 0.0, num_bins), False),<br>
right: “Solarize”: (torch.linspace(255.0, 0.0, num_bins), False),</p>

username: ptrblck
cooked: <p>This issue seems to be already fixed in <a href="https://github.com/pytorch/vision/pull/4806">this PR</a> so you might want to update <code>torchvision</code> to the latest nightly or build from the current <code>main</code> branch.</p>
Losses in different ranks donot occur simultaneously when training with ddp

username: ginobilinie
cooked: <p>Hi, I’m now training a point cloud analysis model with distributed data parallel (ddp). I follow all the rules to create the ddp training. It is normal at the beginning, but the losses in different gpus donot occur consistently when it comes to the end of the 1st epoch, for example:</p>
<p>come to epoch: 0, step: 429, loss: 0.046092418071222385<br>
come to epoch: 0, step: 429, loss: 0.046092418071222385<br>
come to epoch: 0, step: 429, loss: 0.046092418071222385<br>
come to epoch: 0, step: 429, loss: 0.046092418071222385<br>
come to epoch: 0, step: 430, loss: 0.04677124587302715<br>
come to epoch: 0, step: 430, loss: 0.04677124587302715<br>
come to epoch: 0, step: 430, loss: 0.04677124587302715<br>
come to epoch: 1, step: 0, loss: 0.04677124587302715<br>
come to epoch: 0, step: 431, loss: 0.03822317679159113<br>
come to epoch: 0, step: 431, loss: 0.03822317679159113<br>
come to epoch: 1, step: 1, loss: 0.03822317679159113<br>
come to epoch: 0, step: 431, loss: 0.03822317679159113<br>
come to epoch: 0, step: 432, loss: 0.0431362825095357<br>
come to epoch: 0, step: 432, loss: 0.0431362825095357<br>
come to epoch: 0, step: 432, loss: 0.0431362825095357<br>
come to epoch: 1, step: 2, loss: 0.0431362825095357<br>
come to epoch: 0, step: 433, loss: 0.04170320917830233<br>
come to epoch: 1, step: 3, loss: 0.04170320917830233<br>
come to epoch: 0, step: 433, loss: 0.04170320917830233<br>
come to epoch: 0, step: 433, loss: 0.04170320917830233<br>
come to epoch: 0, step: 434, loss: 0.042295407038902666<br>
come to epoch: 0, step: 434, loss: 0.042295407038902666<br>
come to epoch: 1, step: 4, loss: 0.042295407038902666<br>
come to epoch: 0, step: 434, loss: 0.042295407038902666<br>
come to epoch: 0, step: 435, loss: 0.040262431528578634<br>
come to epoch: 1, step: 5, loss: 0.040262431528578634<br>
come to epoch: 0, step: 435, loss: 0.040262431528578634<br>
come to epoch: 0, step: 435, loss: 0.040262431528578634<br>
come to epoch: 0, step: 436, loss: 0.04188207677967013<br>
come to epoch: 0, step: 436, loss: 0.04188207677967013<br>
come to epoch: 0, step: 436, loss: 0.04188207677967013<br>
come to epoch: 1, step: 6, loss: 0.04188207677967013</p>
<p>Can anyone help?</p>
<p>Thanks.</p>

username: ginobilinie
cooked: <p>I think it is because the data partitioned to each gpu is not even.</p>
AMP for two optimizers

username: oasjd7
cooked: <p>Hi, all.<br>
I want to try apply AMP for the two optimizers .<br>
I just added all losses to <code>total_loss</code>.<br>
Below codes are correctly optimize the two optimizers?</p>
<pre><code class="lang-auto">optimizer.zero_grad()
optimizer2.zero_grad()
scaler.scale(total_loss).backward()
scaler.step(optimizer)
scaler.step(optimizer2)
scaler.update()
</code></pre>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>Yes, your code should work. The <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#working-with-multiple-models-losses-and-optimizers">docs</a> also give an example which fits your use case.</p>
Minimizing time moving tensors between CPU and GPU

username: jxmorris12
cooked: <p>I have a very large buffer of images (~1 million). They are all different. I can’t fit all of the images on the GPU, so they’re all on the CPU originally. Then, I sample from the buffer, do some things with the sampled images, and repeat. Something like (pseudocode):</p>
<pre><code class="lang-auto">buffer = torch.ones(10**6, 32, 32, 3) # images are 32x32x3
for step in range(10**8):
  images = sample(buffer, 256).to('cuda')
  update_model_with_images(images)
</code></pre>
<p>There’s no way to predict which images will be sampled, so I can’t pre-fetch them. When I profile my code using <code>torch.utils.bottleneck</code>, I find that almost all of the runtime is spent on moving tensors from CPU to GPU. This is frustrating. How can I increase performance without fitting the entire buffer on my GPU?</p>

username: ptrblck
cooked: <p>If the workload in your model is small, you would be limited by the memory bandwidth between the host and device.<br>
I would recommend to manually profile the code and check, if the data transfer is really slower than the model execution via:</p>
<pre><code class="lang-python">nb_iters = 100
torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    images = buffer[batch_size].to('cuda')
torch.cuda.synchronize()
t1 = time.time()
print((t1 - t0) / nb_iters)
</code></pre>
<p>and the same for the model.</p>

username: pengzhi1998
cooked: <p>Sorry to bother you. But I have a very strange issue when using pytorch with cuda.</p>
<p>The code is used to predict multiple rgb-images’ depth information in a for loop. And during testing, in each loop, the network 1. predicts the depth information and then 2. transfer the tensor to cpu. However, either the first step or the second step will cost a lot of time ( almost 0.5 seconds). But they won’t take a long time at the same time. And more strangely, if I comment the transfer code, there might some other steps which take a rather long time to compute (seems the bottleneck is changing from time to time).</p>
<p>It seems not the problem of the code but might be an issue caused by pytorch. And I have tried <code>torch.cuda.synchronize()</code> which didn’t work. Do you have any idea about this wired situation?</p>

username: ptrblck
cooked: <p>Based on your description it sounds as if you are indeed not properly synchronizing the code while trying to profile it and are thus “moving” the time allocations around. To profile operations you need to synchronize the code before starting and stopping each timer. Otherwise your timers could only profile the kernel launch and would accumulate the entire runtime in the next blocking call.</p>
New version opacus without dataloader

username: Dibbla_XU
cooked: <p>Hi!</p>
<p>I’m using Opacus to train my model. The version before 1.0.0 works fine without dataloader. However, it seems that the new version privacy engine requires one at initialization. Is there any way to avoid this?</p>
<p>Thx!</p>

username: ffuuugor
cooked: <p>Hi!<br>
Thanks for using opacus.</p>
<p>There’s certainly a way to use opacus without DataLoader - please refer to our <a href="https://github.com/pytorch/opacus/blob/main/Migration_Guide.md#no-dataloader" rel="noopener nofollow ugc">Migration Guide, section “No DataLoader”</a></p>
<p>Feel free to ask any further questions here</p>

username: Dibbla_XU
cooked: <p>thx for reply!<br>
However, as I’m following the guide, something seems to get wrong:</p>
<pre><code class="lang-auto">optimizer.attach_step_hook(…
</code></pre>
<p>However, there’s an error claim that SGD has no such attribute. I use SGD from torch.optim.SGD. Any idea about the situation?</p>

username: ffuuugor
cooked: <p>Whoops, thanks for pointing this out - that’s actually a mistake on migration guide. You need to do</p>
<pre><code class="lang-auto">dp_optimizer.attach_step_hook(...
</code></pre>
<p>See <a href="https://github.com/pytorch/opacus/pull/307" rel="noopener nofollow ugc">PR #307</a></p>

username: Dibbla_XU
cooked: <p>Thx, the guide works fine for the problem. However, there’s a new one. I attached the privacyengine to the optim, but when I ran the code, it reported a CUDA out of memory.</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 6.10 GiB (GPU 0; 23.70 GiB total capacity; 13.27 GiB already allocated; 3.20 GiB free; 18.51 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
</code></pre>
<p>So I then check the my graphics using nvidia-smi. I got 2 rtx3090 on the server, both available with 24gb memory. Then I ran the code with older opacus, basically the same but the way to attach privacyengine. The older one works.</p>
<p>Trace back reported something about backward() having trouble. Any idea?</p>

username: ffuuugor
cooked: <p>One potential reason this could be happening is poisson sampling. If you’re using default setup with Opacus 1.0, then your dataloader is using a non-standard sampler (<code>UniformWithReplacementSampler </code>). (it was available before, but had to be manually enabled).</p>
<p>With this sampler batches are of variable size: the average size is still the same, but some batches are larger than average. This leads to an increased memory requirement, as your memory is limited by the peak batch size, not average.<br>
We have a utility to handle this: <code>BatchMemoryManager</code> (see example <a href="https://github.com/pytorch/opacus/blob/main/Migration_Guide.md#if-youre-using-virtual-steps" rel="noopener nofollow ugc">here</a>)</p>
<p>If it’s not that, then it’s interesting and we’ll need to double check and see what could’ve caused it.</p>
Test Predictions are very random

username: jojojo
cooked: <p>Hi guys,<br>
I am working on a OCR model. My vaidation accuracy is near 80 percent but still test predictions are very wrong. Not able to predict even one. Any suggestions?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41d28a10145d995894e41def14fcc6c7161a4063.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/41d28a10145d995894e41def14fcc6c7161a4063" title="IMG_20211210_154348"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41d28a10145d995894e41def14fcc6c7161a4063_2_375x500.jpeg" alt="IMG_20211210_154348" data-base62-sha1="9oi745gEWIqttGwJAyVUgjdbAMr" width="375" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41d28a10145d995894e41def14fcc6c7161a4063_2_375x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41d28a10145d995894e41def14fcc6c7161a4063_2_562x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41d28a10145d995894e41def14fcc6c7161a4063_2_750x1000.jpeg 2x" data-dominant-color="525962"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20211210_154348</span><span class="informations">1920×2560 337 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
0 I have some problem with getting the output gradient with respect to input. It is simple mnist model

username: kHan
cooked: <p>I have some problem with getting the output gradient w.r.t input. It is simple mnist model.<br>
I already add .reuires_grad to input.<br>
but it doesn’t work.<br>
sample_img.grad  is None</p>
<pre><code class="lang-auto">    sample_img, sample_label = mnist_test[0]

    sample_img = sample_img.to(device)
    sample_img.requires_grad = True
    prediction = model(sample_img.unsqueeze(dim=0))
    cost = criterion(prediction, torch.tensor([sample_label]).to(device))
    optimizer.zero_grad()
    cost.backward()
    print(sample_label)
    print(sample_img.shape)

    plt.imshow(sample_img.detach().cpu().squeeze(),cmap='gray')
    plt.show()

    print(sample_img.grad)
</code></pre>
<p>Thank you!</p>

username: kHan
cooked: <p>Please anyone help me</p>

username: mMagmer
cooked: <p>i have no proplem with following code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

model = nn.Sequential(nn.Linear(28*28,10))
optimizer = torch.optim.SGD(model.parameters(), 0.03)
criterion = nn.CrossEntropyLoss()

sample_img, sample_label = torch.randn(1,28,28) , 0

sample_img.requires_grad = True
prediction = model(sample_img.view(-1).unsqueeze(dim=0))
cost = criterion(prediction, torch.tensor([sample_label]))
optimizer.zero_grad()
cost.backward()
print(sample_label)
#print(sample_img.shape)

print(sample_img.grad)
</code></pre>
<p>maybe it’s your model that cuasing problem??</p>

username: kHan
cooked: <p>I solve this problem.</p>
<p>I don’t know why it didn’t work.<br>
but now it works…haha<br>
Thank you!!</p>
How can i concatenate or ensemble two features separately?

username: prog_asker
cooked: <p>I got the below runtime error while I tried to ensemble resnet50 with another model using ensemble class</p>
<pre><code class="lang-auto">return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 5.80 GiB total capacity; 191.43 MiB already allocated; 16.56 MiB free; 202.00 MiB reserved in total by PyTorch)
</code></pre>
<p>how can I concatenate between them if I have their features separately? does this should work</p>
<pre><code class="lang-auto">class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB):
        x1 = self.modelA(x.clone())  
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x)
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        dump(x, open('result.pkl', 'wb'))      
        return x
with open(r"v1.pkl", "rb") as input_file:
    modelA=cPickle.load(input_file)
with open(r"v2.pkl", "rb") as input1_file1:
    modelB=cPickle.load(input1_file1)
model = MyEnsemble(modelA, modelB)
</code></pre>
How to translates torch::tensor type into base type in c++?

username: MartinZhang
cooked: <p><strong>A question about how to translate torch::tensor type into base type in c++?</strong></p>
<p>Such as:</p>
<pre><code class="lang-auto">
torch::Tensor(False) -&gt; bool(False);

torch::Tensor(123) -&gt; int(123);  

torch::Tensor(3.1415) -&gt; double(3.1415);

</code></pre>
<p>or like tensor function:</p>
<pre><code class="lang-auto">int a = torch.tensor(123).data  // there is not data elemenet
</code></pre>
<p>please help me, Thanks. <img src="https://discuss.pytorch.org/images/emoji/apple/handshake.png?v=10" title=":handshake:" class="emoji" alt=":handshake:"></p>

username: ptrblck
cooked: <p><code>int a = torch.tensor(123).item&lt;int&gt;();</code> should work.</p>

username: MartinZhang
cooked: <p>awesome！！！awesome！！！<br>
Thank you very much！！！ <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=10" title=":+1:" class="emoji" alt=":+1:"></p>
Need help：RuntimeError: CUDA out of memory

username: oneluxyou
cooked: <p>RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 4.00 GiB total capacity; 2.22 GiB already allocated; 94.64 MiB free; 2.30 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>

username: ptrblck
cooked: <p>The error message explains that your GPU is running out of memory and you would thus need to reduce the memory usage by e.g. lowering the batch size.</p>

username: oneluxyou
cooked: <p>Thanks for ur reply.<br>
enn，I wonder why would pytorch reserve 2.3GB memory?</p>

username: ptrblck
cooked: <p>PyTorch allocated memory to store the model parameters, the input tensors, intermediate forward activations etc. Depending on the used model, input shape etc., this reserved memory would be higher or lower.</p>

username: oneluxyou
cooked: <p>Many thanks <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=10" title=":grinning:" class="emoji" alt=":grinning:"> for your detailed answer</p>

username: ptrblck
cooked: <p>Also, in case I misunderstood the question: if you are wondering about the difference between “memory allocated” and “reserved”: the latter reports the allocated and cached memory. Take a look at <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management">these docs</a> for more information.</p>
Ordering of elements when using torch.flatten() on 4D arrays

username: Mole_m7b5
cooked: <p>I have a 4D tensor of shape <code>[32,64,64,3]</code> which corresponds to <code>[batch, timeframes, frequency_bins, features]</code> and I do <code>tensor.flatten(start_dim=2)</code>. I understand the shape will then transform to <code>[32,64,64*3] --&gt; [batch,timeframes,frequency_bins*features]</code> -  but in terms of the actual ordering of the elements within that  new flattened dimension of <code>64*3</code> are the first <code>64</code> indexes relating to what would have been <code>[:,:,:,0]</code> the second 64 <code>[:,:,:,1]</code> and the final 64<code> [:,:,:,2]</code></p>

username: ptrblck
cooked: <p>The ordering can probably be best seen in this example:</p>
<pre><code class="lang-python">x = torch.arange(1*4*4*3).view(1, 4, 4, 3)
print(x)
&gt; tensor([[[[ 0,  1,  2],
            [ 3,  4,  5],
            [ 6,  7,  8],
            [ 9, 10, 11]],

           [[12, 13, 14],
            [15, 16, 17],
            [18, 19, 20],
            [21, 22, 23]],

           [[24, 25, 26],
            [27, 28, 29],
            [30, 31, 32],
            [33, 34, 35]],

           [[36, 37, 38],
            [39, 40, 41],
            [42, 43, 44],
            [45, 46, 47]]]])
y = torch.flatten(x, start_dim=2)
print(y)
&gt; tensor([[[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11],
           [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
           [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35],
           [36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]]])
</code></pre>
<p>As you can see, the last two dimensions (the “squares” in <code>x</code>) will be flattened to rows in <code>y</code>.</p>
Confusions regarding Normalization and Loading the model

username: Sumera_Rounaq
cooked: <p>Question 1: In many research papers, researchers tend to suggest normalization of images from [-1 to 1] . How can we achieve it in code?</p>
<p>Question 2: What is the meaning of following line of code?<br>
<code>transforms.Normalize(0.5, 0.5)</code></p>
<p>Question 3: Let say I trained my model with learning rate: 0.0001 and then save the state of my model. Now I want to change learning rate to 0.00002 so should I start training from scratch? Or I could load the previous model ?</p>

username: ptrblck
cooked: <ol>
<li>Something like this should work:</li>
</ol>
<pre><code class="lang-python">x = torch.randn(10, 10, 10)
y = x - x.min()
y = y / y.max() * 2 - 1
print(y.min(), y.max())
&gt; tensor(-1.) tensor(1.)
</code></pre>
<ol start="2">
<li>
<p><a href="https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.Normalize"><code>Normalize</code></a> subtracts the mean and divides by the <code>stddev</code> to create normalized/standardized outputs, which are also known as z-scores.</p>
</li>
<li>
<p>It depends on your use case and yes, you can also load the already pretrained model and continue the training with the lower learning rate.</p>
</li>
</ol>
CTC Loss - for all samples

username: Vishak_Raj
cooked: <p>Hello,</p>
<p>I am learning CTC Loss from <a href="https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CTCLoss — PyTorch 1.10.0 documentation</a>,</p>
<p>If, I have 100 samples, then I got ctc loss for all samples or all samples in the batch</p>
<p><code>ctc_loss = torch.nn.CTCLoss()</code></p>
<p><code>loss = ctc_loss(input, target, input_lengths, target_lengths)</code></p>
<p><code>loss.backward()</code></p>
<p><code>print(loss) -&gt; tensor(7.8303, grad_fn=&lt;MeanBackward0&gt;)</code></p>
<p>But, How should I get CTC Loss for every sample instead of getting CTC Loss for all samples or batch ?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Use <code>torch.nn.CTCLoss(reduction='none')</code> to get the loss for each sample.</p>

username: Vishak_Raj
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for the solution,</p>
<p>Do you know how to add weights to the loss when calculating CTC loss?</p>

username: ptrblck
cooked: <p>After using <code>reduction='none'</code> you would get the unreduced loss and could multiply it with your weights before reducing it to a scalar loss value.</p>

username: Vishak_Raj
cooked: <p>yes, I figured that, thanks for helping <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Mixed precision training is so slow when deterministic=True

username: MeiMei_Liu
cooked: <p>Hi folks, is there a way to set deterministic=True without get slow training using mixed precision ? is it a bug?</p>
<p>Here are my benchmarks using RTX 3070 (pytorch 1.10 - cuda 11.3):<br>
deterministic=False     43.032ms per iter  vs<br>
deterministic=True      354.005ms per iter</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.cuda.amp import autocast, GradScaler
import time
from torchvision.models import mobilenet_v2

class model(nn.Module):
    def __init__(self):
        super(model, self).__init__()
        self.mobilenet = mobilenet_v2()
        self.linear = nn.Sequential(nn.Linear(1000, 250),
                                    nn.Linear(250, 64),
                                    nn.Linear(64, 32),
                                    nn.Linear(32, 10),
                                    nn.ReLU())

    def forward(self, x):
        out = self.mobilenet(x)
        out = self.linear(out)
        return out

x = torch.randn(32, 3, 224, 224).cuda(0)
y = torch.randint(0, 10, (32,)).cuda(0)
m = model().cuda(0)
criterion = nn.CrossEntropyLoss()
optim = torch.optim.Adam(m.parameters(), 1e-4)
scaler = GradScaler()

nb_iters = 100

def train16(nb_iters=100, deterministic=False):
    torch.backends.cudnn.deterministic = deterministic
    torch.cuda.synchronize()
    t0 = time.time()
    for _ in range(nb_iters):
        optim.zero_grad()
        with autocast(enabled=True):
            output = m(x)
            loss = criterion(output, y)
        scaler.scale(loss).backward()
        scaler.step(optim)
        scaler.update()
    loss.detach()
    torch.cuda.synchronize()
    t1 = time.time()
    print('{:.3f}ms per iter'.format((t1 - t0)/nb_iters * 1000.))

train16(nb_iters, deterministic=True)
train16(nb_iters)
</code></pre>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>No, as deterministic algorithms could yield a slower performance to guarantee deterministic results due to the internal kernel implementation.</p>
LSTM gradient shape

username: test123
cooked: <p>Hi,</p>
<p>I’m trying to understand LSTMs and backpropagation with LSTMs. I have a simple architecture.</p>
<pre><code>def __init__(self, input_dim=100, hidden_dim=100, output_dim=216, num_layers=2):
    super(LSTMGenerator, self).__init__()
    self.input_dim = input_dim
    self.hidden_dim = hidden_dim
    self.output_dim = output_dim
    self.num_layers = num_layers
    self.layer1 = nn.LSTM(self.input_dim, self.hidden_dim, self.num_layers, bidirectional=False, dropout=0.5)
    self.out = nn.Linear(self.hidden_dim, self.output_dim)
</code></pre>
<p>I’m using the pytorch register_backward_hook function. I’m trying to modify the gradient in the backward pass. Should I use grad_input or grad_output? Which gradient is being used to update the previous weights?</p>
<p>I printed out the gradient shapes. What do the values in grad_input and grad_output mean?</p>
<p>length of grad input  3<br>
grad input 0 shape  torch.Size([216])<br>
grad input 1 shape  torch.Size([128, 100])<br>
grad input 2 shape  torch.Size([100, 216])</p>
<p>length of grad output  1<br>
grad ouput 0 shape  torch.Size([128, 216])</p>
<p>Any help would be appreciated.</p>