akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to convert a list of cuda tensors to a list of cpu tensors?

Suraj: <p>So I have a big list of cuda tensors containing loss values of each iteration but I can’t even scatter plot it. How can I convert it to a list of cpu tensors or even numpy array without iterating over the indices? Thank You!</p>

Nikronic: <p>Hi,</p>
<p>In this post I have written a code to transfer a list of CPU tensors to GPU which all have the same shape (in your case all have shape of <code>(1, )</code>).</p>
<aside class="quote" data-post="1" data-topic="88336">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/omid_shams/48/16113_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/input-a-list-of-tensors-to-a-model-without-the-need-to-manually-transfer-each-item-to-cuda/88336">Input a list of tensors to a model without the need to manually transfer each item to cuda</a> 
  </div>
  <blockquote>
    Hi, 
I’d like to input a list of tensors to my model. However, pytorch doesn’t automatically transfer each item in this list to cuda and I need to manually do that each time by .cuda() . Is there a better more elegant way of dealing with this?
  </blockquote>
</aside>

<p>Bests</p>

Suraj: <p>Thank you so much <a class="mention" href="/u/nikronic">@Nikronic</a> .</p>
<p>Just creating a new tensor with torch.tensor() worked. Then simply plotted the scatter plot on torch tensor (with device =  cpu).</p>
<p><code>new_tensor = torch.tensor(list_of_cuda_tensors, device =  'cpu')</code></p>

chethanjjj: <p>But, what if you want to keep it as a list of tensors after the transfer from gpu to cpu</p>

Thilani: <p>Thanks! This helped me with my code which ran on cpu but running a plot in gpu. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

nlgranger: <p>In a context where performance is a concern, you’d be better off stacking the scalar tensors first then moving to cpu: <code>torch.stack(list_of_losses).cpu()</code>. Otherwise <code>[t.item() for t in list_of_losses]</code> seems more idiomatic to me if you want the result as a list of floats.</p>
How can we rebuild images from a Generator in a GAN model?

setsailforfail: <p>I followed the paper : <a href="https://arxiv.org/pdf/1511.06434.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1511.06434.pdf</a> to build a GAN model from scratch. The version is very similar to the tutorial available on PyTorch: <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">DCGAN Tutorial — PyTorch Tutorials 2.0.0+cu117 documentation</a></p>
<p>There is one thing that confuses me though, the Generator transforms noise into a fake image, however, the Generator uses the tanh activation function, thus the values of the fake image are in the range between -1 and 1. Displaying this as an RGB image, don’t we require values between 0 and 255? On the tutorial I cannot really detect that step from (-1, 1) to (0, 255).</p>
<p>After training the Generator for some time, how can we recover the pixels from those values to actually see a real image? I think I lack some knowledge here, so sorry if that question is rather stupid or basic!</p>

ptrblck: <p>The tutorial uses <code>torchvision.utils.make_grid</code> with <code>normalize=True</code>, which will - according to the <a href="https://pytorch.org/vision/stable/generated/torchvision.utils.make_grid.html#torchvision.utils.make_grid">docs</a> - normalize the image to the range <code>[0, 1]</code> using the <code>min</code> and <code>max</code> values from <code>value_range</code>. By default <code>value_range</code> will be set to the <code>min/max</code> values of the input tensor unless its set to specific values:</p>
<blockquote>
<ul>
<li><strong>normalize</strong> (<a href="https://docs.python.org/3/library/functions.html#bool"><em>bool</em></a><em>,</em> <em>optional</em>) – If True, shift the image to the range (0, 1), by the min and max values specified by <code>value_range</code>. Default: <code>False</code>.</li>
<li><strong>value_range</strong> (<a href="https://docs.python.org/3/library/stdtypes.html#tuple"><em>tuple</em></a><em>,</em> <em>optional</em>) – tuple (min, max) where min and max are numbers, then these numbers are used to normalize the image. By default, min and max are computed from the tensor.</li>
</ul>
</blockquote>
<p>Here is a code example showing this behavior:</p>
<pre data-code-wrap="python"><code class="lang-python">output = (torch.rand(3, 224, 224)  * 2.) - 1.
print(output.min())
# tensor(-1.0000)
print(output.max())
#tensor(1.0000)

out = make_grid(output)
print(out.min())
# tensor(-1.0000)
print(out.max())
# tensor(1.0000)

out = make_grid(output, normalize=True)
print(out.min())
# tensor(0.)
print(out.max())
# tensor(1.0000)
</code></pre>
How to use pytorch Transformer to do categorical classification

LXR: <p>Hi, I intend to build a multi-class classifier using a transformer.<br>
The input data is an audio spectrogram for example MFCC in (batch, steps, features).<br>
The labels are just 0,1,2,3.</p>
<p>Can this be achieved using torch.nn.Transformer?<br>
Or should I write the transformer by myself especially the decoder?<br>
This is because the example in <a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html" rel="noopener nofollow ugc">Pytorch Transformer</a> suggests the ‘tgt’ is also a 3D array.  I believe this is for NLP.</p>
Libtorch tutorial : Fixing resizing warning / reshape 101

firelight: <p>While completing the first c++ tutorial with recent libtorch, I hit the resize warning described here : <a href="https://github.com/pytorch/examples/issues/819" class="inline-onebox" rel="noopener nofollow ugc">DCGAN C++ warning after PyTorch update · Issue #819 · pytorch/examples · GitHub</a></p>
<p>One of the comments mentions reshaping with <code>fake_labels.size()</code> (instead of batch_size), which I promptly tried. However, the training results seem underwhelming compared to the expected results posted in the guide (trained 60 epochs). Here are some beginner questions related to this.</p>
<ul>
<li>It’s unclear to me how the batch size differs from fake_labels.size(). Are we talking only portability / code robustness here, or would they ever differ?</li>
<li>Assuming I got it right and I can simply think of tensors’ data as multi-dimentional arrays (to start), how does one reshape their array dimensions without data loss?</li>
<li>Can I investigate the loaded dataset schema? This tutorial uses MNIST (characters), but the memory layout / loaded data is a black-box, which makes it really difficult to understand how the batching works. Can you print / reflect / debug dataset schemas?</li>
</ul>
<p>Any links to other documentation appreciated, thank you!</p>
<p>The tut : <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" class="inline-onebox" rel="noopener nofollow ugc">Using the PyTorch C++ Frontend — PyTorch Tutorials 2.0.0+cu117 documentation</a></p>
<p>My reshape changes :</p>
<pre><code class="lang-auto">			// Train discriminator with real images.
			discriminator-&gt;zero_grad();
			torch::Tensor real_images = batch.data.to(device);
			torch::Tensor real_labels = torch::empty(batch.data.size(0), device)
												.uniform_(0.8, 1.0);
			torch::Tensor real_output = discriminator-&gt;forward(real_images)
												.reshape({ batch_size });
			torch::Tensor d_loss_real
					= torch::binary_cross_entropy(real_output, real_labels);
			d_loss_real.backward();

			// Train discriminator with fake images.
			torch::Tensor noise = torch::randn(
					{ batch.data.size(0), k_noise_size, 1, 1 }, device);
			torch::Tensor fake_images = generator-&gt;forward(noise);
			torch::Tensor fake_labels
					= torch::zeros(batch.data.size(0), device);
			torch::Tensor fake_output
					= discriminator-&gt;forward(fake_images.detach())
							  .reshape(fake_labels.sizes());
			torch::Tensor d_loss_fake
					= torch::binary_cross_entropy(fake_output, fake_labels);
			d_loss_fake.backward();

			torch::Tensor d_loss = d_loss_real + d_loss_fake;
			d_optimizer.step();

			// Train generator.
			generator-&gt;zero_grad();
			fake_labels.fill_(1);
			fake_output = discriminator-&gt;forward(fake_images)
								  .reshape(fake_labels.sizes());

</code></pre>
Pre-trained ViT as backbone for segmentation network

Dimitris_Z: <p>Hi,<br>
I am trying to replicate the Segmenter architecture [1] and use it over the TuSimple lane detection dataset. The segmenter model i adapted to my needs consists of a ViT B-16 Backbone network and an MLP (3 layers) classification head, which i use to identify the presence of the lane class on the output patches (as embeddings) from the ViT. Then i planned to interpolate the patch level predictions to pixel level as the authors did in the original paper.</p>
<p>As pre-processing i resized the TuSimple dataset to 640x640 images (from the original 1280x720) along with their respective ground truths by bilinear interpolation. For ViT i did load pre-trained weights  and interpolated(using the ‘nearest’ mode) the pretrained positional embeddings to match my new image input dimensions.</p>
<p>To my understanding, I then had to fine-tune the pipeline by training the patch classification head at first to a reasonable performance and then if needed unfreeze some transformer layers and fine tune the pre-trained ViT also. However, I am struggling to even get a 0.01 F1 or IoU score after 10-20 epochs of training the classification head using a learning rate of 0.1 at first (as i did attempt to overfit the model as a first experiment to see if i am on the right track).</p>
<p>What could I be doing wrong?</p>
<p>Thanks in advance</p>
<p>p.s. for any code dont hesitate to ask me<br>
ps2. I am using the BCELosswithLogics() loss function with pos weight = 1900 for tha lane class(after calculation of total amount of background pixels/lane pixels)<br>
ps3. Pre-trained weights from <a href="https://github.com/huggingface/pytorch-image-models/releases/download/v0.1-vitjx/jx_vit_base_p16_224-80ecf9dd.pth" rel="noopener nofollow ugc">https://github.com/huggingface/pytorch-image-models/releases/download/v0.1-vitjx/jx_vit_base_p16_224-80ecf9dd.pth</a></p>
<p>[1] <a href="https://arxiv.org/abs/2105.05633" class="inline-onebox" rel="noopener nofollow ugc">[2105.05633] Segmenter: Transformer for Semantic Segmentation</a></p>
Dice score for multi class segmentation on medical images

ankita.buntolia: <p>Hi</p>
<p><strong>Background:</strong><br>
I am working on the highly imbalanced medical image (OCT Dataset) where the ground truth for each pixel is one of these values (0 for background, and 1,2, and 3 for type of fluid). so it’s a multiclass segmentation problem.</p>
<p><strong>My question is:</strong><br>
For the model performance check, I am using a dice score. Should I include background for dice score calculation for all (mean overall classes) or not?<br>
If I do not include background for the dice score calculation then how should handle those pixels whose ground truth value was 0 (they only had background) but the model predicted those pixels to one of the fluids (values 1,2, or 3)?</p>
__getitem__ forces return of a different datatype

Michael_Sovich: <p>Hello,</p>
<p>Could someone help me understand why the outputs are not the same? The point is that I want to do all the necessary processing in the getitem itself and not outside, but no matter what I do, it returns not the data type I wanted.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/9/39caf190983b893b8c77c16650de5ca23503ecef.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/39caf190983b893b8c77c16650de5ca23503ecef" title="problem"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39caf190983b893b8c77c16650de5ca23503ecef_2_582x500.jpeg" alt="problem" data-base62-sha1="8fg1gNMTq7rOuUeRrc8yczDW5Ub" width="582" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39caf190983b893b8c77c16650de5ca23503ecef_2_582x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39caf190983b893b8c77c16650de5ca23503ecef_2_873x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/9/39caf190983b893b8c77c16650de5ca23503ecef.jpeg 2x" data-dominant-color="EFF0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">problem</span><span class="informations">893×767 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks.</p>

ege_b: <p>When you enumerate the loader it returns you a tuple <code>(&lt;index&gt; , (&lt;data&gt;, &lt;label&gt;))</code>. So if you unpack the loader like this: <code>for i, (batch, _) in enumerate(val_loader):</code>,  you will get the the data you want.</p>
[Skorch] How can I make a score be logged in TensorBoard?

GeorgeIrwin: <p>I am using <a href="https://github.com/skorch-dev/skorch" rel="noopener nofollow ugc">Skorch</a>.</p>
<p>I created a new score logging using the <a href="https://skorch.readthedocs.io/en/stable/callbacks.html#skorch.callbacks.EpochScoring" rel="noopener nofollow ugc"><code>EpochScoring</code></a> callback.<br>
While it appears on the history log, It doesn’t show on TensorBoard (I am using the <code>TensorBoard</code> callback).</p>
<p>Is there a way to make the TensorBoard logger be aware of that key?</p>
Keras predict number, please code

ChrisTsao: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d152c6d3d489bc8fd7c33d9fc9cf81ac9a6d4d3.png" data-download-href="https://discuss.pytorch.org/uploads/default/5d152c6d3d489bc8fd7c33d9fc9cf81ac9a6d4d3" title="1679213194512"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d152c6d3d489bc8fd7c33d9fc9cf81ac9a6d4d3_2_690x356.png" alt="1679213194512" data-base62-sha1="dhrLFtUjSZiSjuy2SpgrMbRIlPl" width="690" height="356" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d152c6d3d489bc8fd7c33d9fc9cf81ac9a6d4d3_2_690x356.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d152c6d3d489bc8fd7c33d9fc9cf81ac9a6d4d3_2_1035x534.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d152c6d3d489bc8fd7c33d9fc9cf81ac9a6d4d3.png 2x" data-dominant-color="2D2D28"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1679213194512</span><span class="informations">1239×641 268 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>as graph shown , input xs and output ys<br>
xs=2 FOR ys=100<br>
xs=3 FOR ys=200<br>
xs=4 FOR ys=300</p>
<p>NOW, I want a create new array， 7 numbers in xs FOR 1 number in ys, all number in range 1-49</p>
<p>for example<br>
xs=[1,3,11,15,16,28,32] FOR ys=29<br>
xs=[2,6,33,42,13,22,39] FOR ys=8<br>
xs=[35,26,6,40,48,17,10] FOR ys=35<br>
…<br>
…<br>
…<br>
//300 set of data for taining</p>
<p>and machine should predict  24 numbers as ys in range 1-49 according to xs (7 numbers)</p>
<p>for example<br>
we have xs=1，2，5，16，22，32，40，ys=5<br>
and model  predict ys=3，5，7，8，9，10，11，12，13，18，20，22，25，26，32，30，33，35，36，37，40，41，42，43（24 numbers，and 5 is predicted，success, if no 5 in 24 numbers, then fail ）</p>
<p>meanwhile ,  calculate the success rate of prediction</p>
<p>PLEASE do a little help to code for this …thanks</p>

ptrblck: <p>Since you are using Keras and Tensorflow I would recommend asking in their discussion board since you will find the experts there. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ChrisTsao: <p>OK，thanks, i’m new …</p>
cuDNN error with cuda 11.7

tinaalask: <p>Hi everyone. I should run a python program and I should create a VENV as follows:</p>
<pre><code>    python=3.6, 
    torch==1.0.1.post2
</code></pre>
<p>enviromnet:</p>
<p>NVIDIA-SMI 515.86.01<br>
Driver Version: 515.86.01<br>
CUDA Version: 11.7<br>
rtx 3090</p>
<p>I install other packages with</p>
<pre><code class="lang-auto">pip install -r requirements.txt
</code></pre>
<p>all of the package install successfully.</p>
<p>but when I run that program with below code</p>
<pre><code class="lang-auto">    python main_test.py
</code></pre>
<p>I encounter below error.</p>
<pre><code class="lang-auto">RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
</code></pre>
<p>I think it doesn’t match between cudnn and cuda but I don’t know how to match them.</p>
<p>Or I may be wrong and the error lies elsewhere.</p>

ptrblck: <p>Your installed PyTorch version is <code>1.0.1.post2</code> which was built with an older CUDA toolkit and cuDNN version. Also note that the PyTorch binaries ship with their own CUDA runtime dependency and will use these libs. The locally installed CUDA toolkit will be used if you build PyTorch from source or custom CUDA extensions.<br>
Update PyTorch to the latest stable <code>2.0.0</code> release and check if you are still seeing the issue, since <code>1.0.1</code> was released in early 2019.</p>

tinaalask: <p>Thank you so much for your great guide. My problem was one of the packages named spacy==2.0.18 and I couldn’t install on python 3.10,3.9. With try installing on different version of pythons 3.10,3.9,3.8,3.6 I could install that package on python==3.8 and could install Pytorch=11.7 on it too.</p>
Jit.trace not working for ViT model

ssh9919: <p>torch.jit.trace is not working for torchvision vision transformer models, i.e. vit_b_16, vit_b_32, and vit_l_16, vit_l_32. It made a TracingCheckError.</p>

AtraHasis: <p>This is expected for ViT models. Just use jit.trace() instead. For example:<br>
<strong>ViT_b_16_script = torch.jit.script(ViT_b_16)</strong></p>
Erroneous in normal runs but bug-free in debug mode

jasperhyp: <p>The error is: <code>RuntimeError: Mask shape should match input shape; transformer_mask is not supported in the fallback case</code>.</p>
<p>I am using TransformerEncoder for a very very simple task – encoding a batch of sequences only 6 tokens long.<br>
Some of the tokens are noninformative, so I am masking them out with <code>src_key_padding</code>.  The layers are defined as simple as:</p>
<pre><code class="lang-python">encoder_layer = nn.TransformerEncoderLayer(d_model=self.latent_dim, nhead=transformer_att_heads, dim_feedforward=transformer_ffn_dim, dropout=dropout, activation=transformer_actn, norm_first=norm_first, batch_first=batch_first)  # We set batch_first to False to disable "fast path" (which leads to NestedTensor that can't be hooked) and enable the use of hook
self.transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=transformer_num_layers)
</code></pre>
<p>and in <code>forward()</code>, it’s also as simple as:</p>
<pre><code class="lang-python">out = self.transformer_encoder(src = sequence_embedded, src_key_padding_mask = mask)
</code></pre>
<p>where sequence_embedded is of [1300, 6, 256] and mask is of [1300, 6] (a bool matrix, I make sure no rows sum to 6, so that each sample has at least one token to be passed into Transformer).</p>
<p>And then the error comes:<br>
<code>RuntimeError: Mask shape should match input shape; transformer_mask is not supported in the fallback case</code>.</p>
<p>The weird thing is that when I use VSCode debug mode and run this command again when the bug pops, it works perfectly!!  But I just couldn’t make it work natively.  Any thoughts?</p>
Update weights due to gradients coming from one path and not from another

spabho: <p>Hello,</p>
<p>I have a network with 2 parallel heads and each head predicts a different entity and hence they have different losses. I want the backbone to get updated by gradients coming from one head but not the other. How can I do this?</p>
<p>(Attaching a figure explaining the same)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee1c9281f73fcc2efcbfd7882daa152de2906e40.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee1c9281f73fcc2efcbfd7882daa152de2906e40" title="grad_flow"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee1c9281f73fcc2efcbfd7882daa152de2906e40_2_517x291.png" alt="grad_flow" data-base62-sha1="xYqQW76cAn0fRvFCm2LslAy4Io0" width="517" height="291" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee1c9281f73fcc2efcbfd7882daa152de2906e40_2_517x291.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee1c9281f73fcc2efcbfd7882daa152de2906e40_2_775x436.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee1c9281f73fcc2efcbfd7882daa152de2906e40_2_1034x582.png 2x" data-dominant-color="F9F9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grad_flow</span><span class="informations">1280×720 10.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks!</p>

ptrblck: <p>You could <code>.detach()</code> for output of <code>backbone</code> before sending it to <code>Head2</code>, which would cut the computation graph there.</p>
When i trained a neural network i found something interesting!

Teddy_Salas: <p>When i trained a neural network ,<br>
code:<br>
class linear_regression(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.weight = nn.Parameter(torch.rand(1,<br>
requires_grad = True,<br>
dtype=torch.float))<br>
self.bias = nn.Parameter(torch.rand(1,<br>
requires_grad=True,<br>
dtype=torch.float))<br>
in dtype i give dtype=float instead of torch.float.<br>
when i trained it didn’t showed any error instead it accurately predicts the value rather than torch.float. why pytorch showing this behavior?<br>
predicting value with dtype=torch.float<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/445e34da21428994181eb0d0805f2041a6c5b8c0.png" data-download-href="https://discuss.pytorch.org/uploads/default/445e34da21428994181eb0d0805f2041a6c5b8c0" title="new"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/445e34da21428994181eb0d0805f2041a6c5b8c0.png" alt="new" data-base62-sha1="9KOiS3J1CR4rM30k4ygyKHQzrJC" width="655" height="500" data-dominant-color="C6C6C6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">new</span><span class="informations">699×533 17.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I don’t fully understand the issue, but note that Python’s <code>float</code> is a C <code>double</code> and PyTorch will thus use <code>torch.float64</code> or <code>torch.double</code> as seen here:</p>
<pre><code class="lang-python">weight = nn.Parameter(torch.rand(1, requires_grad = True, dtype=torch.float))
print(weight.dtype)
# torch.float32

weight = nn.Parameter(torch.rand(1, requires_grad = True, dtype=float))
print(weight.dtype)
# torch.float64
</code></pre>

Teddy_Salas: <p>it was a logical error i train with wrong model</p>
Why do I not get an output for vit pretrained model?

Goldname: <p>I am trying to use torchvision.models.vit_b_32(). However, when I pass in some arbitrary data I get all zeros in the output</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Resize(224),
     transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])])

batch_size = 4

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                         shuffle=False, num_workers=2)

newmodel = torchvision.models.vit_b_32().to(device)
images, labels = next(iter(testloader))
images = images.to(device)
labels = labels.to(device)
print(torch.sum(newmodel(images)[0]))
</code></pre>
<p>Why is the last print statement printing zero?</p>

ptrblck: <p>By default it seems the <code>VisionTransformer</code> model will initialize its <code>head.weight</code> and <code>head.bias</code> to zeros as seen <a href="https://github.com/pytorch/vision/blob/b403bfc771e0caf31efd06d43860b09004f4ac61/torchvision/models/vision_transformer.py#L264-L266">here</a>.<br>
If you use a pretrained model, the outputs would show different values.</p>
Datapipe warning: Is this a problem?

FALL_ML: <p>Hi all, I am constructing a vocab from the Multi30k dataset using building_vocab_from_iterator and the spaCy tokenizer. However, I’m getting a warning saying that “some child datapipes are not exhausted.”  Is this an issue with my implementation? Here is the snippet of code where the error first appears (the error sporadically re-appears while running epochs):</p>
<pre><code class="lang-auto">import torch
from torchtext.datasets import Multi30k
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator

train_data, validation_data, test_data = Multi30k()

tokenizer_en = get_tokenizer('spacy', language="en_core_web_sm")
tokenizer_de = get_tokenizer('spacy', language="de_core_news_sm")

def yield_tokens_en(data_iter):
    for _, text in data_iter:
        yield tokenizer_en(text)

def yield_tokens_de(data_iter):
    for text, _ in data_iter:
        yield tokenizer_de(text)

vocab_en = build_vocab_from_iterator(yield_tokens_en(train_data), specials=["&lt;pad&gt;", "&lt;bos&gt;", "&lt;eos&gt;", "&lt;unk&gt;"])
vocab_de = build_vocab_from_iterator(yield_tokens_de(train_data), specials=["&lt;pad&gt;", "&lt;bos&gt;", "&lt;eos&gt;", "&lt;unk&gt;"])
</code></pre>
<p>And the corresponding output:</p>
<pre><code class="lang-auto">UserWarning: Some child DataPipes are not exhausted when __iter__ is called. We are resetting the buffer and each child DataPipe will read from the start again.
  warnings.warn("Some child DataPipes are not exhausted when __iter__ is called. We are resetting "
</code></pre>
<p>Relevant library versions:</p>
<blockquote>
<p>pytorch==1.12.0<br>
torchtext==0.13.0</p>
</blockquote>

IamMohitM: <p>What I have noticed is that when I get this warning, while iterating through the corresponding dataloader, I get the same batch for multiple iterations:</p>
<pre><code class="lang-auto">for sample in dataloader:
    source_batch, _, _, _, _, _ = sample
</code></pre>
<p>When I compare the source_batch from the first iteration upto 5th (?) iteration source_batch, they are exactly the same. I wonder if it’s relevant to the above warning.</p>
Is it OK to run PyTorch 2.0 docker image on a host with driver 460?

hsfzxjy: <p>Our GPU cluster is shipped with nvidia driver version 460.67 and CUDA 11.2 now and, for some reasons, we are not able to upgrade the underlying driver version.</p>
<p>I want to try out PyTorch 2.0 in such a condition, but saw that the PyTorch 2.0 docker image <code>ghcr.io/pytorch/pytorch:2.0.0-devel</code> was built with CUDA 11.7. I’ve tried to run it and it works well, but I am not sure if there will be implicit problems concerning the CUDA version mismatch.</p>
<p>Could somebody explain is it OK to do so? Or should I build PyTorch from source with CUDA 11.2?</p>

ptrblck: <p>Driver 460.67 will allow you to run CUDA 11.x applications in “Minor Version Compatibility” mode as explained <a href="https://docs.nvidia.com/deploy/cuda-compatibility/#minor-version-compatibility">here</a>.</p>
<p>I don’t know which user-mode driver the PyTorch <code>2.0.0</code> docker image uses, but would guess a newer one.</p>

hsfzxjy: <p>Thanks for reply! So are you suggesting there could be issues when running PyTorch 2.0.0 in such case?</p>

ptrblck: <p>No, I would not expect to see any issues running the PyTorch <code>2.0.0</code> binaries in such a container.</p>
Issue with Dataloader Batching and Edge Index in PyTorch Geometric

abdulhafeezco: <p>I am working with geometric dataset in Pytorch using pytoch geometric library. I want to do train my data in batches. I observed that when I loaded my data to DataLoader in batches. When I loaded data to the dataloader in batches, it automatically increments the edgeindex tensor by the cumulated number of nodes of all graphs .</p>
<p><code>train_loader = DataLoader(train_dataset, batch_size=32, shuffle=Train)</code></p>
<p>train_dataset is a graph data containing node features and edge indices.</p>
<p>My data is a time series data.</p>
<p>I am concerned that by the modification of the edge index tensor across each batch might impact my performance. Does anyone has an idea about this on how to stop automatic increment. Otherwise, could I just train my graph data model and not bother about the automatic increment.</p>
<p>Thanks</p>
Masked Transformer for any CSV format data

Blake_Martin: <p>I am trying to develop a Masked Transformer for play-by-play football data. All the data is stored in a CSV file where each csv file is a game, each row is a play, and each column is an attribute about that play.</p>
<p>Example of the data turned into a tensor:<br>
input_tensor = torch.tensor([[ 1.,  0.,  0.,  0.,  0.],<br>
[ 1.,  0.,  9.,  3.,  3.],<br>
[ 1.,  0., 52.,  2.,  6.],<br>
[ 1.,  1., 32.,  1., 10.]])</p>
<p>Each of the rows in this tensor would be a play and the values in this row are the attributes/features. For example, this is one row/play with features [ 1.,  0.,  0.,  0.,  0.].</p>
<p>I have a Transformer architecture of the following format.</p>
<pre><code>    def __init__(self, input_dim, output_dim, nhead=2, num_layers=2):
         super(FootballTransformer, self).__init__()
         self.encoder_layer = nn.TransformerEncoderLayer(input_dim, nhead)
         self.encoder = nn.TransformerEncoder(self.encoder_layer, num_layers)
         self.decoder_layer = nn.TransformerDecoderLayer(output_dim, nhead)
         self.decoder = nn.TransformerDecoder(self.decoder_layer, num_layers)
         self.linear = nn.Linear(input_dim, output_dim)
</code></pre>
<p>I want to implement several training techniques. The first is to randomly mask different attributes in different rows to allow the model to learn about the data. The next technique would be to finetune the model by masking only the last row/play in the input data so it learns to then predict the next play/row.</p>
<p>I am currently confused about how to properly mask. I would assume that the model would want a mask the same size as the input, where whatever features/rows you want to mask would be the same location in both tensors.</p>
<p>When I use the example data above it wants a mask shape (4,4) which is (sequence_len, sequence_len) I do not understand this. How would I either alter my model to use a mask of the same size so I can randomly select features / rows to mask or how would I make a mask that allows me to do this?</p>
<p>If masking specific features is not an option and I can only mask a whole row/play, how would I create a mask so I know exactly what rows/plays I am masking. With bigger sequences of say 100+ I want to be able to mask a ton of random rows/features.</p>
<p>I am willing to pay for working code that gets me what I want.</p>
Getting indices of a histogram's elements with pytorch

Yaron_Heiser: <p>Hi,<br>
Is there a way to return a <code>Tensor</code> representing the indices of a histogram into which each element of <code>values</code> would be binned, like with tensorflow.histogram_fixed_width_bins</p>

ptrblck: <p>I’m not aware of a built-in method, but something like this might work:</p>
<pre><code class="lang-python">x = torch.tensor([1., 2, 1, 8, 6, 6, 8, 12])
hist = torch.histogram(x, bins=4, range=(0., 10.))
print(hist.hist)
# tensor([3., 0., 2., 2.])
print(hist.bin_edges)
# tensor([ 0.0000,  2.5000,  5.0000,  7.5000, 10.0000])

idx = (x.unsqueeze(1) - hist.bin_edges.unsqueeze(0)).cumsum(1).argmax(1)
print(idx)
# tensor([0, 0, 0, 3, 2, 2, 3, 4])
</code></pre>
Torch.cuda.is_available() is false on windows when everything installed

Blake_Martin: <p>I just got a new desktop. It has a rtx 3080 gpu. I have torch 1.12.0+cpu installed. cuda 11.6 installed. visual studio 2019 installed. everytime I do torch.conda.is_available() it returns false. I have been trying for hours with no luck</p>

Blake_Martin: <p>I just installed torch that is not the cpu only version so the torch version is now <code>1.12.0</code>. Still no luck</p>

ptrblck: <p>I cannot reproduce the issue on a Windows Laptop with a GPU and can properly use the CUDA runtime:</p>
<pre><code class="lang-auto">pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116
...
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0+cu116'
&gt;&gt;&gt; torch.version.cuda
'11.6'
&gt;&gt;&gt; x = torch.randn(1).cuda()
&gt;&gt;&gt; x
tensor([0.0287], device='cuda:0')
</code></pre>

AlphaBetaGamma96: <aside class="quote no-group" data-username="Blake_Martin" data-post="1" data-topic="157472">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blake_martin/48/51773_2.png" class="avatar"> Blake_Martin:</div>
<blockquote>
<p>visual studio 2019 installed.</p>
</blockquote>
</aside>
<p>Have you selected the correct interpreter? Otherwise, Visual Studio will default to the default env where it’s not installed. I think the command is <code>Ctrl + Shift + P</code> to select the interpreter.</p>
<aside class="quote no-group" data-username="Blake_Martin" data-post="1" data-topic="157472">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blake_martin/48/51773_2.png" class="avatar"> Blake_Martin:</div>
<blockquote>
<p>torch.conda.is_available()</p>
</blockquote>
</aside>
<p>Also, it’s <code>torch.cuda.is_available()</code> not <code>torch.conda</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Blake_Martin: <p>problem solved. Unsure of the issue. remade the environment and installed using pytorch commands.</p>
I am having doubt when i trained a linear regression model?

Teddy_Salas: <p>After i testing the model , like test loss ,loss ,&amp; epoch into a graph after first training test loss &amp; loss getting lower according to the epoch but after i trained 2 or 3 times graphs behaving very differently, can any one explain this ?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/f/2f73cddb85c8621d6b640976aeb95ece7ef627e6.png" data-download-href="https://discuss.pytorch.org/uploads/default/2f73cddb85c8621d6b640976aeb95ece7ef627e6" title="Screenshot from 2023-03-18 14-10-30"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/f/2f73cddb85c8621d6b640976aeb95ece7ef627e6.png" alt="Screenshot from 2023-03-18 14-10-30" data-base62-sha1="6LMy7q1aqnqdqbZ5oyN8sfzrPUi" width="690" height="264" data-dominant-color="998A7C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-03-18 14-10-30</span><span class="informations">752×288 34.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Using Pandas `DataFrame` as Data with Skorch

GeorgeIrwin: <p>I want to use Skorch as a simple wrapper of my model.<br>
My data is a set of <code>80x80</code> RGB images.<br>
Each image as a row in a Pandas data frame.</p>
<p>I want o build a process to use the data frame yet I need to have a pre processing step which rearrange the row data into a batch.</p>
<p>How can I do that in Skorch?</p>
How to change transfer learning model for a lot of classes

bezdarnost: <p>Hello. I use efficient net v2 model pre-trained on imagenet-1k for my task, but I have around 4k classes. Feature block has 1280 outputs, and I think it is a moment, that ruined my results.<br>
I found imagenet 22-k, but I can’t find how experts change last layers for the task with so many classes.<br>
I just change last classifier block to one linear layer, and get around 30% of accuracy. Do you know, what I can do in this situation?</p>

Michal_Bogacz: <p>Hi,<br>
Here is an example for transfer-learning from pytroch documentation.<br>
In general, you should change the size of last classification layer. And retrain the model.<br>
<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 1.13.1+cu117 documentation</a></p>

bezdarnost: <p>Yes, I read this documentation. I guess I have problem with bottleneck.<br>
Because feature block has 1280 outputs, but I need classify 4 000 classes</p>

Mahi: <p>You can alter the particular layer of pretrained model once it is loaded.</p>
<p>Refer this answer for more information.<br>
<a href="https://discuss.pytorch.org/t/how-to-delete-layer-in-pretrained-model/17648/6">https://discuss.pytorch.org/t/how-to-delete-layer-in-pretrained-model/17648/6?u=mahi</a></p>
<p>And if the accuracy is lower, you can add some layers before the last one and train it few more epochs by freezing the pretrained layers.</p>
<p>Hope it helps!.</p>
Generate Hessian diagnal matrix for every sample in parallel

HaoKang-Timmy: <p>Hi, I noticed that there are some repositories that allow us to get gradients of each sample in a batch.<br>
I wonder if there is any way that I could get the hessian diagonal matrix of each sample in every batch?</p>

soulitzer: <p>If there’s no particular structure to your function (element-wise), you might as well compute the whole hessian. And if you want that for every sample, you’d do <code>vmap(hessian(fn))(...)</code>.</p>
<p>You can do that with <a href="https://pytorch.org/functorch/stable/" class="inline-onebox" rel="noopener nofollow ugc">functorch — functorch 2.0 documentation</a> (JAX-like API in PyTorch).</p>

HaoKang-Timmy: <p>Thank you, I have found the solution of my question.<br>
There is a reposity that has a function to get approximation hessian matrix of every sample in the batch.<br>
<a href="https://backpack.pt/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://backpack.pt/</a></p>
Understanding Dataloading2 and MultiProcessingReadingService

fehrlich: <p>I have this minimal example:</p>
<pre><code class="lang-auto">from torchdata.datapipes.iter import IterDataPipe

class Dataset(IterDataPipe):
    def __len__(self):
        return 2

    def __getitem__(self, idx):
        return np.arange(5)+idx

    def __iter__(self) -&gt; int:
        for i in range(len(self)):
            yield self[i]

datapipe = Dataset()
reading_service = MultiProcessingReadingService(num_workers=2)
data = DataLoader2(datapipe, reading_service=reading_service)

for x in data:
  print(x)
</code></pre>
<p>This is a simple dataset with 2 items to load (“0” and “1”). When num_workes is 1 it works as expected I get the output “0 1”. When the number of workers is &gt; 1 the output is replicated by the number of workers so for num_workes = 2 it would be “0 0 1 1”, and for 5 “0 0 0 0 0 1 1 1 1 1”. I tried to make sense out of it, but the prefetching seem to start the threads an all do the same. Is there something wrong in my example or do i have a misunderstanding how it supposed to work. Since it’s suggested to use IterDataPipe instead of the MapDataPipe, how is it even supposed to work with multiple threads ?</p>

ptrblck: <p>This behavior might be expected if I’m understanding the <a href="https://pytorch.org/data/main/torchdata.datapipes.iter.html#torchdata.datapipes.iter.IterDataPipe">docs</a> correctly:</p>
<blockquote>
<p>When <code>num_workers &gt; 0</code>, each worker process will have a different copy of the DataPipe object, so it is often desired to configure each copy independently to avoid having duplicate data returned from the workers. <code>get_worker_info()</code>, when called in a worker process, returns information about the worker. It can be used in either the dataset’s <code>__iter__()</code> method or the <code>DataLoader</code> ‘s <code>worker_init_fn</code> option to modify each copy’s behavior.</p>
</blockquote>
<p>You might thus want to use the worker info to e.g. define the start index and the stride.</p>

fehrlich: <p>Thank you for pointing this out.</p>
<ul>
<li>If i call get_worker_info() (like in <a href="https://github.com/pytorch/data/blob/84587ff57575fd47fcae61635a3f4ffc1e639941/benchmarks/torchvision_classification/helpers.py" class="inline-onebox" rel="noopener nofollow ugc">data/helpers.py at 84587ff57575fd47fcae61635a3f4ffc1e639941 · pytorch/data · GitHub</a>) it returns None.</li>
<li>If i use the _worker_init_fn the datapipe is overwritten by “PrefetcherIterDataPipe”, so i can’t pass any information to my datapipe to distribute the data.</li>
<li>If I rewrite everything to MapDataPipe i get “AttributeError: ‘Dataset’ object has no attribute ‘prefetch’”</li>
</ul>
<p>Is there a working example with prefetch and multiprocessing, whether MapDataPipe or IterDataPipe, or is the state still too “beta”?</p>
Checkpoint file manipulation

takis17: <p>Hi there,</p>
<p>Hope everyone is doing alright.<br>
So I have a ckpt file ( checkpoint file) containing these informations:</p>
<pre><code class="lang-auto">OrderedDict([('model.encoder.0.conv.0.weight',
              tensor([[[[ 1.2614e-01,  1.1951e-01,  1.6265e-01],
                        [ 2.1022e-01,  3.4994e-01,  5.9035e-01],
                        [ 1.3048e-01,  3.1809e-01,  4.4730e-01]],
              
                       [[-2.4336e-01, -1.3665e-01, -1.4839e-01],
                        [-1.2742e-01, -1.7422e-01, -6.1454e-02],
                        [-2.0389e-01, -1.4220e-01, -1.5854e-01]],
              
                       [[-5.5171e-02, -1.1455e-02, -3.8867e-02],
                        [ 1.5383e-02,  2.0011e-02,  1.7529e-01],
                        [-4.2312e-02, -9.9361e-03,  3.7781e-02]],
              
                       ...,
</code></pre>
<p>I would like to know:</p>
<ol>
<li>From what I have understand, checkpoints contain the model weights (I am using a pre-trained model)?<br>
But my model contains more layers than the ones found in the checkpoint file.</li>
<li>How do I manipulate this weights found in the checkpoint file?</li>
</ol>
<p>Thanks<br>
2)</p>
How to add adaptive noise to model parameters?

Guncuke: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d4fbf3cf43e383a7d711f9a8885cf60e6df9164.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d4fbf3cf43e383a7d711f9a8885cf60e6df9164" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d4fbf3cf43e383a7d711f9a8885cf60e6df9164.png" alt="image" data-base62-sha1="1TL3ZgMNhJlg7H6wsTavxXe8Wz2" width="690" height="323" data-dominant-color="373737"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">763×358 27.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The parameters of different layers have different orders of magnitude, if I add noise with the same sigma, the model will not convergence, I only have one globally calculated sigma, so how do I add noise adaptively depending on the magnitude of the data at each level, multiplication?</p>
How can I find unique values in a tensor quickly in libtorch c++?

Anders_G: <p>I want to use unique in libtorch and I use unique_dim. However, finding 20 unique elements in a vector of 100 million elements take half a minute. In pytorch the same operation takes a fraction of a second using the Torch.unique function.  So how should I go about to do fast unique in libtorch c++?</p>
<p>(Im only interested in the CPU version)</p>

Outage4778: <p>I had this same question. If you want to use equivalent of torch.unique, it is in <code>at</code> namespace, <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1ad67ed9d188a9f3688b2ecdf06f316650.html#exhale-function-namespaceat-1ad67ed9d188a9f3688b2ecdf06f316650" rel="noopener nofollow ugc">at::_unique</a>.</p>
How to operate tensor from custom class in registered hook?

zhouxian: <p>Hi!</p>
<p>I implemented some subclass of pytorch tensor in my applications, and need to operate such custom tensors in a call back function hooked to a tensor’s backward pass. However, i found that any operation of such custom tensor would yield torch tensor inside such hooked callbacks, while i am hoping them to return tensor of the custom class.</p>
<p>A minimal example:</p>
<pre><code class="lang-auto">import torch

class CTensor(torch.Tensor):
    pass

def hook(grad):
    print('within hook:', type(a[:1]))

a = CTensor([1,2,3]).requires_grad_()
b = CTensor([1,2,3]).requires_grad_()
b.register_hook(hook)

c = b.sum()
print('outside hook:', type(a[:1]))

c.backward()
</code></pre>
<p>This prints</p>
<pre><code class="lang-auto">outside hook: &lt;class '__main__.CTensor'&gt;
within hook: &lt;class 'torch.Tensor'&gt;
</code></pre>
<p>However, I would need the slicing operation inside the hook() also return tensor of my custom class CTensor. What should I do? Or this is something not possible?</p>
<p>Thanks!!</p>

zhouxian: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Any help would be much appreciated!</p>

ptrblck: <p>I don’t know if PyTorch allows you to use custom tensors in Autograd.<br>
You can extend the tensor class as described <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch">here</a>, but I’m unsure if you could hook it somehow also into the backward pass. <a class="mention" href="/u/alband">@albanD</a> might know more about this use case and if it’s doable.</p>

albanD: <p>Yeah there is something fishy happening here, we should look into this in more details in the issue that is already opened with this example.</p>