akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Installation from source on aarch64 meet ‘-march=armv8.2-a+fp16+dotprod’ failure

username: linnuannuan
cooked: <p>cc1: error: invalid feature modifier in ‘-march=armv8.2-a+fp16+dotprod’<br>
[194/2715] Linking CXX static library lib/libprotobuf.a<br>
ninja: build stopped: subcommand failed.</p>

username: adamjstewart
cooked: <p>Looks like the same error as <a href="https://github.com/google/XNNPACK/issues/1551" class="inline-onebox" rel="noopener nofollow ugc">cmake build error · Issue #1551 · google/XNNPACK · GitHub</a>, solution was to use a newer compiler.</p>
Autograd.grad throws runtime error in DistributedDataParallel

username: Kevinkevin189
cooked: <p>To train a WGAN-GP in DistributedDataParallel,I met several errors in coding.</p>
<pre><code class="lang-python">fake_g_pred=self.model['D'](outputs)
gen_loss=self.criteron['adv'](fake_g_pred,True)
loss_g.backward()
self.optimizer['G'].step()
self.lr_scheduler['G'].step()

#hat grad penalty
epsilon=torch.rand(images.size(0),1,1,1).to(self.device).expand(images.size())
hat=(outputs.mul(1-epsilon)+images.mul(epsilon))
hat=torch.autograd.Variable(hat,requires_grad=True)
dis_hat_loss=self.model['D'](hat)
grad=torch.autograd.grad(
                outputs=dis_hat_loss,inputs=hat,
                grad_outputs=torch.ones_like(dis_hat_loss),
                retain_graph=True,create_graph=True,only_inputs=True
            )[0]
grad_penalty=((grad.norm(2,dim=1)-1)**2).mean()
grad_penalty.backward()
</code></pre>
<p>While the GANis able to run one iter (only one batch in dataloader),the second iter reports an error.</p>
<pre><code class="lang-auto">RuntimeError: Expected to have finished reduction in the prior 
iteration before starting a new one.This error indicates that your module 
has parameters that were not used in producing loss. You can enable 
unused parameter detection by (1) passing the keyword argument 
`find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; 
(2) making sure all `forward` function outputs participate in calculating loss. 
If you already have done the above two steps, then the distributed data parallel 
module wasn't able to locate the output tensors in the return 
value of your module's `forward` function. 
Please include the loss function and the structure 
of the return value of `forward` of your module
 when reporting this issue (e.g. list, dict, iterable).
</code></pre>
<p>It throws an error at line</p>
<pre><code class="lang-auto">fake_g_pred=self.model['D'](outputs)
</code></pre>
<p>Seems that’s an intrinsic bug in DDP model,Can anyone tell me how to debug it or any alternatives to<br>
train a WGAN with Gradient Penalty in distributeddataparallel way？</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am afraid this is expected as DDP only works with <code>.backward()</code> and does not support autograd.grad.<br>
You will have to use <code>.backward()</code> for it to work properly.</p>

username: LuoXin-s
cooked: <p>Could you please provide a working example?<br>
I counter similar issues when implementing WGAN-GP. However, I can not combine the real image loss, fake image loss, and gradient penalty with a summation and call a single <code>backward()</code>. Since I use a discriminator with spectral normalization, it updates parameters in every <code>forward()</code> call. In order to correctly calculate gradient for each loss, I have to call backward() three times separately(single <code>backward()</code> will use the parameters updated by spectral normalization in the last <code>forward()</code> call.).</p>
Embedding bag with concatenation

username: ysig
cooked: <p>Hi,</p>
<p>Imagine for a single data point you have a LongTensor e.g. [0, 1, 1, 2, 1, 3] of length k and you want to embed each of them (similar with a codebook) so that each entry contributes an embedding vector to the total representation.</p>
<p>For now <a href="https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html#torch.nn.EmbeddingBag" rel="noopener nofollow ugc">EmbeddingBag</a>  allows you only to have max,mean or average of elements.<br>
Why doesn’t it support concatenation, so that if each element outputs a vector of dimension d the whole output will be k<em>d? (</em>, k → *, k.d)</p>
<p>Is there an efficient workaround this?</p>
<p>Thanks,</p>
How to modify the computed gradient of middle variables and then backpropagate new grandients

username: Yuanbo_Liu
cooked: <p>When I traning a model, I want to change some non-leaf node’s gradient and see how this affect the model’s behavior. For example:<br>
The gradient backpropagates chain is z-&gt;y-&gt;x,<br>
I wanna get the <code>y.grad</code> and <strong>set it to new values</strong>,then backpropagating to x<br>
the <code>x.grad</code> should be affected by the new value<br>
How can I do that with code?</p>

username: soulitzer
cooked: <p>You can register a backward hook on your tensor. See: <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.register_hook — PyTorch 1.13 documentation</a></p>
fwAD source code dual numbers

username: panos
cooked: <p>I am working on a project to use forward differentiation using dual numbers.  Is there a C++ way to access fwAD?  Was fwAD implemented in pure Python?  If not could you please point me to the implementation source code?  Thank you.</p>

username: soulitzer
cooked: <p>There are some implementation notes in <code>torch/csrc/autograd/forward_grad.h</code></p>
<p>The analogous functions in cpp are:</p>
<ul>
<li>dual_level: `torch::autograd::forward_ad::enter_dual_level()</li>
<li>make_dual: <code>at::_make_dual</code>
</li>
<li>unpack_dual: <code>at::_unpack_dual</code>
</li>
</ul>
<p>Forward AD was mainly implemented with Python in mind, so these are mostly private APIs that we don’t officially support.</p>
How to fix size mismatch for embeddings.weight

username: Sirui_Li
cooked: <p>Hello,</p>
<p>I saved my model by running <code>torch.save(model.state_dict(), 'mytraining.pt')</code>. When I try to load it, I got the error:</p>
<blockquote>
<p>size mismatch for embeddings.weight: copying a param with shape torch.Size([7450, 300]) from checkpoint, the shape in current model is torch.Size([7469, 300]).</p>
</blockquote>
<p>I find it is because I use <code>build_vocab</code> from <code>torchtext.data.Field</code>. <code>TEXT.build_vocab(train_data, vectors=Vectors(w2v_file))</code> would give different vocabularies each time, but I have to get the vocabulary to construct my model: <code>def __init__(self, config, vocab_size, word_embeddings)</code> .How can I fix it?</p>

username: MaveriQ
cooked: <p>Hi <a class="mention" href="/u/sirui_li">@Sirui_Li</a></p>
<p>Did you find a solution to the problem? I am getting similar problem, though in my case the mismatch is smaller.</p>
<p>Thanks!</p>

username: adissa
cooked: <p>Hi <a class="mention" href="/u/maveriq">@MaveriQ</a></p>
<p>Did you get any solution to the problem. i have the similar issue.</p>
<p>Thanks</p>

username: dreidizzle
cooked: <p>I think this might be old torchtext? Did Field not get removed? Maybe this is a bug. Also, you might try TorchScript as a way to serialize, unless you want the model this way.</p>
How to run a backward in a backward hook

username: just_dance
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn

def hook_out(module, grad_in, grad_out):
    print("backward hook out")

    def hook_in(module, grad_in, grad_out):
        print("backward hook in")
    model_in = nn.Conv2d(3, 3, 1, 1)
    input_in = torch.randn([1, 3, 16, 16])
    model_in(input_in).sum().backward()

model_out = nn.Conv2d(3, 3, 1, 1)
input_out = torch.randn([1, 3, 16, 16])
back_hook_out = model_out.register_backward_hook(hook_out)
model_out(input_out).sum().backward()

</code></pre>
<p>I get this erro</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "regist_bh.py", line 21, in &lt;module&gt;
    model_out(input_out).sum().backward()
  File "/data/application/xiw/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/data/application/xiw/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "regist_bh.py", line 16, in hook_out
    model_in(input_in).sum().backward()
  File "/data/application/xiw/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/data/application/xiw/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>

username: soulitzer
cooked: <p>The inputs you are getting your hook don’t require grad, because grad has been disabled. Your outer backward needs to run with create_graph=True for gradients to be enabled during backward.</p>
Difference between FullyShardedDataParallel and ZeroRedundancyOptimizer?

username: marsggbo
cooked: <p>Is my understanding of <code>FullyShardedDataParallel</code> and <code>ZeroRedundancyOptimizer</code> correct?</p>
<p><code>FullyShardedDataParallel</code> is to shard the model’s parameters, which can save GPU memory during the forward process.<br>
<code>ZeroRedundancyOptimizer</code> is to shard state parameters of the optimizer, which is to save GPU memory during backward and gradient update processes</p>

username: agu
cooked: <p>Both <code>ZeroRedundancyOptimizer</code> and <code>FullyShardedDataParallel</code> are PyTorch classes based on the algorithms from the <a href="https://arxiv.org/abs/1910.02054" rel="noopener nofollow ugc">“ZeRO: Memory Optimizations Toward Training Trillion Parameter Models”</a> paper.</p>
<p>From an API perspective, <code>ZeroRedunancyOptimizer</code> wraps a <code>torch.optim.Optimizer</code> to provide ZeRO-1 semantics (i.e. P_{os} from the paper). In contrast, <code>FullyShardedDataParallel</code> wraps a <code>torch.nn.Module</code> to provide ZeRO-3 semantics (i.e. P_{os+g+p} from the paper) (when using <code>sharding_strategy=ShardingStrategy.FULL_SHARD</code>).</p>
<p>ZeRO-1 only shards optimizer states. ZeRO-3 shards optimizer states, gradients, and parameters.  ZeRO-1 saves memory during the optimizer step, and ZeRO-3 saves memory throughout forward, backward, and optimizer steps.</p>
<p>I can provide more details if you have further questions.</p>
Neural Feature Importance

username: banikr
cooked: <p>Is there any PyTorch-supported work for ‘Neural Feature Importance’ extraction?<br>
I have a trained encoder for 1D spectral data. Now I want to know which features are essential in determining the latent representation of the data in low dimensional space.</p>
<p>I have come across the following:</p>
<ol>
<li>SHAP</li>
<li>DeepLift.</li>
</ol>
<p>and other computationally expensive methods for box models like feature permutation.</p>
<p>But SHAP does not support PyTorch fully.</p>
<p>Any idea what could be the workaround?</p>
Errors using Deep Speed 3

username: Taconic
cooked: <p>To the superstars that have worked on py torch and deep speed, I need<br>
help can anyone help me with</p>
<p>This is what we tried,<br>
Set multiple GPUS as accelerator in pl.Trainer<br>
Add DeepSpeed Strategy to pl.Trainer<br>
Update optimizer in TemporalFusionTransformer to be compatible with DeepSpeed<br>
Runs successfully in DeepSpeed stage=2<br>
Returns an error in DeepSpeed stage=3:</p>
<p>This is the actual error,<br>
File “/home/ubuntu/.venv/pl-forecast/lib/python3.10/site-packages/deepspeed/runtime/zero/parameter_offload.py”,<br>
line 39, in _apply_to_tensors_only<br>
return outputs.<strong>class</strong>(touched_outputs)<br>
TypeError: output.<strong>new</strong>() missing 7 required positional arguments:<br>
‘encoder_attention’, ‘decoder_attention’, ‘static_variables’,<br>
‘encoder_variables’, ‘decoder_variables’, ‘decoder_lengths’, and<br>
‘encoder_lengths’</p>
<p>–</p>
Does settings β1 = 0 or β2 = 0 means that ADAM behaves as RMSprop or Momentum?

username: laro
cooked: <p>I read on <strong>ADAM</strong> optimizer, and I saw multiple quotes which say that <strong>ADAM</strong> is a combination of <strong>Momentum</strong> and <strong>RMSprop</strong> optimizers.</p>
<p>So if we:</p>
<ol>
<li>Set β1 = 0 does it means that <strong>ADAM</strong> behaves exactly as <strong>RMSprop</strong> optimizer?</li>
<li>Set β2 = 0 does it means that <strong>ADAM</strong> behaves exactly as <strong>Momentum</strong> optimizer?</li>
</ol>
How to avoid this "UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future."?

username: smathewmanuel
cooked: <p>Hi, I am trying to save checkpoints during training using the following code:</p>
<pre><code class="lang-auto">info_dict = {
                    'epoch' : epoch,
                    'numpy_random_state' : np.random.get_state(),
                    'torch_random_state' : torch.random.get_rng_state(),
                    'net_state' : net.module.state_dict(),
                    'optimizer_state' : optimizer.state_dict(),
                    'train_hist' : history_plot['train'],
                    'val_hist' : history_plot['val'],
                    
                }
torch.save(info_dict, chkpnts)
</code></pre>
<p>And loading them, using:</p>
<pre><code class="lang-auto">net.load_state_dict(chkpnt['net_state'])
optimizer.load_state_dict(chkpnt['optimizer_state'])
</code></pre>
<p>The code is working fine, however, throws up the UserWarning mentioned above. I am new to PyTorch any help is appreciated.</p>

username: ptrblck
cooked: <p>I don’t see the issue in your code, but the warning is raised if positional instead of keyword arguments are used as seen here:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)

# no warning
state_dict = model.state_dict()

# still no warning if keyword argument is used
d = dict()
model.state_dict(destination=d)

# warning is positional argument is used
d = dict()
model.state_dict(d)
# UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
</code></pre>

username: smathewmanuel
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you for the response. Based on your reply, I checked the error again. I am running 1.12 and in <code>torch.nn.modules.module</code> positional arguments were used: <code>module.state_dict(destination, prefix + name + '.', keep_vars=keep_vars)</code>. I believe this was the reason warnings were thrown up.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a36d583c25457021124363ae37d24224aeb2a08c.png" data-download-href="https://discuss.pytorch.org/uploads/default/a36d583c25457021124363ae37d24224aeb2a08c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a36d583c25457021124363ae37d24224aeb2a08c.png" alt="image" data-base62-sha1="njK62ZHUC8V4cK2IVIhu0QwnCxK" width="690" height="96" data-dominant-color="2C2B2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1171×164 10.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Need help in weighted-loss calculation

username: import_all
cooked: <p>I am working on a tumor segmentation problem and need help in calculating the weighted loss in BCEWithLogitsLoss function.<br>
So, I am having image (240,240,155) and mask (240,240,155) with labels - 0,1,2,4</p>
<p>Now in training i have image (1,4,144,240,240) and mask (1,3,144,240,240) shape with labels 0,1<br>
How to calculate the weighted loss here.</p>
CNN + padding - what do you do about padding w a CNN for text classification?

username: dreidizzle
cooked: <p>Hi,</p>
<p>I have a question about padding and the effects of this on a CNN text classification model. Let’s say I have a sentence with 4 words but because I want all my tensors in a batch to be the same size I use padding. So, for the example I might have a padded and unpadded sequence looking like: [1,2,3,4] and [1,2,3,4,0,0,0]; each int here represents some token, let’s say words, and I’ve padded by 3 zeros to have a size of 7 across all tensors in the batch. Suppose the vocabulary is of size 100 and each word vector gets dimension 5. Suppose I have 1 filter with a kernel of size 2 and then I want to apply max pooling across all the (sentence length) dimensions. It seems that if I don’t remove pooling, I’ll have some extra junk at the end, and this might screw up max pooling or mean pooling. Should you remove the pooling indices in such a model? Basically m != m_padded generally below. Is this something to be concerned about? I know that for RNN’s there’s pad sequence logic and utils, so my questions is similar but for CNN. It seems like technically speaking I am introducing padding without wanting to …</p>
<p>Thank you!</p>
<pre><code class="lang-auto">e = nn.Embedding(100, 5, padding_idx=0)
# 4 X 5 matrix
x = e(torch.tensor([1,2,3,4]))

# 7 X 5 matrix
x_padded = e(torch.tensor([1,2,3,4,0,0,0]))

# The filter.
f = nn.Conv1d(5, 1, 2)

# 1 X 3 matrix
z = f(x.t())

# 1 X 6
z_padded = f(x_padded.t())

# 1 X 1
m = nn.AvgPool1d(3)(z)

# 1 X 1
m_padded = nn.AvgPool1d(6)(z_padded)
</code></pre>

username: mygithubid1
cooked: <p>In my experience, it’s okay to pad when using a CNN. Remember to pad to maximum sentence length in a batch. Assuming you use a dense layer as the final layer of the model, things should work fine.</p>

username: dreidizzle
cooked: <p>Thank you! So you are saying that the performance is about the same as this unwanted effect gets washed out. But technically, this is an effect right? For example if you have a batch with a few long sentences and a lot more short sentences and you do average pooling at the end, final values (arrived by way of averaging) of the short sentences will get dragged down by the 0’s quite a bit, no? So, should we not remove the 0’s?</p>

username: mygithubid1
cooked: <p>You’re welcome. The training phase should take care of those minor variations. The performance of a bigger batch is going to be better than a batch size of 1.</p>
IndexError: too many indices for tensor of dimension

username: joel2
cooked: <p>IndexError: too many indices for tensor of dimension 2</p>

username: ptrblck
cooked: <p>As the error message describes, you are trying to index a tensor with too many indices. Here is a small example:</p>
<pre><code class="lang-python">x = torch.randn(3, 4)
x[0, 0, 0]
# IndexError: too many indices for tensor of dimension 2
</code></pre>
RuntimeError: shape '[10, 32, 32]' is invalid for input of size 30720

username: joel2
cooked: <p>Am trying to do quanvolution with Cifar dataset how am geeting error,my models are below,any assistance am grateful</p>
<p>class HybridModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.conv0 = nn.Sequential(<br>
QuanvolutionFilter(),<br>
nn.Conv2d(3, 64, kernel_size=3, padding=1),<br>
nn.BatchNorm2d(64),<br>
nn.ReLU(inplace=False),<br>
nn.MaxPool2d(2),<br>
)<br>
self.fc1 = nn.Linear(16<em>16</em>64,10)</p>
<pre><code>  #forward propagation function
def forward(self,x,use_cirq=False):
    x = self.conv0(x)
    x = self.QuanvolutionFilter(x,use_cirq)
    x = x.view(x.size(0),-1)
    x = self.fc1(x)
    return x
</code></pre>
<p>class HybridModel_without_qf(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.conv1 = nn.Sequential(<br>
nn.Conv2d(3, 64, kernel_size=3, padding=1),<br>
nn.BatchNorm2d(64),<br>
nn.ReLU(inplace=False),<br>
nn.MaxPool2d(2),<br>
)<br>
self.conv2 = nn.Sequential(<br>
nn.Conv2d(64, 128, kernel_size=3, padding=1),<br>
nn.BatchNorm2d(128),<br>
nn.ReLU(inplace=False),<br>
nn.MaxPool2d(2),<br>
)<br>
self.conv3 = nn.Sequential(<br>
nn.Conv2d(128, 256, kernel_size=3, padding=1),<br>
nn.BatchNorm2d(256),<br>
nn.ReLU(inplace=False),<br>
nn.MaxPool2d(2),<br>
)</p>
<pre><code>    self.fc = nn.Linear(4*4*256,10)
</code></pre>
<p><span class="hashtag">#forward</span> propagation function<br>
def forward1(self,x):<br>
x = self.conv1(x)<br>
x = self.conv2(x)<br>
x = self.conv3(x)<br>
x = x.view(x.size(0),-1)<br>
x = self.fc(x)<br>
return x</p>

username: ptrblck
cooked: <p>I don’t see where the failing <code>view</code> operation is used in your code, but here is a code snippet reproducing the error:</p>
<pre><code class="lang-python">x = torch.randn(30720)

# fails
y = x.view(10, 32, 32) 
# RuntimeError: shape '[10, 32, 32]' is invalid for input of size 30720

# works
y = x.view(30, 32, 32)
y = x.view(-1, 32, 32)
</code></pre>
<p>You might be trying to create a view with a shape of <code>[10, 32, 32]</code> which is incompatible with the number of elements, which is <code>30720</code>.</p>
Importing torch causes segfault when using python installed with conda

username: Vedant_Roy
cooked: <p>I’m trying to debug a segfault caused by importing torch.<br>
Using GDB gives the following:</p>
<pre><code class="lang-auto">(dev) root@myhost:/workspace/project/distributed# gdb python3
[GDB license]

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3...
(gdb) r -c "import torch"
Starting program: /opt/conda/envs/dev/bin/python3 -c "import torch"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
type_qualname (type=0x555558aad9e0, context=0x0)
    at /usr/local/src/conda/python-3.9.15/Objects/typeobject.c:481
481	/usr/local/src/conda/python-3.9.15/Objects/typeobject.c: No such file or directory.
</code></pre>
<p>I’m using a conda environment with python=3.9.<br>
Has anyone else run into this error?</p>
Usage of PYTORCH_NO_CUDA_MEMORY_CACHING flag in prod and stream awareness

username: abhishek179
cooked: <p>Hi pytorch Team,<br>
reviewing the CUDACachingAllocator.cpp i see it provides a recordStream() functionality to help insert the correct synchronization when allocations are used on multiple streams. This will ensure that the block is not reused before each recorded stream completes work.</p>
<p>We have a scenario where we want to run with “PYTORCH_NO_CUDA_MEMORY_CACHING” set to true in Production workload. I want to understand what will happen to record_stream functionality if we enable “PYTORCH_NO_CUDA_MEMORY_CACHING” by setting it to true. Does pytorch handle the stream aware allocation in different way when “PYTORCH_NO_CUDA_MEMORY_CACHING” is enabled?</p>
PyTorch: how to use torchvision.transforms.AugMIx with torch.float32?

username: StarryGui
cooked: <p>Hello!</p>
<p>I am trying to apply data augmentation in image dataset by using torchvision.transforms.AugMIx, but I have the following error:</p>
<pre><code>  File "/vol/ideadata/oc69ubiw/conda/env/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py", line 813, in posterize
raise TypeError(f"Only torch.uint8 image tensors are supported, but found {img.dtype}")
</code></pre>
<p>TypeError: Only torch.uint8 image tensors are supported, but found torch.float32</p>
<p>I tried to convert it to int, but I have another error:</p>
<pre><code>  File "/vol/ideadata/oc69ubiw/conda/env/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py", line 83, in convert_image_dtype
raise RuntimeError(msg)
</code></pre>
<p>RuntimeError: The cast from torch.float32 to torch.int32 cannot be performed safely.</p>
<p>Any help is appreciated!</p>

username: massisenergy
cooked: <p>This might help you: <a href="https://stackoverflow.com/a/73755323/9592557" rel="noopener nofollow ugc">PyTorch: how to use torchvision.transforms.AugMIx with torch.float32?</a></p>
Intialising cls_token and bert sep_token in pytorch for multimodal classification

username: vishnu_vardhan1
cooked: <p>Hi,</p>
<p>I hope your doing well, I have a question regarding initialising cls tokens and Bert separator tokens. While building a network, can I initialise both of them in the following way in PyTorch:</p>
<p>Cls_tok = sep_tok = nn.Parameter(torch.randn(1,1, embedding_size))</p>
<p>I’m unable to find any resources on these in PyTorch, could anyone please tell me whether it’s correct or not</p>
<p>Thank you</p>
About copy a model~

username: sy_s
cooked: <p>I want to copy a model and using the following code:</p>
<pre><code class="lang-auto">model_b = copy.deepcopy(model_a)
</code></pre>
<p>However, this operation only copy the weights/bias of the model_a, when model_a includes dropout, there is no guarantee that the outputs of model_a and model_b are exactly the same, and I want to know if pytorch has an operation that can be copied to the dropout of model a at the same time.</p>

username: ptrblck
cooked: <p>The dropout operation uses the global pseudo-random number generator to sample the mask, which won’t be stored in the <code>deepcopy</code> command.<br>
If you need to “restore” the same behavior, you might need to seed the code and make sure the same order of operations is used in the PRNG.</p>
Speed up for loop

username: Tianming_Qu
cooked: <p>Hi all, I have a loop as following:<br>
for i in range(1000000):<br>
SK1 = random.choice(binc1)<br>
SK2 = random.choice(binc2)<br>
SL1 = random.choice(binc1)<br>
SL2 = random.choice(binc2)<br>
while SK1 == SL1:<br>
SL1 = random.choice(binc1)<br>
while SK2 == SL2:<br>
SL2 = random.choice(binc2)<br>
<span class="hashtag">#print</span>("PC1_S.index(SK1) = ",PC1_S.index(SK1))<br>
<span class="hashtag">#print</span>("PC2_S.index(SK2) = ",PC2_S.index(SK2))<br>
F[binc2.index(SK2)][binc1.index(SK1)] = gaussian(SK1,binc1,SK2,binc2)<br>
F[binc2.index(SL2)][binc1.index(SL1)] = gaussian(SL1,binc1,SL2,binc2)<br>
W_SK = Wn[binc2.index(SK2)][binc1.index(SK1)] * F[binc2.index(SK2)][binc1.index(SK1)]<br>
W_SL = Wn[binc2.index(SL2)][binc1.index(SL1)] * F[binc2.index(SL2)][binc1.index(SL1)]<br>
if W_SK &lt; W_SL:<br>
SK1 = SL1<br>
SK2 = SL2<br>
else:<br>
a = random.random()<br>
if W_SL/W_SK &gt; a:<br>
SK1 = SL1<br>
SK2 = SL2<br>
else:<br>
SK1 = SK1<br>
SK2 = SK2<br>
<span class="hashtag">#print</span>(‘SK =’,SK)<br>
count[binc2.index(SK2)][binc1.index(SK1)] = count[binc2.index(SK2)][binc1.index(SK1)] + 1</p>
<p>where binc1 and binc2 are two np.arrays, is there anyway I can use pytorch to speedup this for loop?</p>
Semantic segmentation on big images

username: Sylvain_Ard
cooked: <p>Hello,<br>
I am looking for a program of semantic segmentation for example U-Net which take images of size 3500x2500 pixels at input of neural network please.<br>
Thank you very much for answer<br>
Best regards</p>

username: Sylvain_Ard
cooked: <p>Hello noone has an answer please ?</p>

username: KFrank
cooked: <p>Hi Sylvain!</p>
<aside class="quote no-group" data-username="Sylvain_Ard" data-post="2" data-topic="167504" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sylvain_ard/48/30714_2.png" class="avatar"> Sylvain_Ard:</div>
<blockquote>
<p>Hello noone has an answer please ?</p>
</blockquote>
</aside>
<p>I suspect that people tend not to answer your questions because you ask<br>
the same question over and over again without seriously engaging with the<br>
answers you were given when you asked it earlier.</p>
<p>To summarize and answer I gave you previously, U-Net is designed to work<br>
on images of essentially any size using the tiling strategy discussed in the<br>
original U-Net paper.</p>
<p>Here is my earlier answer:</p>
<aside class="quote quote-modified" data-post="2" data-topic="152522">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/semantic-segmentation-on-very-big-images/152522/2">Semantic segmentation on very big images</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hi Sylvain! 

One reasonable approach would be to use a U-Net (a common and 
solid semantic-segmentation network architecture) with the so-called 
“overlap-tile strategy” (that is, apply the U-Net to patches of the large 
input images).  This is discussed in detail in the <a href="https://arxiv.org/abs/1505.04597" rel="noopener nofollow ugc">original U-Net paper.</a> 
Best. 
K. Frank
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>
Regarding creating the ImageNet validation set

username: joohyunglee
cooked: <p>I am doing a hyper-parameter tuning for self-supervised learning using ImageNet. BYOL [<a href="https://arxiv.org/abs/2006.07733" rel="noopener nofollow ugc">link</a>] states (in Appendix D.1) that they used ‘the last 10009 last images of the official tensorflow ImageNet split’.</p>
<p>When I use torchvision.datasets.ImageNet, the last 10009 images only consist of images from couple classes. I don’t find any image_id for training set in devkit for ImageNet. How can I extract the ‘last’ 10009 images from ImageNet training set? Thank you ahead.</p>

username: tiramisuNcustard
cooked: <p>Please see the here: <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.random_split" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html#torch.utils.data.random_split</a><br>
and this post: <a href="https://discuss.pytorch.org/t/how-to-split-dataset-into-test-and-validation-sets/33987">https://discuss.pytorch.org/t/how-to-split-dataset-into-test-and-validation-sets/33987</a></p>
Latency difference on two different training

username: antoine_mlbry
cooked: <p>I have a set of PyTorch classification models. I trained them with the same parameters on two different datasets:</p>
<ul>
<li>A first time around 5 months ago with <code>torch==1.9.1</code>
</li>
<li>A second time now with <code>torch==1.12.1</code>
</li>
</ul>
<p>I tested inference for both models inside the same environment with <code>torch==1.12.1</code> and somehow the newly trained models have double the latency of the older ones (15ms to 30ms).</p>
<p>It doesn’t seem to be the version with which I train the models as I tested a retrain with version 1.9.1 and got a slower model there too.</p>
<p>Specifically, I looked into one of the models with a combination of 1D convolutional, LSTM, and linear layers. Profiling this model on both trainings I saw that mostly the convolution and lstm operations got much slower. The weights have comparable averages, but the older ones have a bigger standard deviation (2 to 10x).</p>
<p>Another strange thing I noticed, is when retraining on the new dataset for a single epoch I am getting the same latency as the previous training while the code for model training hasn’t been updated between the two.</p>
<p>I there anything I might be missing, or forgot to upgrade when upgrading the torch packages?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="antoine_mlbry" data-post="1" data-topic="166637">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/antoine_mlbry/48/47125_2.png" class="avatar"> antoine_mlbry:</div>
<blockquote>
<p>I there anything I might be missing, or forgot to upgrade when upgrading the torch packages?</p>
</blockquote>
</aside>
<p>I don’t think you’ve missed anything and even if so it wouldn’t explain why retraining in <code>1.9.1</code> also now yields the slower results.<br>
In case you are using the CPU for your inference use case, check if <code>torch.set_flush_denormal(True)</code> makes a difference.</p>

username: antoine_mlbry
cooked: <p>Thank you for your answer <a class="mention" href="/u/ptrblck">@ptrblck</a>! I did try using <code>torch.set_flush_denormal(True)</code> but it was not the cause here.<br>
The cause was the way we were saving and processing the data for training. We used to preprocess our data directly within python before training which by default will use the type <code>int64</code>. We switched to using Spark to preprocess our data and by default Spark will save integers as <code>int32</code>. Somehow, even though we were converting the <code>int32</code> back to <code>LongTensor</code> before training this resulted in longer inference time. The root cause here is still kind of a mystery to me but made it work by forcing Spark to use <code>LongType</code>.</p>
Program for detect multiple keypoints per class

username: Sylvain_Ard
cooked: <p>Hi,<br>
all the programs I found on Internet are for one point per class and per object but I am looking for a program for detecting several points per class and per object, do you know one please ?<br>
Thank you<br>
Best regards</p>

username: Sylvain_Ard
cooked: <p>nobody has an answer please ?</p>

username: Sylvain_Ard
cooked: <p>Hello ! I up please !</p>