akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is there an efficient way to convert a model with float32 params to float64?

AlphaBetaGamma96: <p>Hi All,</p>
<p>I was wondering if it’s at all possible to efficiently convert all my model parameters from float32 to float64? I’ve pretrained my model in float32, but when running it I get a NaN error but I know that model works within float64. So, I’d ideally like to just reformat my dtype.</p>
<p>Is there a way to easily change my model parameters from float32 to float64?</p>
<p>Any help is appreicated! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

zetyquickly: <p>Hey <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a></p>
<p>Have you tried to do <code>model.double()</code> that should convert model parameters to <code>float64</code>?</p>
<p>After that be aware that inputs of the model should also be tensors of double precision</p>

AlphaBetaGamma96: <p>Great that works for my model, but there are one or two other variables that I need to convert (that are outside my <code>model</code>).</p>
<p>I’ve tried applying this to my optimizer, and it yields <code>AttributeError: 'Adam' object has no attribute 'double'</code> (when calling <code>optim.double()</code>)· I’d assume this is because <code>model</code> subclasses from <code>nn.Module</code> whereas <code>optim</code> subclasses from <code>torch.optim.Optimizer</code>. Is there a way to apply this to the optimzier as well?</p>
<p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

zetyquickly: <p>I believe there’s no need to do that with optimizer. To reinitialize optimizer on new version of the model do <code>optimizer = optim.Adam(model.parameters())</code></p>

AlphaBetaGamma96: <p>Valid point, I did ‘solve’ that issue by using <code>torch.set_default_dtype(torch.float64)</code> but it does make sense to reinitialize the optimizer after pre-training!</p>
ERROR conda.core.link:_execute(699)

iLikeBuns: <p>I’m currently trying to install pytorch 1.9.0 on windows 10 through conda on python3 with cuda 11.3</p>
<p>Everything installed but at the end of installation I get these errors:</p>
<p>Preparing transaction: done<br>
Verifying transaction: done<br>
Executing transaction: done<br>
ERROR conda.core.link:_execute(699): An error occurred while installing package ‘conda-forge::cudatoolkit-11.1.1-heb2d755_7’.<br>
Rolling back transaction: done</p>
<p>LinkError: post-link script failed for package conda-forge::cudatoolkit-11.1.1-heb2d755_7<br>
location of failed script: C:\Users\iLikeBuns\anaconda3\Scripts.cudatoolkit-post-link.bat<br>
==&gt; script messages &lt;==<br>
<br>
==&gt; script output &lt;==<br>
stdout:<br>
stderr: Access is denied.</p>
<p>return code: 1</p>
<p>()</p>
<p>I’ve been googling but I couldn’t find issues similar to mine.</p>

111543: <p>Did you resolve it? I have a same problem.</p>

Liuxy94: <p>Hi,</p>
<p>I guess you installed the torch in the Windows platform. You can try run the Prompt in the admin mode.</p>

Samuel_Bachorik: <p>For me this is working.</p>
<p>Open CMD and write - conda create -n somename python = 3.8<br>
Then - activate somename<br>
Then - copy command from this page <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a><br>
Then push that command into CMD.</p>
<p>EDIT:<br>
If you dont know your cuda version, then push this command to CMD -<br>
<code>nvcc --version</code></p>

Majidur_Rahman: <p>I was also facing the same issue. It didn’t solve with the conda but worked with the pip command. Here it goes:</p>
<p>pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio===0.9.0 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>
Using Pytorch GPU with Numpy and Sklearn

Asad_Jeewa: <p>Hi</p>
<p>I have been using PyTorch for a while but since I switched over to the GPU I have realised that I made quite a few assumptions and would therefore like clarity:</p>
<ul>
<li>
<p>Since Numpy is not designed for GPU, should one always aim to use an alternate like Cupy? The tutorials state that numpy can be used to process the data and then convert to tensor, does this mean that only AFTER the data has been sent to the GPU, numpy should not be used?</p>
</li>
<li>
<p>Similarly, since SKLearn is not for GPU, should one always look for Pytorch implementations for the standard stuff like classification report, confusion matrix, gridsearchcv… (I have been getting TypeError: Implicit conversion to a NumPy array is not allowed)</p>
</li>
</ul>
<p>Thanks for your help.</p>

AlphaBetaGamma96: <p>Most linear algebra from numpy should be include with PyTorch already, but if you’re looking for a GPU supported version of sklearn, try <code>skorch</code>! <a href="https://github.com/skorch-dev/skorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - skorch-dev/skorch: A scikit-learn compatible neural network library that wraps PyTorch&lt;/t</a></p>
How to finetune a model after pruning?

garyyyccc: <p>I used <code>prune.global_unstructured()</code> to prune my CNN model. The Cifar10 dataset testing accuracy dropped from 85.6 to 82.9. Can I finetune the model to increase the accuracy?</p>
<p>I freeze the zero weight in the pruned model but I found that the loss will diverge after several training batches.</p>
<p>Does anyone know how to fix this?</p>
<p>Thanks.</p>
How to store flow and rgb images of Video Dataset on disk under prescribed inode limit

pytorch_learner1: <p>I am working on video understanding for which  I need to store video datasets such as UCF101, HMDB on the server. My server has an inode limit for each user say <code>L</code> number of files.<br>
If I store the RGB images and flow images of UCF-101; it overshoots the limit. I want to know if there is a better way to store these RGB images and flow images and process these during video training so that I don’t overshoot the inode limit.<br>
Have anyone faced this issue ?</p>
Passing Iteration number to LambdaLR

Jaideep_Valani: <p>How i can pass iteration to Lambda LR instead of Epoch</p>
<p>I want to varry my LR  using this function after running it with flat LR for certain number of iterations</p>
<p><code>def sched_cos(start, end, pos): return start + (1 + math.cos(math.pi*(1-pos))) * (end-start) / 2</code></p>
<p>Where pos=iteration number</p>
<p>My objective is to train with Flat LR for certaion portion of training cycle say 30 percent then there after decay the LR as per cosine function … Some thing like that as shown in image. Green Line.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b4ba014ea8ecde0bcc1518487097bc511790c3d.png" alt="image" data-base62-sha1="jSgotXGNa2x5QhpCv7eaffno4HP" width="378" height="248"></p>

Jaideep_Valani: <p>I think i found solution</p>
<pre><code class="lang-auto">model = nn.Linear(10, 2)
optimizer = optim.SGD(model.parameters(), lr=1e-3)
steps = 10

scheduler=CosineAnnealingLR(optimizer, T_max=15*2240//2, eta_min=0, last_epoch=-1, 
                            verbose=False)
lrs=[]
for epoch in range(15):
    for idx in range(2240):
        if epoch&gt;15//2:
            
            scheduler.step()
            #print(scheduler.get_last_lr()[-1])
        lrs.append(scheduler.get_last_lr()[-1])
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/4/94d4fd0422c98ca2e8c08c23f374f6953f715e53.png" alt="image" data-base62-sha1="leCZrATkqKcRaUZ7boGIivcpePF" width="461" height="309"></p>
Add a New Loss Term and a Gradient Term which are inconsistent

ZhichaoWang970201: <p>Question: I want to embed a new loss term into the neural network. However, its gradient is the modification of the derivative of the newly added loss term. I want to know if there is any reference.</p>
<p>The loss, gradient and updated gradient are shown in the following equations.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b461906da45ed314a66bbb53cd5de3bef073a82.png" alt="image" data-base62-sha1="aJTZD6pWqp9jRxEBiEyHKC9h0hs" width="488" height="246"></p>
Weight sharing within one layer

fteufel: <p>Hi,</p>
<p>I’m interested in implementing SortNet (<a href="https://ieeexplore.ieee.org/document/5957304" class="inline-onebox" rel="noopener nofollow ugc">SortNet: Learning to Rank by a Neural Preference Function | IEEE Journals &amp; Magazine | IEEE Xplore</a>). It’s an architecture designed for pairwise learning-to-rank that operates on a concatenation of two feature vectors. To enforce symmetrical behaviour, it adopts an interesting weight-sharing technique:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d67fabe197fae008f6367511863e264cb1a31891.png" alt="image" data-base62-sha1="uBxIemdXTiFCCPyKIVGrggGQuQ1" width="550" height="337"><br>
For each neuron h_i, there exists a <em>dual neuron</em> hi’ with the weights swapped, so that</p>
<blockquote>
<ol>
<li>v_k,i′=v_yk,i and vyk,i′=vxk,i hold, i.e., the weights from xk, yk to i are swapped in the connections to i′;</li>
<li>wi′,≻=wi,≺ and wi′,≺=wi,≻ hold, i.e., the weights of the connections from the hidden i to the outputs N≻, N≺ are swapped in the connections leaving from i′;</li>
<li>bi=bi′ and b≻=b≺ hold, i.e., the biases are shared between the dual hiddens i and i′ and between the outputs N≻ and N≺.</li>
</ol>
</blockquote>
<p>I was not able to find a reference implementation. As I understand it, this setup would require me to share elements of a nn.Linear’s weight tensor. Any way to achieve this in Pytorch?</p>
Does the official conda install include / require cudnn?

Data-drone: <p>I notice that the official <code>conda install</code> doesn’t include cudnn. Is that required separately? Or it is included in the <code>cudatoolkit</code>?</p>
<p>FYI the website says to use:</p>
<p><code>conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia</code></p>

ptrblck: <p>cuDNN is statically linked into PyTorch, as no <code>conda</code> binaries or <code>pip</code> wheels are available for it.</p>

Data-drone: <p>so that mean I need to manually make sure they are available or all it is all good?</p>
<p>If that conda command runs successfully and I <code>torch</code> is showing <code>cuda</code> capabilities then all good?</p>

ptrblck: <p>Yes, the binaries will ship with cuDNN and you can check the version via <code>print(torch.backends.cudnn.version())</code>.</p>

Data-drone: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="130713">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>print(torch.backends.cudnn.version())</p>
</blockquote>
</aside>
<p>Cheers! Thanks for clarifying</p>
Modifying a model

Ammara: <p>I am initializing a model and modifying it like this.</p>
<pre><code class="lang-auto"># create model
print("=&gt; creating model '{}'".format(args.arch))
model = models.__dict__[args.arch]()

# freeze all layers but the last fc
for name, param in model.named_parameters():
        if name not in ['fc.weight', 'fc.bias']:
            param.requires_grad = False
    
# init the fc layer
if args.multi_label:
      model.fc = nn.Linear(model.fc.in_features, args.num_classes)
      model = nn.Sequential(
                            model,
                            nn.Sigmoid()
                           )

print(model)
</code></pre>
<p>Is this the correct approach? Or should the modifications always have to be in <code>__init__</code> and <code>forward</code> method of an <code>nn.Module</code> class?</p>
Compile a CUDA compatible version on a non CUDA computer?

Data-drone: <p>Can I compile a cuda compatible version of pytorch on a machine with no GPUs available?</p>
<p>I have access to a computer with high CPU count but no GPUs and was wanting to leverage it to compile PyTorch faster. I am compiling within an nvidia docker container which I am then using on my machine that has GPUs. <code>nvidia-smi</code> is returning the gpus properly <code>nvcc</code> seems to be working properly</p>
<p>But when I run <code>torch.cuda.is_available()</code> in python I get:</p>
<pre><code class="lang-auto">/opt/conda/envs/computer_vision/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  /opt/pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() &gt; 0

</code></pre>
<p>Any guidance?</p>

ptrblck: <p>It should be possible to use <code>nvcc</code> without a GPU.<br>
The error message might be raised, if your NVIDIA driver or local CUDA toolkit isn’t properly installed or found. Did you execute the docker container via <code>nvidia-docker</code> or with the <code>--gpus=all</code> option?</p>

Data-drone: <p>I did run with <code>--gpus=all</code> and I set <code>--ipc=host</code> as required</p>

Data-drone: <p>I am using driver <code>460.32.03</code> with the nvidia-cuda container <code>cuda:11.2.1-cudnn8-devel</code> and I installed <code>magma-cuda112</code> could it be that <code>magma-cuda112</code> and <code>cuda:11.2.1-cudnn8-devel</code> don’t mix?</p>

ptrblck: <p>How did you install <code>magma-cuda112</code>? If you’ve installed a conda packagge, did the install logs show that PyTorch would be downgraded to a CPU-only version?<br>
If not, then I don’t think that <code>magma-cuda112</code> should have influence in PyTorch being able to find a GPU.</p>

Data-drone: <p>Sorry missed the reply. It seems variable sometimes whether conda wants to downgrade or not. I never dug deep enough to work out why. But I did get it working without downgrade</p>
Question about FrozenBatchNorm for finetune

jhp: <p>Hi, I have a quick questions about the FrozenBatchNorm.</p>
<p>When using a pretrained model for fintuning, I’ve heard that it is common for the BatchNorm to freeze. But my data is much different from the existing pre-trained ones, so I want to train this as well. Is it better to freeze only the running statistics? Or should I change all the parameters in BatchNorm if I want to unfreeze it?</p>
Depthwise and Separable convolutions in Pytorch?

Bixqu: <p>Anyone have an idea of how I can implement Depthwise convolutions and Separable Convoltuons in pytorch? The definitions of these can be found <a href="https://arxiv.org/pdf/1706.03059.pdf" rel="nofollow noopener">here.</a></p>
<p>Can one define those using just regular conv layers somehow?</p>

smth: <p>for Depthwise / Separable, you can use Conv’s <code>groups</code> parameter.</p>
<p><a href="http://pytorch.org/docs/master/nn.html#conv2d" class="onebox" target="_blank">http://pytorch.org/docs/master/nn.html#conv2d</a></p>
<p>If groups = nInputPlane, then it is Depthwise. If groups = nInputPlane, kernel=(K, 1), (and before is a Conv2d layer with groups=1 and kernel=(1, K)), then it is separable.</p>

Bixqu: <p>Thanks so much, I will try it out.</p>

hellodrx: <p>Hi, Smith, is the Depthwise / Separable convolutions still very slow on Pytorch as before?</p>

Tejan_Mehndiratta: <p>Hi,</p>
<p>In Spatial Separable Convolution:<br>
why is groups=nInputPlane?</p>

eslammofreh: <p>Hi, hellodrx, I am facing this issue. Do you solve it ?</p>

leimao: <p>According to the definition, I think this implementation is not the separable conv.</p>
Transforming inside CustomDataset or inside CustomDataloader - which is better? pytorch

DanteTemplar: <p>Do I want to apply transformations to augment data inside the <strong>CustomDataloader</strong> class in Pytorch, or should I do it in the <strong>CustomDataset</strong> class? I would be grateful if you could show an example of how to use it correctly.</p>

ptrblck: <p>You would usually create a custom <code>Dataset</code> and would not change anything in the <code>DataLoader</code>.<br>
<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">This tutorial</a> shows how to create a custom <code>Dataset</code> and how to apply the transformations in the <code>__getitem__</code> method.</p>
Index of the last timestep LSTM

CarlosValleA: <p>Hi! I’m working with LSTM and the output tensor has a shape of (batch_size, seq_length, hidden_size). To get only the last output of the network, I need to get the last index of the tensor  <code>output[:, seq_length -1, :]</code> or the first one <code>output[:, 0, :]</code> ?</p>
<p>Thanks!</p>

ptrblck: <p>You can get the last time step via <code>output[:. -1]</code> assuming you’ve used <code>batchj_first=True</code> during the setup of the <code>nn.LSTM</code>.</p>
How to give input to an intermediate layer

Xia_Zhengmi: <p>Hi everyone,</p>
<p>Assume that we have a model which takes x as input and outputs y at the end, now I want to train it again but give the input to an intermediate layer rather than at the input of the model. May youp please help me how I need to do it with a simple snippet code.</p>
<p>Thank you in advance,</p>

zetyquickly: <p>Hi <a class="mention" href="/u/xia_zhengmi">@Xia_Zhengmi</a></p>
<p>First of all the intermediate level might not have the same input size as the first level, I’d propose to check the input of your <code>layer_of_interest</code> with pre forward hook that outputs the size of input.</p>
<pre><code class="lang-auto">def print_input_size(self, input):
  print('Inside ' + self.__class__.__name__ + 'pre forward')
  print(input.shape)

net.layer_of_interest.register_forward_pre_hook(print_input_size)
</code></pre>
<p>Secondly, you can shorten your model using Python slicing and <code>nn.Sequential</code> (this option will work only if your <code>net</code> is already a <code>Sequential</code>, if not find a container in it and slice it)</p>
<pre><code class="lang-auto">from = 3 # number of layer_of_interest
new_net = nn.Sequential(*net[from:])
</code></pre>
<p>Hope that will help!</p>

Xia_Zhengmi: <p>Hi <a class="mention" href="/u/zetyquickly">@zetyquickly</a>,</p>
<p>Many thanks for your response. Quite helpful and exactly what I need for my project.</p>
Automatic mixed precision result in NaN

ginobilinie: <p>Hi, I’d like to ask if anyone can help about using of torch.cuda.amp for float16 training. When I use it to train (only forward pass and loss computation, config.use_fp16_bp=False), I observe NaN even though I tried many times.</p>
<p>Environment:<br>
Pytorch 1.6 and cuda10.1</p>
<p>Snippet:</p>
<pre><code>      with torch.cuda.amp.autocast():
            loss = model(imgs, gts)
            # reduce the whole loss over multi-gpu
            if engine.distributed:
                dist.all_reduce(loss, dist.ReduceOp.SUM)
                loss = loss / engine.world_size
            else:
                if len(loss.shape)&gt;1:
                    loss = Reduce.apply(*loss) / len(loss)

        optimizer.zero_grad()
        current_idx = epoch * config.niters_per_epoch + idx
        lr = lr_policy.get_lr(current_idx)

        for i in range(0, len(optimizer.param_groups)):
            optimizer.param_groups[i]['lr'] = lr


        if config.use_fp16_bp:
            # Scales the loss, and calls backward()
            # to create scaled gradients
            scaler.scale(loss).backward()

            # Unscales gradients and calls
            # or skips optimizer.step()
            scaler.step(optimizer)

            # Updates the scale for next iteration
            scaler.update()
        else:
            loss.backward()
            optimizer.step()</code></pre>

ptrblck: <p>A NaN in the loss would point towards an overflow in the model.<br>
Are you training the model from scratch and do you know, if the current config is converging properly in <code>float32</code>? Note that an exploding loss would cause NaN outputs in all numerical formats, but way earlier in <code>float16</code> due to the smaller range compared to <code>float32</code>.</p>

ginobilinie: <p>Thank you. Yes, I’m training from scratch. I’m sure the current config works in in float32.</p>

nesterd: <p>Hi, is there any solution?</p>
LSTM units in Pytorch

chericha: <p>Hello, I am trying to implement this architecture (<a href="https://www.hindawi.com/journals/jcse/2021/8785636/fig6/" class="inline-onebox" rel="noopener nofollow ugc">Figure 6 | COVID-19 Pandemic Forecasting Using CNN-LSTM: A Hybrid Approach</a>) using torch.nn.LSTM but I am not sure what 200 units LSTM corresponds to in nn.LSTM parameters. I would guess it is <strong>num_layers</strong> but then how will <strong>hidden_size</strong> be determined?</p>

zetyquickly: <p>Hey <a class="mention" href="/u/chericha">@chericha</a></p>
<p>Let me try to clarify these two <code>LSTM</code> parameters:</p>
<ol>
<li>
<strong>num_layers</strong> is the parameter responsible for the number of <code>LSTM</code> cells stacked together, in other words how deep is recurrence of that RNN</li>
<li>
<strong>hidden_size</strong> is the number a hyperparameter that correspond to a length of hidden vector that is passed from cell to cell</li>
</ol>
<p>Hope it helps</p>
How do you use the weights/trained data in an actual custom script from yolov5?

Michael_Jensen: <p>Hey,</p>
<p>Read the getting started guides etc, and got to creating a custom dataset to detect some specific object in a video (mp4), I cloned the repo, found lots of images, and annotated them in yolov5 format, and used the train.py script and got a pretty good weight? file (the one in \yolov5\runs\train\yolov5s_resultsxxx\weights\best.pt), that was pretty accurate when using detect.py on a mp4 file.</p>
<p>Looked at countless blogs, and videos, everyone just shows how to use detect.py, not actual custom python code that uses the files generated by train.py, I even tried to dissect the detect.py file, but it’s really a terrible mess to understand for a torch beginner, does anyone know an example of how to load that best.pt file that was created from train.py, basically I want to run the detection on live video, frame by frame, so can’t use detect.py, the program needs to work on the bounding boxes that is detected, but there doesn’t seem to be any examples of actual use, outside the simple train.py → detect.py, usually done in codelab, which isn’t practical to real world use, really confused how this is supposed to work.</p>
<p>I read the parts about saving/loading state etc, but isn’t that what train.py already did for the model, when i try to load best.py that was generated from train.py it just complains it’s not a valid file to load (despite it working fine with detect.py).</p>
<p>Hope someone can point me in the right direction, on how to use the weights/model that was generated from the yolo5 stuff, docs just make no sense to me, on how to move that trained data into a real world standalone script/app, outside the provided scripts/codelabs.</p>

zetyquickly: <p>Hey <a class="mention" href="/u/michael_jensen">@Michael_Jensen</a></p>
<p>Your confusion of a beginner is totally understandable. Could you provide which repo do you try to execute?</p>
<p>Without it I’d say that you need to find in the code following:</p>
<ol>
<li>a class that implements PyTorch model</li>
<li>understand where is the preprocessing of images is done (inside a model or in some external function)</li>
<li>then write your own script that takes frames, preprocess them and pass it to the model one by one.</li>
</ol>
Torch.histc along an axis?

sbelharbi: <p>hi,<br>
is there a way to compute histogram for each row of matrix of size (n, m) using <a href="https://pytorch.org/docs/stable/generated/torch.histc.html" rel="noopener nofollow ugc"><code>torch.histc</code></a><br>
so far, <code>histc</code> computes the histogram of the the entire tensor.</p>
<p>thanks</p>

HDCharles: <p>not conveniently, you’d have to do something like</p>
<p>x = torch.randn(3,3,3,3)*10<br>
a = torch.tensor([torch.histc(x[i], 100, 0, 20).tolist() for i in range(3)])</p>

sbelharbi: <p>sure<br>
but i was looking for batchable operation with no loop.<br>
thanks</p>
Does pytorch support quantized model conversion to onnx

bigtree: <p>I am trying to convert a quantied model trained in pytorch to onnx. And then got</p>
<pre><code class="lang-auto">  File "test_QATmodel.py", line 276, in test
    torch.onnx.export(model_new, sample, 'quantized.onnx')#, opset_version=11, operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK)
  File "/monly/workspaces/bigtree/miniconda3/envs/color_detcet/lib/python3.7/site-packages/torch/onnx/__init__.py", line 280, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/monly/workspaces/bigtree/miniconda3/envs/color_detcet/lib/python3.7/site-packages/torch/onnx/utils.py", line 94, in export
    use_external_data_format=use_external_data_format)
  File "monly/workspaces/bigtree/miniconda3/envs/color_detcet/lib/python3.7/site-packages/torch/onnx/utils.py", line 695, in _export
    dynamic_axes=dynamic_axes)
  File "monly/workspaces/bigtree/miniconda3/envs/color_detcet/lib/python3.7/site-packages/torch/onnx/utils.py", line 459, in _model_to_graph
    _retain_param_name)
  File "monly/workspaces/bigtree/miniconda3/envs/color_detcet/lib/python3.7/site-packages/torch/onnx/utils.py", line 422, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "monly/workspaces/bigtree/miniconda3/envs/color_detcet/lib/python3.7/site-packages/torch/onnx/utils.py", line 370, in _trace_and_get_graph_from_model
    orig_state_dict_keys = _unique_state_dict(model).keys()
  File "monly/workspaces/bigtree/miniconda3/envs/color_detcet/lib/python3.7/site-packages/torch/jit/_trace.py", line 71, in _unique_state_dict
    filtered_dict[k] = v.detach()
AttributeError: 'NoneType' object has no attribute 'detach'
</code></pre>
<p>Does the quantized model-&gt;onnx supported today?</p>

HDCharles: <p>I believe this thread goes through the existing support for ONNX: <a href="https://discuss.pytorch.org/t/onnx-export-of-quantized-model/76884/23" class="inline-onebox">ONNX export of quantized model - #23 by ZyrianovS</a></p>
Standardize whether logits and probs get normalized internally in distribution functions?

johnstjohn: <p>Hi there,<br>
I have been writing some wrappers for loss functions that use various probability distributions in pytorch. I’m noticing some inconsistencies in the meaning of a few arguments of the same name that have slowed me down, and unless I am missing something seems like a good area for cleanup. Here is one example I have found which highlights this point:</p>
<pre><code class="lang-auto">torch.distributions.Multinomial(logits=...)
</code></pre>
<p>vs</p>
<pre><code class="lang-auto">torch.distributions.Binomial(logits=...)
</code></pre>
<p>torch.distributions.Multinomail (which farms the initialization to torch.distributions.Categorical) normalizes both parameters on intialization:</p>
<pre><code class="lang-auto">            self.probs = probs / probs.sum(-1, keepdim=True)
            ...
            self.logits = logits - logits.logsumexp(dim=-1, keepdim=True)
</code></pre>
<p>On the other hand torch.distributions.Binomial doesn’t do this:</p>
<pre><code class="lang-auto">        if probs is not None:
            self.total_count, self.probs, = broadcast_all(total_count, probs)
            self.total_count = self.total_count.type_as(self.probs)
        else:
            self.total_count, self.logits, = broadcast_all(total_count, logits)
            self.total_count = self.total_count.type_as(self.logits)
</code></pre>
<p>Would it create any issues if I do something similar to what is done with categorical in other distribution functions? It seems like if your input is properly normalized already, the above wouldn’t cause issues (unless probs.sum(-1) == 0), but likely there are already issues downstream if that is the case.</p>
<p>Thanks for letting me know if this is an agreeable change, and if there are already tickets/etc for it. If not it seems like a nice, small-ish thing to contribute, unless I am missing something about typical usages etc.</p>
<p>Thanks,<br>
John</p>
How to get rid of for loops to make the loss function faster?

seyeeet: <p>Would it be possible to get rid of these two for loops to make my computation faster?</p>
<pre><code class="lang-auto">import torch
target = torch.ones(5,1)
target[1]=0
target[4]=0

batch_size = 5
embed_feat_1 = torch.rand(batch_size,512)
embed_feat_2 =torch.rand(batch_size,512)
inputs = torch.rand(batch_size,3,1)
MSELoss = torch.nn.MSELoss()

loss = 0
for q in (range(inputs.shape[0])):
  for j in range(inputs.shape[0]):
      if q == j or target[q]!=target[j]:
          continue
      dij_plus = MSELoss(embed_feat_1[q,:], embed_feat_1[j,:]) ** 2
      dij_minus = MSELoss(embed_feat_1[q,:], embed_feat_2[q,:]) ** 2
      dij_minus = dij_minus + MSELoss(embed_feat_1[j,:], embed_feat_2[j,:]) **2
      dij_minus = dij_minus/2
      loss = loss + torch.log(torch.exp(-dij_plus)/(torch.exp(-dij_plus) + torch.exp(-dij_minus))) * (-1)
    
loss
</code></pre>
<p>the formula for the loss function is technically:<br>
<code>-torch.log(torch.exp(-dij_plus )/(torch.exp(-dij_plus )+torch.exp(-dij_minus )))</code></p>
Select-average-insert spans from a multi-dimensional tensor to another multi-dimensional tensor

gorjanradevski: <p>Let’s assume that I have a 3D tensor <code>x = torch.rand(4, 17, 128)</code>, from which I want to average a selected span of elements (of variable length) for each row:</p>
<pre><code class="lang-python">spans = [[[1, 3], [6,10], [11, 15]],
         [[1, 10]],
         [[1, 5]],
         [[1,7], [8,16]]]
</code></pre>
<p>For example, in this case, I want to average elements <code>1</code> to <code>3</code>, <code>6</code> to <code>10</code>, and <code>11</code> to <code>15</code> from row one, then elements <code>1</code> to <code>10</code> from row two, etc. Each of the averaged tensors, in this case, will be of shape <code>(128,)</code>. Then, I want to insert the span average to another tensor, defined as:</p>
<pre><code class="lang-python"># 4
num_elements = len(spans)
# 3
max_spans = max([len(span) for span in spans])
# [128]
remaing_dims = x.size()[2:]
# [4, 3, 128]
output = torch.zeros(num_elements, max_spans, *remaing_dims)
</code></pre>
<p>Currently, in order to do what I want I do the following:</p>
<pre><code class="lang-python">for i, element_spans in enumerate(spans):
    for j, span in enumerate(element_spans):
        output[i, j] = x[i, span[0]:span[1]].mean(0)
</code></pre>
<p>which is a really brute-force kind of approach. Therefore, I am wondering how can I speed-up the whole select-average-insert.</p>
<p>Thanks in advance!</p>
Out Of Memory Error - for batch size 32 & greater. Not completing a single forward pass!

Abhishek_K: <p>Hello All,</p>
<p>I am trying to implement a U-Net Architecture for Image Segmentation task.<br>
<strong>Problem</strong>:  : <em><strong>Out of Memory Error</strong></em> thrown even before completing a single forward pass for a batch size of 32 and greater. But for a batch size of 5, I am able to train the model without any errors.</p>
<p><strong>Model :</strong>  Taken from <a href="https://github.com/mateuszbuda/brain-segmentation-pytorch/blob/master/unet.py" class="inline-onebox" rel="noopener nofollow ugc">brain-segmentation-pytorch/unet.py at master · mateuszbuda/brain-segmentation-pytorch · GitHub</a></p>
<pre><code class="lang-auto">def _batch_trainer(net,lossFn,optimFn,batch_size=64,val_percentage = 0.1,device='cpu'):
  net.train()
  val_len = int(len(Caravan_dataSet)*val_percentage)
  train_len = len(Caravan_dataSet) - val_len
  train,val = random_split(Caravan_dataSet,[train_len,val_len])
  train_iterable = DataLoader(train,batch_size=batch_size,shuffle=True,drop_last=True)
  val_iterable = DataLoader(val,batch_size=batch_size,shuffle=True,drop_last=True)
  train_cum_loss,val_cum_loss = 0,0

  print("Iterator generated.......")
  
  
  for data in tqdm(train_iterable):
    imgData,maskData = data[0].detach().to(device),data[1].detach().to(device)
    print(imgData)
    #print(torch.cuda.memory_stats(device=torch.cuda.current_device()))
    optimFn.zero_grad()
    #with torch.set_grad_enabled():
    #print(torch.cuda.memory_stats(device=torch.cuda.current_device()))
    
    #torch.cuda.empty_cache()
    pred_masks = net(imgData)
    #except:
    #print(torch.cuda.memory_stats(device=torch.cuda.current_device()))

    # print("done with forward pass")
    # print("Pred size : ",pred_masks.size())
    loss = lossFn(pred_masks,maskData)
    loss.backward()
    train_cum_loss+=float(loss)
    optimFn.step()
    

  # net.eval()
  # for ind,data in enumerate(val_iterable):
  #   imgData,maskData = data['Images'].to(device),data['Masked Images'].to(device)
  #   imgData,maskData = imgData.to(device),maskData.to(device)
  #   pred_masks = net(imgData)
  #   loss = lossFn(pred_masks,maskData) ## Loss Per Batch 
  #   val_cum_loss+=float(loss)

  train_loss_per_epoch = train_cum_loss/len(train_iterable)
  #val_loss_per_epoch = val_cum_loss/len(val_iterable)
  print("Completed an Epoch")
  #return train_loss_per_epoch,val_loss_per_epoch ## Loss per Epoch  
  return train_loss_per_epoch
</code></pre>
<p>Though multiple queries have been raised on the same issue, I was not able to resolve it. I have used loss.item() as well to append the loss values into a list.</p>
<p>Thanks in advance</p>

ptrblck: <p>Your GPU doesn’t seem to have enough memory for the used batch size, so you would need to reduce the memory usage e.g. by lowering the batch size or by trading compute for memory via <code>torch.utils.checkpoint</code>.</p>
Torch.autograd keeps on increasing GPU usage on backward pass

Gul_Zain: <p>Hello,<br>
I am working on <a href="https://github.com/tamarott/SinGAN" rel="noopener nofollow ugc">SinGAN</a> and they use a gradient penalty loss which just keeps on increasing GPU usage to the extent that I can not train even on A100(40 GB). I am not sure where the problem could be or what are the ways around it. I have gone through previous related posts but they were either had a problem in their own logic or were talking about <strong>memory leak</strong>(which may happen if you set create_graph=False). I am also pasting the chunk of code below where the memory increase happens continuously.</p>
<pre><code class="lang-auto">LAMBDA=opt.lambda_grad
device=opt.device
alpha = torch.rand(1, 1)
alpha = alpha.expand(real.size())
alpha = alpha.to(device)#cuda() #gpu) #if use_cuda else alpha

interpolates = alpha * real + ((1 - alpha1) * fake)


interpolates = interpolates.to(device)#.cuda()
interpolates = torch.autograd.Variable(interpolates, requires_grad=True)
disc_interpolates = netD(interpolates)
            
gradients = torch.autograd.grad(outputs=disc_interpolates, inputs=interpolates,
                                      grad_outputs=torch.ones(disc_interpolates.size()).to(device),#.cuda(), #if use_cuda else torch.ones(
                                          #disc_interpolates.size()),
                                      create_graph=True, retain_graph=True, only_inputs=True)[0]
            #LAMBDA = 1            
gradient_penalty = ((gradients.norm(2, dim=1) - 1) ** 2).mean() * LAMBDA

            # gradeint penalty calculated
            # gradient_penalty = functions.calc_gradient_penalty(netD, real, fake, opt.lambda_grad, opt.device)            
**gradient_penalty.backward()**

</code></pre>
<p>The increase in GPU usage happens precisely at gradient_penalty.backward() which I have also made bold. I have already tried detaching and deleting the gradients and gradient_pentalty variables. It doesn’t work.</p>

Gul_Zain: <p>The problem goes away as soon as I switch to pytorch 1.4.1. However, I am not sure why the problem exists in Pytorch 1.7.0.</p>

Gul_Zain: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> any insights here would be great <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=10" title=":smile:" class="emoji" alt=":smile:"> This seems to be a pytorch related issue. I just checked code works fine on 1.5.0 as well.</p>
Access to attributes of model wrapped in DDP

sbelharbi: <p>hi,<br>
i have a model that is wrapper within a ddp (DistributedDataParallel).</p>
<p>what is the right way to access to all model attributes?<br>
i recall i had similar issue with <code>DataParallel</code>.</p>
<p>in a ddp, the model is stored in <code>ddp.module</code> <a href="https://github.com/pytorch/pytorch/blob/371c6612b39c1fa18d6c2e22613e292ae87b686f/torch/nn/parallel/distributed.py#L536" rel="noopener nofollow ugc">here</a>.<br>
so far, i use <code>ddp_model.module.attribute</code>.<br>
is there a better way? because i have to go through entire code to change this…<br>
also, i want to make sure that i am using the right model because i need to get other things from it: grad, and other stuff from some layers.</p>
<p>is it safe to use the same trick for <a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html" rel="noopener nofollow ugc">dataparallel</a> ? i want to avoid <code>if isinstance(model, ddp) / isinstance(model, MYMODEL)</code> as well.</p>
<pre><code class="lang-python">class myDDP(torch.nn.parallel.DistributedDataParallel):
   def __getattr__(self, name):
        try:
            return super().__getattr__(name)
        except AttributeError:
            return getattr(self.module, name)
</code></pre>
<p>this goes with the risk of undetected name clash… until later.</p>
<p>thanks</p>

gcramer23: <blockquote>
<p>in a ddp, the model is stored in <code>ddp.module</code> <a href="https://github.com/pytorch/pytorch/blob/371c6612b39c1fa18d6c2e22613e292ae87b686f/torch/nn/parallel/distributed.py#L536" rel="noopener nofollow ugc">here </a>.<br>
so far, i use <code>ddp_model.module.attribute</code> .</p>
</blockquote>
<p>That seems correct since you are accessing a python object attribute.</p>
<blockquote>
<p>is it safe to use the same trick for <a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html" rel="noopener nofollow ugc">dataparallel</a> ?</p>
</blockquote>
<p>Yes, it should be safe since you are accessing the attributes of a python object.</p>

sbelharbi: <p>one can also reference the model inside ddp and work on the reference to avoid <code>ddp.modul.attr</code>.<br>
so, the optimizer can work on ddp, but you can do everything else over the reference.<br>
see <a href="https://discuss.pytorch.org/t/model-and-ddp-wrapped-model/123023">this</a>.<br>
the <a href="https://github.com/allenai/allennlp/blob/c5bff8ba0d835eb03931f10f4f427ffe936cf796/allennlp/training/gradient_descent_trainer.py#L302" rel="noopener nofollow ugc">example</a>.</p>
Github pages are not recognized as websites

Sayed_Nadim: <p>Hi, I tried to update my profile here (<a href="http://discuss.pytorch.org">discuss.pytorch.org</a>). But when I tried to update the website section,  it says invalid whenever I put my github pages link. I tried with https as the beginning. Yet, it can’t recognize the github pages as valid website link. Is it expected behavior i.e. the site only recognizes .com or .org domain as website postfix?</p>
Time Series Classification, Not Improving Beyond First Epoch

jj77: <p>Hello,</p>
<p>Full disclosure, I am new to PyTorch and machine learning in general.</p>
<p>I am trying to follow <a href="https://www.youtube.com/watch?v=PCgrgHgy26c&amp;t=2076s" rel="noopener nofollow ugc">this tutorial</a> to preform Time Series Classification. However, instead of using the data set provided in the tutorial, I substituted it with my own data set.</p>
<p>However, when I run the code with my data set it does not seem to improve beyond the very first epoch. Does anyone have any ideas as to what I could experiment with to try and fix this? I’ve tried experimenting with different drop out rates but that did not seem to any have noticeable effect. Perhaps I forgot to make and obvious change somewhere.</p>
<p>You can view the modified code with my data set here,</p>
<p><a href="https://colab.research.google.com/drive/1H048DMQlpSCMn7fh_1HsGa6RhdAcU233?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1H048DMQlpSCMn7fh_1HsGa6RhdAcU233?usp=sharing</a></p>
<p>Thanks!</p>

ptrblck: <p>You could use a small subset of the dataset (e.g. just 10 samples) and try to overfit your model on it by playing around with the hyperparameters etc. Once this is done you could scale up the use case again.</p>

jj77: <p>Hello! I’ve been experimenting with different parameters for the past few days, but I haven’t made any significant progress.</p>
<p>Are there any PyTorch tutorials that show how to do multivariate time series classification?</p>
<p>Thanks!</p>
[Urgent] - RAM Memory Leak iterating a short-lived dataloader

gilbertocunha: <p>Hi, I am experiencing a RAM (not VRAM) memory leak issue while using <code>next(iter(dataloader))</code> on a short lived dataloader.</p>
<p>I am training a model in the context of Meta-Learning Few-Shot Classification and I have to randomly sample different classes AND images of those classes for each step of my training loop.</p>
<p>I am therefore creating a new Dataloader for each training step which only contains a single batch.</p>
<p>I already saw this thread that addresses a similar problem, but I think it does not apply to mine because I have to create a new DataLoader per training step: <a href="https://github.com/pytorch/pytorch/issues/1917#issuecomment-433698337" class="inline-onebox" rel="noopener nofollow ugc">Get a single batch from DataLoader without iterating · Issue #1917 · pytorch/pytorch · GitHub</a></p>
<p>I do the following per training step:</p>
<pre><code class="lang-auto"># Create my task (collection of randomly sampled classes)
task = task_type(meta_train_classes, ...)
# Fetch DataLoader for that task
dataloader = fetch_dataloaders('train', task)
...
# Iterate dataloader (causes memory leak)
X_sup, Y_sup = next(iter(dataloader))
</code></pre>
<p>I also use numpy arrays in my custom Dataset class (and from memory profilings, memory does not seem to leak from <code>__getitem__()</code>).</p>
<p>I am wondering if I could be doing this in any other way to prevent this memory leak. I have to train my model for a huge number of steps and my 64GB of RAM fills ups slowly, but fast enough that I can’t fully train the model.</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="gilbertocunha" data-post="1" data-topic="116888">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gilbertocunha/48/27690_2.png" class="avatar"> gilbertocunha:</div>
<blockquote>
<p>I have to create a new DataLoader per training step</p>
</blockquote>
</aside>
<p>Do you make you sure the old DataLoader is deleted? That sounds like that could be a potential leak!</p>

gilbertocunha: <p>Yes, I have already tried using both <code>del dataloader</code> and the garbage collector, but none have worked, unfortunately</p>

Sayed_Nadim: <p>Are you using TensorBoard or TensorBoardX in your code? I had a similar issue where I couldn’t find bug in the dataloader. However,  the tensorboardX was consuming RAM slowly.</p>
Re-load and re-train model with self defined layers

Ardeal: <p>Hi,</p>
<p>I defined a new layer as the following code.<br>
after the training of the network with the following layer, I would like to re-load the model to retrain.</p>
<p>My questions are:</p>
<ol>
<li>torch.load() and model.load_state_dict() will load thoss parameters in the layer, will those layer be tensor type or parameter type after the re-loading?</li>
<li>if it is tensor type, the mask_weight in my following code couldn’t be optimized, right?</li>
<li>how to make the reloaded model trainable?</li>
</ol>
<pre><code class="lang-python">class SMConv2d(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, padding=1, stride=1):
        super(SMConv2d, self).__init__()
    
        self.mask_weight = nn.Parameter(torch.Tensor(out_channels, in_channels, kernel_size, kernel_size))
        nn.init.constant_(self.mask_weight, 1)
    
    def compute_mask(self,temp):
        self.mask = self.mask_weight*temp
        # self.mask_weight = self.mask_weight * temp
       
        return self.mask 

    def forward(self, x, temp=1):
        masked_weight = self.compute_mask(temp)
        out = F.conv2d(x, masked_weight, stride=self.stride, padding=self.padding)        
        return out
</code></pre>

ptrblck: <p>The recommended way to save and load a model is to use the <code>state_dict</code> and to recreate the model before loading it as described <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html">here</a>.<br>
This will make sure all parameters stay as such and are trainable.</p>
Pybind for Function?

soulslicer: <p>Looking at this tutorial here:<br>
<a href="https://pytorch.org/tutorials/advanced/cpp_autograd.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_autograd.html</a></p>
<p>How exactly do i write a pybind binding for this? Eg.</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;

using namespace torch::autograd;

class MulConstant : public Function&lt;MulConstant&gt; {
 public:
  static torch::Tensor forward(AutogradContext *ctx, torch::Tensor tensor, double constant) {
    // ctx is a context object that can be used to stash information
    // for backward computation
    ctx-&gt;saved_data["constant"] = constant;
    return tensor * constant;
  }

  static tensor_list backward(AutogradContext *ctx, tensor_list grad_outputs) {
    // We return as many input gradients as there were arguments.
    // Gradients of non-tensor arguments to forward must be `torch::Tensor()`.
    return {grad_outputs[0] * ctx-&gt;saved_data["constant"].toDouble(), torch::Tensor()};
  }
};
</code></pre>
<p>How do i make this work with Python / Pytorch</p>
Speedup forward in model partially frozen

sbelharbi: <p>hi,<br>
is there a way to further speed up the forward (in train or eval modes) of a model that has part of its layers frozen?<br>
the freezing is done by setting:</p>
<pre><code class="lang-python">param.requires_grad = False
BatchNorm2d_layer.eval()
Dropout_layer.eval()
</code></pre>
<p>the frozen layers are included in the forward.</p>
<p>thanks</p>