akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Child Module as "weak reference"

PiotrDabkowski: <p>I have a large pretrained <code>torch.nn.Module</code> that is being used by many different <code>torch.nn.Module</code>s.</p>
<p>For the purpose of example assume it is a pretrained &amp; fixed ResNet image model, that I use for feature generation in many different image classifiers.</p>
<p>How can I best store such model within <code>torch.nn.Module</code> if I do not want the weights to be stored as part od the model state?</p>
<p><strong>Option 1:</strong><br>
Simply storing it as a child module:</p>
<pre><code class="lang-auto">class MyModel(torch.nn.Module):
    def __init__(self, resnet: ResNet):
       self.resnet = resnet

</code></pre>
<p>Would result in its parameters being stored as part of the <code>Module</code> and increase the checkpoint size.</p>
<p><strong>Option 2</strong></p>
<p>Store the “reference”</p>
<pre><code class="lang-auto">class MyModel(torch.nn.Module):
    def __init__(self, resnet: ResNet):
       self._resnet = [resnet]
</code></pre>
<p>With this approach the Module just stores the reference to the ResNet and does not really manage it. The syntax is a bit annoying.</p>
<p>Is there a better way to store weak references?<br>
Also, if I want to only enable device management of the child module, but not weight ownership, is overriding <code>apply</code> method the best option?</p>
Pytorch C++ support on RIOT platform

MakGulati: <p>Hello Pytorch lovers!</p>
<p>I am facing some problems porting c++ torch code to low-level embedded devices for very level on-board training on RIOT platform. It would be nice if somebody in the community help me in addressing  following issues:</p>
<ul>
<li>
<p>PyTorch is using a lot of exceptions. For RIOT this is an issue as a CPP exception needs dynamic allocation and on embedded this causes issues with the low amount of memory available. RIOT compiles CPP code with -fno-exceptions because of this and pytorch doesn’t compile with this compiler flag enabled.</p>
</li>
<li>
<p>Multi-threading is heavily used in PyTorch with the threadpool. On the bare metal target this is not available and usually not useful because of the single core nature of the system.</p>
</li>
</ul>
<p><strong>Hardware used: Arduino ble 33</strong></p>
<p>Thanks in advance!</p>
Assertion error when using resnet_18 as backbone with faster_rcnn

ajaychawda58: <p>File “train.py”, line 219, in <br>
train_one_epoch(model, optimizer, traindata, device, epoch, print_freq=100)<br>
File “/home/ajay/Desktop/ID_CNN/engine.py”, line 30, in train_one_epoch<br>
loss_dict = model(images, targets)<br>
File “/home/ajay/id-env/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/ajay/id-env/lib/python3.6/site-packages/torchvision/models/detection/generalized_rcnn.py”, line 100, in forward<br>
detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)<br>
File “/home/ajay/id-env/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/ajay/id-env/lib/python3.6/site-packages/torchvision/models/detection/roi_heads.py”, line 752, in forward<br>
box_features = self.box_roi_pool(features, proposals, image_shapes)<br>
File “/home/ajay/id-env/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/ajay/id-env/lib/python3.6/site-packages/torchvision/ops/poolers.py”, line 207, in forward<br>
self.setup_scales(x_filtered, image_shapes)<br>
File “/home/ajay/id-env/lib/python3.6/site-packages/torchvision/ops/poolers.py”, line 173, in setup_scales<br>
scales = [self.infer_scale(feat, original_input_shape) for feat in features]<br>
File “/home/ajay/id-env/lib/python3.6/site-packages/torchvision/ops/poolers.py”, line 173, in <br>
scales = [self.infer_scale(feat, original_input_shape) for feat in features]<br>
File “/home/ajay/id-env/lib/python3.6/site-packages/torchvision/ops/poolers.py”, line 157, in infer_scale<br>
assert possible_scales[0] == possible_scales[1]</p>
<p>The code is working for vgg backbone but while using resnet backbone I get assertion error.</p>
<p>anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256),), aspect_ratios=((0.5,1.0),))</p>
<p>roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[‘0’], output_size=7, sampling_ratio=2)</p>
<p>For vgg -  output_channels=512, resnet_18 - output_channels=512, model = FasterRCNN(backbone, num_classes=num_classes, rpn_anchor_generator= anchor_generator, box_roi_pool=roi_pooler)</p>

ajaychawda58: <p>Can anyone please help me with this. I am stuck while running my code for KITTI dataset. It works fine for COCO and PASCAL VOC datasets.</p>

gkalliatakis: <p>Have you managed to find a solution? I’m facing the same problem.</p>
Computer vision, ResNet, Sharing weights

Aymen_Sadraoui: <p>Hi,</p>
<p>I wanna create a model that takes 3 inputs and each input will pass through a resnet. How to share weights between the 3 resnets?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a4ee6eafb6c67a279fb36e1172e9c54e9e6b43c.png" data-download-href="https://discuss.pytorch.org/uploads/default/8a4ee6eafb6c67a279fb36e1172e9c54e9e6b43c" title="Screenshot from 2022-04-08 10-24-10"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a4ee6eafb6c67a279fb36e1172e9c54e9e6b43c_2_690x395.png" alt="Screenshot from 2022-04-08 10-24-10" data-base62-sha1="jJwW8CwPKXgmVBDw2NOATS5g6T2" width="690" height="395" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a4ee6eafb6c67a279fb36e1172e9c54e9e6b43c_2_690x395.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a4ee6eafb6c67a279fb36e1172e9c54e9e6b43c_2_1035x592.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a4ee6eafb6c67a279fb36e1172e9c54e9e6b43c.png 2x" data-dominant-color="E0EAF5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-04-08 10-24-10</span><span class="informations">1138×652 84.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I might misunderstand the question, but wouldn’t passing all 3 inputs to a single model work?</p>

Aymen_Sadraoui: <p>No the 3 inputs are images of 3 different magnifications so I want each input to be processed by one ResNet and all these resnets share the weights.</p>
Check any one value in tensor is zero or not

cbd: <p>I want to check any one value in tensor is zero or not. For below program it shows “RuntimeError: Boolean value of Tensor with more than one value is ambiguous”</p>
<pre><code class="lang-auto">import torch
a=torch.tensor((torch.rand(4)))
a[1]=0
print(a)
if(a!=0):
  print("All value is non zero")
else:
  print("Atleast one value is zero")</code></pre>

Matias_Vasquez: <p>You can use <code>.all()</code></p>
<pre><code class="lang-python">import torch
a=torch.rand(4, 4)
a[1]=0

if a.all():
    print("All values are non zero")
else:
    print("At least one value is zero")
</code></pre>
Increasing the Number of layers in lstm give me the error "RuntimeError: CUDA error: unspecified launch failure" "

saran: <p>I want to implement lstms with CNN in my model with pytorch frame work, when I add two lstms layers one after another and the number of layers in each of them increased more than one it gives me this error<br>
“CUDA error: unspecified launch failure”</p>
<p>self.lstm1=nn.LSTM(input_size=1,hidden_size=256,num_layers=1,batch_first=True,bidirectional=False,) self.lstm2=nn.LSTM(input_size=1,hidden_size=256,num_layers=1,batch_first=True,bidirectional=False) <strong>works with me</strong></p>
<p>self.lstm1=nn.LSTM(input_size=1,hidden_size=256,num_layers=2,batch_first=True,bidirectional=False,) self.lstm2=nn.LSTM(input_size=1,hidden_size=256,num_layers=2,batch_first=True,bidirectional=False)</p>
<p><strong>give me the error "RuntimeError: CUDA error: unspecified launch failure</strong>"</p>

ptrblck: <p>Could you update to the latest nightly binary and check if you are still hitting the error?<br>
If so, could you post a minimal, executable code snippet as well as the output of <code>python -m torch.utils.collect_env</code> here, please?</p>

saran: <p>thank you for the reply sorry I don’t know much about nightly binaries does it mean i should update torch vision ?</p>

saran: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4aa910dcb8fb0a7d93fc6a5fb6c7da17cb205329.png" data-download-href="https://discuss.pytorch.org/uploads/default/4aa910dcb8fb0a7d93fc6a5fb6c7da17cb205329" title="pyt"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4aa910dcb8fb0a7d93fc6a5fb6c7da17cb205329.png" alt="pyt" data-base62-sha1="aEtyovYPJSdEsVopaCNBTj7vMgN" width="639" height="500" data-dominant-color="141414"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pyt</span><span class="informations">806×630 15.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Thanks for the update. You are currently using PyToch <code>1.6.0</code>, which is a bit old by now, so you might be running into an already fixed issue.<br>
Check the install instructions <a href="https://pytorch.org/get-started/locally/">here</a> and select either the stable or nightly release. The box will then show the install command which you could execute to update PyTorch (as well as <code>torchvision</code> etc.).</p>
Creating a copy of a layer with identical parameters

Samuele_Bolotta: <p>Is this function a good way of creating a copy of a layer with identical parameters?</p>
<pre><code class="lang-auto">`def _copy_layer(layer):
 """Create a copy of a layer with identical parameters.
The new layer will not share weights with the old one"""

 return type(layer).from_config(layer.get_config())`
</code></pre>

AlphaBetaGamma96: <p>Have you tried using <code>deepcopy</code>? I know that can be used for <code>nn.Module</code> objects, so I assume it can be used for layers too.</p>
How to normalize a tensor inplace?

lamrin: <p>Given a embedding layer initialised as below:<br>
<code>embedding_layer = nn.EmbeddingBag(subvocab_size, embedding_dim, mode="mean", sparse=True, max_norm=1)</code><br>
The subvocab_size is 10M and embedding_dim is 100</p>
<p>If I normalize like below:<br>
<code>embedding_layer.weight.data = F.normalize(embedding_layer.weight.data, p=2,dim=1)</code><br>
Code throws gpu out of memory error.</p>
<p>I’d like to normalize the embeddings every n batches.</p>

ptrblck: <p>Assuming the normalization needs additional memory you might be able to normalize the embedding before pushing it to the device.</p>

lamrin: <p>I want to normalize it every n batches.<br>
PS: Just updated the question</p>

AlphaBetaGamma96: <p>Make sure to not use <code>.data</code> as it’s deprecated. You use <code>.copy_()</code> to update your weight instead</p>
Out of memory with DRAGAN

shkiper325: <p>Hello! I’m trying to reproduce <a href="https://arxiv.org/abs/1708.05509" rel="noopener nofollow ugc">this</a> paper in code. It uses combination of losses for training GAN. One of the losses I need is <a href="https://arxiv.org/abs/1705.07215v1" rel="noopener nofollow ugc">DRAGAN</a> loss. On every iteration of GAN lerning process I compute this loss, add it other losses and finally run backwards. Code for loss is:</p>
<pre><code>            images = FloatTensor(get_data_minibatch()[0])
            p_images = images + 0.5 * torch.std(images) * torch.rand(images.size()).cuda()
            p_images = torch.clamp(p_images, -0.99, 0.99)
            p_images.requires_grad_(True)
            D_p_images = discriminator(p_images)[0]
            grad = torch.autograd.grad(outputs=D_p_images.sum(), inputs=p_images, create_graph=True)[0]
            grad = torch.flatten(grad, start_dim=1)
            grad_norm = torch.norm(grad, p=2, dim=1)
            target_value = torch.subtract(grad_norm, 1)
            L_gp_D = torch.square(target_value).mean()
</code></pre>
<p>When I add this loss to code, after some training time I recieve out of memory on RAM, not GPU’s memory. I think that there is memory leak in this code, but I don’t understand how to fix it.</p>
<p>I can send full notebook if needed.</p>
<p>Here is the DRAGAN loss:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7583c780c4452272ebd7f39d96d7c20911816a5.png" alt="изображение" data-base62-sha1="sru3PDyY3AsjeGNyHRttfOubfRb" width="272" height="27"></p>
<p>So, how to fix it?</p>
Assign memory blob to py-torch output tensor (C++ API)

koparasy: <p>I am training a linear model using py-torch and I am saving it to a file with the “save” function call. I have another code that loads the model in C++ and performs inference.<br>
I would like to instruct the Torch CPP Library to use a specific memory blob at the final output tensor. Is this even possible? If yes, how? Below you can see a small example of what I am trying to achieve.</p>
<pre><code class="lang-auto">
#include &lt;iostream&gt;
#include &lt;memory&gt;

#include &lt;torch/script.h&gt;

int main(int argc, const char* argv[]) {
  if (argc != 3) {
    std::cerr &lt;&lt; "usage: example-app &lt;path-to-exported-script-module&gt;\n";
    return -1;
  }
  long numElements = (1024*1024)/sizeof(float) * atoi(argv[2]);

  float *a = new float[numElements]; 
  float *b = new float[numElements];
  float *c = new float[numElements*4];

  for (int i = 0; i &lt; numElements; i++){
    a[i] = i;
    b[i] = -i;
  }

  //auto options = torch::TensorOptions().dtype(torch::kFloat64);
  at::Tensor a_t = torch::from_blob((float*) a, {numElements,1});
  at::Tensor b_t = torch::from_blob((float*) b, {numElements,1});
  at::Tensor out = torch::from_blob((float*) b, {numElements,4});

  at::Tensor c_t = at::cat({a_t,b_t}, 1);
  at::Tensor d_t = at::reshape(c_t, {numElements,2});

  torch::jit::script::Module module;
  try {
    module = torch::jit::load(argv[1]);
  }
  catch (const c10::Error&amp; e) {
    return -1;
  }


  out =  module.forward({d_t}).toTensor();
  std::cout&lt;&lt; out.sizes() &lt;&lt; "\n";

  delete [] a;
  delete [] b;
  delete [] c;

  return 0;
}
</code></pre>
<p>So, I am allocating memory into “c” and then I am casting creating a tensor out of this memory. I store this memory into a tensor named “out”. I load the model when I call the forward method. I observe that the resulted data are copied/moved into the “out” tensor. However, I would like to instruct Torch to directly store into “out” memory. Is this possible?</p>
Heterogeneous graph and edge attributes

Yusuf_Basyouni: <p>I am new with graph models, and I appreciate any help.<br>
I have my custom heterogeneous graph with edge attributes, and I want to use HEATCONV . My question, How to feed the graph to that model?<br>
I get error when the hetrolinear layer transforms nodes features, which needs tensor, but I feed the forward with x_dict  the dictionary of nodes features tensors.</p>
Trying to covert a .pth file into CSV

Anirban: <p><em>(Comparatively new to PyTorch)</em></p>
<p>I have a standalone .pth (PyTorch) file. I want to convert the tensor weights into CSV to understand the file. What is the step-by-step procedure to convert it into a CSV file?</p>

huahuanZ: <p>CSV file only stores table-liked object, so basically it depends on the data content in the .pth file.<br>
Can you give an example of the content of the .pth file? You could load it with <code>torch.load("mytorchfile.pth")</code></p>

Anirban: <p>Yes, I loaded the data on the terminal. This array appeared. I am not able to interpret it as I do not have the model’s knowledge (attaching a screenshot if that is fine) -</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/f/2fa85aeaede7e805d591bce7c4d4ed6bf6fc78f6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2fa85aeaede7e805d591bce7c4d4ed6bf6fc78f6" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2fa85aeaede7e805d591bce7c4d4ed6bf6fc78f6_2_690x458.jpeg" alt="Capture" data-base62-sha1="6NB8HvsDNJwRpXHGjF980NO0KpM" width="690" height="458" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2fa85aeaede7e805d591bce7c4d4ed6bf6fc78f6_2_690x458.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/2/f/2fa85aeaede7e805d591bce7c4d4ed6bf6fc78f6.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/f/2fa85aeaede7e805d591bce7c4d4ed6bf6fc78f6.jpeg 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">866×575 96 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mxahan: <p>Hi, although it may be exhaustive, you can make sense of the model by looking into the serialized layers and weights shape.</p>

Anirban: <p>Yes, I was trying to make sense out of it. Wish I was able to see the parameters but I only have the Tensor weightages.</p>

huahuanZ: <p>So I guess your .pth file is a model checkpoint, isn’t it?<br>
Your screenshot just shows the parameters. I don’t understand why you want to convert the parameters into csv. Or if you don’t want the massive output, maybe use this</p>
<pre><code class="lang-auto">checkpoint = torch.load('mypth.pth')
for key, param in checkpoint.items():
    print(f"param name: {key} | param shape: {param.size()}")
</code></pre>

Anirban: <p>It does show the parameters (mistook it) but I wanted to have the massive output just to match my ideas with it. I was checking if it was going along with any mathematical expressions of my thinking. As I have other two models and those are working finely but a particular one has some problems with arguments in it. So, I was trying to understand the same.</p>
Creating conv layers only if some conv layer parameters are specified

Samuele_Bolotta: <p>I am trying to create a general feedforward net with CNN and FNN layers. What I am trying to do is: if some conv_layer_params are passed in the class, then I want to create conv layers according to how config is specified.</p>
<p>Would this be a good way of doing it inside a class EncodingNetwork(nn.Module):?</p>
<pre><code class="lang-auto">    layers = []

    if conv_layer_params:
      conv_layer_type = nn.Conv2d

      for config in conv_layer_params:
        if len(config) == 5:
          (in_channels, out_channels, kernel_size, stride, dilation_rate) = config
        elif len(config) == 4:
          (in_channels, out_channels, kernel_size, stride) = config
          dilation_rate = (1, 1)

        layers.append(
            conv_layer_type(
                in_channels=in_channels,
                out_channels=out_channels,
                kernel_size=kernel_size,
                stride=stride,
                dilation_rate=dilation_rate,
                activation=activation_fn,
                kernel_initializer=kernel_initializer,
                dtype=dtype))

    layers.append(torch.flatten())
</code></pre>

ptrblck: <p>Is this a double post from <a href="https://discuss.pytorch.org/t/creating-fully-connected-layers-inside-a-class-depending-on-various-parameters/148590">here</a> or could you describe the difference between these use cases?</p>

Samuele_Bolotta: <p>It is actually a double post, but this has to do with CNNs and the other one with FNNs!</p>
Creating fully connected layers inside a class, depending on various parameters

Samuele_Bolotta: <p>I am trying to create a general feedforward net with CNN and FNN layers. What I am trying to do is: depending on the parameters that are passed in the class, then I want to create fully connected layers according to how config is specified.</p>
<p>Would this be a good way of doing it inside a class EncodingNetwork(nn.Module)?</p>
<pre><code class="lang-auto">      for num_units, dropout_params, weight_decay in zip(
          fc_layer_params, dropout_layer_params, weight_decay_params):
        kernal_regularizer = None
        if weight_decay is not None:
          kernal_regularizer = ??
        layers.append(
            nn.Linear(
                num_units,
                activation=activation_fn,
                kernel_initializer=kernel_initializer,
                kernel_regularizer=kernal_regularizer,
                dtype=dtype))
</code></pre>
<p>Also, I am confused as to how I should specify L2 regularization in kernal_regularizer. I couldn’t find a way to do this in PyTorch, but for sure there is one.</p>

ptrblck: <p>Yes, creating the modules in the <code>__init__</code> method sounds like the right approach.<br>
Make sure to use <code>nn.ModuleList</code> for <code>layers</code> to properly register these modules or wrap the <code>list</code> into e.g <code>nn.Sequential</code> if this fits your use case.</p>
<p><code>nn.Linear</code> doesn’t have a <code>kernel_regularizer</code> argument so I’m unsure where this is coming from.<br>
You could check the <a href="https://pytorch.org/tutorials/intermediate/parametrizations.html">Parametrizations tutorial</a> to see if this could be useful.</p>
"If your network has multiple losses, you must call scaler.scale on each of them individually"

divinho: <p>This is from the documentation on amp: <a href="https://pytorch.org/docs/stable/notes/amp_examples.html" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision examples — PyTorch 1.11.0 documentation</a></p>
<p>This is talking about the scenario when one has multiple tensors that one is calling backwards on correct?</p>
<p>Not the scenario where one adds the different losses together into one <code>total_loss</code> and calls backwards on that right?</p>

ptrblck: <p>Yes, your understanding is correct. You don’t need to use separate scalers if you are accumulating both losses first.</p>
Exporting the operator bucketize to ONNX opset version 14 is not supported

xajdlg: <p>when i convert model of torch to onnx, the issue exist, which is so sad,</p>

marksaroufim: <p>onnx exporter has an argument called <code>opset_version</code> which you can use to see if this feature was supported in an older or newer opset <a href="https://pytorch.org/docs/stable/onnx.html#torch.onnx.export" class="inline-onebox" rel="noopener nofollow ugc">torch.onnx — PyTorch 1.11.0 documentation</a></p>
<p>You can also manually scan this folder here to see if your op exists <a href="https://github.com/pytorch/pytorch/tree/master/torch/onnx" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/onnx at master · pytorch/pytorch · GitHub</a></p>
<p>And if it doesn’t please make a feature request on pytorch/pytorch or implement your own custom op</p>
From_numpy vs as_tensor

cheng: <p>According to my understanding, when <code>a</code> is ndarray, <code>torch.from_numpy(a)</code> is same as <code>torch.as_tensor(a)</code>, and they don’t copy the data just share the same memory, so I think they are same speed when applied, but I test that <code>from_numpy</code> faster than <code>as_tensor</code>. Could someone explaine for me? Thanks a lot first</p>

ptrblck: <p>Yes, both approaches share the underlying memory.<br>
<code>torch.as_tensor</code> accepts a bit more that <code>torch.from_numpy</code>, such as <code>list</code> objects, and might thus have a slightly higher overhead for these checks.</p>

cheng: <p>Thank you for your reply！</p>

jumlauft: <p><code>as_tensor()</code> returns a tensor on the default device whereas <code>from_numpy()</code> returns a tensor on the cpu:</p>
<pre><code class="lang-auto">torch.set_default_tensor_type(torch.cuda.FloatTensor)
torch.as_tensor(np_array).device
&gt;&gt;&gt; device(type='cuda', index=0)
torch.from_numpy(np_array).device
&gt;&gt;&gt; device(type='cpu')
</code></pre>
Download backbone of YOLOv1

hyman9090: <p>I have built a YOLO version 1 model, following the research paper. In the middle of building the model, I had googled a lot of materials and found that most people don’t follow the exact original model (convolution layers). Instead, they would rather use ResNet or VGG. I thought it might be about the accuracy. However, I could not find any original YOLOv1 backbone, even on GitHub or the <a href="https://pjreddie.com/darknet/imagenet/#extraction" rel="noopener nofollow ugc">official website</a> written on the research paper (I hope I didn’t overlook it).</p>
<p>My question is, does anyone know where I can download the weights of the original backbone?</p>
<p>And the follow-up question is how to load them into the model if I build the model by pytorch? Simply use <code>model.load_state_dict(torch.load('.pth file'))</code>?</p>
Gdb usage: setting more breakpoints whilst running

Brent1: <p>Following<br>
<code>https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#gdb-integration</code><br>
I am able to hit a break point.</p>
<p>Executing <code>import torch</code> runs for 10s of seconds. If I <code>Ctrl-c</code> to stop execution; I have to re-import on the next run; which again takes some time.</p>
<p>Is there a way to be able to set more breakpoints without having to stop the <code>run</code>?</p>

ptrblck: <p>You can set new breakpoints in <code>gdb</code> while you are debugging as described <a href="https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_28.html#SEC29">here</a>. E.g. <code>break filename:linenumber</code> will add a new breakpoint which will be hit if you <code>continue</code> the execution in <code>gdb</code>.</p>

Brent1: <p>Sorry for my slow response; and probably for the very basic question; but to double check I’m not missing something. The below console output shows the error I get trying to set a “break” after run.</p>
<p>I think the <code>&gt;&gt;&gt;</code> indicates I’m inputting to a python interpreter; so makes sense the gdb commands aren’t going to have an affect there. But if I <code>ctr-c</code> out of the running program; I have to re-import; which is just a bit slow</p>
<pre><code class="lang-auto">root@brent:/home/brent/repos/pytorch# gdb python
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
(gdb) break at::Tensor::neg
(gdb) run
Starting program: /opt/conda/bin/python 
&gt;&gt;&gt; break at::Tensor::neg
  File "&lt;stdin&gt;", line 1
    break at::Tensor::neg
          ^
SyntaxError: invalid syntax
&gt;&gt;&gt; import torch
&gt;&gt;&gt; 
</code></pre>

Brent1: <p>AHHHHH - but once I’ve hit a breakpoint; I’m back in the hands of <code>gdb</code> – and then seem able to set more breakpoints.</p>
Example code to put matplotlib graph to Tensorboard X

DuaneNielsen: <p>While working on timeseries data, wanted to save plots as images in tensorboard.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ede13a57f229b321686c5067d9299c92fb3a7295.jpg" data-download-href="https://discuss.pytorch.org/uploads/default/ede13a57f229b321686c5067d9299c92fb3a7295" title="plot_tensorboard"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ede13a57f229b321686c5067d9299c92fb3a7295_2_658x499.jpg" alt="plot_tensorboard" data-base62-sha1="xWnHXBWTTTQNzMLN3fmPtpmXW29" width="658" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ede13a57f229b321686c5067d9299c92fb3a7295_2_658x499.jpg, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ede13a57f229b321686c5067d9299c92fb3a7295_2_987x748.jpg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/e/ede13a57f229b321686c5067d9299c92fb3a7295.jpg 2x" data-dominant-color="F4EEE7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">plot_tensorboard</span><span class="informations">1060×805 132 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the code.</p>
<pre><code class="lang-auto">import io
import matplotlib.pyplot as plt
from tensorboardX import SummaryWriter
import PIL.Image
from torchvision.transforms import ToTensor

def gen_plot():
    """Create a pyplot plot and save to buffer."""
    plt.figure()
    plt.plot([1, 2])
    plt.title("test")
    buf = io.BytesIO()
    plt.savefig(buf, format='jpeg')
    buf.seek(0)
    return buf


# Prepare the plot
plot_buf = gen_plot()

image = PIL.Image.open(plot_buf)
image = ToTensor()(image).unsqueeze(0)


writer = SummaryWriter(comment='hello imaage')
#x = torchvision.utils.make_grid(image, normalize=True, scale_each=True)
for n_iter in range(100):
    if n_iter % 10 == 0:
        writer.add_image('Image', image, n_iter)
</code></pre>

sidak: <p>Thanks, this was helpful.</p>
<p>Also, in case you are running from Jupyter, setting the backend to ‘Agg’ at the time of matplotlib helped. Otherwise, it kept giving me errors like, KeyError on PNG.</p>

FarisHijazi: <p>Thanks for the example.</p>
<p>For me it got an AssertionError</p>
<pre><code class="lang-auto">AssertionError: size of input tensor and input format are different.         tensor shape: (1, 432, 288, 3), input_format: CHW
</code></pre>
<p>I ended up using <code>writer.add_figure()</code> which is much more concise</p>
Complex-valued CNN layers

zerozy: <p>Hello,</p>
<p>I’m building a complex-valued CNN whose weights are complex numbers. Initially I wanted to test a simple network, something like this:</p>
<pre><code class="lang-python">class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Conv2d(3,16,3).to(torch.cfloat)
        self.conv2 = nn.Conv2d(16,16,3).to(torch.cfloat)
        
        self.fc = nn.Linear(16*600, 2).to(torch.cfloat)
    
    def forward(self, x):
        x = self.conv1(x)
        x = complex_relu(x)
        
        x = self.conv2(x)
        x = complex_relu(x)
        
        x = x.contiguous().view(x.size(0), -1)
        x = self.fc(x)
        x = x.abs()
        return x
</code></pre>
<p>However, I see this error when training this:</p>
<pre><code class="lang-python">RuntimeError: "slow_conv2d_cuda" not implemented for 'ComplexFloat'
</code></pre>
<p>I have cucnn disabled already. Does it mean the conv2d layer is currently not supported for complex float/double data and weights? Is there any workaround? Before, I built a DNN the same way and no errors were returned. Thank you.</p>

ptrblck: <p>Yes, the error message suggest that complex tensors aren’t supported in convs yet.<br>
The feature request is tracked <a href="https://github.com/pytorch/pytorch/issues/52983">here</a> as well as <a href="https://github.com/pytorch/pytorch/issues/71108">here</a>. The first implementations also seem to land already in e.g. <a href="https://github.com/pytorch/pytorch/pull/75013">this PR</a>.</p>

zerozy: <p>Thanks for the info! looking forward to seeing this feature implemented.</p>
Model with unused parameters does not work in DDP

TaojiannanYang: <p>Hi, I have a model with conditioned output, for example,</p>
<pre><code class="lang-auto">if CONDITION_A:
    x = self.fc1(x)
else:
    x = self.fc2(x)
</code></pre>
<p>The training works well on single GPU. However, on multi-gpu with DDP it gives the error</p>
<pre><code class="lang-auto">RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. Thi
s error indicates that your module has parameters that were not used in producing loss.
</code></pre>
<p>I see some people suggest to collect all outputs and multiply the unwanted one with 0 and add it to the loss. I am wondering is there a better way to solve this problem? Thanks.</p>

kumpera: <p>You can try to change how you <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" rel="noopener nofollow ugc">construct your DDP module</a> and pass <code>find_unused_parameters=True</code>.</p>
Not able to export large model with onnx

DeepakSaini119: <p>I am trying to export a model(which when dumped on disk as a pt file, is 4GB). The code I use is:</p>
<pre><code class="lang-python">net.eval()
torch.onnx.export(net,               # model being run
                  (doc_embeddings, shortlist),                         # model input (or a tuple for multiple inputs)
                  "/path/to/a/folder/net.onnx",   # where to save the model (can be a file or file-like object)
                  opset_version=11,          # the ONNX version to export the model to
                  input_names = ['doc_embeddings', 'shortlist'],   # the model's input names
                  output_names = ['scores'], # the model's output names
                  dynamic_axes={'doc_embeddings' : {0 : 'batch_size'},    # variable length axes
                                'shortlist' : {0 : 'batch_size', 1: 'num_shortlist'},
                                'scores' : {0 : 'batch_size', 1: 'num_shortlist'}},
                  use_external_data_format=True,
                  verbose=True)
</code></pre>
<p>But this doesn’t work and I get the error: <code>RuntimeError: Exporting model exceed maximum protobuf size of 2GB. Please call torch.onnx.export with use_external_data_format=True.</code> I would ideally expect that the model would be exported with external files after specifying <code>use_external_data_format=True</code></p>

JulesBelveze: <p>+1 I’m having the same problem over here</p>
nn.Embedding layer in categorical features

erkinovic: <p>Hi everyone,</p>
<p>These forums have helped me greatly but unfortunately i can’t find a problem similar to the one i have run into now.</p>
<p>For context, I am creating a VAE in order to learn data distribution and create a mock dataset with similar statistical properties of the original dataset (synthetic data generation). Now I am working with a heavily categorical value based dataset (20 out of 27 variables are categorical) and I have learned I can use nn.Embedding to deal with these categorical values. I have tried one-hot-encoding them all but this resulted in an explosion of features (resulted in over 120 features which made my data very sparse).</p>
<p>However, I have noticed that nn.Embedding is primarily used in NLP and vocabularies with a set amount of words. Now I am dealing with features that all have different “vocabularies” AKA amount of categories and I am wondering on how I can correctly implement the nn.Embedding layer and correctly let the network know which features are categorical and should not be treated continuously.</p>
<p>Example: one feature has 2 categories (binary) and one has 10 categories (different positions in a company). Is there a way to satisfy all category sizes with nn.Embedding?</p>
<p>Thank you in advance!</p>