akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.autograd.profiler doesn't save much

f10w: <p>I have a training script that I launch with <code>python -m torch.distributed.launch --nproc_per_node=1 --use_env train.py</code>. I would like to profile it, so I did something like this:</p>
<pre><code class="lang-python">import torch.autograd.profiler as profiler
...
    with profiler.profile() as prof:
        with profiler.record_function("training"):
            print("Start training")
            for epoch in range(epochs):
                ...
    if is_main_process():
        prof.export_chrome_trace(output_dir / 'trace.json')
</code></pre>
<p>The resulted json contains almost nothing:</p>
<p><code>[{"name": "training", "ph": "X", "ts": 140.203, "dur": 139.73299999999998, "tid": 1, "pid": "CPU functions", "args": {}}]</code></p>
<p>What should I do to have the detailed running time of all the operations of my training?</p>
<p>Thank you very much in advance for your help!</p>

hhaoao: <p>The reason lies in this code, you put too much code in this block.</p>
<p><code>with profiler.record_function("training"):</code></p>

f10w: <p><a class="mention" href="/u/hhaoao">@hhaoao</a> Thanks. But how do we know how much is too much?</p>

hhaoao: <p>I personally summarized the use of <code>record_function</code> in the following points:</p>
<ol>
<li>If you think it is redundant, only the code segment with general statistics is needed.</li>
<li>Code snippets that need comment help to view.</li>
</ol>
<p>Of course, there are more than these uses, you can dig by yourself.</p>
How to convert a pytorch tensor into a byte string?

Shisho_Sama: <p>in libtorch we can do :</p>
<pre><code class="lang-cpp">    std::stringstream stream;
    torch::save(tensor, stream);
    return stream.str();
}
</code></pre>
<p>and get a byte-str and then can serialize it using protobuf for example. How can I do its equivalent in Pytorch?<br>
would using <code>io.BytesIO</code> suffice?</p>
<pre><code class="lang-python">import io 
 buffer = io.BytesIO()
torch.save(my_tensor, buffer)
bytes = buffer.read()
# use bytes ...
</code></pre>
<p>Any help is greatly apprecaited</p>

ChenchenJT: <p>Maybe you can use .to() function.</p>
<p>Hope this information can help you.</p>

Shisho_Sama: <p>Thanks a lot. but <code>.to</code> is not applicable here. it works in libtorch only if the underlying data is a string (usually used with IValue if I’m not mistaken) other than that, types and devices are other usecases of <code>.to</code> which again are not applicable here. since here we want to turn the whole tensor into a byte representation an not individual elements in it. .to(torch.float, or the likes, only change the underlting data representation and not the tensor itself.<br>
The way to go was to use <code>io.BytesIO()</code> and get the bytes needed.</p>
Define neural network layer

Mi_Rak: <p>Good morning every one,<br>
I am new in neural network and I try to experiment on, I am a bit confused of the meaning of these different way to define neural network layer.<br>
Could anyone help me to understand the difference of this different line please.<br>
<strong>1</strong>```<br>
<code>self.relu(self.fc1(x))</code></p>
<pre><code class="lang-auto">`**2**``
`self.relu(self.hidden(x))`
</code></pre>
<p>``<strong>3</strong>`<br>
self.fc1(self.hidden(x))</p>
<pre><code class="lang-auto">``**4**`
self.fc1(self.relu(self.hidden(x)))
</code></pre>

zcajiayin: <pre><code class="lang-auto">1. self.relu(self.fc1(x))
</code></pre>
<p>It first pass <code>x</code> to a fully connected layer <code>fc1()</code>, then the output of <code>fc1()</code> is passed to a <code>relu()</code></p>
<pre><code class="lang-auto">2. self.relu(self.hidden(x))
</code></pre>
<p>It seems it defined a <code>hidden()</code> layer somewhere in the model, and just like <code>self.relu(self.fc1(x))</code>, the output of this <code>hidden()</code> layer is passed to a <code>relu()</code></p>
<pre><code class="lang-auto">3. self.fc1(self.hidden(x))
</code></pre>
<p>Similar</p>
<pre><code class="lang-auto">4. self.fc1(self.relu(self.hidden(x)))
</code></pre>
<p>Now it is a three-layer NN, <code>hidden()-&gt;relu()-&gt;fc1()</code></p>

Mi_Rak: <p><a class="mention" href="/u/zcajiayin">@zcajiayin</a>  Thank you for your response.</p>
Saving GAN Output

mmfroid: <p>Hi everyone. Total n00b question. I am just getting started at learning how to use PyTorch to generate images and train neural networks. I have been using this tutorial <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a><br>
as a way to learn about PyTorch. I would like to save all of the images generated during the last epoch, but as individual files, not in a grid. I am really having a hard time figuring this out. Would anyone have any advice/tips/tricks to accomplish this? Thank you in advance.</p>

Hari_Krishnan: <p>You can loop though <code>img_list</code> and save every image using <code>plt.savefig</code>. Something along the below code.</p>
<pre><code class="lang-auto">for i in range(len(img_list)):
    plt.savefig(str(i)+'.png', img_list[0])
</code></pre>

mmfroid: <p>Thanks for the tip. Will this save every image the model generates or will this just save the image for the last epoch?</p>

pfloat: <p>To select the last epoch, you need to get the index of the epoch loop and then test if the index is indeed the last epoch.</p>
<pre><code class="lang-auto">for epoch in range(epochs):
    if epoch == epochs - 1:
        # save your images here
</code></pre>
No module named '__torch__'

SungmanHong: <p>In python3.7 and windows10, pytorch 1.5, I saved my model<br>
and…<br>
I loaded</p>
<p>import torch<br>
model = torch.load(‘D:\CR_DOWN\CRAFT-pytorch-master\CRAFT-pytorch-master\model_OCR.pt’)</p>
<p>but… I got a error of<br>
No module named ‘<strong>torch</strong>’<br>
why I got this error??</p>
<hr>
<p>ModuleNotFoundError                       Traceback (most recent call last)<br>
 in <br>
----&gt; 1 model = torch.load(‘D:\CR_DOWN\CRAFT-pytorch-master\CRAFT-pytorch-master\model_OCR.pt’)</p>
<p>~\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module, **pickle_load_args)<br>
582                     opened_file.seek(orig_position)<br>
583                     return torch.jit.load(opened_file)<br>
–&gt; 584                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)<br>
585         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)<br>
586</p>
<p>~\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py in _load(zip_file, map_location, pickle_module, **pickle_load_args)<br>
840     unpickler = pickle_module.Unpickler(data_file, **pickle_load_args)<br>
841     unpickler.persistent_load = persistent_load<br>
–&gt; 842     result = unpickler.load()<br>
843<br>
844     return result</p>
<p>ModuleNotFoundError: No module named ‘<strong>torch</strong>’</p>

albanD: <p>Hi,</p>
<p>Did you saved that file with a different version of pytorch by any chance? Especially a more recent version than the one you use for loading?</p>

SungmanHong: <p>I’ve recognized that the model what i trying to convert to mobile is actually not pytorch model…<br>
(it was python function name, not a pytorch model)<br>
I changed the model correctly, and it worked!<br>
Thank you.</p>
Repeating tensor different number of times fully on GPU

Gabriel_Barello: <p>I am trying to implement a model in which there are categories (genera) and subcategories (species) which I am trying to classify images into. I want to build a model which explicitly separates the output into <code>P[species|data] = P[genus | data] * P[species | genus, data]</code>. I am implementing this by having one layer compute <code>Log[P[genus|data]]</code> and adding that to another layer which computes <code>Log[P[species | genus, data]]</code>. Unfortunately the various genera have different numbers of species in the dataset. So, in order to compute the final output I need to transform <code>Log[P[genus|data]]</code> into a tensor where each entry is repeated a number of times equal to the number of species in each genus. I am achieving this right now in a very inefficient way which relies on a python list comprehension as follows: (<code>G</code> is a tensor of size <code>[batch_size, n_genera]</code> giving the genus log-probabilities and <code>S</code> is a tensor of size <code>[batch_size, n_species]</code> giving the species log-probabilities with <code>n_species &gt; n_genera</code>):</p>
<pre><code class="lang-auto">genus_expand = torch.transpose(torch.stack([g for s,g in zip(n_species_list,torch.transpose(G,0,1)) for i in range(s)],0),0,1)

final_prob = S + G
</code></pre>
<p>where <code>n_species_list</code> is a list of length <code>n_genera</code>, of integers giving the number of species in each genera.</p>
<p>Anyway, all that is just to explain what I am doing. Implemented this way I only seem to be able to get about 70% GPU utilization (an implementation without this line, where I only use <code>P[species|data]</code> so its entirely on the GPU and no python list comprehension) gets more like 90-95% utilization). I want to use something like <code>tile</code> but since the number of repetitions is different for different genera I don’t think I can make it work without essentially reproducing the above line of code.</p>
<p>Let me know if my explanation of my goal here is not fully clear.</p>

Gabriel_Barello: <p>It just occurred to me that I can solve this by constructing a matrix on the GPU to convert <code>G</code> to its expanded version. This seems like a good solution, though it seems rather inelegant… if anyone has a better idea let me know!</p>
Curiousity of seq2seq model

Henry_Chibueze: <p>After reading the seq2seq google paper it was stated that they used a fixed vector length of 160000 and 80000 for source and target sequence respectively.<br>
I’m wondering how google vectorized their data, whether as one-hot encoding ([0,1,0,0,0…0]) or as zero padded vector of unique token index sequence ([10, 17, 2, 3, 0, 0 …0, 0, 0]) where each interger &gt; 0 represents an index of word in the vocabulary.<br>
Can someone please throw more light on this.</p>
How do I store a batch of distributions in a distribution object?

ellacataluna: <p>My neural network tries to get the loss (torch.distributions.kl_divergence) between two Beta distributions (Pytorch distribution objects). I know I can do this for a single example, but can I do this for a whole batch of distributions in a single object? Right now the code looks like:</p>
<pre><code class="lang-auto">alpha_actual = torch.Tensor(output_scores[:,0])
beta_actual = torch.Tensor(output_scores[:,1])
actual_distrib = Beta(alpha_actual, beta_actual)

alpha_y = torch.Tensor(y_batch[:,0])
beta_y = torch.Tensor(y_batch[:,1])
y_distrib = Beta(alpha_y, beta_y)

loss = torch.distributions.kl_divergence(actual_distrib, y_distrib).mean()
</code></pre>
<p>…where <code>alpha_actual</code> and <code>beta_actual</code> are <code>torch.Size([batch_size])</code>. My code doesn’t break or anything when I do this, but I’m having trouble wrapping my head around how the distribution object works if I feed it a batch of parameters instead of a single parameter. Does it actually store a batch of distributions in the single object? Or am I doing this wrong?</p>
How to detach a certain hidden state during RNN training

dlzzk: <p>I am training an RNN (LSTMs, more precisely) network with input video frames. The idea is to let a number of frames, for example, t=[0,8], through the LSTMs and use the last (t=8) hidden state to predict label associated with only t=4 (center t). I could’ve pre-processed the sequential data in a way where each t is padded with neighboring frames, but to be more efficient, I would like to design a training loop where I can “reuse” the previous computation results - for example, when I am training with t=[1,9], I would like to “reuse” the results of t=[1,8] from the previous forward pass, with only t=9 frame being added. (I understand that this is not exactly the same as the pre-processed way, since now the t=1 hidden state is not initialized, instead it is taking output from t=0, but this is not harmful in my opinion.) And before calling loss.backward(), I can just detach the t=0’s variables from the compute graph.</p>
<p>My main problem is how to detach these specific variables from the compute graph. I’ve tried to keep the pointers to those variables so later I could call detach() on those pointers to detach the actual variables. But that did not work. May I know if there is a way to do this?</p>
<p>Thanks in advance.</p>
PyTorch Geometric TopKPooling with ValueError: too many values to unpack (expected 5)

Papertiger_Hunter: <p>When I run this Notebook <a href="https://github.com/khuangaf/PyTorch-Geometric-YooChoose/blob/master/YooChooseBuy.ipynb" rel="nofollow noopener">https://github.com/khuangaf/PyTorch-Geometric-YooChoose/blob/master/YooChooseBuy.ipynb</a> to learn Graph Neural Networks, I get the error ValueError: too many values to unpack (expected 5). I am new to PyTorch and graph neural networks. It seems this code worked before. I navigated the code and found the output is not right. It’s not a tuple or list with five elements. Can anyone tell me how to fix this?</p>
<pre><code class="lang-auto">torch.Size([2093, 256])
torch.Size([2093, 128])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-49-fb44712a6f41&gt; in &lt;module&gt;
      1 for epoch in range(1, 2):
----&gt; 2     loss = train()
      3     train_acc = evaluate(train_loader)
      4     val_acc = evaluate(val_loader)
      5     test_acc = evaluate(test_loader)

&lt;ipython-input-47-d6d407b09375&gt; in train()
      6         data = data.to(device)
      7         optimizer.zero_grad()
----&gt; 8         output = model(data)
      9 
     10         label = data.y.to(device)

C:\ProgramData\Miniconda3\envs\ptg\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--&gt; 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

&lt;ipython-input-45-312d18271bec&gt; in forward(self, data)
     38         x = F.relu(self.conv1(x, edge_index))
     39         print(x.shape)
---&gt; 40         x, edge_index, _, batch, _ = self.pool1(x, edge_index, None, batch)
     41         # x = self.pool1(x, edge_index, None, batch)
     42         x1 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)

ValueError: too many values to unpack (expected 5)
</code></pre>

Papertiger_Hunter: <p>I found the answer in issues by Diego Koz.</p>
<p>It is probably a change in <a href="https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/nn/pool/topk_pool.html#TopKPooling" rel="nofollow noopener">TopKPooling</a>, now the function returns</p>
<p><code>return x, edge_index, edge_attr, batch, perm, score[perm]</code></p>
<p>So it needs to be added another  <code>_</code>  in the  <code>forward</code>  function. e.g.</p>
<p><code>x, edge_index, _, batch, _, _ = self.pool1(x, edge_index, None, batch)</code><br>
instead of</p>
<p><code>x, edge_index, _, batch, _ = self.pool1(x, edge_index, None, batch)</code></p>
Threading and Dataloader

terro: <p>Loading data from dataloader requires too much time. I would like to have two processes running in parallel. One that load data into batches and put them into a shared queue and the other one that performs the training using GPU. In this way I could fully utilize the GPU without waiting for the loading of the data. Is it possible?</p>

kshitij: <p>The <code>torch.utils.data.Dataloader</code> class should do the trick.</p>
<p><a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader</a></p>
<p>Following blog could be useful : <a href="https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel" rel="nofollow noopener">https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel</a></p>

terro: <p>When I iterate the dataloader:</p>
<p>'for idx, sample in enumerate(dataloader)</p>
<p>Training steps<br>
’</p>
<p>The first line of code takes one minute to load the data. I would like to have loading of the data and training in parallel</p>
Stacking lstm layers with nn.lstm

granth_jain: <p>Hi,</p>
<p>I want to retain information of my data using lstm.</p>
<p>However on debugging I am finding that model is not performing well and it should retain more info from the past.</p>
<p>I would like to know whether stacking lstm with num_layers will help in this regard and what is the primary purpose of adding layers to lstm.</p>
Does pytorch models require particular kind of class labels?

Sassan_Owssia: <p>I am trying to take a subset of my dataset using Pandas .sample method and due to the highly unbalanced structure of my data’s class distribution, I lose some of the classes which is ok. However, when I want to train the model I run into this:</p>
<blockquote>
<p>RuntimeError: cuda runtime error (710) : device-side assert triggered at</p>
</blockquote>
<p>I have run into this error in the past, and it always meant that something was wrong with my model architecture especially the last layer. Is this error showing up because of the lost classes during sampling?</p>
<pre><code class="lang-auto"># sampling
sample_csv = train_csv.sample(frac = .2)
sample_csv.reset_index(drop=True, inplace = True)
sample_landmark_count = sample_csv.landmark_id.value_counts()
# resnet model
num_classes = len(sample_landmark_count) 
resnet = models.resnet18()
num_ftrs = resnet.fc.in_features
resnet.fc = nn.Linear(num_ftrs, num_classes)    

</code></pre>
<p>The labels are in the landmark_id and the data are from kaggle’s google landmark competition.</p>
<p>And this is how I made my costum dataset</p>
<pre><code class="lang-auto">class landmark_dataset(Dataset):
   def __init__(self,csv_file,mode , tfms = None):
       self.mode = mode
       self.csv_file = csv_file
       self.transforms = tfms
       
   def __len__(self):
       
       return len(self.csv_file)
       
   def __getitem__(self, idx):
       
       img_path = get_folder_names(self.csv_file.id[idx])
       img = Image.open(img_path)
       img = self.transforms(img)
       if self.mode == 'train':
           labels = self.csv_file.landmark_id[idx]
           return img, labels 
       else:
           return img
</code></pre>

ptrblck: <p>Which PyTorch version are you using? If not <code>1.6</code>, could you update and rerun the code?<br>
Also, could you run the code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the stack trace here?</p>
Norm cannot backpropagate for p<1

neovand: <p>Hi I use <code>tensor.norm</code> somewhere in my code which works fine for all values of p (including fractional ones e.g. p=1.5) but runs into problems with p&lt;1. With anomaly detection on, I get this error:</p>
<blockquote>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
11             # ===================backward====================<br>
12             optimizer.zero_grad()<br>
—&gt; 13             loss.backward()<br>
14             optimizer.step()<br>
15         # ===================log========================</p>
<p>c:\users\windows.conda\envs\normnet\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)<br>
183                 products. Defaults to <code>False</code>.<br>
184         “”"<br>
 → 185         torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
186<br>
187     def register_hook(self, hook):</p>
<p>c:\users\windows.conda\envs\normnet\lib\site-packages\torch\autograd_<em>init</em>_.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
125     Variable._execution_engine.run_backward(<br>
126         tensors, grad_tensors, retain_graph, create_graph,<br>
 → 127         allow_unreachable=True)  # allow_unreachable flag<br>
128<br>
129</p>
<p>RuntimeError: Function ‘NormBackward1’ returned nan values in its 0th output.</p>
</blockquote>
<p>any idea how to fix this?</p>

ptrblck: <p>Could you check, if the input to <code>torch.norm</code> could be a tensor containing all zeros?</p>
TypeError: slice indices must be integers or None or have an __index__ method

Khawar_Islam: <p>Code</p>
<pre><code>for file in zip(frames_list[:-round(0.2 * len(frames_list))], masks_list[:-round(0.2 * len(masks_list))]):
    # Convert tensors to numpy arrays # khawar

    frame = frame_batches.next().numpy().astype(np.uint8)
    mask = mask_batches.next().numpy().astype(np.uint8)

    # Convert numpy arrays to images
    frame = Image.fromarray(frame)
    mask = Image.fromarray(mask)

    # Save frames and masks to correct directories
    frame.save(DATA_PATH + '{}_frames/{}'.format(dir_name, dir_name) + '/' + file[0])
    mask.save(DATA_PATH + '{}_masks/{}'.format(dir_name, dir_name) + '/' + file[1])
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/khawar/Desktop/Khawar_Seg/main.py", line 181, in &lt;module&gt;
    generate_image_folder_structure(frame_tensors, masks_tensors, frames_list, masks_list)
  File "/home/khawar/Desktop/Khawar_Seg/main.py", line 148, in generate_image_folder_structure
    for file in zip(frames_list[:-round(0.2 * len(frames_list))], masks_list[:-round(0.2 * len(masks_list))]):
</code></pre>

ptrblck: <p>This error seems to be raised by Python in the used loop.<br>
Could you post an executable code snippet to reproduce this issue?</p>
Error Analysis on Binary classification

Jona: <p>Hi</p>
<p>Suppose you have a NN model which predicts a probability (<code>Sigmoid</code> in the last layer and <code>BCELoss</code> as a loss function), and the target column has the values <code>True</code> or <code>False</code>, a you use roc_auc as a accuracy metric.</p>
<ol>
<li>How can you perform an Error Analysis? I mean … take a look at the missclassified samples</li>
<li>How can I know if a sample was missclassified? I don’t have any probability threshold to classify a sample as <code>target=True</code> or <code>target=False</code>
</li>
</ol>
<p>Best regards</p>

ptrblck: <p>You could use the ROC to pick a threshold value and create the predictions using this threshold.<br>
Once you have the predictions you could check all misclassified samples, compute a confusion matrix etc. for further investigation.</p>
How to combine 3 pytorch model?

Zila_Tajudin: <p>I have 2 dataset.<br>
1 dataset is preprocess and the other is actual.<br>
The preprocess data will go through model 1 and being classify as one of 2 classes.<br>
for example the class name for model 1 is A and B.<br>
If image 1 classify as Class A it will go through model 2.<br>
However I want only image successful being classify as class A go through model 2.<br>
Class B will go through model 3 .<br>
In summary input( image) feed to model 1 and 3 is preprocess.<br>
Input for model 2 is the actual one.<br>
Can someone suggest or had any idea to do it?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9eda3cee9a54070ddade531cc9f470b65833b36b.png" data-download-href="https://discuss.pytorch.org/uploads/default/9eda3cee9a54070ddade531cc9f470b65833b36b" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9eda3cee9a54070ddade531cc9f470b65833b36b_2_690x424.png" alt="Capture" data-base62-sha1="mFh0OrF9Vx8ymQ4tG6eEeihSrTd" width="690" height="424" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9eda3cee9a54070ddade531cc9f470b65833b36b_2_690x424.png, https://discuss.pytorch.org/uploads/default/original/3X/9/e/9eda3cee9a54070ddade531cc9f470b65833b36b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/e/9eda3cee9a54070ddade531cc9f470b65833b36b.png 2x" data-dominant-color="F8F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1023×629 36.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

hhaoao: <p>It’s easy to get answers by following your thoughts and continuing to write…</p>

ptrblck: <p>I’m not fully understanding the use case.<br>
How would you know, if <code>model1</code> correctly predicted <code>classA</code> during the validation and test phase, where no labels are available?</p>
ModuleNotFoundError: No module named 'tools'

dawachat: <p>I am trying to compile pytorch from source for static setting. I am getting this error</p>
<blockquote>
<p>Blockquote<br>
Error while finding module specification for ‘tools.codegen.gen’ (ModuleNotFoundError: No module named ‘tools’)<br>
–<br>
CMake Error at cmake/Codegen.cmake:202 (message):<br>
Failed to get generated_cpp list<br>
Call Stack (most recent call first):<br>
caffe2/CMakeLists.txt:2 (include)</p>
</blockquote>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/blob/05c1f1d974910736275b4c4572095a106499c298/tools/codegen/gen.py"><code>tools/codegen/gen.py</code></a> was added quite recently, so could it be that you have updated some files but not the complete repository?<br>
Could you clean the build via <code>python setup.py clean</code>, update it, and rebuild?</p>
Recreate CNN from state dict

add023: <p>Hi, I save my model periodically with relevant information. The model is <code>CNN -&gt; transformer encoder-&gt; CTC loss</code>. The CNN is Pytorch resnet18 which is finetuned as the model collectively trains (the softmax layer is removed and a fc layer of 512 length is added).</p>
<pre><code class="lang-auto">model_path = "{}/{}.ckpt".format(self.model_dir, self.steps)
state = {
            "steps": self.steps,
            "model_state": self.model.state_dict(),
            "optimizer_state": self.optimizer.state_dict(),
            "scheduler_state": self.scheduler.state_dict() \
            if self.scheduler else None,
}
torch.save(state, model_path)
</code></pre>
<p>Now, I need to use the CNN to generate embeddings in new images. Essentially forward pass a new image just through the CNN and get the [1,512] tensor output.</p>
<p>torch.save(state, model_path)</p>
<pre><code class="lang-auto">file = '128000.ckpt'
best_model = torch.load(file, map_location=torch.device('cpu'))
state_dict = best_model['model_state']
</code></pre>
<p>The state_dict is an <code>OrderedDict</code> and has weights and biases of everything in the collective model. I can see the layers associated with cnn in the dictionary keys, however, I am not sure how to order them to recreate the CNN.</p>
<p>Any ideas how to do this is appreciated.</p>

add023: <p>Ok. I found a way to do this.</p>
<ol>
<li>Get state dict of the original resnet18 that was used in training.</li>
<li>Get state dict of the collectively trained model.</li>
<li>Create a new state dict from collectively trained model where keys == keys in step 1.</li>
</ol>
<pre><code class="lang-auto">from torchvision import models
from torch.hub import load_state_dict_from_url
from collections import OrderedDict


# get best model state dict
file = 'best.ckpt'
best_model = torch.load(file, map_location=torch.device('cpu'))
best_state_dict = best_model['model_state']

# get original resnet18
cnn_model = "resnet18"
cnn = models.resnet18(num_classes=num_classes)
cnn_state_dict = load_state_dict_from_url(model_urls[cnn_model])

# create new tuned resnet using best_state_dict
tuned_cnn_state_dict = OrderedDict({k:v for k, v in \
                       best_state_dict.items() if k in cnn_state_dict})

# finally load the weights and biases
cnn.load_state_dict(tuned_cnn_state_dict, strict=False)
</code></pre>
Cannot build pybind11/libtorch code with cmake

coincheung: <p>Hi,</p>
<p>My <code>play.cpp</code> code is like this:</p>
<pre><code class="lang-auto">#include "pybind11/pybind11.h"
#include &lt;torch/torch.h&gt;
#include &lt;torch/extension.h&gt;

torch::Tensor get_ten() {
    return torch::ones({3, 4});
}


PYBIND11_MODULE(play, m) {
    m.doc() = "load image with c++";
    m.def("get_ten", &amp;get_ten, "get single image float32 array");
}
</code></pre>
<p>My <code>CMakeLists.txt</code> file is like this:</p>
<pre><code class="lang-auto">cmake_minimum_required (VERSION 3.0 FATAL_ERROR)
project (play)

set (CMAKE_PREFIX_PATH /root/build/libtorch)
find_package(Torch REQUIRED)
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")
message (${CMAKE_CXX_FLAGS})

list(APPEND LIGHTSTEP_LINK_LIBRARY glog pthread )

add_subdirectory(pybind11)
pybind11_add_module(play SHARED play.cpp)
target_link_libraries(play PRIVATE ${TORCH_LIBRARIES} ${LIGHTSTEP_LINK_LIBRARY})
set_property (TARGET play PROPERTY CXX_STANDARD 14)
</code></pre>
<p>I compile it like this:</p>
<pre><code class="lang-auto">mkdir -p build &amp;&amp; cd build
cmake .. &amp;&amp; make
</code></pre>
<p>The compile process succeed, but when I import it, there is an error:</p>
<pre><code class="lang-auto">import torch
import play
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
ImportError: /root/repo/play/build/play.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC1ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
</code></pre>
<p>I am using pytorch1.6 installed from cuda pytorch channel, and libtorch downloaed from the link: <code>https://pytorch.org/get-started/locally/</code>. My local cuda versin is 10.1.243, and in my conda environment, I use cudatoolkit of version 10.1.243.</p>
<p>What is the problem here please ?</p>

classner: <p>Hi <a class="mention" href="/u/coincheung">@coincheung</a>,</p>
<p>You should be able to just define <code> -D_GLIBCXX_USE_CXX11_ABI=0</code> for your library and you should be fine (in CMake add the line <code>add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0)</code>).</p>
<p>More context: the pytorch core library and your extension must be built with the same CXX11 API settings. This used to be _GLIBCXX_USE_CXX11_ABI=1 for older versions ans is =0 now for newer ones. I thought that for a while it was possible to build extensions without setting this flag at all, but it seems it’s explicitly required again.</p>
Object Detection Quantization in PyTorch

Nilakshan_Kunanantha: <p>How quantization for object detection models varies from that of classification models?<br>
Since detection models need to handle the bbox coordinates(multiple objects in an input), there must be some scaling trick in quantization.<br>
Is there any implementation sources?</p>

dskhudia: <p>We have usually quantized the backbone part for detection models while leaving the rest in fp32 and gotten good speedups. For the other part, <a class="mention" href="/u/zafar">@Zafar</a> has tried quantizing but the accuracy is usually bad.</p>

Nilakshan_Kunanantha: <p>If we employ MinMax observer for calibrating the floating model for quantization,how are the bounding box coordinates quantized? Does it follow same way as of feature extraction?</p>

supriyar: <p>Yes, MinMax observer will operate the same way if you’re using it for bounding box co-ordinates. It calculates the scale and zero-point of the tensor based on min and max values.</p>
Declarative specification of the Torchscript IR

powderluv: <p>Hi - We are trying to implement a lowering of Torchscript to MLIR/ATen dialect as part of the LLVM npcomp incubator effort [1]. Is there a declarative specification of the Torchscript IR we could leverage for this work ?  Thanks</p>
<p>[1] <a href="https://llvm.discourse.group/t/npcomp-next-steps-for-torch-ir-aten-dialect/1777" rel="nofollow noopener">https://llvm.discourse.group/t/npcomp-next-steps-for-torch-ir-aten-dialect/1777</a></p>

SplitInfinity: <p>I think the closest thing to what you are looking for is the <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/OVERVIEW.md#core-program-representation" rel="nofollow noopener">core program representation section of the JIT overview documentation.</a></p>

Stella_Laurenzo: <p>Thanks - that is a really helpful and nicely written doc.</p>
<p>Are there any facilities for systematically enumerating the ops and how they are materialized in the IR? (i.e. I’ve got a harness that traces each op and then extracts the corresponding IR to make such a mapping, but it would be better if this was in some machine-usable source form somewhere)</p>

SplitInfinity: <p>If by ops, you mean things like convolutions, not really. Most ops of that sort show up in the IR as <code>aten::op</code>, but they are not a formally defined part of the JIT IR. There is no formal restriction on which ops are “allowed” and which ops are not. They map to operators in ATen, some of which can be user-defined and registered outside of the core PyTorch library.</p>

Stella_Laurenzo: <p>Thanks - that was my understanding but was just double checking that I wasn’t missing something obvious.</p>
[install torch from source] what third-party can I add to make torch faster?

xi_xi: <p>[install torch from source] what third-party can I add to make torch faster?<br>
e.g. MKL  or MKL-DNN  ?<br>
I has no GPU. I use torch-CPU.<br>
Thanks.</p>

ptrblck: <p>Sorry for not being very helpfup, but the conda binaries should already ship with MKL, if I’m not mistaken, so you might try these out directly.</p>

xi_xi: <p>Thanks. I couldn’t use conda for some reasons.  I will try conda again.<br>
In addition, do you konw, is pytorch with mkldnn and mklml slower than pytorch with mkl?</p>

ptrblck: <p>No, I don’t know it, as I’m not deeply experienced with CPU accelerators.</p>
Build on CPU then run on another computer with GPU

Boyuan_Liang: <p>Hi, I am using libtorch, I only have a laptop which has no gpu. I have access to a server with strong GPUs. Therefore, I want to compile (with cmake) using gpu versions of libtorch locally on my laptop, then move the compiled file to the remote server for execution. Note I cannot compile on that remote server since it does not have cmake installed and I am not root on that remote server.</p>
<p>Hence, I wonder if it is possible to compile with gpu version of libtorch on a computer without cuda and GPU. Then move the compiled file to another machine with GPUs for execution.</p>

alanzhai219: <ol>
<li>Maybe it doesn’t work for cuda toolkit is needed during compilation with GPU. When you import torch, related xxx.so will be called.</li>
<li>No cmake is not a big question. You can still pip install cmake though you are not in sudoers. I am sure it works well.</li>
</ol>

ptrblck: <p>You can install CUDA on your laptop (even without a GPU) and try to cross-compile PyTorch for your GPU architectures.<br>
However, I’m not sure what the best use case would be (building wheels?), as I’ve only done it in docker containers, which were executed on the servers with GPUs.</p>

Boyuan_Liang: <p>Do you mean it is better if I can compile on the remote server with gpus instead of my laptop?</p>

ptrblck: <p>If that’s possible, sure as you wouldn’t have to cross-compile on a potentially slower laptop.<br>
However, you’ve mentioned you cannot compile on the server.</p>
Variable column length 2D Tensors?

Rio1210: <p>Hello all,</p>
<p>I have an algorithm in MATLAB that uses “Cell-Datastructure” to store 1D column vectors of different lengths. Now, I want to code this in PyTorch. However, I did not find a way to use variable column length tensors for 2D matrices, as it seems like 2D PyTorch Tensors has to be strictly of “m x n” shape.</p>
<p>The only way that I can think of is using Python dictionaries, and holding various columns in each Dictionary entry.</p>
<p>My prime concern is of processing speed, as I would have lots of columns and python for loops are pretty slow when I apply them in this task. Is there a better way to do it without dictionaries and prefereably with PyTorch?</p>

ptrblck: <p>I don’t think there is any other valid approach besides manually looping over the <code>dict/list</code> or padding all tensors.<br>
Nested tensors should solve this issue and might provide a clean way without padding. The progress can be tracked <a href="https://github.com/pytorch/pytorch/issues/22169">here</a>.</p>
Adding an STN to prevent network from learning

Torchuser95: <p>Hey,</p>
<p>I want to build a CNN that contains several Spatial transformer networks (STNs). As a first step I wanted to add one STN, but unfortunately this makes the CNN perform very poorly.The same CNN without the STN (commenting out the line “x = self.stn(x)”) reaches 99% validation accuracy within 5 epochs. As soon as the line is added, the result is only about 6 percent after the second epoch at the latest, and here it remains, no matter how many epochs are trained.</p>
<p>So I assume that I have some bug in the code that makes the CNN get stuck in such a bad area, but I can’t find it.</p>
<p>I hope one of you can help me. Here is my code:</p>
<pre><code class="lang-auto">from functools import partial

import torch
from deconvolution.models.deconv import FastDeconv
from torch import nn as nn
from torch.nn import functional as F
import PIL

class Stn(nn.Module):
    def __init__(self):
        super(Stn, self).__init__()
        # Spatial transformer localization-network
        self.loc_net = nn.Sequential(
            nn.MaxPool2d(2,2),
            nn.Conv2d(3, 250, kernel_size=5, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(2,2),
            nn.Conv2d(250, 250, kernel_size=5, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(2,2),
            nn.Flatten(),
            nn.Linear(250*6*6, 250),
            nn.ReLU(),
            nn.Linear(250, 6)
        )

    def forward(self, x):
        xs = self.loc_net(x)
        theta = xs.view(-1, 2, 3)
        grid = F.affine_grid(theta, x.size())
        x = F.grid_sample(x, grid)
        return x


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        deconv = partial(FastDeconv)
        stn = partial(Stn)
        self.stn = stn()
        self.deconv = deconv(3, 3, kernel_size=3, stride=1, padding=1)
        self.conv1 = nn.Conv2d(3, 200, (7, 7), stride=1, padding=2)
        self.conv2 = nn.Conv2d(200, 250, (4, 4), stride=1, padding=2)
        self.conv3 = nn.Conv2d(250, 350, (4, 4,), stride=1, padding=2)
        self.fc1 = nn.Linear(12600, 400)
        self.fc2 = nn.Linear(400, 43)
        self.lcn1 = LocalContextNorm(200)
        self.lcn2 = LocalContextNorm(250)
        self.lcn3 = LocalContextNorm(350)
        self.dropout = nn.Dropout2d(0.2)


    def forward(self, x):
     #### Hidden 1 ####
      x = self.deconv(x)
      #x = self.stn(x)
      
      x = F.max_pool2d(F.relu(self.conv1(x)), 2)
      x = self.dropout(x)
      x = self.lcn1(x)
      x = self.dropout(x)

      #### Hidden 2 ####
      x = F.max_pool2d(F.relu(self.conv2(x)), 2)
      x = self.lcn2(x)
      x = self.dropout(x)

      #### Hidden 3 ####
      x = F.max_pool2d(F.relu(self.conv3(x)), 2)
      x = self.lcn3(x)
      x = self.dropout(x)

      #### Hidden 4 ####
      x = torch.flatten(x, 1)
      x = self.fc1(x)
      x = F.relu(x)

      #### Out ####
      x = self.fc2(x)
      return x
</code></pre>
When using distribute, how to write log?

wangyilan: <p>I implement my code modified from <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a> ,and now I have 2 problems:<br>
1.Dose this code is now the best way to do distribution?<br>
2.How to to write logs? In my opinion, We should find the major process and wrtie, but could the mp.spawn can do this?<br>
Many thanks to your replies!</p>

mrshenli: <blockquote>
<p>1.Dose this code is now the best way to do distribution?</p>
</blockquote>
<p>This depends on the application requirements. For available tools, see this: <a href="https://pytorch.org/tutorials/beginner/dist_overview.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Distributed Overview — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
<blockquote>
<p>2.How to to write logs? In my opinion, We should find the major process and wrtie, but could the mp.spawn can do this?</p>
</blockquote>
<p>You can use the rank to control which process does the log, e.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.distributed as dist

if dist.get_rank() == 0:
    # do log
</code></pre>
<p>If you are using RPC, the counterpart API is <a href="https://pytorch.org/docs/master/rpc.html#torch.distributed.rpc.get_worker_info" rel="noopener nofollow ugc"><code>get_worker_info</code></a>.</p>
Speed up fully-separable 1D convolution?

Raj_Bhattacharjea: <p>Suppose I have a 1D convolutional layer with 2 input channels, 32 output channels, and length 9 kernels. My weight tensor has a very special structure: it can be expressed as an “outer product” of three tensors as seen below, where I generate a dummy weight matrix and some dummy data of this form and calculate the convolution using conv1d:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

in_channels = 2
out_channels = 32
kernel_size = 9
nsamples = 2**12
batch_size = 1
padding = kernel_size//2
x = torch.randn(batch_size, in_channels, nsamples)

outspace = torch.randn(out_channels,1,1)
inspace = torch.randn(1,in_channels,1)
kernelspace = torch.randn(1,1,kernel_size)
w = outspace*inspace*kernelspace
y = F.conv1d(x,w,padding=padding)
</code></pre>
<p>The space of all weight tensors of the same shape as <code>w</code> shape requires <code>in_channels*out_channels*kernel_size parameters</code> to describe, but the restricted space of tensors that factor as above only requires <code>in_channels+out_channels+kernel_size parameters</code> to describe. Correspondingly, there should be a sequence of simpler operations that calculates <code>F.conv1d(x,w)</code> as above with far fewer multiplies when <code>w</code> has the special structure above. Something like</p>
<pre><code class="lang-auto">y = op1(kernelspace,x)
y = op2(inspace, y)
y = op3(outpsace, y)
</code></pre>
<p>For some <code>op1</code>, <code>op2</code>, and <code>op3</code>. I can write out the math for how to do it but I’m wondering what’s the fast way to do this in pytorch? I imagine group convolution and clever use of matmul would do it, but I’m not familiar enough with the API to formulate it.</p>
Torch.rfft() with onnx

hashbrown: <p>Question about onnx exporting a model which uses torch.rfft() in the forward function. How would this work as ONNX opset doesn’t support FFT yet.</p>
<p>Do let me know. Thanks!</p>

tom: <p>I think you’ll just get a funny-looking operator (aten::…) in ONNX.<br>
The other question is, of course, which backend would than execute your operator.</p>
<p>Best regards</p>
<p>Thomas</p>

hashbrown: <p>Would use x86_64 CPU to run this using onnxruntime.</p>

tom: <p>From a cursory look, it seems to be a contributed operator shown as <code>com.microsoft.rfft</code> there. Maybe you can pretend that aten::rfft is a custom op and <a href="https://pytorch.org/docs/stable/onnx.html#custom-operators" rel="nofollow noopener">register a custom ONNX operator for it</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
Realistically how many simultaneous training runs can be ran to train with the same data

add023: <p>I have ~40GB compressed images for training, separated by folders. Each folder has about 150 images, and there are ~10000 folders in total. I load N number of folders at a time where N is the batch size. Loading the whole dataset in memory is not an option for me.</p>
<p>I have 5 simultaneous training running with various hyper parameters. They are running in isolated docker containers, but they are using the same data folder. I have noticed the training is getting slower as I add new training runs. It is possible more than 1 training run is trying to access the same data folder. When does this start becoming a problem? I need to run many more for experimentation.</p>

ptrblck: <aside class="quote no-group" data-username="add023" data-post="1" data-topic="95587">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/87869e/48.png" class="avatar"> add023:</div>
<blockquote>
<p>It is possible more than 1 training run is trying to access the same data folder. When does this start becoming a problem?</p>
</blockquote>
</aside>
<p>I don’t think the problem is that multiple processes are accessing the same data folder, but that multiple processes are generally accessing the storage at the same time thus sharing the bandwidth of e.g your SSD.</p>

add023: <p>Ok, makes sense. Thanks.</p>
Dynamic batch size learning rate

carmocca: <p>I have implemented a custom DataLoader batch_sampler to have dynamic batch sizes during training. The first batch has a fixed size but the rest do not. e.g:</p>
<p>original_batch_size = 5</p>
<ul>
<li>iteration 1: original_batch_size samples</li>
<li>iteration 2: 8 samples</li>
<li>iteration 3: 13 samples</li>
</ul>
<p>Does that mean that I should scale the learning rate every iteration before calling <code>optimizer.step()</code>?</p>
<pre><code class="lang-python"># pseudocode:
prev_lr = optimizer["lr"]
# scale the learning rate depending on the current batch size
# for example, one batch may have 2 samples but another may have 100
# and we dont want to use the same learning rate in both cases
optimizer["lr"] = prev_lr / original_batch_size * current_batch_size
# update parameters
optimizer.step()
# keep original value
optimizer["lr"] = prev_lr
</code></pre>
<p>This would also apply when using a DataLoader with drop_last=False, where the last batch might have less samples than the rest of the batches, correct?</p>

googlebot: <p>Usually, loss is divided by batch size (i.e. reduction=“mean”) to have the same effect.</p>

carmocca: <p>I do not understand how that would have the same effect. It would if all of the batches have the same batch_size (normal case)</p>
<p>But when every batch can have a different length, you would be using the same learning rate for a loss averaged over 1 sample than one averaged over a 1000 samples. But we want a larger learning rate in the second case, don’t we?</p>

googlebot: <aside class="quote no-group" data-username="carmocca" data-post="3" data-topic="95944">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/carmocca/48/28679_2.png" class="avatar"> carmocca:</div>
<blockquote>
<p>But when every batch can have a different length, you would be using the same learning rate for a loss averaged over 1 sample than one averaged over a 1000 samples.</p>
</blockquote>
</aside>
<p>For both batches you’ll get unbiased estimates of global dataset’s gradients directions. Estimators from small batches will have higher variance, but I don’t think that this is a problem with realistic batch sizes, as if I’m not mistaken, sampling errors reduce as 1/sqrt(batch_size).</p>
<p>There is also shuffling to smooth steps taken in the long term.</p>