akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Error in the PyTorch GRU Doc Page

abedshantti: <p>I have noticed that there might be an error in the docs for the GRU layer.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.GRU.html</a></p>
<p>Shouldn’t the hidden layer in the 4th line of the definition be presented as:</p>
<p>h(t)  = ( (1 - z(t) ) * h(t-1) ) + ( z(t) * n(t) )</p>
<p>instead of: h(t)  = ( (1 - z(t) ) * n(t) ) + ( z(t) * h(t-1) )</p>
<p>as it is currently written in the docs?</p>
<p>Here is the architecture of the GRU layer:</p>
<aside class="onebox githubgist">
  <header class="source">

      <a href="https://gist.github.com/sey-kh/34cd13a4139b30776ff697b431c3c370" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/sey-kh/34cd13a4139b30776ff697b431c3c370" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/sey-kh/34cd13a4139b30776ff697b431c3c370</a></h4>

  <h5>gru_recurrent_network.md</h5>
  <pre><code class="Markdown"># GRU recurrent neural network  #

GRU (Gated Recurrent Unit) aims to solve the `vanishing gradient problem` (The problem is that in some cases, the gradient will be small, effectively preventing the weight from changing its value then the network stop learning) which comes with a **standard recurrent neural network**.

## Standard recurrent neural network ##
![reccurrent-network-arch (1)](https://user-images.githubusercontent.com/49018140/59248727-c1309f80-8c12-11e9-82ea-fb6b6c70ed53.png)

**RNN network** can predict output base on previous output or predict output by taking external input + previous output. It kind of ilterate process by taking previous output to generate new output in terms of processing sequence data.

As above diagram mentioned, there are external input (I0, I1, I2) and sequence output (O0, O1, O2)  </code></pre>
    This file has been truncated. <a href="https://gist.github.com/sey-kh/34cd13a4139b30776ff697b431c3c370" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>Thanks for raising this issue. Would you mind creating a <a href="https://github.com/pytorch/pytorch/issues">GitHub issue</a> so that the code owners can take a look there, too? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
#IndexError: too many indices for tensor of dimension 1

Ragnor: <pre><code>def compute_mask_loss(self, mask_predict, positive_gt_idx, box_predicts, targets):
    mask_gt = targets['mask'].split(targets['batch_len'])
    box_gt = targets['target'].split(targets['batch_len'])
    loss_mask_predicts = list()
    loss_mask_target = list()
    for mg, bg, mp, pgi, bp in zip(mask_gt, box_gt, mask_predict, positive_gt_idx, box_predicts):
        cls_idx = bg[:, 0].long()[pgi]
        mg_t = mg[pgi]
        mp = mp[range(len(mp)), cls_idx, :]
        bp_extend = torch.cat([torch.arange(len(bp), device=bp.device)[:, None], bp], dim=-1)
        mt = roi_align(mg_t[:, None, :, :], bp_extend, (mp.shape[-1], mp.shape[-1]), 1.)[:, 0]
        loss_mask_target.append(mt)
        loss_mask_predicts.append(mp)
    loss_mask_predicts = torch.cat(loss_mask_predicts)
    loss_mask_target = torch.cat(loss_mask_target)
    mask_loss = self.bce(loss_mask_predicts, loss_mask_target)
    return mask_loss</code></pre>

ptrblck: <p>I don’t know, which operation is raising this issue, but it could be the indexing of <code>mg_t</code>, in case it has a single dimension as seen here:</p>
<pre><code class="lang-python">mg_t = torch.randn(10)
mg_t[:, None, :, :]
&gt; IndexError: too many indices for tensor of dimension 1
</code></pre>
Difference between torchscript and onnx?

tuttelikz: <p>Could anybody please kindly explain difference between <strong>torchscript</strong> and <strong>onnx</strong>? As far as I understand, both are the scripted formats to export PyTorch models for faster inference on devices/environments without Python dependency (please correct me if I am wrong). In which real-world use case one would prefer over the other.<br>
Thank you!</p>
Child process randomly throws an error every once in a while

DarQ: <p>I’m generating a bunch of random data on the fly in an IterableDataset<br>
loading the data in the main thread works just fine, And although setting num_workers=2 STILL works fine, the received data is uncorrupted and the training loop continues. But every now and then it throws this exception:</p>
<pre><code class="lang-auto">Exception ignored in: &lt;function _MultiProcessingDataLoaderIter.__del__ at 0x7f5ae7bc4d40&gt;
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1328, in __del__
    self._shutdown_workers()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1320, in _shutdown_workers
    if w.is_alive():
  File "/usr/lib/python3.7/multiprocessing/process.py", line 151, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
Exception ignored in: &lt;function _MultiProcessingDataLoaderIter.__del__ at 0x7f5ae7bc4d40&gt;
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1328, in __del__
    self._shutdown_workers()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1320, in _shutdown_workers
    if w.is_alive():
  File "/usr/lib/python3.7/multiprocessing/process.py", line 151, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
Exception ignored in: &lt;function _MultiProcessingDataLoaderIter.__del__ at 0x7f5ae7bc4d40&gt;
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1328, in __del__
    self._shutdown_workers()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1320, in _shutdown_workers
    if w.is_alive():
  File "/usr/lib/python3.7/multiprocessing/process.py", line 151, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
</code></pre>
<p>Should I worry about this? and what do I have to do to stop it?</p>
CUDA Available, moved model to Cuda:0. Still runs on CPU

Chris_Rollings: <p>Hi I’m trying to get my model to run on the GPU.</p>
<p>I am using this example code: <a href="https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler With TensorBoard — PyTorch Tutorials 1.9.0+cu102 documentation</a><br>
except in mine I call torch.cuda.is_available() to check it is available which returns True.</p>
<p>In Tensorboard however I can see it is running on CPU and my CPU usage goes up…</p>
<p>What am I missing here? I’m using a GTX 1080TI on windows 10. Thanks</p>

ptrblck: <p>If you’ve moved the model as well as the inputs to the GPU, it would be used (I don’t know what exactly TensorBoard is showing).<br>
You could verify it by checking the <code>device</code> argument of some parameters of the model and check the memory usage as well as the utilization of the GPU via <code>nvidia-smi</code>.</p>

Chris_Rollings: <p>Tensor board shows<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41ecde4839551e0bc5dab421de2492e6a4484345.png" alt="image" data-base62-sha1="9pcwswowOlCJdOPJK2tUWRNEHc1" width="499" height="384"><br>
my model and training data are placed onto cuda using cuda:0</p>
<p>Printing the parameters device outputs cuda:0.</p>
<p>Thanks for your help</p>

Chris_Rollings: <p>sorry I replied but now sure you have seen it <img src="https://discuss.pytorch.org/images/emoji/apple/open_mouth.png?v=9" title=":open_mouth:" class="emoji" alt=":open_mouth:"></p>

ptrblck: <p>Unfortunately, I don’t know what Tensorboard tries to show as the “Device Type”. Were you able to check <code>nvidia-smi</code> for the memory usage and utilization?</p>
Dataloading with Homogeneous Batches from Pools of Data

sauhaardac: <p>I have several “pools” of data. For my given training task, I’d like to load data from these pools, but need to ensure that in a given batch only data from the same “pool” is returned.</p>
<p>The way I currently have this implemented is to create a separate dataloader for each pool and create a master dataloader that wraps the individual dataloaders. I’ve noticed this is pretty slow though, as it loses out on a lot of the optiization the pytorch dataloader does. Are there any suggestions to optimize this behvavior?</p>
<p>For reference, here is the code I have for the “MasterDataLoader”:</p>
<pre><code class="lang-python">class MasterLoader:
    def __init__(self, data_files, max_active_loaders=10, **data_loader_args):
        self.current_loader = 0
        self.max_active_loaders = max_active_loaders
        self.active_loaders = []
        self.data_files = data_files
        self.data_loader_args = data_loader_args

        batch_size = data_loader_args['batch_size']
        self.len = 0

        for file in tqdm(self.data_files, desc='Loading and verifying files'):
            try:
                self.len += np.ceil(len(torch.load(file)) / batch_size)
            except EOFError:
                self.data_files.remove(file)
        self.len = int(self.len)
        print(f'There are a total of {self.len} batches in an epoch.')

    def __len__(self):
        return self.len

    def __iter__(self):
        self.data_files_queue = self.data_files[:]  # reset queue
        self.active_loaders = [iter(DataLoader(AIDDataset(self.data_files_queue.pop()), **self.data_loader_args))
                               for _ in range(min(len(self.data_files_queue), self.max_active_loaders))]
        return self

    def __next__(self):
        while True:  # iterate until valid sample is returned
            if len(self.active_loaders) == 0 and len(self.data_files_queue) == 0:  # finished an epoch
                raise StopIteration

            try:
                sample = next(self.active_loaders[self.current_loader])
                break
            except StopIteration:  # if the dataloader ran out of samples
                if len(self.data_files_queue) &gt; 0:
                    self.active_loaders[self.current_loader] = iter(DataLoader(AIDDataset(self.data_files_queue.pop()),
                                                                               **self.data_loader_args))
                else:
                    self.active_loaders.pop(self.current_loader)  # remove loader at current index

        self.current_loader += 1
        self.current_loader = self.current_loader % len(self.active_loaders)

        return sample
</code></pre>
About change of way to store tensors in GPU

vuhoangminh: <p>Dear All,</p>
<p>At the end of an intermediate layer in the forward pass, I want to store a modified version of the output of this layer instead of the original one. Then in the backward pass, I would like to use the modified version to compute the gradient for the next layer. Let’s assume that the operation of that intermediate layer (ReLU, conv. …) is unchanged.</p>
<p>My questions are</p>
<ul>
<li>How to ask PyTorch to save the modified tensor and detach the original one from the graph?</li>
<li>If PyTorch allows to do so, how will the GPU allocation be affected? Will it be different by the amount of the difference between the original and modified tensors?</li>
</ul>
<p>Help is much appreciated!</p>
How to change loss between epochs

mrubio-chavarria: <p>Hello everyone,</p>
<p>I have a model in which I use a loss function for the first epoch, and a second loss the second epoch onwards. In pseudocode:</p>
<pre><code class="lang-auto">for batch in batches:
    model.zero_grad()
    output = model(batch)
    if epoch == 0:
        loss_function = loss1
    else:
        loss_function = loss2
    loss = loss_function(output)
    loss.backward()
    optimiser.step()
</code></pre>
<p>Is there any problem? Should I change something to connect both losses? I thought that because of the backward computes the gradients for all the tensors with <code>requires_grad=True</code> and the <code>zero_grad()</code>, no connection is needed. Am I right?</p>
<p>The reason is that I have a model to classify sequences of characters in a speech recognition problem. Therefore,  I am using the CTC loss. However, I fall in a local minimum around loss=1.50 (all blanks). I wanted to use another loss during the first <em>n</em> epochs (e.g. cross entropy) to reach a loss value &lt; 1.50 to change then to the CTC and go on with the training.</p>
<p>Unfortunately, when I do this, I reach a value of 1.38 with the first loss (cross entropy) and then, when I change to CTC the loss, it drastically increases to 8. Therefore, I fall in the same minimum afterwards. I understood that I was doing something wrong.</p>
<p>Thank you very much!</p>

pascal_notsawo: <p>Regarding the loss functions, you can even use a different one at each epoch without problem: if I’m not mistaken the only condition is that they are differentiable (=&gt;and do not break the computation graph)</p>
Modèle ResNet18

Achrouf: <p>Bonjour, je souhaite faire la classification de tumeurs d’une image de pathologie numérique, j’utilise un modèle CNN déja près que j’ai récupéré du site:(<a href="https://www.frontiersin.org/articles/10.3389/fmolb.2020.00183/full" rel="noopener nofollow ugc">https://www.frontiersin.org/articles/10.3389/fmolb.2020.00183/full</a>)<br>
j’obtiens l’erreur suivante: ‘ReLU’ object has no attribute ‘threshold’<br>
il semble qu’il y ait une incohérence dans le modèle pré-entrainé, car un des activations<br>
essaye de passer une valeur seuil à une activation de type “relu”, alors que cette fonction d’activation n’a pas de paramètre de seuil.<br>
merci d’avance pour votre réponse</p>

ptrblck: <p>From Google translate:</p>
<blockquote>
<p>Hello, I want to do the classification of tumors from a digital pathology image, I am using a CNN model already close that I got from the site: (<a href="https://www.frontiersin.org/articles/10.3389/fmolb">https://www.frontiersin.org/articles/10.3389/fmolb</a>. 2020.00183 / full)<br>
I get the following error: ‘ReLU’ object has no attribute ‘threshold’<br>
there seems to be an inconsistency in the pre-trained model, because one of the activations<br>
tries to pass a threshold value to a “reread” type activation, when this activation function does not have a threshold parameter.<br>
Thank you in advance for your reply</p>
</blockquote>
<p><a class="mention" href="/u/achrouf">@Achrouf</a> could you use an online translator for further posts, please? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>For the issue:<br>
The <code>nn.ReLU</code> module doesn’t have a <code>threshold</code> method, so you might want to use e.g. <code>torch.threshold</code>.</p>

Achrouf: <p>please can tell me what to do exactly, i’m new to cnn network with python, thanks</p>

ptrblck: <p>Could you post a minimal and executable code snippet, which would reproduce this issue, so that we could help debugging?<br>
Based on the error it seems you are trying to use:</p>
<pre><code class="lang-python">relu = nn.ReLU()
relu.threshold()
&gt; AttributeError: 'ReLU' object has no attribute 'threshold'
</code></pre>
<p>while most likely you want to use:</p>
<pre><code class="lang-python">relu = nn.ReLU()
x = torch.randn(10, 10)
out = relu(x)
</code></pre>

Achrouf: <p>Thank you for your answer, there was no problem in the resnet model that I used, I just installed a version 1.1.0 pyrtorch and it worked! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Outputs are predicted to be same class for neural network with multiclass classification

rajasai_b: <p>Hi, I am a student, I have written this code for neural network and I’m trying to classify image data using below neural network as covid or viral or normal. But my model, even running on several epochs, it predicts entire images to be either only one of the three classes i.e, if it predicts ,it predicts all images to be covid, or viral or normal. This happens even during the training phase, I think i’m missing something , please help.<br>
the dataset link: <a href="https://www.kaggle.com/tawsifurrahman/covid19-radiography-database" class="inline-onebox" rel="noopener nofollow ugc">COVID-19 Radiography Database | Kaggle</a><br>
The dataset is imbalanced, but i have also tried taking subset of dataset with equal number of images for each class. Please correct me</p>

eqy: <p>Can you share the code and some sample outputs/labels of the model? It might help to get a sense of whether the predictions are not changing during training or something else is happening.</p>
Restore optimizer for sharded model

Eti49: <p>Hi,<br>
I am using a sharded model (part of model on gpu 0 and part of model on gpu 1). (As suggested here <a href="https://discuss.pytorch.org/t/sharding-model-across-gpus/1065/2" class="inline-onebox">Sharding model across GPUs - #2 by ajdroid</a> )</p>
<p>Since i need to stop and re-start training sometimes, i save both the state_dict of the model and the optimizer to disk. The problem arises when i restore the model and the optimizer.<br>
This is the code that loads the states</p>
<pre><code class="lang-auto">state = torch.load(path, map_location='cpu')
model.load_state_dict(state["model_state_dict"])
optimizer.load_state_dict(state["optimizer_state_dict"])
</code></pre>
<p>Afterwards i run a function that moves the child modules of my model to the approriate gpus:</p>
<pre><code class="lang-auto">model.move_to_gpu()  
</code></pre>
<p>Previously i relied on the following code to move the optimizers parameters to the gpu</p>
<pre><code class="lang-auto">for param in optim.state.values():
        if isinstance(param, torch.Tensor):
            param.data = param.data.to(device)
            if param._grad is not None:
                param._grad.data = param._grad.data.to(device)
        elif isinstance(param, dict):
            for subparam in param.values():
                if isinstance(subparam, torch.Tensor):
                    subparam.data = subparam.data.to(device)
                    if subparam._grad is not None:
                        subparam._grad.data = subparam._grad.data.to(device)
</code></pre>
<p>This obviously failed with an error that elements are on different cuda devices.</p>
<p>Afterwards i tried to leave out the part where the optimizers parameters where moved to gpu and my program started working again.</p>
<p>My question:</p>
<ol>
<li>Am i missing something? When i do <code>state = torch.load(path, map_location='cpu')</code> i load the tensors to cpu, so how can the parameters of the optimizer can now be located on the right device?</li>
<li>When i understood something wrong: What is the appropriate way to handle optimizer state when checkpointing a model, both in the un-sharded and sharded scenario?</li>
</ol>
<p>For reference: I am using Adam optimizer</p>
<p>Thanks in advance</p>
What changes we need to make in metrics calculation when we are using Distributed Data Parallel for Multi-GPU training

Bilal_Yousaf: <p>I am updating my training script to use Distributed Data Parallel to do Multi-GPU training.<br>
I am done with most of the steps as mentioned in PyTorch Guidelines.<br>
But I am confused about how to handle metrics calculation and visualization:<br>
For example,<br>
I need to calculate accuracy and I have 4 samples in total and 2 GPUs.<br>
When I run testing I will have predictions and ground truths for 2 2 samples in each process.<br>
Now if I want to calculate accuracy do I need to call dist.reduce or it is not needed and I can directly calculate accuracy in rank 0 process.</p>

gcramer23: <blockquote>
<p>I need to calculate accuracy and I have 4 samples in total and 2 GPUs.<br>
When I run testing I will have predictions and ground truths for 2 2 samples in each process.<br>
Now if I want to calculate accuracy do I need to call dist.reduce or it is not needed and I can directly calculate accuracy in rank 0 process.</p>
</blockquote>
<p>Do you need to calculate local metrics or global metrics? To calculate global metrics, you would need to communicate between the processes <a href="https://pytorch.org/docs/stable/distributed.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 2.1 documentation</a>.</p>

Bilal_Yousaf: <p>Thanks a lot for the quick reply. It was really helpful. I have understood how to use reduce functions to solve the issue and I can get correct metrics as I was getting for single GPU.</p>
<p>Could you please share if there is any way I can gather dictionaries created on multiple processes for multi-gpu on rank 0 process as for now I think all such reduce/gather functions work for only tensors?</p>

CBGxd: <p>Could I just use the rank 0 to calculate everything? I know that the speed will be slower (which is not obvious in my case), but the accuracy should be same, right?</p>

Bilal_Yousaf: <p>For calculating global accuracy on rank0, don’t change sampler of TestDataloader to DDP sampler. And only use rank0 to run the testing loop.</p>
How to load models wrapped with SyncBatchNorm?

oasjd7: <ol>
<li>Save model which is wrapped with <code>DataParallel()</code>.</li>
</ol>
<pre><code class="lang-auto">torch.save(model.module.state_dict(), save_folder + '/' + 'model.pt')
</code></pre>
<ol start="2">
<li>Load this model &amp; Train with <code>SyncBatchNorm + DDP</code>.</li>
</ol>
<pre><code class="lang-auto"># Define Model
model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model)
model.cuda(args.gpu)
model = torch.nn.parallel.DistributedDataParallel(net, device_ids=[args.gpu], find_unused_parameters=True)
# Load Model
loc = 'cuda:{}'.format(args.gpu)
net.module.load_state_dict(torch.load(save_folder + '/model.pt', map_location=loc), strict=False)

</code></pre>
<pre><code class="lang-auto">Error(s) in loading stage_dict for DistributedDataParallel:
Missing key(s) in state_dict: "module.con1.weight", "module.bn1.weight", ...
</code></pre>
<p>How can I load my models trained with <code>DataParallel()</code> after warp with <code>SyncBatchNorm + DDP</code> ?</p>

Yanli_Zhao: <p>just do like this:</p>
<h1>
<a name="define-model-1" class="anchor" href="#define-model-1"></a>Define Model</h1>
<p>model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model)<br>
model.cuda(args.gpu)<br>
model_DDP = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu], find_unused_parameters=True)</p>
<h1>
<a name="save-and-load-model-2" class="anchor" href="#save-and-load-model-2"></a>save and load Model</h1>
<p>torch.save(model_DDP, tmp.name)<br>
model_DDP = torch.load(tmp.name)</p>
Tensorboard for Distributed Training using DistributedDataParallel

hugo_thim: <p>Hello,</p>
<p>I have seen some questions related to using tensorboard with <code>DistributedDataParallel</code>(DDP) on the forum but I haven’t found a definitive answer to my question.</p>
<p>For instance, I wish to log loss values to tensorboard. When not considering DDP my code looks like the following for a loss item</p>
<pre><code class="lang-auto">loss_writer.add_scalar('Overall_loss', overall_loss.item(), total_iter)
</code></pre>
<p>where <code>loss_writer = torch.utils.tensorboard.SummaryWriter(loss_dir)</code>.<br>
Now, when considering DDP I included as recommended <a href="https://discuss.pytorch.org/t/using-tensorboard-with-distributeddataparallel/102555/2?">here</a> to include</p>
<pre><code class="lang-auto">if args.rank == 0 :
    loss_writer.add_scalar('Overall_loss', overall_loss.item(), total_iter)
</code></pre>
<p>However, when doing so, I obtain something that looks like this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07c3ef36b26e85ef052bbf9d5e4ee7e1cf5e3cbe.png" alt="Selection_706" data-base62-sha1="16H83dCLhFysT2n3DQlTcYF7hSe" width="372" height="320"></p>
<p>which has been evoked in <a href="https://discuss.pytorch.org/t/using-tensorboard-with-distributeddataparallel/102555/8?">here</a>.</p>
<p>Any idea on how to obtain a smooth loss graph as on single-GPU training ?<br>
Thanks !</p>

cbalioglu: <p>I don’t have an immediate answer. Are you also making sure that <code>SummaryWriter</code> is only instantiated by rank 0 (as suggested by the link you referred to in your question)?</p>

hugo_thim: <p><a class="mention" href="/u/cbalioglu">@cbalioglu</a> Hello, thank you for your answer!<br>
Unfortunatly, my <code>SummaryWriter</code> is in deed instanciated only by rank 0.</p>
Why does the following give me a concurrency error?

mrakgr: <pre><code class="lang-auto">    with torch.cuda.stream(s):
        w = sample_indices.cpu().numpy()
    q = action_probs.detach().cpu().numpy()
    s.synchronize()
</code></pre>
<p>Later when I try to access <code>w</code> I get an out of bounds error which does not happen if I do the <code>.cpu()</code> transfers on the same stream. Looking at the value of <code>w</code>, it is clearly corrupted. Is the transfer to host not being synchronized for some reason?</p>

mrakgr: <p>I forgot to wait on the default stream. <code>s.wait_stream(torch.cuda.default_stream())</code> fixes it.</p>
Encrypt the model file

wowdeeply: <p>Currently I need to deploy the trained model on a public computer, but I don’t want others to see the structure of my model. Is there a way to encrypt the model file (.pth)?</p>

chhaya_kumar_das: <p>Crypten has some functionality around this. <a href="https://github.com/facebookresearch/CrypTen" rel="noopener nofollow ugc">GIthub link</a></p>
Tqdm bar is not changing

YoonhoRoh: <p>Hello, I’m trying to create progress bar using tqdm package.<br>
Here is my code</p>
<pre><code class="lang-auto">import numpy as np
import os
from glob import glob
from PIL import Image
from tqdm.notebook import tqdm

path = glob('dataset/mnist_png/training/*/*.png')

height = []
width = []

for i in tqdm(path):
    img_pil = Image.open(path)
    image = np.array(img_pil)
    h, w = image.shape
    height.append(h)
    width.append(w)

</code></pre>
<p>However, progress bar does not changing and stops like this, even though the calculation is done<br>
0%|          | 0/60000 [00:00&lt;?, ?it/s]</p>
<p>Is there any way to make progress bar changing?<br>
I am looking forward to see any help. Thanks in advance.<br>
Kind regards,<br>
Yoon Ho</p>

pascal_notsawo: <p>Already, your code seems to have a problem : you still use path in the loop</p>
<pre><code class="lang-auto">
for p in tqdm(path):

    #img_pil = Image.open(path) 

    img_pil = Image.open(p) 

    # ...

</code></pre>
<p>The code works for me without any problem.</p>
<pre><code class="lang-auto">
import numpy as np

import os

from glob import glob

from PIL import Image

#from tqdm import tqdm

from tqdm.notebook import tqdm

path = glob('/content/*.png')

height = []

width = []

print("")

for p in tqdm(path):

    img_pil = Image.open(p)

    image = np.array(img_pil)

    print(image.shape)

    h, w, _ = image.shape

    height.append(h)

    width.append(w)

</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d07b76568cc0a85a102993db21389fa07ff90fb9.jpeg" alt="1" data-base62-sha1="tKjPuwxlZUKbzSaTJJ7HoniMgSB" width="517" height="90"></p>

YoonhoRoh: <p>Thanks for your answer.<br>
However, even though I replaced path to p, my bar still stops at<br>
0%|          | 0/60000 [00:00&lt;?, ?it/s]</p>
Use AdaptiveMaxPool indices for training

mhtrinh: <p>Hi,</p>
<p>I would like to know if it is possible to use Indices returned by AdaptiveMaxPool to feed into FC layers (not for Unpool) ?</p>
<p>The idea is to provide to the network what features is present, but also where is it. That may help for Embedding or Metric learning ?</p>
<p>Is this a crazy idea ?</p>

ptrblck: <aside class="quote no-group" data-username="mhtrinh" data-post="1" data-topic="124774">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/c6cbf5/48.png" class="avatar"> mhtrinh:</div>
<blockquote>
<p>Is this a crazy idea ?</p>
</blockquote>
</aside>
<p>I don’t think any idea is too crazy to run some experiments. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
Note that the returned indices from this pooling layer are “detached”, i.e. they do not have a <code>grad_fn</code> and thus Autograd won’t backpropagate through them, so you should keep this information in mind when experimenting with it.</p>

mhtrinh: <p>That what I am afraid of: the indice returned do not have <code>grad_fn</code><br>
Does this mean that backward will have some weird effect ? Would there be any trick to have the indices “attached” ?</p>

ptrblck: <p>Not directly, since you won’t be able to calculate gradients for these indices (unless you can come up with a valid backward method to do so and would implement it manually). Autograd will properly backpropagate to the max. values, but not the indices.</p>
Instance segmentation on custom dataset with COCO annotations

AndriPi: <p>I would like to train an instance segmentation model on a custom dataset, for which I converted the annotations in COCO format. Which frameworks would you suggest me to use? I know about <a href="https://github.com/pytorch/vision" rel="noopener nofollow ugc">torchvision</a>, <a href="https://github.com/open-mmlab/mmdetection" rel="noopener nofollow ugc">MMDetection</a> and <a href="https://github.com/facebookresearch/detectron2" rel="noopener nofollow ugc">Detectron2</a>. I need a framework which</p>
<ol>
<li>supports instance segmentation (clearly <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:">)</li>
<li>consumes COCO annotations</li>
<li>is a good compromise between flexibility and robustness/ease of use.</li>
<li>has a Docker image. Bonus points if the Docker image is available on <a href="https://ngc.nvidia.com/" rel="noopener nofollow ugc">https://ngc.nvidia.com/</a>, or if it’s compatible with the NVIDIA PyTorch image available there.</li>
</ol>
<p>Which one would you suggest to use? Thanks,</p>
<p>Andrea</p>
AttributeError: 'Tensor' object has no attribute 'items' in swin

Mr.L: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1235ca78cd2ffae777e468430b01b755a0783e1.png" data-download-href="https://discuss.pytorch.org/uploads/default/c1235ca78cd2ffae777e468430b01b755a0783e1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1235ca78cd2ffae777e468430b01b755a0783e1_2_690x147.png" alt="image" data-base62-sha1="ryzUGTINRHWtIeIbJVNxKaDw7iV" width="690" height="147" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1235ca78cd2ffae777e468430b01b755a0783e1_2_690x147.png, https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1235ca78cd2ffae777e468430b01b755a0783e1.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1235ca78cd2ffae777e468430b01b755a0783e1.png 2x" data-dominant-color="1C1B1D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">742×159 35 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
hello, i encounter this problem in line: for name, x in xs.items(), how to solve it, thanks</p>

ptrblck: <p>Tensors don’t have the <code>items()</code> method (which is a method of e.g. the <code>dict</code> class) and I’m unsure which method you are looking for, as it seems you would like to get a <code>name</code> and tensor from <code>xs</code> (I assume <code>xs</code> should be a <code>dict</code> or any other container?).</p>
Save torchsummary

Florentino: <p>I know how to print out the model by</p>
<blockquote>
<p>torchsummary.summary()</p>
</blockquote>
<p>,So I want to know is there any way to save the text into a txt file that I don’t know how to save it? Thank you</p>

ptrblck: <p>Assuming you are using <a href="https://github.com/sksq96/pytorch-summary/blob/345d898d84507b848e92dab4629e03405e19afce/torchsummary/torchsummary.py#L9-L14">this method from <code>torchsummary</code></a> you could call:</p>
<pre><code class="lang-python">result, params_info = summary_string(
        model, input_size, batch_size, device, dtypes)
</code></pre>
<p>and save the <code>result</code> into a file instead of printing it out.</p>

Florentino: <p>Thank you very much!</p>
Error in data loader

ekosman: <p>I created my own torch geometric graph classification dataset. However, I encounter the following problem:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/snap/pycharm-community/240/plugins/python-ce/helpers/pydev/pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/snap/pycharm-community/240/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/eitank/GraphVid/train_kinetics.py", line 207, in &lt;module&gt;
    args=args
  File "/home/eitank/GraphVid/train_kinetics.py", line 119, in train_video_recognition
    batch_splitter=classification_batch_splitter,
  File "/home/eitank/GraphVid/utils/PackageUtils/TorchUtils.py", line 168, in fit
    batch_splitter=batch_splitter, )
  File "/home/eitank/GraphVid/utils/PackageUtils/TorchUtils.py", line 263, in do_epoch
    for iteration, batch in enumerate(data_iter):
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch_geometric/data/dataloader.py", line 36, in __call__
    return self.collate(batch)
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch_geometric/data/dataloader.py", line 31, in collate
    return [self.collate(s) for s in zip(*batch)]
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch_geometric/data/dataloader.py", line 31, in &lt;listcomp&gt;
    return [self.collate(s) for s in zip(*batch)]
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch_geometric/data/dataloader.py", line 17, in collate
    self.exclude_keys)
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch_geometric/data/batch.py", line 151, in from_data_list
    batch[key] = torch.cat(items, cat_dim)
TypeError: expected Tensor as element 40 in argument 0, but got NoneType
</code></pre>
<p>How can I find the cause? I checked if the value returned with my dataset is None, and it isn’t.</p>

ptrblck: <p>Did you check the entire <code>items</code> content, as it seems that “element 40” is causing the issue and is a <code>None</code>?</p>
CUDA error when I call a preprocess function within a DataLoader

MauroGentile: <p>Hello,</p>
<p>I have a network which is the combination of a CenterTrack network (<a href="https://github.com/xingyizhou/CenterTrack" class="inline-onebox" rel="noopener nofollow ugc">GitHub - xingyizhou/CenterTrack: Simultaneous object detection and tracking using center points.</a>) and a custom network that is fed with the embedding from the CenterTrack net. Original images goes through CenterTrack and I branch out a couple of heads before  CenterTrack  starts upsampling.</p>
<p>The non efficient implementation of this model works fine, meaning that I get the output I expect. The only problem is that the whole process takes too long.</p>
<p>The inefficiency is due to the fact that the only way to make CenterTrack net working is through a for loop that iterates through every single image of a batch, applying the Centertrack network and extracting the embeddings.  Indeed Centertrack seems to work only on single images. This is what aurthors do in “demo.py”.</p>
<p>In order to get rid of the for loop, I tried to call the CenterTrack detector within <strong>getitem</strong> in the DataLoader. This is the most natural solution in my head. But such an implementation  I get the following error:</p>
<p>““RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method””</p>
<p>I get this error both if I initialise the CenterTrack in the DataLoader itself and also if I initialize it outside the dataloader, passing the CenterTrack instance to teh DataLoader as a parameter.</p>
<p>Does anyone know why I get this error and how I can fix it?</p>
<p>Than you</p>

ptrblck: <p>The error is raised, since you are most likely using multiprocessing in the <code>DataLoader</code> by specifying <code>num_workers&gt;0</code>, which would then try to re-initialize the CUDA context.<br>
Did you try the suggested workaround using the <code>'spawn'</code> start method?</p>
How to download torchtext datasets once?

Ayman_Al_Jabri: <p>Hi,</p>
<p>I’m experimenting with torchtext datasets e.g.:</p>
<pre><code class="lang-auto">train, test = torchtext.datasets.IMDB(PATH, split=('train', 'test'))
</code></pre>
<p>Unfortunately there is no <code>download=True</code> or <code>False</code> option in the set, which means every time I run the script it downloads the entire set from the internet again!</p>
<p>Is there a way to open the set from local folder once it’s there i.e download once?</p>
<p>Thanks,<br>
Ayman</p>

mmg: <p>Use <code>root</code> as argument e.g.</p>
<p><code>train, test = torchtext.datasets.IMDB(root=/home/user/datasets, split=('train', 'test'))</code></p>
<p>If it exists, it won’t download again.</p>

Ayman_Al_Jabri: <p>Are you sure?<br>
Because I’m already passing <code>PATH</code> as root in the code above, but we I run the line it takes longer than expected and it changes the timestamp on the PATH folder. That makes me think it’s either downloading again or re-building the train,test split … it’s definitely doing something!</p>

mmg: <p>Try disconnecting from the internet and see if that line runs fine <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Integrated gradients for negative labeled examples

chsher: <p>How do I compute the feature attributions for negative labeled examples (i.e., target = 0) when the model output is a scalar? Currently, integrated gradients gives me the feature attributions for all examples wrt target = 1.</p>

hoangle_tttm: <p>I guess your problem is a binary classification and you are using sigmoid for your output layer.<br>
Currently, it seems that specifying a specific output if you use sigmoid in the output layer is not supported yet. If you want integrated gradient (Captum) give you the feature attributions wrt a specific target, you should use softmax instead of sigmoid.</p>

chsher: <p>Thanks for your reply! I opted for sigmoid instead of softmax for binary classification to avoid doubling the number of parameters in the final layer. I wonder if there is a simple workaround, e.g., 1 - sigmoid(x) so target 1 becomes target 0, and vice versa.</p>

hoangle_tttm: <p>In fact, in the case of binary classification, a positive(+) attribution will contribute to the positive class and vice versa for the negative class. Hope this helps.</p>
Libtorch Tensor cannot use cuda

LF2: <p>VS2019 Manual build, not using cmake.</p>
<p>I tried two versions<br>
1.8.1 1.9.0</p>
<pre><code class="lang-auto">auto aRet = torch::cuda::is_available(); // 1.8.1 = false ,1.9.0 = true
auto adRet = torch::cuda::device_count(); // 1.8.1 = 0 ,1.9.0 = 1
torch::Tensor b = torch::randn({ 2, 2 });
b.to(torch::kCUDA);    //All errors
</code></pre>
<p>1.9.1 Tips  c10::NotImplementedError</p>
<p>Here are the libraries I included<br>
opencv_world410.lib<br>
c10.lib<br>
c10_cuda.lib<br>
torch.lib<br>
torch_cpu.lib<br>
torch_cuda.lib<br>
torch_cuda_cu.lib<br>
torch_cuda_cpp.lib<br>
pytorch_jni.lib<br>
cudart_static.lib</p>
<p>Save me…</p>
Two sparse matrix multiplication but one of them is diagonal matrix

zhao: <p>Hi. I have been looking for the sparse matrix multiplication: mm(S, S)-&gt;S for a long time and it seems there is no implementation for this. But my problem can be reduced to the matrix multiplication of a sparse matrix S and a sparse diagonal matrix D, and only D requires gradient. A doable solution is to define a forward and backward methods, using scipy to do the multiplication and manually compute the gradient (since it is just diagonal matrix it’s quite straightforward). However, doing so doesn’t allow me to use GPU. What are possible methods to make the multiplication more efficient? I want to try some math tricks but I cannot find one.</p>
<p>BTW my sparse matrices are all complex and the dense dimension can easily be too large to be converted to a dense matrix on my local machines… Thanks for all you guys! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Pytorch 1.5+ is slower than pytorch 1.3

elepherai: <p>Hi, I use <a href="https://github.com/WXinlong/SOLO" rel="noopener nofollow ugc">solov2</a> for instance segmentation and I found that the inference speed of using pytorch 1.5+ is about 3 times slower than pytorch 1.3.</p>
<p>The codes are almost the same except changing AT_CHECK to TORCH_CHECK in pytorch 1.5+.</p>
<p>The base docker images are <strong>pytorch/pytorch:1.3-cuda10.1-cudnn7-devel</strong> , <strong>pytorch/pytorch:1.9.0-cuda11.1-cudnn8-devel</strong> and 1.6.0-cuda10.1-cudnn7-devel.</p>
Safely taking Log of probabilities

legoh: <p>Hi all, I’m computing a bistochastic matrix, whereby each entry represents some action. When trying to decode my solution for a trajectory in the REINFORCE algorithm, we need log probabilities to find the loss. Hence, if each entry in my bistochastic matrix is a probability, is there a safe way to take logs for this? I’m not really sure how to do it as directly taking <code>torch.log</code> causes instabilities.</p>

omarfoq: <p>Hi,</p>
<p>I think adding a positive epsilon (e.g. 1e-10) to your matrix before applying the log will solve your problem. Check also that your matrix doesn’t have negative entries (normally the columns of your matrix will be the result of some softmax operation so this shouldn’t be a problem I guess)</p>

KFrank: <p>Hi Legoh and Othmane!</p>
<aside class="quote no-group" data-username="omarfoq" data-post="2" data-topic="124232" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/omarfoq/48/27473_2.png" class="avatar"> omarfoq:</div>
<blockquote>
<p>normally the columns of your matrix will be the result of some softmax operation</p>
</blockquote>
</aside>
<p>If your probabilities are, in fact, the result of <code>softmax()</code>, you can pretty<br>
much avoid any instabilities by using pytorch’s <code>torch.log_softmax()</code><br>
to combine the two steps together.</p>
<p>Best.</p>
<p>K. Frank</p>

raharth: <p>Just a minor slightly off-topic warning:<br>
One thing I stumbled upon using REINFORCE, is that, if you run it often enough, you will by chance pick a highly unlikely action at some point in time. Since it uses the log of this highly unlikely action the gradient explodes end might entirely mess up your agent, especially when using a momentum based optimizer. So you should definitely use a gradient clipping to avoid that (sadly it took my quite a while to figure that out…)</p>
Parameter Passing: Bool Value of Tensor with more than one value is ambiguous

eshamittal: <p>Hi all, I have an issue with bool value of tensor with more than one value in this notebook: N835242</p>
<p>I am trying to pass two parameters: indices and offsets into an embeddingbag proxy module (emb3) that is supposed to return a torch.Tensor  and the I want to pass the output of this node into the intermediate node along with the same arguments of offsets and per_sample_weights from before. The return of this should be a tuple of Tensors.</p>
<p>Then finally I want to pass the output of the intermediate node into another embedding bag proxy module</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40d41537cc2bdccdc862657fb2eff0fec3f59440.png" data-download-href="https://discuss.pytorch.org/uploads/default/40d41537cc2bdccdc862657fb2eff0fec3f59440" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40d41537cc2bdccdc862657fb2eff0fec3f59440_2_690x259.png" alt="image" data-base62-sha1="9fuWv9Jmmxb5UN01h7zNndg4yxW" width="690" height="259" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40d41537cc2bdccdc862657fb2eff0fec3f59440_2_690x259.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40d41537cc2bdccdc862657fb2eff0fec3f59440_2_1035x388.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40d41537cc2bdccdc862657fb2eff0fec3f59440_2_1380x518.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3244×1222 364 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I don’t include offsets or per_sample_weights as input to the emb3 node then I no longer get this bool value of tensor error however my model then no longer recognizes the parameter offset.</p>
<p>I’m not entirely sure why this happens and would appreciate any advice, thank you!</p>
Parallel execution of Agents of an Ensemble

raharth: <p>I’m using an ensemble of several agents in parallel in a RL setting. Currently I’m doing it in the most naive way possible, having a list of agents predicting the same input (a single state) with all of them.<br>
This is fairly slow, to an extend that you can visibly see the environment slowing down.</p>
<p>What is the most elegant way of doing that? Is anybody having experience with a similar problem? It is not about the training, this can be done efficiently, just the trajectory generation is enormously slow.</p>