akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Loading Tensorflow checkpoints with Pytorch

username: Krag
cooked: <p>I have some pre-existing code that uses Pytorch to interact with the generator from a trained GAN.</p>
<p>The problem is I have a new set of checkpoints I want to load but there are from a Tensorflow implementation of the model.  Rather than going through all the code and changing it to work with TF, I’d rather just load the model as if it were Torch.</p>
<p>The examples I looked at online were different from what I have so I’m not sure how to go about doing this.  Here’s what the folder with the TF checkpoints looks like:<br>
                    <a href="https://cdn.discordapp.com/attachments/770714118646661140/975507195988942878/Screen_Shot_2022-05-15_at_2.17.05_PM.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://cdn.discordapp.com/attachments/770714118646661140/975507195988942878/Screen_Shot_2022-05-15_at_2.17.05_PM.png" width="433" height="500">
          </a>

</p>
<p>Here’s how I’ve been loading other models with Torch:</p>
<pre><code class="lang-auto"># model name
gan_model_name = "gan_{}.tar".format(model_prefix)

# model checkpoints
checkpoint = torch.load(gan_model_name, map_location="cpu")

# model parameters are imported from params.py (don't change these)
wavegan_generator = WaveGANGenerator(
            slice_len=window_length,
            model_size=model_capacity_size,
            use_batch_norm=use_batchnorm,
            num_channels=num_channels,
        ).to(device)

# load saved checkpoints into initialized model
wavegan_generator.load_state_dict(checkpoint["generator"])
wavegan_generator.eval()
</code></pre>
<p>I know how to initiate the model with the correct parameters but loading the checkpoints into the state_dict is where I’m stuck.</p>
<p>I’ve seen examples of ONNX but they look different from what I have.  Is there a way to load these TF model checkpoints using the same Torch code?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>You could transform the TF checkpoint to a PyTorch <code>state_dict</code> manually as <a class="mention" href="/u/tom">@tom</a> and I did for StyleGAN in <a href="https://github.com/lernapparat/lernapparat/blob/master/style_gan/pytorch_style_gan.ipynb">this notebook</a>.</p>
Google Colab ToTensor2 albumentations

username: Nurkhan_Laiyk
cooked: <p>Hello, I tried to import ToTensor2 as a “from albumentations import ToTensor2” in colab, but it shows an error "cannot import name ‘ToTensor2’ from ‘albumentations’ " Could someone help?</p>

username: ptrblck
cooked: <p>I guess you have a typo and want to use <code>ToTensorV2</code>?</p>

username: Nurkhan_Laiyk
cooked: <p>Yes, but still i cannot  fix it.</p>

username: E4CE
cooked: <p>Hello, is there any solution for this problem?<br>
I have tried</p>
<pre><code class="lang-auto">!pip install -q -U albumentations

import albumentations 
from albumentations.pytorch import ToTensorV2
</code></pre>
<p>and</p>
<pre><code class="lang-auto">!pip install albumentations==1.1.0
import albumentations 
from albumentations.pytorch import ToTensorV2
</code></pre>
<p>both of them do not work. Is there any other way to use albumenations ?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Could you describe your issue with more details please, as:</p>
<pre><code class="lang-python">!pip install albumentations
import albumentations
from albumentations.pytorch import ToTensorV2
albumentations.__version__
&gt; '1.1.0'
</code></pre>
<p>works in my setup.</p>
`torch.gather` doesn't work as expected

username: ecolss
cooked: <p>From the documentation of <a href="https://pytorch.org/docs/stable/generated/torch.gather.html" rel="noopener nofollow ugc"><code>torch.gather</code></a>, it can gather values along an axis specified by dim, which is exactly what I needed.</p>
<p>However, I just tried some very simple examples, <code>backward()</code> cannot work if <code>sparse_grad=True</code>, why?</p>
<pre><code class="lang-python">In [1]: import torch as th

In [2]: x = th.rand((3,3), requires_grad=True)

# sparse_grad = False, the backward could work as expetecd
In [3]: th.gather(x @ x, 1, th.LongTensor([[0], [1]]), sparse_grad=False).sum().backward()

# sparse_grad = True, backward CANNOT work
In [4]: th.gather(x @ x, 1, th.LongTensor([[0], [1]]), sparse_grad=True).sum().backward()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
----&gt; 1 th.gather(x @ x, 1, th.LongTensor([[0], [1]]), sparse_grad=True).sum().backward()

~/miniconda3/lib/python3.9/site-packages/torch/_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    305                 create_graph=create_graph,
    306                 inputs=inputs)
--&gt; 307         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    308
    309     def register_hook(self, hook):

~/miniconda3/lib/python3.9/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    152         retain_graph = create_graph
    153
--&gt; 154     Variable._execution_engine.run_backward(
    155         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    156         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag

RuntimeError: sparse tensors do not have strides
</code></pre>
Dynamic Graphs - Retain Graph

username: Harman_Singh
cooked: <p>I need some advice.<br>
So I am currently using retain graph = True in my code, simply because I need all the gradients of previous tensors in my code.<br>
The reason for this is, that my variables come from a recursion, where one variable depends on another which in turn depends on another, all the way till the first variable.<br>
(I hope I was able to explain this).</p>
<p>I want to get rid of the retain graph = True as it takes a lot of time in computation.<br>
Is there a way I could combine all gradients and pass it on to the next iteration or something like this which will enable me to get rid of retain Graph = True<br>
Thanks</p>

username: ptrblck
cooked: <p>I’m not sure I understand the use case completely, so please correct me if I’m wrong.</p>
<p><code>retain_graph</code> is used to keep the computation graph in case you would like to call <code>backward</code> using this graph again.<br>
A typical use case would be multiple losses, where the second <code>backward</code> call still needs the intermediate tensors to compute the gradients.</p>
<aside class="quote no-group" data-username="Harman_Singh" data-post="1" data-topic="78468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/harman_singh/48/15452_2.png" class="avatar"> Harman_Singh:</div>
<blockquote>
<p>simply because I need all the gradients of previous tensors in my code.</p>
</blockquote>
</aside>
<p>Each backward call computes the gradients for the parameters. If you don’t zero out the gradients, each following <code>backward</code> will accumulate the gradients in all parameters.<br>
So I’m not sure, if you really need to retain the graph or if you just want to accumulate the grads.</p>

username: LEO_XiLiu
cooked: <p>I have the same problem. How did you fix it? I’m trying to backprop gradients through NLMS filter which parameters are recursively updated.</p>

username: LEO_XiLiu
cooked: <p>Thanks for your reply. I believe we have the same issue, the code can be written as the follow:</p>
<pre><code class="lang-auto">for batch in self.trainset:
      for i in range(seq):
          out1, out2 = net(input1[i], input2[i])
          out3, W = adapt_filter(out1, out2, W)
          loss = mse_loss(out3, traget)
          optimizer.zero_grad()
          loss.backward(retain_graph=True)
          optimizer.step()
</code></pre>
<p>if seq is large like 1000, huge amount of computaion graphs will be saved which can cause CUDA out of memory and slow down the training process. Do you have any suggestions to avoid it? Hope I make it clear. Thanks again.</p>

username: ptrblck
cooked: <p>The increasing memory usage is expected, since you are not deleting the computation graph via <code>retain_graph=True</code>. Could you explain your use case a bit more and why you are setting this argument?</p>
Is Pytorch have any asynchronous inference API?

username: Forceless
cooked: <p>Wonder if Pytorch could cooperate with other coroutine and functions like</p>
<pre><code class="lang-auto">async def infer():
  await return model(data)
async def wait():
  await asyncio.sleep(1)
asyncio.run([infer(),wait()])
</code></pre>

username: tom
cooked: <p>It does not out of the box.<br>
However, JITed models release the GIL for their processing, so launching the model in a background thread and waiting for completion is quite efficient.</p>
<p>We do this in Chapter 15 of <a href="https://www.manning.com/books/deep-learning-with-pytorch" rel="noopener nofollow ugc">our book</a>. While the free download on the <a href="http://PyTorch.org" rel="noopener nofollow ugc">PyTorch.org</a> website has ended, the example code is available on github: <a href="https://github.com/deep-learning-with-pytorch/dlwpt-code/blob/master/p3ch15/request_batching_jit_server.py" class="inline-onebox" rel="noopener nofollow ugc">dlwpt-code/request_batching_jit_server.py at master · deep-learning-with-pytorch/dlwpt-code · GitHub</a></p>
<p>Best regards</p>
<p>Thomas</p>

username: Forceless
cooked: <p>Sorry about bothering you again and replying too late,<br>
I read and tested many cases and read many documents recently and I have a guess.<br>
Is pytorch’s async inference worked like asyncio.sleep()?<br>
Because I can’t feel it’s async in the main thread.<br>
It needs you manually put it in an event loop.<br>
Otherwise, it will not yield.<br>
So I designed two situations and correspond with guess</p>
<pre><code class="lang-python">def cpu_task():
    2*5
def cpu(result):
    a = result.mean()
# 1
for i in range(100):
  cpu_task()
  result = model(data[i])
  cpu(result)

# 2
for i in range(100):
  cpu_task()
result = []
for i in range(100):
  result.append(model(data[i]))
for i in range(100):
  cpu(result[i])
</code></pre>
<p>If my suppose is true, situation 1 worked like blocked<br>
situation2 worked like async.<br>
When I test this code, situation2 is 10x faster than <span class="hashtag">#1</span>.<br>
Is async the reason why?<br>
Best wish</p>

username: Forceless
cooked: <p>And I have an another suppose:</p>
<pre><code class="lang-python">queue = queue()
thread1: queue.push(model(data))
thread2: 
while(!queue.empty()):
  task(queue.pop())
</code></pre>
<p>will this code work appropriately?</p>

username: tom
cooked: <p>I don’t know what queue is for you, but the async primitives can be touchy w.r.t. multithreading.</p>

username: Forceless
cooked: <p>So is my first <a href="https://discuss.pytorch.org/t/is-pytorch-have-any-asynchronous-inference-api/151019/2#:~:text=Sorry%20about%20bothering%20you%20again%20and%20replying%20too%20late%2C">suppose</a> correct?</p>

username: tom
cooked: <p>I’m not entirely sure what you’re trying to achieve / what the semantics of that code should be.</p>
About Minmax Normalization and its Gradient

username: oytun_demirbilek
cooked: <p>Hello there,</p>
<p>I need and trying to normalize my input using minmax normalization. I know this easily can be done with:</p>
<pre><code class="lang-auto">X_norm = (X - X.min() ) / ( X.max() - X.min())
</code></pre>
<p>But I also want to add learnable parameters to it like (assume register_parameter etc. already handled in <strong>init</strong>):</p>
<pre><code class="lang-auto">X_norm = ( X - X.min() ) / ( X.max() - X.min() ) * self.weight + self.bias
</code></pre>
<p>I am asking two things:</p>
<p>Is there already a built-in wrapper in pytorch that does this?<br>
How much does it make sense to do it in terms of gradients? The differentiation would work well?</p>

username: Matias_Vasquez
cooked: <p>Using your first equation, we get values between <code>[0, 1]</code>.</p>
<aside class="quote no-group" data-username="oytun_demirbilek" data-post="1" data-topic="151722">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oytun_demirbilek/48/30241_2.png" class="avatar"> oytun_demirbilek:</div>
<blockquote>
<p><code>X_norm = (X - X.min() ) / ( X.max() - X.min())</code></p>
</blockquote>
</aside>
<p>However, with the learnable parameters <code>self.weight</code> and <code>self.bias</code> this will not always be true. The values can be shifted and scaled.</p>
<p>Assuming you do <strong>not</strong> care about this. The you can use the <code>BatchNorm2d</code> (or <code>1d</code> or <code>3d</code> depending on your case).</p>
<p>As you can see, the equation is given as.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be286796763210b397bbc1955fce7c7b29ae76a1.png" alt="image" data-base62-sha1="r8dhJ6hjmE0WwPg5FMNlqUkJvON" width="228" height="67"></p>
<p>Where gamma and beta are your learnable parameters. The only difference to your original equation is that the <code>mean</code> is centered towards <code>0</code> and the division by the <code>standard deviation</code> will give you values between <code>[-1, 1]</code>. Instead of just resizing the values between <code>[0, 1]</code> without looking at the <code>mean</code> or <code>std</code>.</p>
<p>You can have values greater than <code>1</code> and lower than <code>-1</code>, but these will be mostly outliers.</p>
<p>In your implementation it will be prone to noise. This means that if you have one single outlier that is much greater than every other value, then the main information will be crammed together, in order to fit this value.</p>
<p>I think this would be a more robust solution.<br>
Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
Prediction of Specific Section of Target Data in LSTM

username: Passisimple
cooked: <p>Hello,<br>
I am trying to obtain a wave model which requires simultaneous solutions of multiple ODE’s.<br>
My input is has 2 input features. First one is Stimuli which only initiates the wave, and mostly 0’s.</p>
<p>However, my model can’t produce a specific part of the wave.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50601292f26f19dd21909992df74f707087b65d0.png" alt="image" data-base62-sha1="bt21ZwHEEo0J94Ne75Tjcw2T8ME" width="333" height="223"></p>
<p>As you can see, most of the wave is predicted very accurately.<br>
What I tried is,</p>
<ul>
<li>Changing the hidden dimension</li>
<li>Changins the sequence length</li>
<li>Changing the batch size</li>
<li>Adding additional FC layers</li>
<li>Increasing and decreasing LSTM Layer numbers</li>
<li>Using MSE and L1 Loss functions</li>
<li>Increasing data sizes</li>
</ul>
<p>When I try to make my model more complex (such as increasing number of layers &amp; size of hidden dimension) It tends to overfit,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9cdf107d7b334d20d1280fa84186e66d0986cf6.png" alt="image" data-base62-sha1="zDS7iVh8fKHG5iQXGhQYX57mCai" width="349" height="215"></p>
<p>Exact opposite effect occurs when model is simplified or regularization is added.</p>
<p>How can it learn the area after initial sharp increase ? Is there a known approach for this ? I am at a loss at this point.</p>
<p>Thank you kindly.</p>

username: Andre_Amaral_IST
cooked: <p>Hey,<br>
I am also applying LSTMs for circuit behavior modelation.<br>
Did you pay attention to exploding and vanishing gradients?<br>
Do you shuffle the data in the DataLoader?<br>
Do you reduce the learning rate?</p>

username: Passisimple
cooked: <p>Hello Andre,<br>
It sounds your problem is a regression problem as well ? This here is an electrophysiological wave model, we might be working on similar models.<br>
I am using adaptive learning rate starting around e-5 with gamma = 0.1<br>
I do use the shuffle = True for my training data loader. I guess that is what you meant?<br>
I didn’t pay much attention to gradients since, my sequence lengths were relatively short and I was not sure on how to modify LSTM in that regard.</p>
<p>When I make the model even slightly more complex it starts making way shorter waves like in the first post which I assume due to overfitting.<br>
I also have only 2 inpus namely stimuli and the electrical potential (which I am predicting).</p>

username: Andre_Amaral_IST
cooked: <p>It is an amplifier wave</p>
Finding benchmark dataset - node prediction + binary classes + high number of graphs

username: Cedric_Baron
cooked: <p>Hello everybody,</p>
<p>For my research I would like to test my models and my training pipeline with a benchmark dataset with similar characteristics as my own dataset to be sure that the model architecture / training pipeline has no error bug. I looked into various torch_geometric.datasets but it is difficult to match what i need.</p>
<p>Does anybody knows about a benchmark dataset from torch geometric that matches these criterias?</p>
<p>node prediction, binary classes, high number of graphs</p>

username: Cedric_Baron
cooked: <p>I found the <code>PATTERN</code> datasets from the <code>GNNBenchmarkDataset</code> from pytorch geometric that matches my criterias</p>
ROI Polling for unequal dimensions

username: sidnetopia
cooked: <p>Hi,</p>
<p>How do you specify the scale in <a href="https://pytorch.org/vision/stable/generated/torchvision.ops.roi_pool.html" rel="noopener nofollow ugc">roi_pooling</a> with unequal dimensions?</p>
<p>E.g. image size - (640, 480)</p>

username: tom
cooked: <p>I think the thing that is not foreseen is to have inhomogenous scaling. Having a non-square image size while keeping the aspect ratio should work…</p>
<p>Best regards</p>
<p>Thomas</p>

username: sidnetopia
cooked: <p>Thank you <a class="mention" href="/u/tom">@tom</a>. I just realized that 640 and 480 have the same spatial scales on VGG16.</p>
Calculate Model Accuracy for one of the Kaggle Notebook

username: spotdar
cooked: <p>How to calculate accuracy for PyTorch model used in  <a href="https://www.kaggle.com/code/andradaolteanu/whales-dolphins-effnet-train-rapids-clusters" class="inline-onebox" rel="noopener nofollow ugc">🐳Whales&amp;Dolphins: EffNet Train &amp; RAPIDS Clusters | Kaggle</a></p>
<p>I tried using some methods like -</p>
<ol>
<li>
<p>Tried this → <br>
correct += (out == targets).float().sum()<br>
accuracy = 100 * correct / len(test)<br>
This is giving weird errors and</p>
</li>
<li>
<p>Tried this → <br>
num_corrects = num_corrects  + torch.sum(out.argmax(1) == targets)<br>
print(“Accuracy = {}”.format(train_accuracy))<br>
This is giving 0 as accuracy</p>
</li>
</ol>
<p>Please could anyone help me on it</p>
Save only part of model

username: jhp
cooked: <p>Hi I have a question about saving the model.<br>
I currently doing some project regard to knowledge distillation, so I load the pretrained teacher model in the student model.<br>
When I finish the training, only thing that I want to save is the parameters of student model, but it seems that teacher model also included in the model save file.</p>
<p>How can I remove those teacher model automatically? Seems that register_buffer do this job but cant find the details of how to use it for complex model</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>It depends on your currently used workflow and I assume you are storing the <code>state_dict</code> of a model containing both submodels. If so you might want to store the <code>state_dict</code> of the student model only by accessing it. If this description doesn’t fit your use case could you describe it a bit more?</p>
Running out of memory with 8 gpus?

username: NearsightedCV
cooked: <p>Hi</p>
<p>I am training a model using the UNET++ architecture… I have large images that I crop down to make this more manageable but I feel I should be able to handle more data using the dataparallel setup.</p>
<p>Per the title I am running on eight 8GB gpus on a cluster. I am trying to run a model with an image size of 1024x1024 and batch size of 3. I get an out of memory error. I estimate with these settings I would need 3gb per batch not including all the layers in the model. Still I have ~50-60gb of memory to play with…</p>
<p>What is going on ?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="NearsightedCV" data-post="1" data-topic="148823">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/ac8455/48.png" class="avatar"> NearsightedCV:</div>
<blockquote>
<p>Still I have ~50-60gb of memory to play with…</p>
</blockquote>
</aside>
<p>I don’t quite understand this statement as it seems you claim to be using these 8 GPUs as “one”?<br>
Data parallel will execute the same model on each device so the memory limitations would still be the same, but you could increase the “global” batch size.</p>

username: NearsightedCV
cooked: <p>Thanks for that clarification</p>
Train a network to get the noise

username: Andre_Amaral_IST
cooked: <p>Hey,</p>
<p>I am wondering if it is possible to, given the input waves, train a network to learn the noise in the data.</p>
<p>Regards,<br>
André</p>

username: Sadegh_ab
cooked: <p>You can test deep autoencoders.</p>
Problem with Computing Loss in DDP Setup

username: Imahn
cooked: <p>I have a problem with computing the loss in the DDP setup. This is how I do it:</p>
<pre><code class="lang-python">loss = ... # per batch
train_loss_batch.append(loss) # per batch
if (batch_idx % 60) == 0: 
    print(f'Training Loss: {loss}')        
train_losses_epoch.append(np.sum(train_loss_batch)/len(train_loader.dataset)) # per epoch
</code></pre>
<p>So basically, per batch, I calculate the loss, append it to a list, and per epoch, I sum over the losses in the list and take the mean over the number of samples.</p>
<p>But here are two problems I’m facing:</p>
<ul>
<li>Problem 1:</li>
</ul>
<pre><code class="lang-python">Train Epoch: 0 [61440 / 760000 (08.08 %)]	Training Loss: 3016.5581	Elapsed Time: 162.10 s
Train Epoch: 0 [122880 / 760000 (16.17 %)]	Training Loss: 2030.2073	Elapsed Time: 309.48 s
Train Epoch: 0 [184320 / 760000 (24.25 %)]	Training Loss: 1733.3897	Elapsed Time: 458.85 s
Epoch 00: 499.45 sec ...
Averaged training loss: 2334.3673, validation loss: 430.6891
</code></pre>
<p>Basically, the problem is that there is a huge discrepancy between the validation loss using DDP and not using DDP. So when not using DDP, these are the values I had gotten:</p>
<pre><code class="lang-python">Averaged training loss: 3419.7038, validation loss: 1355.6035
</code></pre>
<ul>
<li>Problem 2:</li>
</ul>
<pre><code class="lang-python">Train Epoch: 1 [61440 / 760000 (08.08 %)]	Training Loss: 1553.6359	Elapsed Time: 158.11 s
Train Epoch: 1 [122880 / 760000 (16.17 %)]	Training Loss: 1494.2148	Elapsed Time: 308.32 s
Train Epoch: 1 [184320 / 760000 (24.25 %)]	Training Loss: 1396.2194	Elapsed Time: 454.41 s
Epoch 01: 502.54 sec ...
Averaged training loss: 383.5921, validation loss: 354.7226
</code></pre>
<p>Here, there is around a factor <code>4</code> difference between the training losses in the batch updates and in the final printout after the epoch training is finished. This makes sense, as every DDP process gets only <code>1/4</code> of the data, but how can I add the loss terms of all processes to get the correct loss?</p>
<p>I’d really appreciate some help <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>

username: Yanli_Zhao
cooked: <ol>
<li>
<p>looks like the discrepancy is expected, because params are synced and updated correctly using DDP?</p>
</li>
<li>
<p>average the training loss among processes for each iteration?</p>
</li>
</ol>

username: Imahn
cooked: <p>Thank you for your answers.</p>
<p>1.) But why is there a discrepancy only for the validation loss, not for the training loss?<br>
2.) Sounds good! Any idea how to achieve this?</p>
How to traverse the tensor elements in the list and concatenate them all

username: Gorgen
cooked: <p>hello everyone:<br>
If the list elements are all composed of tensors of the same dimension,<br>
such as list = [a,b,c],a=b=c = torch.randn(6, 3, 10), how can I expand every element in the list  adding each element in the last dimension, for example a convert to  (6,1,3,10), a = b =c = torch.randn(6,1,3,10). finally， I can acquire (6,3,3,10)</p>

username: Gorgen
cooked: <p>sorry。 the torch.stack can be down</p>
Element wise addition with temporary tensors

username: mopi47
cooked: <p>I am trying to implement the following part of a model<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/2/32edf67648e6f0ef82333fe00c3f6859b4dc4fe3.png" alt="image" data-base62-sha1="7gxHNcbVgv8Xh6S9Cth6IxKQNgv" width="459" height="291"></p>
<p>this is the code I wrote</p>
<pre><code class="lang-auto">            x1, x2, x3, x4 = level(x1, x2, x3, x4)

            tempx2 = torch.add(x2, x1)

            tempx3 = torch.add(x3, x2)

            tempx4 = torch.add(x4, x3)

            x2 = tempx2

            x3 = tempx3

            x4 = tempx4
</code></pre>
<p>do you find it right,<br>
I am a beginner to pytorch, but I think the “temp” tensors I create could be a source to memory leakage because this calculation is inside a loop</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a72c396a8bb03f783ceebe1266643bdb81bed77.png" data-download-href="https://discuss.pytorch.org/uploads/default/2a72c396a8bb03f783ceebe1266643bdb81bed77" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a72c396a8bb03f783ceebe1266643bdb81bed77_2_239x499.png" alt="image" data-base62-sha1="63vVWlibxfT3DkjRliEjRV92XyL" width="239" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a72c396a8bb03f783ceebe1266643bdb81bed77_2_239x499.png, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a72c396a8bb03f783ceebe1266643bdb81bed77.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a72c396a8bb03f783ceebe1266643bdb81bed77.png 2x" data-dominant-color="CBCFD6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">333×696 34.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="mopi47" data-post="1" data-topic="151648">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ed655f/48.png" class="avatar"> mopi47:</div>
<blockquote>
<p>I think the “temp” tensors I create could be a source to memory leakage</p>
</blockquote>
</aside>
<p>It can’t be the case, as python implements a <a href="https://towardsdatascience.com/memory-management-and-garbage-collection-in-python-c1cb51d1612c#:~:text=Garbage%20collection%20is%20implemented%20in,cleans%20up%20the%20object%20immediately">reference count based garbage-collection</a>. i.e., When the reference count of an object reaches 0, reference counting garbage collection algorithm cleans up the object.</p>
<p>In python, for simplicity, you can do as below:</p>
<pre><code class="lang-auto">x2, x3, x4 = x1+x2, x2+x3, x3+x4
</code></pre>
Is there any way to do use num_workers > 0 and avoid using pickle?

username: ThibaultGROUEIX
cooked: <p>Hi,</p>
<p>I am computing a scipy SuperLU object in my preprocessing. However this object cannot be pickled nor dilled. Hence, when using num_workers &gt; 0, I get as expected <code>AttributeError: Can't pickle local object</code></p>
<p>Is there any way to still paralellize my preprocessing ?</p>
<p>Thanks<br>
Thibault</p>
Efficiently get first N numbers that satisfy a condition in each row in a pytorch tensor

username: tczj
cooked: <p>Given a tensor <code>b</code>, and I would like to extract <code>N</code> elements in each row that satisfy a specific condition. For example, suppose <code>a</code> is a matrix that indicates whether an element in <code>b</code> satisfy the condition or not. Now, I would like to extract <code>N</code> elements in each row whose corresponding value in <code>a</code> is <code>1</code>.</p>
<p>And there can be two scenarios. (1) I just extract the first <code>N</code> elements in each row in order. (2) among all the elements that satisfy the condition, I randomly sample <code>N</code> elements in each row.</p>
<p>Is there an efficient way to achieve these two cases in pytorch? Using a for loop will be very slow when <code>b</code> has many rows. Thanks!</p>
<p>Below I give an example that shows the first case.</p>
<pre><code class="lang-auto">import torch
# given
a = torch.tensor([[1, 0, 0, 1, 1, 1], [0, 1, 0, 1, 1, 1], [1,1,1,1,1,0]])
b = torch.arange(18).view(3,6)

# suppose N=3
# output:
c = torch.tensor([[0, 3,4],[7,9,10], [12,13,14]])
</code></pre>

username: Matias_Vasquez
cooked: <p>This can be a way to do what you want.</p>
<p>However, you need to take into account that you will <strong>NOT</strong> notice if a row from  <code>a</code> has less than <code>N</code>.</p>
<p>In this example I have put a fourth row with only two <code>1</code>'s, but you still get <code>3</code> values.</p>
<pre><code class="lang-python">a = torch.tensor([[1, 0, 0, 1, 1, 1], [0, 1, 0, 1, 1, 1], [1,1,1,1,1,0], [1,0,0,1,0,0]])
b = torch.arange(24).view(4,6)

N = 3

mode = 1

if mode == 1:
    # Choose in order from left to right
    idx = a * torch.arange(start=a.shape[1]+1, end=1, step=-1)
if mode == 2:
    # Choose randomly
    idx = a * torch.randint(low=1, high=a.shape[1]+1, size=(a.size()))

_, ind = torch.topk(input=idx, k=N, dim=1)
c = b[torch.arange(b.shape[0]), ind.T].T

print(c)
</code></pre>
<pre><code class="lang-python"># Output:
tensor([[ 0,  3,  4],
        [ 7,  9, 10],
        [12, 13, 14],
        [18, 21, 22]]) # ← Here are actually only two True values in the mask.
</code></pre>
<p>If you are sure that you will always have at least <code>N</code> values, then it should be no problem.<br>
Else, you can <code>a.sum(dim=1)</code> to see how many values you can trust from this new matrix.</p>

username: tczj
cooked: <p>Thanks! this works well!</p>
Randomize ImageFolder

username: cerebrou
cooked: <p>Torchvision dataset ImageFolder reads images sequentially from the directories. Is it possible to shuffle them before loading them into the DataLoader. That is using a different way to shuffle dataset, not the DataLoader shuffle option.</p>

username: InnovArul
cooked: <p>You can use a custom <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler">Sampler</a> to specify the sampling order to the data loader. Specifically, write the shuffling order in <code>__iter__()</code> method of the Sampler.</p>
<pre><code class="lang-auto">import numpy as np
from torch.utils.data import Dataset, DataLoader
import torch
import torch.utils.data as data

class IndexSampler(data.Sampler):
    r"""Custom ampler

    Arguments:
        dataset (Dataset): dataset to sample from
    """

    def __init__(self, dataset):
        self.dataset = dataset

    def __iter__(self):
        # write sampling logic here
        return iter(range(len(self.dataset)))

    def __len__(self):
        return len(self.dataset)

if __name__ == '__main__':
    inps = torch.arange(20)
    dataset = data.TensorDataset(inps)
    customsampler = IndexSampler(dataset)
    dataloader = DataLoader(dataset, batch_size=2, sampler=customsampler)
    print([d for d in dataloader])
</code></pre>