akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
It seems that nvidia-smi preserves more GPU memory if the model class is inherited from another nn.Module class?

raining_day513: <p>As title. It seems that nvidia-smi preserves more GPU memory if the model class is inherited from another nn.Module class.</p>
<p>I create a class to implement my algorithm that uses <code>torch.utils.checkpoint</code>, then I found that I need to share the implementation for a new model. So I extract it into a base class which is inherited from <code>nn.Module</code>. My original model class now has been simplified a lot, but I noticed that the GPU memory usage reported by nvidia-smi is much higher than before: 14450MiB to almost full 22868MiB on Nvidia GeForce RTX 3090.</p>
<p>What could be the reason(s)? Is it possible to make the value lower?</p>

raining_day513: <p>I just realized that this is caused by the env setting:</p>
<pre><code class="lang-bash">export 'PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512'
</code></pre>
<p>But I don’t know why.</p>

eqy: <p>Measuring memory usage via <code>nvidia-smi</code> when you are using the CUDA caching allocator is going to be noisy, as you are really observing the total CUDA memory <em>reserved</em> (which means it is not necessarily in use by live tensors) and this reserved memory is also subject to fragmentation effects (which would be affected by your <code>PYTORCH_CUDA_ALLOC_CONF</code> setting).</p>
<p>If you want to accurately measure GPU memory usage I would recommend taking a look at <a href="https://pytorch.org/docs/stable/generated/torch.cuda.memory_allocated.html" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda.memory_allocated — PyTorch 2.0 documentation</a></p>
Is it possible to calculate MACs and Throughput per layer during the inference on GPU?

Apisteutos84: <p>I am trying to find a tool for pytorch which supports the estimation of the MACs and Throughput per layer on GPU. I am interesting only for the convolution layers. Is any tool except thop?</p>

eqy: <p>There are a wide variety of user-developed tools for that you’re describing <a href="https://www.google.com/search?channel=fen&amp;client=firefox-b-1-d&amp;q=pytorch+model+flops+github" class="inline-onebox" rel="noopener nofollow ugc">pytorch model flops github - Google Search</a></p>
<p>If you’re interested only in convolution layers, it would also be straightforward to e.g., create your own convolution layer that wraps the built-in convolution layer and does instrumentation for computing FLOPs/throughput in its own <code>forward</code> method.</p>
Manually moving data to GPU becomes much slower after using multiprocessing, how to solve it?

remi: <p>I have a large tabular dataset with sampling number of 300,000,000 and dimension of 400. I first copy all data into RAM which costs about 500GB, and then sequentially move data batch by batch to GPU for the model training.</p>
<p>The problem is that training one batch takes <strong>much less</strong> time than moving one batch from RAM to GPU, so it means that it will not work if we simply use a separate thread and queue to asynchronously keep saving the batch data moved to GPU and get data from the queue for the model training in the main thread (refer to <a href="https://discuss.pytorch.org/t/how-do-i-copy-data-to-gpu-in-parallel/48762" class="inline-onebox">How do I copy data to GPU in parallel?</a>)</p>
<p>So my solution is that I use multiprocessing to get the batch and move them to GPU with 10+ cores.  I simulate this solution and find it will not work. Could you please make it work?</p>
<pre><code>import ctypes
import torch.multiprocessing as mp
import numpy as np

import os
os.environ["CUDA_VISIBLE_DEVICES"]= '0'
import time

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

print(device)

train_df_np = np.random.normal(0, 1, size=(799200, 400)).astype(np.float32)
train_df_np = np.ravel(train_df_np)
train_df_np = mp.Array(ctypes.c_float, train_df_np) # **share data between different cores without copying** 

def move_data(df_np):
    x = np.array([df_np[i:i+400] for i in range(10)])
    inputs = torch.from_numpy(x).to(device)
    return inputs


if __name__ == "__main__":
    #**parallel takes 90s**
    ctx = mp.get_context('spawn')
    pool = ctx.Pool(10)
    time_start=time.time()
    data = {}
    for i in range(20):
        res = pool.apply_async(move_data, args=(train_df_np))
        data[i] = res  
    pool.close()
    pool.join()
    time_end=time.time()
    print('time cost',time_end-time_start,'s') # 

    #**sequential takes 2s**
    time_start=time.time()
    data = {}
    for i in range(20):
        res = move_data(train_df_np)  
        data[i] = res
    time_end=time.time()
    print('time cost',time_end-time_start,'s')
</code></pre>

eqy: <p>If your goal is asynchronous (nonblocking copies), have you looked into using pinned memory instead (e.g., check the details of the <code>non_blocking</code> keyword here <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.to.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.to — PyTorch 2.0 documentation</a>).</p>
<p>Note that you may need to do some modifications to use pinned memory and do your numpy conversion before the actual data movement (I would also verify that these are not the actual bottlenecks in your code).</p>
Optimizing Transformer bug

Goldname: <p>I created a Transformer with zero decoders. Based on the source code, if you have zero decoders, the output should just be the decoder tgt. This is true until I start optimizing the function</p>
<p>The second print output is false, which doesn’t make sense. Let me know if I am doing something wrong or if it’s a bug</p>
<pre><code class="lang-auto">import torch
from torch import nn
import torch.optim as optim
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
batch=2048

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.tgt = torch.zeros(1, batch, 1).to(device)
        self.trans = nn.Transformer(nhead=1, num_encoder_layers=1, d_model=1, dim_feedforward=1, num_decoder_layers = 0, dropout=0)
    def forward(self, x):
        y = self.trans(x, self.tgt)
        return y


model = MyModel()
model.to(device)
optimizer = optim.Adam(model.parameters(), lr=0.01)
running_loss = 0.0
upper = 10000000

for epoch in range(2):
    training = torch.randint(low=0, high=upper, size=(batch,)).float().to(device) / upper #Get batch of random numbers
    training = torch.unsqueeze(training, 1)
    training = torch.unsqueeze(training, 0)

    print(torch.all(model(training)==model.tgt))



    labels = training + 1
    labels.to(device) # Learn to add 1
    optimizer.zero_grad()
    outputs = model(training)
    loss = nn.MSELoss()(outputs, labels)
    loss.backward()
    optimizer.step()
    # print statistics
    running_loss += loss.item()
    print(f'[{epoch + 1}] loss: {running_loss:.3f}')
    running_loss = 0.0
print('Finished Training')
</code></pre>
Pitch shift an STFT?

potatoportato: <p>I have the result of a <code>torch.stft()</code>.<br>
I want to perform a pitch shift on the audio.<br>
The end result should be an STFT of the pitch-shifted audio.</p>
<p>I don’t want to time stretch, <code>istft()</code>, <code>resample()</code>, then <code>stft()</code> because it seems like that would be slow.</p>
<p>Instead, I wrote some code that is supposed to make a new spectrogram, where the <code>n</code>th frequency bin is just the <code>n * scaling_factor</code>th frequency bin of the original spectrogram, interpolated for fractional indices:</p>
<pre><code class="lang-auto">def interpolate(frequencies: torch.Tensor, sgram: torch.Tensor):
    start = frequencies.int()
    frac = (frequencies - start)[:, None]
    return sgram[start, :] * (1 - frac) + sgram[start + 1, :] * frac

def pitch_shift_spectrogram(sgram: torch.Tensor, semitones: torch.Tensor):
    scaling_factor = 2 ** (-semitones / 12)
    frequencies = torch.arange(0, sgram.shape[0], 1, device=sgram.device)
    shifted_frequencies = frequencies * scaling_factor
    shifted_sgram = interpolate(shifted_frequencies, sgram)
    return shifted_sgram
</code></pre>
<p>The <code>semitones</code> parameter is a tensor because the pitch is what the network is going to learn, so it needs a gradient.</p>
<p>This code sort of works?<br>
It does return a new spectrogram, and the audio from that spectrogram kind of sounds like a pitch-shifted version of the audio, but it doesn’t sound right.<br>
First of all, there is phasiness/distortion/artifacts (was I supposed to do something with the phases?).<br>
But also, the pitches in the bass don’t sound right.</p>
<p>What’s the correct code for this? Is this faster than the inverse STFT and resampling method? Will the autograd still backprop through this?</p>
'DistributedDataParallel' object has no attribute 'my_custom_method'

salidw: <p>I had run my command on Terminal as below<br>
$ python -m torch.distributed.run --nproc_per_node=2 train_ddp.py</p>
<p>and in my code, I had used “module” attribute before accessing my_custom_method like below<br>
“out = model.module.my_custom_method”<br>
but looked like that DDP still can not access my custom method. The error was shown as below</p>
<p>==&gt; ‘DistributedDataParallel’ object has no attribute ‘my_custom_method’</p>
<p>Please help to advise. Thanks.</p>

agu: <p>Could you share a code snippet that can reproduce the error?</p>

salidw: <p>Hi Andrew, thanks… I resolved my problem. Thanks…</p>

Lin-Mao: <p>Hi, could you tell me how did you resolve it? Thanks</p>
Distilling Resnet based model

Johnny_Cash: <p>I am trying to distill the model ModifisedResnet in CLIP.</p>
<p>I use (2,2,2,2) for the convolutional blocks, 64 heads, 32 width and output dimension 1024. That makes the model around 30 MB.</p>
<p>I put the teacher into eval mode. Create the embedding for the image with the teacher and run it through a softmax. Then I do the same with the student but not put it in eval mode and here I do log softmax. I take the results of both and put it into KDL as a loss function and start the training.</p>
<p>Do I need to consider anything else here? The goal is to have a smaller CLIP model that reproduces the orginal results as closely as possible.</p>
<p>Any ideas are welcome.</p>
New custom op API coming?

BeachHut: <p>I saw a mention of a <a href="https://github.com/pytorch/pytorch/issues/102720#issuecomment-1574097329" rel="noopener nofollow ugc">“new custom op API”</a>. I am currently in the process of rewriting (after already rewriting it twice before as the suggested method has changed, and this, third, time I was rewriting the C/CUDA to be independent of PyTorch so that I could provide it as shared libraries that should work without the user having to compile anything), but would like to know if I should wait.</p>

richard: <p>I would recommend not waiting. The existing custom op APIs are powerful enough that one can do a lot with them, but you should just be careful while using them to avoid potential footguns. Though, if you have a moment, I’m curious about your use case:</p>
<ul>
<li>are you writing C++ code and want to use that with PyTorch? Or are you writing Python code?</li>
<li>are you doing training? Do you want your custom operator to work with torch.compile?</li>
</ul>

BeachHut: <p>Hi Richard,</p>
<p>Thank you for your reply and for your interest in what I am working on, and apologies for the delay in replying (I missed the notification).</p>
<p>I have unfortunately never been able to get adequate performance with Python (even with torch.compile) so the computational part of the code is in C/C++ (for the CPU version) and CUDA.</p>
<p>It is used for training, but it is not a standard neural network. It implements a physical simulation using a finite difference discretization of time and space. Backpropagation is generally used to optimize the inputs (the spatial models of physical properties, source time series, etc.), but some users combine the physical simulation with a neural network (the outputs of one are the inputs of the other, for example).</p>
<p>The most recent previous implementation followed the approach necessary for my extension to be compatible with torch.jit (inheriting from the C++ class torch::autograd::Function and handling the dispatch), but I doubt that anyone used that ability, so this time I am just inheriting from the Python class autograd.Function, calling my compiled forward/backward functions, and not worrying (for now) about supporting torch.jit.script or torch.compile being applied to it. After encountering too many users who struggled to install the previous version because of difficulties with the compilation step, my main concern now is achieving the easiest installation that still provides adequate performance, hence my decision to now distribute the native code as precompiled shared/dynamic libraries that have no dependence on Python, PyTorch, etc., with the appropriate one for the user’s platform being loaded by ctypes at runtime.</p>
Optimize pkl model to trt model

kiyaabdi: <p>hi I’m new in using tensorrt. I have a .pkl model and I don’t know how to optimize it with tensorrt. I try to convert it to onnx or torch2trt but it return this error: TypeError: forward() missing 1 required positional argument: ‘x2’<br>
I tried to convert pkl to pth model or json model but this work doesn’t work too and return this error: “_pickle.UnpicklingError”<br>
can you please help me to optimize my model?</p>

ptrblck: <p>Based on the error message you should check the inputs to the <code>forward</code> method of your model and make sure all requires arguments are provided, such as the <code>x2</code> argument in this case.<br>
Besides that I don’t know what else might be causing it as only the error message was posted.</p>
<p>Also, please don’t tag specific users as it could demotivate others to post a valid answer.</p>

kiyaabdi: <p>thanks for your reply, I’ll check it again and my code is here:<br>
"<br>
device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)<br>
net = torch.load(‘./train_model/best.pkl’)<br>
net.to(device)<br>
net.eval()<br>
x = torch.ones(1, 1, 256, 256).cuda()<br>
model_trt = torch2trt(net, [x])</p>
<p>torch.save(model_trt, ‘best_trt.pkl’)<br>
from torch2trt import TRTModule</p>
<p>model_trt = TRTModule()</p>
<p>model_trt=torch.load(‘best_trt.pkl’)</p>
<p>total_score = demo(data_path, net, save_video=args.save,vis=args.vis)<br>
print(f’score: {total_score}')`"</p>
Why are there two different Compute Platform installation options for "CUDA 11.7" and "CUDA 11.8" on the PyTorch official website?

songyuc: <p>Hi, guys!<br>
I want to know why there are two different Compute Platform installation options for “CUDA 11.7” and “CUDA 11.8” on the PyTorch official website.<br>
As “CUDA 11.7” is known to be compatible with “CUDA 11.8”, what is the reason for releasing these two different versions of PyTorch?</p>
<p>Your answer and guidance will be appreciated!</p>

ptrblck: <p>Our current workflow is to release one “stable” binary package with a CUDA runtime (+ their corresponding libs) which were tested for a period of time in the nightly release and add another “experimental” binary with a newer CUDA runtime, which users can start using and should report any issues.<br>
Once a newer version is available and no critical issues were reported for the “experimental” release, we drop the old “stable”, replace it with the old “experimental”, and add the newly built binary.</p>

songyuc: <p>Thanks sincerely for your explanation!</p>

f3ba: <p>So at the moment users wanting the most stable experience should go for CUDA 11.7?</p>
How are the "nan" values outputted by scaled dot production attention dealt with?

RKnowledge: <p>I am trying to understand how masking works with the scaled_dot_product_attention, I’m using the one implemented in <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" rel="noopener nofollow ugc">torch.nn.functional.scaled_dot_product_attention</a></p>
<p>So I wanted to test how the masks work, I create three tensors to simulate the queries, keys and values. In this setting we have a batch size of 3, a sequence length of 7 and a d_model of 5. I want to stick with a very basic setting to understand the output. I create the atto_mask as it is done in the documentation, but I also want to simulate the case where the first and last sentences have two padding tokens (at the end) and the middle sentence only has one padding token (also at the end).</p>
<pre><code class="lang-auto">random_q = torch.rand(3, 7, 5)
random_k = torch.rand(3, 7, 5)
random_v = torch.rand(3, 7, 5)
attn_mask = torch.ones(3, 7, 7, dtype=torch.bool).tril(diagonal=0)
attn_mask[0, -2:] = False
attn_mask[1, -1:] = False
attn_mask[2, -2:] = False
torch.nn.functional.scaled_dot_product_attention(random_q, random_k, random_v, attn_mask)
</code></pre>
<p>We notice that when we apply the scaled dot product attention that the rows associated with the padding tokens have all nan values. Which makes sense because when the mask is applied it is supposed to mask these tokens which makes the attention scores (the logins before applying a softmax) -infinity.</p>
<p>I wanted to know how these rows are dealt with later on within the Transformer module for example. I couldn’t figure that out from the source code so I’m trying to reach out to the experts here.</p>
<p>Thank you in advance!</p>

RKnowledge: <p>My bad, I found out later that the padding mask added to the logits contain $-\infty$ values column wise with respect to the position of padding tokens. So there is no “nan” values outputted by the scaled dot product attention since there is no row that will only contain $-\infty$ values.</p>
Fast Multplication of all Tensors in a list

Kevin_Li1: <p>Hi all,</p>
<p>I have a list of tensors [A_1, A_2, … A_n] where the tensors are conformable but of different sizes. n can be large n &gt; 100. Is there a fast way to multiply A_1 x A_2 x A_3 … A_n without using a for loop or reduce?  I am aware of torch multidot, but I also need batching.</p>
<p>Thanks!</p>
Using a attention layer instead of cat to combine tensors

r00bi: <p>Hello,<br>
I have a naive question:<br>
Since, I have seen in several papers that they used attention or cross attention layer to combine the output of different encoders, my question is that how I can use an attention layer in Pytorch to combine the two or three  tensors instead of concatenating them?<br>
Let’s say I have three embedded tensors t1, t2, and t3 with the shapes of (128, 100, 512), (128, 32, 512), and (128, 16, 512), I used a cat layer to combine:<br>
torch.cat([t1,t2, t3], dim=1)<br>
How can I use a attention layer instead of cat to combine these three tensors?<br>
Thanks!</p>
Is it possible to do one sided padding with a Conv1d?

shamoons: <p>I’m attempting to give me data more temporal understanding. To do that, I want to pad only the left side with each successive layer so that I’ll maintain the same shape, but progressively “roll up” information from the older / earlier elements. Is this possible with PyTorch?</p>

ptrblck: <p>You could leave <code>padding=0</code> in the conv layers and use <code>F.pad</code> in the forward method to pad one-sided.</p>

Thomas_Ahle: <p>You can also use a larger padding, and then throw away the last couple of outputs to maintain the size.</p>
Slow data loading when training a classification model on ImageNet

Wufei_Ma: <p>I’m trying to train a <a href="https://github.com/facebookresearch/deit/blob/main/README_deit.md" rel="noopener nofollow ugc">classification model</a> on ImageNet but it seems that the training is much slower than expected and the reason is that the data loading is very slow. I’m training on a 4-gpu machine with 48 cpu threads.</p>
<p>I tried to increase the <code>num_workers</code> and <code>prefetch_factor</code> but no significant help. CPUs seem fine – only two threads with 100% usage during data loading. All the data are stored in the home directory as well.</p>
<p>I am very confused about this problem as I thought if the data loading is the bottleneck, there must be something wrong with the CPU (e.g., not enough CPU to process the data of the current batch). But now it seems that the CPU are mostly idle. What could possibly be the problem?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">This post</a> might give you some information about data loading bottlenecks.</p>

Wufei_Ma: <p>Thanks for the reference. This post definitely helps to fill in some background regarding the data processing.</p>
<p>However, it’s still a bit hard for me to identify the problem on my machine. What is the limitation? Disk throughput, IOPS, CPU, or memory?</p>
<p>I see that PyTorch has a benchmark to test performance on operations like <code>bmm</code> or neural network inference. I guess it would be great to have some benchmark suite to figure out what is blocking the data loading. It would also be useful to see how the current system compared to others. We could anticipate performance gaps from the GPU benchmarks but currently there is no ways to how my other non-GPU infrastructure is compared to other people’s systems.</p>
Multiple loss function optimization

Padmaksha_Roy: <p>Hi Team, I have a loss function where I am calculating the loss over multiple domains and summing it up to form the final loss, and for each epoch, I am doing gradient descent on it. Can you plz let me know if the implementation is correct?</p>
<blockquote>
<p>def train_AE():</p>
<pre><code>for epoch in range(epochs):
    train_loss = 0
    losses = []

    for domain in range(domains):
        dataset = Dataset(dom=domain)
        dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=False)

        for input, out, targets in dataloader:
            # data = data.to(device)

            input = input.float().to(device) 
            dom_out = out.float().to(device) # ground truth data
 
            targets = targets.long().to(device)
        
            optimizer.zero_grad()
</code></pre>
</blockquote>
<blockquote>
<pre><code>            logits, recon_batch = multitaskAE(input) ## model
</code></pre>
</blockquote>
<blockquote>
<pre><code>            loss_mse = customLoss()
            loss = loss_mse(recon_batch, dom_out, logits, targets)
            losses.append(loss)

    final_loss = torch.sum(torch.stack(losses))
    final_loss.backward()
    train_loss += final_loss.item()
    optimizer.step()

    if epochs % 50 == 0:
        print('====&gt; Epoch: {} Average loss: {:.4f}'.format(
            epochs, train_loss / len(dataloader.dataset)))
        train_losses.append(train_loss / len(dataloader.dataset))
</code></pre>
</blockquote>

Padmaksha_Roy: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , can you please have a look at the block of code above? Here, I am trying to implement a loss over multiple domains and the final loss is the sum of all those. Is it the correct way to implement it or is there any better way to do it? Thank you!</p>

ptrblck: <p>Your code looks correct, but is accumulating the computation graphs of the entire dataset, which would increase the memory usage. The <code>final_loss.backward()</code> call would then free all stored intermediate tensors and would release the memory. If you run out of memory you could call <code>backward()</code> on each loss inside the <code>DataLoader</code> loop and scale the gradients. While this would reduce the memory the computation would be more expensive since you would call <code>backward</code> multiple times.</p>

Padmaksha_Roy: <p>Thank you so much for clarifying this! I found that torch.mean brings the loss values to a different range but the model performance is not degrading. So torch.mean can be used as well, right?</p>

ptrblck: <p>Yes, taking the <code>mean</code> of the final loss would also work as it would just scale the loss and thus gradients.</p>
Spearman correlation very low for one output channel and high for the other

szahan: <p>Hi</p>
<p>I am trying to use ViT as a regressor for my time series data.<br>
My input data size is N * 189 * 150 * 3 [N-batch size, 189- different signal channels, 150 - frame size and 3 is xyz coordinates]. I created an image representation of size N * 224 * 224 * 3 by repeating channels and frames horizontally and vertically respectively.</p>
<p>I put an FC on top of ViT to output N * 900 feature which I reshaped to match the output timeseries shape N * 2 * 150 * 3.</p>
<p>I am just finetuning the last encoder layer (11) and the fc layer. I normalized the input as per ViT requirement and did not normalize the actual output matrix (that I am trying to regress).</p>
<p>Trained for 100 epochs and decreasing learning rate every 10th epoch. Base lr 0.01 adam optimizer. Loss: L2+0.001 * L1</p>
<p>Evaluation of the second channel of the output matrix shows a good enough Spearman correlation (0.67) but for channel one is very poor (0.20). The two output channels represent different sensor signals.</p>
<p>Could you please explain or direct me towards any intuitions, mistakes or training considerations that I am missing? Thanks a lot.</p>
<p>Update: I tested with normalized (sklearn standardscaler) output vector as well. Now the regressions for both channels are worse 0.20 and 0.10.</p>
Rubik’s cube problem

Bluemanrocks: <p>Hi there,</p>
<p>I’m helping a friend of mine who is a graduate mathematics professor design a problem and prospective research paper revolving around machine learning and the Rubik’s cube and am looking for hope. There’s even a stipend for anyone who can help us! We are looking to change pre-existing code for “single state solutions” to “multiple state solutions” (I can forward the rest of the information to anyone interested).</p>
<p>Thanks for reading! Excited to be using the PyTorch forums <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
How apply transform.compose to data augmentation?

Mahdi_Amrollahi: <p>When we use the term of Data Augmentation, I believe that the images like <code>x</code> and <code>rotated x</code> are two different images that will be fed to model. Indeed, when we have a dataset of <code>n</code> images, we will have <code>d*n</code> images with Data Augmentation which <code>d</code> is the number of transformations and their combinations . However, when we use <code>Transform.Compose</code>, it transform any image to one image.<br>
How we can apply range of transformation like resize, rotate, crop … to list of images to get <code>d*n</code> which d in the number of combination of transformation?</p>

ptrblck: <aside class="quote no-group" data-username="Mahdi_Amrollahi" data-post="1" data-topic="184734">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdi_amrollahi/48/62481_2.png" class="avatar"> Mahdi_Amrollahi:</div>
<blockquote>
<p>to get <code>d*n</code> which d in the number of combination of transformation?</p>
</blockquote>
</aside>
<p>I’m not sure if you are looking for a way to create <em>all</em> possible transformations defined in <code>Compose</code> (this would be quite impossible) or e.g. 5 outputs for 5 chained transformations.<br>
In both cases you can just call <code>out = transform(x)</code> multiple times since the transformations will be applied on the fly.</p>

Mahdi_Amrollahi: <p>I thought that we should do this however, I understand that the number of data will increase exponentially !<br>
Thanks</p>
How is the sampling iterator passed to the workers?

Sam_Lerman: <p>Are the indices generated in the main process and sent to the workers one by one or does each worker have its own Sampler that it iterates through independently?</p>

ptrblck: <p>The <code>DataLoader</code> should use one <code>sampler</code> which then sends the indices to each worker. If each worker would have a custom sampler they could load duplicated samples or the sampler would need to have multiprocessing capabilities, which is not the case.</p>

Sam_Lerman: <p>Are you sure? Because this:</p>
<pre><code class="lang-auto">class Sampler:
    def __init__(self, data_source):
        self.data_source = data_source

    def __iter__(self):
        yield from torch.randperm(len(self))

    def __len__(self):
        return len(self.data_source)
</code></pre>
<p>doesn’t return the same expected behavior as the default <code>RandomSampler</code>.</p>
<p>As in, inexplicably, it makes my MacBook crash after about 20k learning steps and the other doesn’t. I’ve ablated them 3 times now.</p>
<p>Well, I’m comparing to no sampler + <code>shuffle=True</code>. Is that the same as <code>RandomSampler</code>?</p>

ptrblck: <p>Yes, not providing a <code>sampler</code> and using <code>shuffle=True</code> will create a <code>RandomSampler</code> as seen <a href="https://github.com/pytorch/pytorch/blob/7b211ff8dd93415cb355871f92fc2d3cf4e11bb8/torch/utils/data/dataloader.py#L349">here</a>.<br>
I don’t understand your doubt as you are just asking if I’m sure without explaining what might indicate I’m wrong.</p>