akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Parzen kernel density estimation for Multivariate time series data

username: ShanakaRG
cooked: <p>I have a data set of shape (N,C,T,V) N samples, C channels, T frames, , V points. I want to estimate the point’s densities using parzen window technique to get the spatial distribution of each frame of each sequence. My expected output size is N,T,V.</p>
<p>I have developed a code where N=1. But when i have multiple samples like batch, the code will too slow since i have use a lot of for loops. So is there any method to improve and optimize this code.</p>
<pre><code class="lang-auto">import os
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import torch
from scipy.stats import multivariate_normal
def density_estimation(self):

        
        print ('data = ',data.shape)
        N,T,V,C = data.shape
        
        frames= self._normal_skeleton(data)
        print(frames[0].shape)
        frames = list(frames[0]) #select only one sequence 
        data = torch.tensor(data)
        # Define the window width (standard deviation of the Gaussian)
        window_width = 0.01

        joint_densities = torch.zeros((len(frames), frames[0].shape[0]))
        for frame_idx, points in enumerate(frames):
            for joint_idx in range(points.shape[0]):
                joint_densities[frame_idx, joint_idx] = self.parzen_window(points[joint_idx], points, window_width)
        print('joint densities 1 ', joint_densities)
        print('joint densities 1 ', joint_densities.shape)
        # Apply Parzen window to each joint
        
        
            
    # Define the Parzen window function (Gaussian kernel)
    def parzen_window(self,x, data, window_width):
        N, d = data.shape
        # print(x.shape , data.shape)
        # lk
        density = 0
        for i in range(N):
            density += multivariate_normal.pdf(x, mean=data[i], cov=window_width)
        return density / N       
</code></pre>
<p>to get process the while batch, i modified it as follows,</p>
<pre><code class="lang-auto">for i in range(N):
            for j in range(T):
                for k in range(V):
                    joint_data = data[i, j, k]
                    frame = data[i,j]
                    # density[i, j, k] = parzen_window2(joint_data, frame, window_width)
                    density[i, j, k] =self.parzen_window(joint_data, frame, window_width)
                # Calculate the average density over time for each joint
        print('joint densities', density.shape)
</code></pre>
<p>but the problem is when the no of points and no of samples get increased, the code is too slow. So how do i optimize this code ?</p>
Trying to backward through the graph a second time with partially modified intermediate tensor

username: Hengguang_Zhou
cooked: <p>Hi! I am writing some PyTorch code. I have a Transformer model that predicts the next word. And I also have an adjunc model (e.g. GNN) that outputs some information to be combined in the Transformer. My model looks like this:</p>
<p>class LMGNN(Transformer):<br>
def <strong>init</strong>(self, LM_params, GNN_params):<br>
super().<strong>init</strong>(LM_params)<br>
self.gnn = GCN(GNN_params)</p>
<pre><code>def forward(self, tokens, graph):
    node_emb.= gnn(graph.x, graph.edge_index)
    (code for passing tokens to Transformer)
    ...
    ...
    ...
    h[0, 0, 0] = h[0, 0, 0,] + node_emb[0, 0] # not the real code, just a simple example
    for layer in self.layers:
        h = layer(h)
    ...
    ...
    return self.output(h)
</code></pre>
<p>As you can see, I try to modify some part of the intermediate parameter h of the Transformer with the output from another model (e.g. GNN). But it comes to a RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). I tried to set the retain_graph=True (I know it is not the solution), then it comes to another error, giving me in_place variable error.</p>
<p>I would appreciate any assistance to this annoying issue.</p>

username: soulitzer
cooked: <p>If it is erroring because a tensor saved for backward is being modified in-place, you can try <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.graph.allow_mutation_on_saved_tensors" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 2.1 documentation</a></p>
In PyG AE, are negative edge indices needed in the test() function?

username: salvdr
cooked: <p>In <a href="https://colab.research.google.com/github/AntonioLonga/PytorchGeometricTutorial/blob/main/Tutorial6/Tutorial6.ipynb#scrollTo=UPSQWiXyBzxI" rel="noopener nofollow ugc">Tutorial6 of PyG</a> the dataset ends up having 455 edges considered negative edges (eg, edges that don’t exist in the original graph). In an AE are these edges needed in the Test set?</p>
<p>I train the Autoencoder on edges with a “negative” label (or 0). The test set will include a subset of “positive” labeled edges and some that are labeled as “negative”. The use of Autoencoder architecture is to use the reconstruction loss on Pos edges to decide if that they are indeed “negative” if the loss is below a threshold and “pos” otherwise.</p>
<p>Do I need to pass negative edges to the test function?</p>
<p>Note that after splitting the dataset, we end up with this profile:</p>
<pre><code class="lang-auto"></code></pre>
<p>And notice that in the epochs loop, we pass the data as is:</p>
<pre><code class="lang-auto">for epoch in range(1, epochs + 1):
    loss = train()

    auc, ap = test(data.test_pos_edge_index, data.test_neg_edge_index)
    print('Epoch: {:03d}, AUC: {:.4f}, AP: {:.4f}'.format(epoch, auc, ap))
</code></pre>
<p>can I just pass: <code>auc, ap = test(data.test_pos_edge_index)</code> ?</p>
Dataloader best practices for large sparse matrices

username: jdenhof
cooked: <p>My data is currently stored in 100 .npz files each with a shape of (275,000, 60664) where (samples, features). My current solution is to implement my own data loader that spins off worker process to load in the chunks in random order and then shuffle the chunks and the main thread pulls in the chunk to splice batches from when it needs a new chunk. The problem is I currently have my data passed through a Queue which means my GPU utilization tanks every time a chunk gets exhausted. I believe a solution would be to have the chunks memory mapped but I have not gone down that path and am wondering if there is a way to leverage the data loader in a way that allows this. I would like to leverage the DataLoader and DistributedDataParallel using torchrun however without implementing my own DataLoader I do not know of a way of having the chunks in memory at all times when needed.</p>
<p>I think a neat solution would be to treat the loaders like Dataset’s where you can stack loaders on top of each other and they will only contain the context of the data loader iteration previous. This could also work the same in Parrellel for multiple data loaders.</p>
Torch::jit::script::module training

username: Jesus
cooked: <p>Hello everyone. Here is a code to train a torch::jit::script::module :<br>
" ```<br>
torch::jit::script::Module module = torch::jit::load(base_path + “vgg11_back.pt”);<br>
module.train();<br>
std::vector<a>at::Tensor</a> parameters;<br>
for (const auto&amp; params : module.parameters()) {<br>
parameters.push_back(params);<br>
}<br>
torch::optim::SGD optimizer(parameters, /<em>lr=</em>/0.1);<br>
int epoch = 0;<br>
std::vector<a>torch::jit::IValue</a> inputs;<br>
inputs.push_back(torch::ones({ 1, 64, 32, 32 }));<br>
torch::Tensor ground_truth = torch::ones({ 1 }).to(at::kLong);<br>
while (epoch &lt; 2) {<br>
optimizer.zero_grad();<br>
at::Tensor output = module.forward(inputs).toTensor();<br>
std::cout &lt;&lt; output.slice(/<em>dim=</em>/1, /<em>start=</em>/0, /<em>end=</em>/5) &lt;&lt; std::endl;<br>
torch::Tensor loss = torch::nll_loss(output, ground_truth);<br>
std::cout &lt;&lt; loss &lt;&lt; std::endl;<br>
loss.backward();<br>
optimizer.step();<br>
epoch++;<br>
}"</p>
<p>I want to know if i should save “module” in a pt file to conserve the updated weights or i don’t need to ?</p>
<p>Please, someone answer me</p>
Can I gather dataloader objects from processes using torch.distributed.all_gather_object?

username: Xgk
cooked: <p>I am new to distributed training on multiple GPUs. Following the tutorial in <a href="https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html" rel="noopener nofollow ugc">here</a>, I believe my model is the same on all GPUs, and the distributed sampler is creating dataloaders for each of the processes.<br>
Assuming that this is correct, then each of my GPUs have access to a subset of my data, and I should have a separate <code>loader_train</code> for each rank.<br>
Now mid training, after (lets say) epoch 10, I need to gather all the dataloader objects into a list. My solution is to use <code>all_gather_object</code>.</p>
<p>When I try the following code:</p>
<pre><code class="lang-auto">dloader_all = [None for _ in range(torch.distributed.get_world_size())]
torch.distributed.all_gather(dloader_all , loader_train[torch.distributed.get_rank()])
</code></pre>
<p>I get the error:</p>
<pre><code class="lang-auto">TypeError: 'DataLoader' object is not subscriptable
</code></pre>
<p>I wrote the code following the <a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">documentation</a>. What am I missing here? How can I gather the objects into a list where <code>list[0]</code> is the dataloader from the first rank and so on?<br>
I would appreciate any help.</p>

username: wconstab
cooked: <p>I suspect you cannot actually send a DataLoader object to a remote process, because it would have some internal state (file handles, etc) which wouldn’t make sense in the context new process even if they supported python’s serialization protocol (pickle).</p>
<p>However the literal error you’re hitting seems to be coming from improper use of subscripting on the loader_train object, which is not actually related to the all_gather call.</p>

username: Xgk
cooked: <p>I managed to fix this using the code and idea in <a href="https://discuss.pytorch.org/t/how-to-concatenate-different-size-tensors-from-distributed-processes/44819">here</a> and <a href="https://github.com/facebookresearch/maskrcnn-benchmark/blob/main/maskrcnn_benchmark/utils/comm.py" rel="noopener nofollow ugc">here</a>.<br>
Basically, I serialized the dataset, converted it to bytestorage, padded it (since my data was not of the same size), gathered and then unpadded and unserialized it.</p>
Does random seed have any effect on the order of data loader when shuffle is set to False?

username: pytorchuser101
cooked: <p>I define the seeds in the following way:</p>
<pre><code class="lang-auto">torch.manual_seed(rs)
torch.cuda.manual_seed(rs)
torch.cuda.manual_seed_all(rs)
torch.backends.cudnn.deterministic=True
torch.backends.cudnn.benchmark = False
np.random.seed(rs)
random.seed(rs)
</code></pre>
<p>And the data loader as:</p>
<pre><code class="lang-auto">testloader = DataLoader(testset, batch_size=16, shuffle=False, num_workers=2, pin_memory=True)
</code></pre>
<p>Will the order of the samples be different if I change the random seed rs ?</p>
<p>Would setting number of workers to 0 be helpful in the case each background process can lead to different order of samples ?</p>

username: ptrblck
cooked: <p>The order should be deterministic and not affected by the seeding if no suffling is used. You could verify it by iterating your <code>DataLoader</code> and comparing e.g. the <code>.abs().sum()</code> values of each sample.</p>
How to do a sort that returns the ranking of the numbers in pytorch

username: timmd
cooked: <p>Hi, everyone. I am wondering how to do a sort that returns the ranking of the numbers?<br>
For example:<br>
<code>a = torch.tensor([[ 1,2,3],[5,6,1]])</code><br>
the output of <code>torch.argsort(a,axis=1)</code> is <code>tensor([[ 0,1,2],[2,0,1]])</code>,  in which the numbers represent the original positions of those cols. But what I want is <code>tensor([[ 0,1,2],[1,2,0]])</code>, in which the number represent the ranking of the original data in that col . Thanks for your attention and time.</p>

username: KFrank
cooked: <p>Hi Tim!</p>
<aside class="quote no-group" data-username="timmd" data-post="1" data-topic="193638" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/ecccb3/48.png" class="avatar"> timmd:</div>
<blockquote>
<p>But what I want is <code>tensor([[ 0,1,2],[1,2,0]])</code>, in which the number represent the ranking of the original data in that col .</p>
</blockquote>
</aside>
<p>Think of the indices returned by <code>argsort()</code> as permutations.  If I understand<br>
what you are asking, you want the <em>inverses</em> of those permutations.  They<br>
can be computed as follows:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.1.1'
&gt;&gt;&gt; a = torch.tensor ([[ 1,2,3], [5,6,1]])
&gt;&gt;&gt; asrt = torch.argsort (a, dim = 1)
&gt;&gt;&gt; asrt
tensor([[0, 1, 2],
        [2, 0, 1]])
&gt;&gt;&gt; torch.empty(2, 3, dtype = torch.long).scatter_ (1, asrt, torch.arange (3).repeat (2, 1))
tensor([[0, 1, 2],
        [1, 2, 0]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Nan backward for scaled_dot_product_attention

username: yuzhenmao
cooked: <p>Hi,</p>
<p>I found scaled_dot_product_attention can return ‘nan’ in its backward pass after a few iterations. But after changing to the self-implemented self-attention, everything is good. I guess there is some overflow or underflow in scaled_dot_product_attention.</p>
<p>I am using 40G A100</p>

username: LewsTherin
cooked: <p>Could you please elaborate on how you solved this?</p>

username: yuzhenmao
cooked: <p>Hi, I have not solved this issue. My guess is that the backward pass of scaled_dot_product_attention might be unstable for now. But I am not sure.</p>
Backpropagate second loss without detach or second forward

username: tjoseph
cooked: <p>Hi everyone,<br>
can the following be done with a single forward pass instead of two forward passes?</p>
<pre><code class="lang-auto">outA = fnA(inputs)
loss = fnB(OutA, OutA) 
optim.zero_grad()
loss.backward()
optim.step()

outA = fnA(inputs)
loss = fnB(OutA.detach(), OutA) 
optim.zero_grad()
loss.backward()
optim.step()
</code></pre>

username: ptrblck
cooked: <p>It depends on the actual use case and what <code>optim.step()</code> updates. E.g. if any parameters used in <code>fnA</code> or <code>fnB</code> are updated, you won’t be able to use a single forward pass as the second forward pass would differ.</p>
Group imgs up by targets in ImageFolder

username: Oberon
cooked: <p>Hi, I’m new to PyTorch. I’m trying to using <code>ImageFolder</code> to load my own dataset.<br>
When I load the folder, the imgs/samples are:</p>
<pre><code class="lang-plaintext">[
    ("img0path.ext",0),
    ("img1path.ext",0),
    ...
]
</code></pre>
<p>For some reasons, I need to group them up by targets, just like:</p>
<pre><code class="lang-plaintext">[
    0=[img list of targets 0],
    1=[img list of targets 1],
    ...
]
</code></pre>
<p>Is there any convenient way to make this? Or shall I need to implement this on hand?</p>

username: ptrblck
cooked: <p>You could create a custom <code>Dataset</code> and use the <code>ImageFolder</code> as a template or base class, which would probably the cleanest approach.</p>

username: Oberon
cooked: <p>I see…Actually, I intended to make a siamese pair/triplet dataset with image folder, so that means I have to implement this by myself?</p>

username: ptrblck
cooked: <p>I’m not aware of any dataset implementations for your siamese use case, so I would assume you need to implement a custom <code>Dataset</code>. However, you can of course check other repositories to see if these would provide the needed classes.</p>
Tensor 3G Mali/Rio support - Samsung & Pixel

username: johan-sightic
cooked: <p>Hi, I want to run a PyTorch model accelerated on new Samsung and Pixel devices.<br>
I see that there is a tutorial for XNNPACK (CPU) and one for Hexagon NPU. However if I understand it correctly Samsung and Pixel phones use something called “Tensor 3G” with Mali GPU and Rio TPU.<br>
Is there currently any support for accelerated Tensor 3G processing?</p>
<p>Thanks!</p>
Multi image embedding

username: Raj_Roy
cooked: <p>Hi,</p>
<p>I wanted to encode multiple images to construct a single 3d object. To do so, I want to use as many images as possible, say around 1000. So, I was wondering if there is any encoding that would help achieve it and wondering if that encoding would be permutation invariant.</p>
<p>I would greatly appreciate if there is any suggestion or direction that may help me achieve that.</p>
Cross entropy shape of input and label

username: andreasrouv
cooked: <p>I am trying to implement a neural tager using roberta. It’s my first go at pretrained models and neural networks in general. This is my code.</p>
<pre><code class="lang-auto">class PosTagger(nn.Module):
    def __init__(self, output_size = 18, lr = 0.001):
        super().__init__()
        self.roberta =  XLMRobertaModel.from_pretrained('xlm-roberta-base')
        self.linear = nn.Linear(768, output_size)
        self.lr = lr
        


    def forward(self, input_ids, attention_mask):
        # Forward pass through the neural network
        seq_h = self.roberta(input_ids, attention_mask)
        
        last_hidden_state = seq_h.last_hidden_state

        print(last_hidden_state.shape) #it is (batch_size, sequence_length, hidden_size)
        
        
        logits = self.linear(last_hidden_state)
        print(logits.shape) #it is (batch_size, sequence_length, output_size)

       
        return logits
</code></pre>
<p>I am using Cross entropy and getting this runtime error :Expected target size [32, 18], got [32, 56].</p>
<p>Transposing the logits seems to solve the problem<br>
<code>logits = logits.transpose(1,2) #now is (batch_size, hidden_size, sequence_length)</code></p>
<p>According to documentation for crossentropy needs for input the shape <code>[batchsize, number of classes]</code> and for label <code>[batchsize]</code>. In contrast my input after transposing it has shape <code>[batchsize, num_of_classes, lenght_sequence]</code> and labels of the shape <code>[batchsize, lensequence]</code>.</p>
<p>I don’t understand if it works properly and if yes why.</p>

username: vdw
cooked: <p>You can check the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">docs</a>. <code>CrossEntropyLoss</code> does support multidimensional cases. I found some code for a RNN-based NER tagger containing the line</p>
<pre><code class="lang-auto">loss = criterion(outputs.permute(0,2,1), targets)
</code></pre>
<p>So this should in general work.</p>

username: andreasrouv
cooked: <p>I wasn’t sure if the output will still the same as having less in dimensions.</p>
<p>Thank you!</p>
Pytorch compatibility with cuda 11.8

username: Bakhtiyor_Jumanazaro
cooked: <p>hi, i am new to pytorch and i am having compatibility issues i tried everything, ran out of options. when i ran this:<br>
pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a><br>
Looking in indexes: <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a><br>
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)<br>
ERROR: No matching distribution found for torch<br>
Python 3.12.0<br>
i tried nightly versions but after 2 hours of continuous run I stopped the process.</p>

username: ptrblck
cooked: <p>Python 3.12 it not supported yet so you would need to downgrade.</p>
Dataset for video is slow

username: retsuh
cooked: <p>Hi! I’m implementing a dataset that loads data from videos. I am following the example from <a href="https://pytorch.org/hub/facebookresearch_pytorchvideo_x3d/" rel="noopener nofollow ugc">X3D pytorch</a>, which use <code>EncodedVideo</code> to load a video and <code>get_clip</code> to segment it to a clip.</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, data_root):
        csv_list = get_csv_list(data_root)
        action_list = create_action_list(csv_list)
        self.data = self._prepare_data(action_list)   # This function returns a list of [
                                                      #  [path, label, start, end],
                                                      #  [path, label, start, end], ...]
    
    def _prepare_data(self, action_list):
        data = []
        for action in action_list:
            filename = action['filename']
            label = action['label']
            start, end = time2sec([action['start'], action['end']])
            data.append([filename, label, start, end])
        return data

    def __getitem__(self, idx):
        filename, label, start, end = self.data[idx]
        start_sec = 0
        end_sec = clip_duration + start_sec
        video_data = EncodedVideo.from_path(filename)
        video_data = video_data.get_clip(start_sec=start_sec, end_sec=end_sec)
        input = video_data["video"]

        return label, input
    
    def __len__(self):
        return len(self.data)

clip_duration = 5
dataset = MyDataset(data_root)
for data in dataset:
    label, input = data
</code></pre>
<p>The <code>get_clip</code> call is very expensive (1~1.5s when the <code>clip_duration</code> is 5). The memory usage is also high, and for some reason the RAM usage doesn’t drop when the code stops (I’m using Colab and the session crashes after a few iterations).<br>
Is there a better way to implement a Dataset (and DataLoader) for datasets contaning videos?</p>
Changing the for loops in my custom convolution layer

username: fatemeh.sepah
cooked: <p>Hello<br>
I have implemented my convolution layer and changed it’s forward function like this. Because I have to perform certain operations on the input matrix layer and weight, I have to use several For loops, but this has made the execution speed of the LeNeT-5 network, which is a small network, very low. How can I remove these Fore loops?</p>
<p>class MyConv2d1(nn.Conv2d):<br>
def <strong>init</strong>(self,in_channels, out_channels, kernel_size, stride=1, padding=0):<br>
super(MyConv2d1, self).<strong>init</strong>(in_channels, out_channels, kernel_size, stride, padding)<br>
self.in_channels = in_channels<br>
self.out_channels = out_channels<br>
self.kernel_size = kernel_size<br>
self.stride = stride<br>
self.padding = padding</p>
<pre><code>    # Initialize weights and biases
    self.weights = layer_weights1
    self.biases = layer_bias1



def forward(self, input):
    batch_size, in_channels, in_height, in_width = input.shape


    # Calculate output dimensions
    out_height = int((in_height + 2 * self.padding - self.kernel_size) / self.stride + 1)
    out_width = int((in_width + 2 * self.padding - self.kernel_size) / self.stride + 1)

    # Pad input
    input_padded = torch.nn.functional.pad(input, (self.padding, self.padding, self.padding, self.padding))

    # Initialize output
    output = torch.zeros((batch_size, self.out_channels, out_height, out_width))

    # Perform convolution

    unfoldedconv1=torch.nn.functional.unfold(input_padded,(self.kernel_size,self.kernel_size))
    inputfold=torch.tensor(unfoldedconv1)
    inputfold=unfoldedconv1.transpose(1,2)
    inputfold.apply_(FloatToBinary)

    outputfold=torch.zeros((batch_size,inputfold.shape[1],self.out_channels))

    weightfold=self.weights.view(self.weights.size(0),-1).t()
    weightfold2=torch.tensor(weightfold)
    weightfold2.apply_(FloatToBinaryThenInt)


    for b in range(batch_size):
        for outchannel in range(self.out_channels):
            for i in range(inputfold.shape[1]):
                sum=0
                for j in range(inputfold.shape[2]):
                    weightValue=abs(int(weightfold2[j][outchannel]))
                    SN=torch.zeros((1,weightValue))
                    SN=BISC(int(inputfold[b][i][j]))
                    partSN=torch.tensor(SN[0:weightValue])

                    if weightfold2[j][outchannel]&lt;0:
                        sum-=torch.count_nonzero(partSN)
                    else:
                        sum+=torch.count_nonzero(partSN)

                outputfold[b][i][outchannel]=sum*(2**length)
        outputfold/=2**(2*length)


    outputfold=outputfold.transpose(1,2)

    output=torch.nn.functional.fold(outputfold,(out_height,out_width),(1,1))

    return output
</code></pre>
<p>These four for loops perform the work of the matmul function, but in such a way that instead of multiplying in matmul, my function is used.<br>
Can I replace the multiplication operation in the matmul function with my own function?</p>
How Does the gradient for torch.nn.Embedding layer works

username: Rahul_Vashisht
cooked: <p>Hi,</p>
<p>If I keep the embedding layer with a very large vocab size, but my training data has only a few tokens from the vocabulary. Does the vector representation of the tokens which are not part of the training also change?</p>

username: ptrblck
cooked: <p>No, the unused vectors do not get any gradients from the backward pass and would thus not be updated unless you add the <code>weight</code> tensor to a weight decay term or any additive loss.</p>

username: Rahul_Vashisht
cooked: <p>Okay, so weight decay will affect the embedding of all the tokens; I understood it. Thanks for the reply</p>
Larger batches not faster

username: mhaines94108
cooked: <p>I’m using Google’s EfficientNet (the small version) on an RTX 3090.  I’ve found that overall epoch time doesn’t vary much whether I train with big batches or small batches.  Similarly, with classification I’m not seeing much difference in images/second whether I use big or small batches.</p>
<p>Am I missing something?  I thought the idea of a batch was that all of the images got processed in parallel, so batch processing time should be relatively constant whether it’s 1 image or 32.</p>

username: ptrblck
cooked: <p>Your training code might suffer from a bottleneck, which might be unrelated to the GPU, e.g. the data loading. You could profile your use case using Nsight Systems or the native PyTorch profiler to check which part of your code is the current bottleneck.</p>
Split train set into datasets for each class

username: pytorchuser101
cooked: <p>I have several datasets define in this fashion:-</p>
<pre><code class="lang-auto">trainset = torchvision.datasets.CIFAR10(root='path_to_CIFAR10', train=True, download=True)
trainset = torchvision.datasets.CIFAR100(root='path_to_CIFAR100', train=True, download=True)
trainset = torchvision.datasets.SVHN(root='path_to_SVHN', split='train', download=True)
trainset = torchvision.datasets.GTSRB(root='path_to_GTSRB', split='train', download=True)
</code></pre>
<p>I apply data transforms separately before creating the dataloaders.</p>
<p>What would be the most efficient way to split each trainset into sets where each set has only data of one particular class ?</p>
<p>I’d prefer to create the sets and store them in a list so that I can iterate over each classes’ set.</p>