akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The location of `zero_grad` at the training loop

DavidHere: <p>In the PyTorch quick start guide it is written that:</p>
<pre><code class="lang-python">def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    model.train()
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to(device), y.to(device)

        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")
</code></pre>
<p>I was wondering, what difference would it make if the <code>zero_grad()</code> operation would be placed at the beginning of the loop:</p>
<pre><code class="lang-python">def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    model.train()
    for batch, (X, y) in enumerate(dataloader):
        optimizer.zero_grad()
        X, y = X.to(device), y.to(device)

        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")
</code></pre>
<p>Should it have the same result?</p>

InnovArul: <p>As far as I know,  they are equivalent.<br>
Do you observe different results?</p>

ptrblck: <p><a class="mention" href="/u/innovarul">@InnovArul</a> is right and both approaches should work, since you are zeroing out the gradients before calculating the new ones from the current iteration.<br>
However, if you want to optimize the code a bit more you could use <code>optimizer.zero_grad(set_to_none=True)</code> which will not fill the <code>.grad</code> attributes with zeros but will delete them (set to <code>None</code>) and could save some memory and avoid an unnecessary sum kernel.<br>
In this case it would be more beneficial to delete the gradients at the end of the training loop or right at the beginning.</p>
Returning a Tensor in a multiprocess DataSet is extremely slow

zareami10: <p>Returning a Tensor is orders of magnitudes (60 times in this example) slower than returning something like a numpy array in a DataSet when used with a DataLoader with <code>num_workers</code>&gt;0.<br>
This is the case even if the Tensor happens to share memory with a numpy array (i.e. <code>from_numpy</code>), but does not seem to happen when we don’t use any workers.</p>
<p>How can I deal with this issue? (the code is run on Linux)</p>
<p>Some example:</p>
<pre><code class="lang-python">import torch
import numpy as np

class MyDataSet(torch.utils.data.Dataset):
    def __init__(self):
        super().__init__()

    def __len__(self):
        return 100000

    def __getitem__(self, idx):
        arr = np.arange(250)
        tensor = torch.arange(250)

        # return arr # 1s
        return tensor # 60s

def collate_wrapper(batch):
    return batch

ds = MyDataSet()
data_loader = torch.utils.data.DataLoader(
    ds,
    num_workers=2,
    shuffle=False,
    batch_size=64,
    collate_fn=collate_wrapper,
    prefetch_factor=1,
)

c = 0
for _ in data_loader:
    c+=1
print(c)
</code></pre>
Multiplication of 4D and 3D tensor using torch.einsum

Jayden9912: <p>How to multiply a 4D tensor with 3D tensor according to code below?<br>
qr = torch.einsum('b h i d, i d j → b h d j ', q, r_q)<br>
b = 64<br>
h = 8<br>
i = 8<br>
d = 64<br>
j = 64</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/jayden9912">@Jayden9912</a>,</p>
<p>When you say “multiply” do you mean matrix-multiplication? Or do you want to use element-wise multiplication instead?</p>
<p>Your code example works and returns a Tensor of shape <code>[b,h,d,j]</code></p>
<pre><code class="lang-auto">import torch
b=64
h=8
i=8
d=64
j=64
 
q=torch.randn(b,h,i,d)
r_q=torch.randn(i,d,j)

qr=torch.einsum("bhid, idj-&gt;bhdj", q, r_q)
print(qr.shape) #returns torch.Size([64, 8, 64, 64])
</code></pre>
<p>I assume you want to multiply a batch of Tensors (the 4D tensor) with a single 3D tensor for all tensors in your batch?</p>
<p>The einsum expression (<code>"bhid, idj -&gt; bhdj"</code>) you have is equivalent to taking a matmul over the i-th index, then taking the trace over the d-th index. There some examples of einsum notation in numpy’s documentation (see <a href="https://numpy.org/doc/stable/reference/generated/numpy.einsum.html" rel="noopener nofollow ugc">here</a>)</p>
Indexing customized contiguous rows of 2d tensor with even/odd starting point

ltang: <p>Hi guys, I am looking for a method to index customized contiguous rows of 2d tensor with even/odd starting point. For example, mat is a 4*4 2d tensor. I know a[1::2,:] and a[::2,:] can index odd/even rows.<br>
But I dont know how to index 2 contiguous rows starting from odd rows (i.e., 1 and 3) for getting rows 1/2 and 3/4, respectively. Any idea? Thanks.</p>
Multiple kernels for single CONV

ltang: <p>Hi, I am still learning PyTorch, so please forgive me for any confusion. I am looking for an implementation that one CONV operation uses multiple kernels. For example on CONV1D, I have an array [1,0,1,1,0] and would like to use 3 1x3 kernels ([0.1,0.2,0.1],[0.1,0,0], [0,0.2,0]) to sweep the array sequentially, and get [1<em>0.1+0</em>0.2+1<em>0.1=<em>0.2, 0</em>0.1+1</em>0+1<em>0=0, 1</em>0+1<em>0.2+0</em>0=0.2]. I couldn’t find any straightforward PyTorch solutions and would like to hear any suggestions. Thanks.</p>

ptrblck: <p>Based on your description it seems as if you would like to apply the kernels to a single location and shift each kernel by 1?<br>
If so, you could e.g. apply a standard convolution and get the diagonal of the result:</p>
<pre><code class="lang-python">x = torch.tensor([[[[1., 0., 1., 1., 0.]]]])
conv = nn.Conv2d(1, 3, kernel_size=(1, 3), bias=False)

with torch.no_grad():
    weight = torch.tensor([[[[0.1, 0.2, 0.1]]], [[[0.1, 0., 0.]]], [[[0, 0.2, 0]]]])
    conv.weight.copy_(weight)

out = conv(x)
out[:, torch.arange(3), :, torch.arange(3)]
</code></pre>
<p>or <code>unfold</code> the input and use a matrix multiplication to get the desired results.</p>

ltang: <p>This is what I need, thanks</p>
Dataloader with expanding dataset

Zahl: <p>I am trying to create a training loop in a reinforcment learning type setting. I have a dataset object which is continously growing as data is being added to it during the execution. I simultaneously want to train a model by sampling batches from this growing dataset using a dataloader however the dataloader seems to not register the new data being added to the dataset. Is there a way to achieve this using the Dataset and DataLoader classes, if not how can I achieve this efficiently?</p>

mxahan: <p>I was just wondering can you prepare a loop and create a dataset loader every time to capture the expanding data?</p>

Zahl: <p>Hi, thanks I did consider this, but while searching for an answer i came across a comment " DataLoader iterators are not meant to be very short lived objects" <a href="https://github.com/pytorch/pytorch/issues/1917#issuecomment-400004494" class="inline-onebox" rel="noopener nofollow ugc">Get a single batch from DataLoader without iterating · Issue #1917 · pytorch/pytorch · GitHub</a>, which made me think this would not be a good idea though im not absolutely sure.</p>

mxahan: <p>What I meant, you can keep two loops, one loop for Dataloader, and a child loop to iterate over the data, once done del the dataloader and reload for new epoch training.</p>
Bug? torch.jit.script(model) loses list attributes with mixed element types

Fei_Liu: <p>I found that the scripted model (by <code>torch.jit.script</code>) silently loses list attributes where the elements have mixed types, e.g. the following code snippet:</p>
<pre><code class="lang-python">import torch
from torch import nn

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.a = [1, 2]
        self.b = [1, 2.2]

m = Model()
print(m.a)  # [1, 2]
print(m.b)  # [1, 2.2]

sm = torch.jit.script(m)
print(sm.a)  # [1, 2]
print(sm.b)  # error, no attribute b, why?
</code></pre>
<p>The attribute <code>a</code> is a list of int and the scripting worked fine, but <code>b</code> (with mixed element types, <code>int</code> and <code>double</code>) is silently lost. The scripted model <code>sm</code> doesn’t have it anymore. I think the biggest concern is that the scripting process didn’t give any error and just silently returned a scripted model.</p>
<p>Is this a known bug (or WAI) ?</p>
<p>torch version: 1.12.0</p>
Combine lstm with transpose CNN

Mariam: <p>is it possible to combine lstm with transpose cnn, I have a time series data where I want to use the combination of lstm and transpose cnn for image recognition from a time series data, can you plz let me know if I can do such thing in pytorch?</p>

carperbr: <p>This depends on the specifics of the problem. Here is a quick example showing one way you could go about this, assuming you’re dealing with a timeseries that has only one dimension per item in the sequence for a shape of [B,W]. If each item has multiple dimensions and is in shape of [B,H,W] or [B,W,H] it changes a little but not much. That being said, not 100% sure how well this would work as my focus has been on transformers rather than LSTM, might take some tweaking to get working well. One thing to note is that in the below snippet I preserved the sequence length by using a stride of (2,1), however this can obviously be changed and downsampled via conv2d if that is your goal. I also expand the unsqueezed height dimension to 2 as you’ll require more than one feature to expand in that direction (this will be less hacky if you have multiple datapoints per timestep)</p>
<p>One note on increasing channels: this is doable, however you’d have to take the route MMDENSELSTM uses in compressing back to a single channel via a 1x1 kernel convolution and then using that for the lstm’s input; after that you could concatenate with the input and decrease channel count to the original via a 1x1 conv2d.</p>
<pre><code class="lang-auto">a = torch.ones((1, 32))
a = a.unsqueeze(1).unsqueeze(1)
conv = nn.ConvTranspose2d(1, 1, kernel_size=3, padding=1, stride=(2,1))
b = conv(a.expand((a.shape[0], a.shape[1], 2, a.shape[3]))) # B,1,2,W -&gt; B,1,3,W
b2 = conv(b) # B,1,5,W

dims = b2.shape[2]
lstm = nn.LSTM(dims, dims, batch_first=True, bidirectional=True)

out, _ = lstm(b2.squeeze(1).transpose(1,2)) # B,W,5 / B,W,10 if bidirectional
print(out.shape)
out = out.transpose(1,2).unsqueeze(1) # B,1,5,W
print(out.shape)
</code></pre>
Is it safe to expose Tensor.data_ptr() to the outside world?

Fei_Liu: <p>I’m working on a library that processes on a Tensor that needs to take values from the external world. The tensor is fixed-size during the whole lifetime of the program. Something like below:</p>
<p>in my library:<br>
A singleton fixed size tensor x, initialized when the program starts.<br>
Some functions that process x periodically.</p>
<p>external users:<br>
fill in / update x every now and then, and then call the processing functions in my library to get the processed results.</p>
<p>Now due to API constraints, I can only expose raw data pointer to the external users. In this case, is it safe to expose <code>static_cast&lt;float*&gt;(x.data_ptr())</code> to the outside world and let the downstream client modify the tensor values by it? Assuming my tensor was initialized with a given size (and dtype=float) and not altered anywhere in my own library.</p>
<p>If this is a bad practice, what should I do instead? Are there any other APIs that I can expose to be more idiomatic? (I know sharing data by exposing mutable memory address is not good in general, but let’s say we really want to do it for performance reasons.)</p>

eqy: <p>As long as you are confident that the lifetime of the Tensor matches its usage, I don’t see an issue here, as something similar has to be done when dispatching to operators in external libraries (e.g. cuDNN).</p>
<p>However, if the mutation is relevant for things like calling <code>.backward</code> later, I would check that versioning semantics are correct for your application: <a href="https://pytorch.org/docs/stable/notes/autograd.html#in-place-correctness-checks" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 1.12 documentation</a>.</p>

Fei_Liu: <p>Thank you for the education! That clears my concerns.</p>
Nesting an autograd graph inside another

polo5: <p>Heya,</p>
<p>There is an increasing amount of research (meta-learning, deep equilibrium models, OptNets etc.) where the forward pass of some model includes solving a seperate optimization problem.</p>
<p>In many cases, we don’t need to track this <em>inner</em> optimization problem as part of the same graph as our main model (e.g. because we derive gradients manually and add them as a hook).</p>
<p>The question is this: how can I use an entire sub <em>inner</em> graph that doesn’t affect or know about the <em>outer</em> graph it is part of. Here is some pseudo code:</p>
<pre><code class="lang-python">class OuterModel(nn.Module):
    def __init__(self):
        super(OuterModel, self).__init__()
        self.tail = nn.Sequential(...)
        self.head = nn.Sequential(...)

    def forward(self, x):
        tail_out = self.tail(x) #tracked in outer graph

        with torch.no_grad(): #everyhting happening here should be tracked in some inner graph only
            inner_params = torch.zeros(requires_grad=True)
            inner_optim = torch.optim.AdamW([inner_params])

            for i in range(inner_steps):
                inner_loss = torch.norm(A@inner_params - tail_out.detach())
                inner_loss.backward() #this should only affect inner params
                inner_optim.step()
                inner_optim.zero_grad()

            inner_params.detach_() #outer graph doesn't need to track these

        out = tail_out + inner_params * self.head(tail_out) 
        
        return out


## forward/ backward pass of outer model 
outer_model = OuterModel()
outer_optim = torch.optim.AdamW(outer_model.parameters())

for x,y in loader:
    x,y = data_batch
    out = outer_model(x)
    outer_loss = (out-y)

    outer_loss.backward() #only does backward for outer params
    outer_optim.step()
    outer_optim.zero_grad()
</code></pre>
<p>The easiest way is to do this manually: <code>inner_grad = torch.autograd.grad(inner_loss, inner_params)</code>  and then <code>inner_params -= 0.1*inner_grad</code> but ideally we want to leverage all the nice optimizers pytorch offers out of the box and and we don’t want to concatenate all our inner_params into one tensor. I’ve also tried to play with <code>inner_loss.backward(retain_graph=True)</code> to retain the outer graph, but this doesn’t train (I suspect because the inner graph is also retained or something)</p>
<p>Thanks for any pointers!</p>
<p>Paul</p>
Tensor - read and write values across batches and channels

MartinPerry1: <p>I have a batched tensor [B, C, H, W], where B is batch, C is a number of channels, H and W are dimensions. I have an array with indices and I would like to read values at those indices from input tensor and write them to another tensor.</p>
<p>For example:</p>
<pre><code class="lang-auto">two tensors A (source) and B (target)

B = 1
C = 3
H, W = 64

indicesRead = [[0, 0], [13, 15], [32, 43]]
indicesWrite = [[7, 5], [1, 1], [4, 4]]
</code></pre>
<p>and I would like to get from tensor A value for channel 0 at [0, 0], channel 1 at [13, 15] and channel 2 at [32, 43].</p>
<p>Once I have these values, I want to write them to tensor B to channel 0 to position [7, 5] (in this sample copy A[0, 0, 0, 0] to B[0, 0, 7, 5]) etc.</p>
<p>Can it be done with torch methods or I have to iterate tensor manually?</p>
<p>Simple 3x3 2-channel examle (1 batch only):</p>
<pre><code>Anp = np.array([[[[0, 1, 2], [3, 4, 5], [6, 7, 8]], [[0, 11, 22], [33, 44, 55], [66, 77, 88]]]])

A = torch.as_tensor(Anp, dtype=torch.float)
B = torch.empty_like(A)

indicesRead = [[[0, 1], [1, 2]]]
indicesWrite = [[[1, 1], [2, 2]]]

#what next?
</code></pre>
<p>Note: read and write indices can be possibly reoriented if needed, they donw have to be in this format.</p>

ptrblck: <p>A direct assignment might work:</p>
<pre><code class="lang-python">B = 1
C = 3
H, W = 64, 64

indicesRead = torch.tensor([[0, 0], [13, 15], [32, 43]])
indicesWrite = torch.tensor([[7, 5], [1, 1], [4, 4]])

a = torch.randn(B, C, H, W)
b = torch.zeros(B, C, H, W)

b[:, torch.arange(C).unsqueeze(1), indicesWrite[:, 0], indicesWrite[:, 1]] = a[:, torch.arange(C).unsqueeze(1), indicesRead[:, 0], indicesRead[:, 1]]
</code></pre>

MartinPerry1: <p>Works great, thank you</p>
Thread safety in Multiprocessing - CUDA tensors dont update asynchronously

Vijay_Giri: <p>Are CUDA tensors always updated synchronously? Is there some inherent mechanism for thread safety?<br>
I am updating a tensor from multiple processes, but I don’t see errors due to lack of locking.</p>
<pre><code class="lang-auto">import torch, torch.multiprocessing as mp
n_processes, n_iterations = 4, 1_000_000
def fn(t):
    for _ in range(n_iterations): t[0] += 1
if __name__ == "__main__":
    context = mp.get_context('spawn')
    processes = []
    shared_array = torch.zeros(1, device = 'cpu')
    shared_array.share_memory_()
    for _ in range(n_processes):
        p = context.Process(target=fn, args=(shared_array,))
        processes.append(p)
        p.start()
    for process in processes: process.join()
    print(shared_array)
</code></pre>
<p>Output: <code>tensor([3857068.])</code> which indicates lack of thread safety. (A thread safe output would be 4_000_000)<br>
However, replacing <code>shared_array = torch.zeros(1, device = 'cuda:0')</code> results in the output <code>tensor([4000000.], device='cuda:0')</code> which indicates thread safety.<br>
I observe this issue persistently over multiple runs on a Tesla V100, CUDA 11.6, PyTorch 1.10.2</p>
Bug with torch.multinomial

Carsten_Ditzel: <p>following some of the discussions with multinomial in the past, there still seems to be some problem with its implementation in torch. 1.8.1.</p>
<p>The code</p>
<pre><code class="lang-auto">sample = torch.multinomial(torch.tensor([[1, 0, 0, 1]], dtype=torch.float), num_samples=3, replacement=False)
print(sample)
</code></pre>
<p>does not throw, nor does it for num_samples=4. However,</p>
<pre><code class="lang-auto">sample = torch.multinomial(torch.tensor([[1, 0, 0, 1]], dtype=torch.float), num_samples=5, replacement=False)
print(sample)
</code></pre>
<p>gives</p>
<blockquote>
<p>RuntimeError: cannot sample n_sample &gt; prob_dist.size(-1) samples without replacement</p>
</blockquote>
<p>something is wrong here I guess</p>

ptrblck: <p>Thanks for raising this issue as it seems to be a valid bug.<br>
<a href="https://github.com/pytorch/pytorch/issues/48273">This GitHub issue</a> seems to already track it.</p>
Dictionary creation

Angelina_Robert: <p>Hi,<br>
The following code works fine, however, if I change the label values it shows the error.<br>
class_samples={}<br>
labels=[0, 1, 2 ,3]<br>
sample_count=[10, 20, 30, 40]<br>
for labl in  labels:<br>
class_samples[labl]=  sample_count[labl]<br>
print (’ Class Samples: ', class_samples)</p>
<p><span class="hashtag">#OUTPUT</span> : Class Samples:  {0: 10, 1: 20, 2: 30, 3: 40}<br>
If I change it:<br>
class_samples={}<br>
labels=[0, 2, 4 ,6]<br>
sample_count=[10, 20, 30, 40]<br>
for labl in  labels:<br>
class_samples[labl]=  sample_count[labl]<br>
print (’ Class Samples: ', class_samples)</p>
<p>Error  :   class_samples[labl]=  sample_count[labl]<br>
IndexError: list index out of range</p>
<p>I want have output as : Class Samples:  {0: 10, 2: 20, 4: 30, 6: 40}</p>

ptrblck: <p>Indexing <code>sample_count</code> with <code>labl</code> won’t work as you need indices in <code>[0, len(sample_count)]</code>.<br>
Use the index create by <code>enumerate</code>:</p>
<pre><code class="lang-python">for idx, labl in enumerate(labels):
    class_samples[labl]= sample_count[idx]
print(' Class Samples: ', class_samples)
</code></pre>
<p>or create the dict directly as:</p>
<pre><code class="lang-python">dict(zip(labels, sample_count))
</code></pre>
What would be the reason the loss does not decrease (in-batch loss)?

kjang0517: <p>Hello. I’m trying to train my model with in-batch loss. Following is the key part of the loss computation.</p>
<ul>
<li>There are three representations for (query, positive, negative).</li>
<li>The shape of <code>in_batch_scores</code> is (bsize, bsize). The diagonal elements are the target scores for cross entropy, which are the dot product between queries and positives.</li>
<li>I additionally compute <code>neg_scores</code> between queries and negatives, concat two scores computed so far, then finally feed the scores to <code>cross_entropy()</code>
</li>
</ul>
<p>The problem is the loss only oscillates and does not decrease. I checked all the <code>parameters()</code> as well as<br>
tensors marked as trainable. What would be the reason?</p>
<pre><code class="lang-python">def _compute_ibl(self, rep_q, rep_pos, rep_neg):
    # scores
    rep_q_cloned = rep_q.clone()
    in_batch_scores = rep_q @ rep_pos.T  # (bs, bs)
    neg_scores = (rep_q_cloned * rep_neg).sum(dim=1).unsqueeze(dim=1)  # (bs, 1)
    scores = torch.cat([in_batch_scores, neg_scores], dim=1)  # (bs, bs+1)

    # compute loss
    # temp = torch.cat([in_batch_scores, neg_scores], dim=1)  # (bs, bs+1)
    # scores = F.log_softmax(temp, dim=1)
    # nb_columns = in_batch_scores.shape[1]
    # return torch.mean(
    #     -scores[
    #         torch.arange(in_batch_scores.shape[0]),
    #         torch.arange(nb_columns),
    #     ]
    # )

    # compute loss
    target = torch.arange(
        in_batch_scores.shape[0], device=scores.device, dtype=torch.long
    )

    return F.cross_entropy(scores, target)
</code></pre>

eqy: <p>Just as a sanity check, have you checked whether the parameters that are expected to change every iteration are indeed changing (e.g., with something like <code>params.sum()</code>)?</p>
<p>And (just because it wasn’t mentioned in your post), have you tried decreasing the learning rate?</p>

kjang0517: <p>Thanks for the suggestion. I actually found a solution! Because of the bug in the code the learning rate schedule was set to ‘inf’, which kept the effective learning rate low, preventing the model from actually training. Anyway it was related to learning rate <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Thanks!</p>
Comparing a Tensor to a group of others

tsaG1337: <p>Hey!</p>
<p>I am quite new here and not really sure if Pytorch is something I am looking for but I found the PyTorch Tensor compare function and it goes into the right direction.</p>
<p>My problem is the following. I do have several (physical) fingerprints in tensor format.<br>
These Tensors have a size of 192x192 values.</p>
<p>I have “known Fingerprints” where I know the owner from, lets say 2 different Fingers a 3 Images. These three Images are similar but differ in the sense that the finger might have been rotated or shifted on the sensor.</p>
<p>I would know need a function that compares the “unknown” fingerprint to the known Fingerprints and rotates and shifts the Tensor to see if it fits and output a confidence factor.</p>
<p>I am programming in Python. Could anyone point me in the right direction?</p>
<p>Thanks!</p>

Karthik_Ganesan: <p>It sounds like what you want is actually an image processing library, not necessarily a ML library, which is what PyTorch is. So you might be able to do what you want using PyTorch, but something like OpenCV might be better suited for it.</p>
<p>Based on your description, it sounds like what you want is a ‘rotation invariant template matching’. OpenCV should have support for that and you should also be able to find a lot of resources online to do this as well.</p>
What data does each worker process hold? Does it hold the full dataset object or only a batch of it?

clarissesimoes: <p>Hi there.</p>
<p>The <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" rel="noopener nofollow ugc">torch.utils.data documentation</a> says that</p>
<blockquote>
<p>When <code>num_workers &gt; 0</code> , each worker process will have a different copy of the dataset object, so it is often desired to configure each copy independently to avoid having duplicate data returned from the workers.</p>
</blockquote>
<p>However, on this other <a href="https://discuss.pytorch.org/t/loading-huge-data-functionality/346/29">topic</a>, @ <a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a> says that</p>
<blockquote>
<p>If you use multiple workers in your <code>DataLoader</code>, each worker will load a batch in the background using multiprocessing while the GPU is busy.</p>
</blockquote>
<p>My question is: what data does each worker process hold? Does it hold the full dataset object or only a batch of it?</p>

ptrblck: <aside class="quote no-group" data-username="clarissesimoes" data-post="1" data-topic="160136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/clarissesimoes/48/51789_2.png" class="avatar"> clarissesimoes:</div>
<blockquote>
<p>Does it hold the full dataset object or only a batch of it?</p>
</blockquote>
</aside>
<p>Each worker will hold a reference to an own <code>Dataset</code>.<br>
Now, if you are using a lazy loading approach in the <code>Dataset</code>, the actual copy will be small, since basically only the <code>Dataset</code> object itself and everything in its <code>__init__</code> will be copied in each worker, which is usually just a reference to a transformation, the image paths etc.</p>
<p>However, if you are eagerly loading the data, i.e. you are pre-loading the entire data in the <code>Dataset.__init__</code>, then each worker will also create a copy of it and will thus increase the overall memory usage significantly.</p>
<p>In both approaches the workers will then create a batch by calling the <code>Dataset.__getitem__</code> using the indices from the sampler.</p>
Segmentation Fault for loss.backward() with batch_size > 1 on M1 Mac

harzenic00: <p>Hello! I have the following simple model and training procedure, but when I run loss.backward() on an M1 Mac with any batch size greater than 1, it produces a segmentation fault.</p>
<pre><code class="lang-auto">class MLP(nn.Module):
    def __init__(self, classes):
        super().__init__()
        self.classes = classes
        
        self.ff1 = nn.Linear(2, 20)
        self.ff2 = nn.Linear(20, classes + 1)
        
        self.relu = nn.ReLU()
        
    def forward(self, x):
        out = self.relu(self.ff1(x))
        out = self.ff2(out)
        
        return out
</code></pre>
<pre><code class="lang-auto">def train(dataloader, models, loss_fn, optimizer):
    for model in models:
        model.train()
    
    total_loss = 0
    batches = 0
    
    for batch, (X, Y) in enumerate(dataloader):
        loss = 0
        optimizer.zero_grad()
        for group in range(len(models)):
            current_y = torch.where(Y[:, 0] == group, Y[:, 1], models[group].classes)
            
            preds = models[group](X)
            loss = loss + loss_fn(preds, current_y)
        
        loss.backward()
        optimizer.step()
        
        total_loss += loss
        batches += 1
        
        if batch % 1000 == 0:
            print(f'Current Loss: {total_loss / batches}')
            total_loss = 0
            batches = 0
</code></pre>
<pre><code class="lang-auto">train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=4, shuffle=True)
val_dataloader = torch.utils.data.DataLoader(val_dataset)

group_1 = MLP(3)
group_2 = MLP(3)
lr = 1e-4
epochs = 15

loss_fn = nn.CrossEntropyLoss(reduction='sum')

params = list(group_1.parameters()) + list(group_2.parameters())

optimizer = torch.optim.SGD(params, lr)

for e in range(epochs):
    print(f'Epoch {e}:')
    train(train_dataloader, [group_1, group_2], loss_fn, optimizer)
</code></pre>
<p>Interestingly, this code does not produce a segmentation error when run on an Ubuntu server. This leads me to believe that it may be an issue related to the M1 chip. Advice on how to resolve this issue would be greatly appreciated.</p>
Classifier returns [0.5, 0.5]

philipp0827: <p>Hi im try to train a SSD-model,<br>
but after faw epochs the classifer returns exactly [0.5, 0.5]…<br>
what could be causing this problem?<br>
i use for loss cross entropy and for optimizer <code>optimizer = torch.optim.Adam(model.parameters(), lr=0.001)</code></p>

srishti-git1110: <p>To avoid any confusion, could you please specify what [0.5, 0.5] exactly represent here? Is it the proportion of samples in both categories, in case you have two classes?</p>
Pytorch Dataloaders and Sampling without Replacement

papoo13: <p>My question is about PyTorch dataloaders.<br>
Suppose I have a data_loader in a function. In calling this function once, I sample a number of batches from it. Then, I leave this function and later call it again. How should I make sure that batches that are sampled include unique values, a.k.a, sampling is done without replacement?<br>
I appreciate your guidance.</p>

ptrblck: <p>One approach would be to create the <code>iterator</code> manually and pass it around.<br>
This would make sure that the <code>next</code> operation will continue the sampling process and will not restart the <code>DataLoader</code> iteration.</p>
<p>Here is a small example:</p>
<pre><code class="lang-python">def fun(loader_iter):
    for i in range(2):
        try:
            a = next(loader_iter)
            print(a)
        except StopIteration:
            print("loader_iter is empty, re-create it!")

dataset = TensorDataset(torch.arange(10))
loader = DataLoader(dataset, batch_size=2)
loader_iter = iter(loader)

fun(loader_iter)
# [tensor([0, 1])]
# [tensor([2, 3])]

fun(loader_iter)
# [tensor([4, 5])]
# [tensor([6, 7])]

fun(loader_iter)
# [tensor([8, 9])]
# loader_iter is empty, re-create it!

# still empty!
fun(loader_iter)
# loader_iter is empty, re-create it!
# loader_iter is empty, re-create it!

loader_iter = iter(loader)
fun(loader_iter)
# [tensor([0, 1])]
# [tensor([2, 3])]
</code></pre>

papoo13: <p>thank you very much. So using next() and iter() is the solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Understanding the loss function in DeepAR

Jose_Peeterson: <p>The loss function looks like below, where N is the number of time series. ignore N for now and say N = 1. T is length of prediction horizon.<br>
During training $t_0$ starts from encoder time step 0 all the way to last decoder time step $T$.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c45293e5bf0993784db3bb5c9c98bd593ca4549.png" data-download-href="https://discuss.pytorch.org/uploads/default/1c45293e5bf0993784db3bb5c9c98bd593ca4549" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c45293e5bf0993784db3bb5c9c98bd593ca4549_2_690x236.png" alt="image" data-base62-sha1="425xRrQ2aopQsWddjnbzlTDspqF" width="690" height="236" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c45293e5bf0993784db3bb5c9c98bd593ca4549_2_690x236.png, https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c45293e5bf0993784db3bb5c9c98bd593ca4549.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c45293e5bf0993784db3bb5c9c98bd593ca4549.png 2x" data-dominant-color="F8F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">702×241 28.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How does the sum of <strong>individual</strong> log of negative binomial (nbinom ) likelihood loss function at each time step contribute to the convergence of nbinom parameters?</p>
<p>How does maximizing <strong>individual</strong> time steps likelihood result in convergence of nbinom paramters?<br>
e.g. say an input to a time step is 1. the paramters (5,0.75) maximize it.</p>
<p>However because the input can come from any sub-sequence it could have come from another distribution (10,0.5) for this time step.</p>
<p>The encoder input comes from any sub-sequence of the time series (i.e. phase shifted) so the nbinom parameters learnt cannot associate with a fixed time step. So during the initial stages of training how does the network know the right nbinom parameter to update by first detecting the phase of the encoder input? Can it detect the phase of the encoder input?</p>
Torch.nn.functional.pad seems to have maximum input length

FredMushZhao: <p>Hi!<br>
To reproduce:</p>
<pre><code class="lang-python">import torch
from torch.nn.functional import pad
pad(torch.randn(4, 65535, 20).cuda(), (0,0), 'reflect') # ok
pad(torch.randn(4, 65536, 20).cuda(), (0,0), 'reflect') # wrong
</code></pre>
<p>traceback<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/opt/conda/lib/python3.9/site-packages/torch/nn/functional.py”, line 4369, in _pad<br>
return torch._C._nn.reflection_pad1d(input, pad)<br>
RuntimeError: CUDA error: invalid configuration argument<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
<p>Env:<br>
cuda 11.3<br>
torch 1.11</p>

ptrblck: <p>Thanks for reporting the issue! I can reproduce it and it seems the kernel launch fails. We’ll take a look at it.</p>
Is there any way to implement stochastic pooling?

kkjh0723: <p>I wonder if it is possible to implement stochastic pooling as in this <a href="https://arxiv.org/abs/1301.3557" rel="nofollow noopener">paper</a>, “Stochastic Pooling for Regularization of Deep Convolutional Neural Networks” or “fractional average pool” as in <a href="https://www.tensorflow.org/versions/r0.12/api_docs/python/nn/pooling" rel="nofollow noopener">Tensorflow</a>?<br>
If possible, what is the most efficient way? Do I need to add custom C module?<br>
(It is nice if it’s possible with only python modules and not too slow…)</p>

tom: <p>Hello,</p>
<p>thank you for asking!</p>
<p>Stochastic pooling as in the paper with stride = pool size  is easy to implement using view (so that the indices to be pooled are in their own dimension e.g. <code>x.view(x.size(0),x.size(1)//2,2)</code>), sampling random coordinates from multinomial and using that for indexing.<br>
The weighting can be done using a standard (“spatial”) convolution in the functional interface and a filter that contains the probability.</p>
<p>You could also use stochastic average  pooling by drawing scores + softmax + convolution similar to what they suggest for test time but with random weights.</p>
<p>I could do an implementation example if that helps.</p>
<p>Someone more knowledgable than me will have to answer fractional average pooling (fraction max pooling is done in pytorch, but I’m not aware of average pooling).<br>
An ad-hoc way of achieving something similar could involve “splitting the activations at the border” of larger kernels by using positive convolutions stencils that sum to one but are not all equal and then keeping a subgrid.</p>
<p>Best regards</p>
<p>Thomas</p>

kkjh0723: <p>Thanks for the answers!</p>
<p>As for your first suggestion, is it possibly extend to 2D with shape (N,C,H,W)?<br>
If its possible, should <code>x</code> be reshape  to <code>x.view(N,C,H//2,W//2,2,2)</code> or <code>x.view(N,C,H//2,2,W//2,2)</code>?<br>
(I’m not sure but the correct one might be <code>x.view(N,C,H//2,2,W//2,2).permute(0,1,2,4,3,5)</code>?)</p>

tom: <p>You need to keep the 2s next to the original dims, so the latter variant. Instead of permuting, I’d probably either sample the two independently (if the desired distribution is just the product) or do the //2 and %2 calclations to get the indices from values 0…3. (Numpy has unravel, but that is CPU only and a bit overkill here.)</p>
<p>Best regards</p>
<p>Thomas</p>

Ashutosh_Kushwaha: <p>Can you please explain the process by giving an example?</p>

smani: <p>Is there any code available for this? I am working on stochastic pooling. However, still not much idea about how to implement it.</p>

jyothi_sri: <p>Did u implement stochastic pooling?If you get can u plz provide with an example</p>
How to efficiently preserve most important DCT coefficients by PyTorch?

111414: <p>I am studying the discrete cosine transform (DCT) and JPEG compression technique. I do not know how to perform DCT for a 2-D image tensor and preserve only the most important coefficients?</p>
<p>I know that I can efficiently perform DCT by referencing <a href="https://github.com/zh217/torch-dct" rel="noopener nofollow ugc">https://github.com/zh217/torch-dct</a>, here is an example:</p>
<pre><code class="lang-python">import torch
import torch_dct as dct

x = torch.randn(8, 8)
x_dct = dct.dct_2d(x)
x_rec = dct.idct_2d(x_dct)
print(torch.abs(x - x_rec).mean())  # x == y within numerical tolerance
</code></pre>
<p>However, I want to preserve only the most important DCT coefficients at the upper left corner. In other words, how can I generate a Zig-Zag binary mask for a given image size and coefficient amount?</p>
<p>The following function is what I want:</p>
<pre><code class="lang-python">&gt;&gt;&gt; f(h=3, w=3, q=1)
[[1, 0, 0],
[0, 0, 0],
[0, 0, 0]]
&gt;&gt;&gt; f(h=3, w=3, q=2)
[[1, 1, 0],
[0, 0, 0],
[0, 0, 0]]
&gt;&gt;&gt; f(h=3, w=3, q=5)
[[1, 1, 0],
[1, 1, 0],
[1, 0, 0]]
&gt;&gt;&gt; f(h=4, w=4, q=10)
[[1, 1, 1, 1],
[1, 1, 1, 0],
[1, 1, 0, 0],
[1, 0, 0, 0]]
</code></pre>
<p>(I will use it for a batch of image with the shape of <code>[b, c, h, w]</code>.)</p>
<p>How can I implement the above function in an efficient form by using PyTorch? Or can I obtain a similar efficient function in another way?</p>
<hr>
<p><strong>P.S.</strong> I know I can implement the above function by some for-loops, but the obtained function may be too inefficient for me. Since I want to generate multiple masks in each network forwarding process, a real-time fast function implementation is appealing for me.</p>

111414: <p>It seems that I resolved this problem by myself. The following code seems to be competent for meeting my requirements:</p>
<pre><code class="lang-python">import torch, random
from functools import lru_cache

@lru_cache(maxsize=None)
def get_zigzag_ordered_indices(h=8, w=8):
    x, y = [], []
    x1, x2, y1, y2 = 0, 0, 0, 0
    flag = True
    while x2 &lt; h or y1 &lt; w:
        if flag:
            x = [*x, *range(x1, x2 - 1, -1)]
            y = [*y, *range(y1, y2 + 1)]
        else:
            x = [*x, *range(x2, x1 + 1)]
            y = [*y, *range(y2, y1 - 1, -1)]
        flag = not flag
        x1, y1 = (x1 + 1, 0) if (x1 &lt; h - 1) else (h - 1, y1 + 1)
        x2, y2 = (0, y2 + 1) if (y2 &lt; w - 1) else (x2 + 1, w - 1)
    return x, y

@lru_cache(maxsize=None)
def get_zigzag_truncated_indices(h=8, w=8, q=6):
    if random.randint(0, 1):
        x, y = get_zigzag_ordered_indices(h, w)
    else:
        y, x = get_zigzag_ordered_indices(w, h)
    return x[:q], y[:q]

@lru_cache(maxsize=None)
def get_mask_DCT(h=8, w=8, q=6, device='cuda'):
    mask_DCT = torch.zeros(1, 1, h, w, device=device)
    DCT_x, DCT_y = get_zigzag_truncated_indices(h, w, q)
    mask_DCT[:, :, DCT_x, DCT_y] = 1
    return mask_DCT

print(get_mask_DCT())
</code></pre>
<hr>
<p>Is there a more efficient PyTorch implementation?</p>
How to keep the same randomness in dropout？

klion: <p>I was hoping that the following two scripts would get the same model, but it seems that due to the randomness of the dropout, I am not able to get the same model.</p>
<p>The code abstraction is as follows</p>
<pre><code class="lang-auto"># script 1
for t in range(100):
    vec = torch.randn([1,500])
    s = compute_score(method="std", vec=vec)
    
    some_functions(s)

    TrainNetwork(model)

# script 2
for t in range(100):
    vec = torch.randn([1,500])
    s = compute_score(method="NoE", vec=vec)
    
    some_functions(s)

    TrainNetwork(model)
</code></pre>
<p>In the function compute_score I set <code>nu=0</code>, so the two different methods, “NoE” and “std”, should give the same result.</p>
<pre><code class="lang-auto">import torch
from torch import nn

class NetworkFC2(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.fc1 = nn.Linear(500, 100)
        self.fc2 = nn.Linear(100, 1)
        self.dropout = nn.Dropout(p = 0.2)
    
    def forward(self, vec):
        """
        vec.shape = (1, 500)
        """
        o1 = self.dropout(self.fc1(vec))
        return self.fc2(o1)

torch.manual_seed(123)

model = NetworkFC2()

def compute_score(method, vec):
    if method == "std":
        model.train()
        score_v = torch.zeros(20)
        for i in range(20):
            score_v[i] = model(vec)
        std = score_v.std()

        model.eval()
        s0 = model(vec)

        nu = 0
        score = s0 + nu * std
    
    elif method == "NoE":
        model.eval()
        score = model(vec)

    return score        

def TrainNetwork(model):
    pass

def some_functions(score):
    pass
</code></pre>
<p>But since the number of dropout runs in the “std” method is more than in the “NoE” method, the randomness of the dropout is different and I can’t get the same model.</p>
<p>Is there a way to solve this problem?</p>
<p>Thanks in advance for any help/suggestion.</p>

ptrblck: <p>You are right that the different number of calls into the pseudorandom number generator is causing the difference in the outputs.<br>
One approach would be to sample the dropout masks manually and select the ones for the corresponding iteration, another one could be to explicitly seed the code before the dropout execution with a specific seed.<br>
However, since your use case forces you to use different number of iterations, it would be interesting to learn more about the actual expectation.<br>
Is one method creating a single output sample in the loop while the other one uses a batched approach?</p>

klion: <p>Thanks for your answer, in fact, I am doing research on uncertainty in networks and would like to use the outputs of dropout to study it.</p>
Installing both tensorflow and pytorch with gpu support

ahabo8: <p>hello.</p>
<p>i want to install both tf and pt on my rtx 3060 laptop, with windows 10. but i don’t know the most efficient approach to achieve this goal. there are three approaches that come to my mind:</p>
<ol>
<li>
<p>first i go to this <a href="https://www.tensorflow.org/install/source#gpu" rel="noopener nofollow ugc">link</a> and check for cuda and cudnn versions. i install cuda 11.2 and cudnn 8.1 locally (after downloading the respective files from their sources from nvidia). then, i go <a href="https://pytorch.org/" rel="noopener nofollow ugc">here</a> and check for versions. i choose cuda 11.3 and pip install with this command:<br>
<code>pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113</code><br>
will my laptop run both tf and pt packages on gpu? do i have to install cuda 11.3 with pip install command for pytorch?</p>
</li>
<li>
<p>i pip install pytorch with the help of the command above. if i do this, how am i going to install tf?</p>
</li>
</ol>
<p>something to note here. i don’t want to have multiple cuda versions. i want one version which both tf and pt support it. please help me install both these packages with gpu support efficiently. because i don’t know how to install.</p>
<p>thank you very much.</p>

ptrblck: <aside class="quote no-group" data-username="ahabo8" data-post="1" data-topic="160087">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ahabo8/48/46127_2.png" class="avatar"> ahabo8:</div>
<blockquote>
<p>something to note here. i don’t want to have multiple cuda versions. i want one version which both tf and pt support it.</p>
</blockquote>
</aside>
<p>This might not be possible if you install the pre-built binaries as it depends how each framework uses the CUDA runtime.<br>
E.g. the PyTorch binaries ship with their own CUDA runtime, which you are selecting during the install process. If you are installing the pip wheels, the PyTorch <code>lib</code> folder will ship with all ibraries while the conda binaries will install the <code>cudatoolkit</code> or <code>cuda</code> conda package. If TF provides conda binaries with the same dependency, both frameworks might be able to share these libs.<br>
However, I also don’t know if TF actually ships with the CUDA runtime or if your system libraries will be used.</p>
<p>If you really depend on the same CUDA toolkit/runtime for both frameworks, the best approach might be to install a CUDA toolkit locally (you wouldn’t need to do it to run PyTorch-only) and build both frameworks from source.</p>

ahabo8: <p>thank you for your response.<br>
so should i install cuda 11.3 with pip command while installing pt after installing cuda 11.2 for tf?</p>
Energy consumption of convolutional neural networks

Saurav_Pawar: <p>Hello!</p>
<p>I want to measure the energy and power consumed by my CNN. Can anyone please help me or suggest any ways of achieving it ?</p>
<p>Thank you in advance!</p>

tom: <p>One very simplistic way - assuming you use a NVidia GPU - could be to repeately call nvidia-smi and record the power consumption reported there and then “integrate” (i.e. if you measure every 100ms, sum and divide by 10 to get the energy in Joule). This is less then scientific and neglects data loading or more generally anyting outside the GPU in addition to inaccuracies of the (indicative) reporting.</p>
<p>Best regards</p>
<p>Thomas</p>

Saurav_Pawar: <p>Okay, thanks for your response : )<br>
But is there any way that I can do it using code like any package or framework that has this functionality ?</p>
Implement OPTIMIZER scheduler in Pytorch Lightning

hoang: <p>Hi , I try to implement the optimizer in this <a href="https://github.com/facebookresearch/swav/blob/main/main_swav.py" rel="noopener nofollow ugc">code</a>. I also specify some variable :</p>
<pre><code class="lang-auto">base_lr = 4.8
final_lr = 0
warmup_epochs = 10
start_warmup = 0 
epochs = 100 
weight_decay = 1e-6
params = model.parameters()
optimizer = torch.optim.SGD(
    params,
    lr= base_lr,
    momentum=0.9,
    weight_decay= weight_decay,
)
optimizer = LARC(optimizer=optimizer, trust_coefficient=0.001, clip=False)
warmup_lr_schedule = np.linspace(start_warmup, base_lr, len(train_loader) * warmup_epochs)
iters = np.arange(len(train_loader) * (epochs - warmup_epochs))
cosine_lr_schedule = np.array([final_lr + 0.5 * (base_lr - final_lr) * (1 + \
                     math.cos(math.pi * t / (len(train_loader) * (epochs - warmup_epochs)))) for t in iters])
lr_schedule = np.concatenate((warmup_lr_schedule, cosine_lr_schedule))
</code></pre>
<p>When training begins, the learning rate is update base on the epoch and iteration numbers :</p>
<pre><code class="lang-auto">for epoch in range(epochs + 1):  # loop over the dataset multiple times
    total_loss = 0.0
    for it, data in enumerate(train_loader):
        # get the inputs; data is a list of [inputs, labels]
        iteration = epoch * len(train_loader) + it
        for param_group in optimizer.param_groups:
            param_group["lr"] = lr_schedule[iteration]
</code></pre>
<p>I wonder can we implement this optimizer with <strong>configure_optimizer</strong> function  in <strong>Pytorch-Lightning</strong> like this :</p>
<pre><code class="lang-auto">def configure_optimizers(self):
    return Adam(self.parameters(), lr=1e-3)
</code></pre>
<p>If possible, can u please give me some pseudo-code for this implementation? I have read many documents about this function but still can’t understand it.</p>
<p>Thanks for reading.</p>