akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is the difference between full_backward_pre_hook and full_backward_hook?

Junhyeok_Lee: <p>hi. I am dealing with <code>register_full_backward_pre_hook</code> and <code>register_full_backward_hook</code> to change gradient values while backward stages.</p>
<p>what I am trying is like below<br>
receive input gradient → multiply value to a input gradient(backward pre hook) → calculate param gradient w.r.t input gradient → devide value to a output gradient(backward hook)</p>
<p>is this a right understanding? is this a correct way to do this? thanks in advance</p>
Create a tensor using torch::from_blob failed

tiberius: <p>I use torch::from_blob in C++ to create a tensor, and failed with "The deleter and context arguments are mutually exclusive.“<br>
it seems that the it is because the tensorMaker is with both deleter and context which i don’t know why.<br>
my code is like below:<br>
‘’’<br>
x = torch::from_blob(data, {shape0, shape1}, torch::TensoorOptions().dtype(torch::kF32));<br>
‘’’<br>
p.s. it seems the error occurs when i update torch from 1.10 to 1.13. And only when building with release will this happen<br>
i’d be grateful if anyone can give some advice</p>

ptrblck: <p>Could you try to create the tensor first using <code>torch::from_blob</code>, <code>clone</code> it afterwards, and finally transform to the desired <code>dtype</code>? <code>torch::from_blob</code> will share the same memory coming from <code>data</code> and in previous versions maybe providing additional options would create a copy of the data for you while it’s now disallowed.</p>

tiberius: <p>hanks for reply!<br>
in the code above, <code>data</code> is from a <code>vector&lt;float&gt;</code>, so i use <code>tensorOptions.dtype(torch::kF32)</code> not to change the dtype of the data but to specify the dtype of the data. i suppose this will not lead to copy of the data.<br>
additionally, i tried your advice, but got the same failure, i’m not sure if i’ve done the correct thing as you suggest.</p>
<pre><code class="lang-auto">x = torch.from_blob(data, {shape0, shape1});
xc = x.clone();
xc.toType(torch::kI64);
</code></pre>

tiberius: <p>supprise to find only calling <code>from_blob(void*, at::IntArrayRef, const at::TensorOption &amp;)</code> alse leads to the same failure. code is like:</p>
<pre><code class="lang-auto">std::vector&lt;float&gt; data(shape0 * shape1 * 10, 1); // only to ensure the data size is enough
from_blob(data.data(), {shape0, shape1});
</code></pre>

ptrblck: <p>Ah OK, in this case I’ve understood the error message wrong as the <code>at::TensorOptions</code> argument it used to specify how to <em>interpret</em> the data. The error is raised from here](<a href="https://github.com/pytorch/pytorch/blob/fbc803df0c420db84429c51599f4fa4354b4493f/aten/src/ATen/templates/Functions.cpp#L14-L16" class="inline-onebox">pytorch/Functions.cpp at fbc803df0c420db84429c51599f4fa4354b4493f · pytorch/pytorch · GitHub</a>) but I’m unsure how both arguments are set.<br>
Are you seeing the same issue using e.g.:</p>
<pre><code class="lang-auto">float data[] = {1, 2, 3, 4, 5, 6};
auto f = from_blob(data, {1, 2, 3});
</code></pre>

tiberius: <p>I’ve found that the error is because that the directory of build is not cleared <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"> , something( perhaps cmake cache etc.) generated when i used the old version of libtorch is still there. thank you anyway.</p>
Running Variance update in torch.nn.functional.batch_norm

hello-fri-end: <p>According to the documentation for batch_norm, running_mean and running_var are updated according to the following equation:</p>
<pre><code class="lang-auto">running_mean = momentum * batch_mean + (1 - momentum) * running_mean
running_var = momentum * batch_var + (1 - momentum) * running_var
</code></pre>
<p>Consider the following input:</p>
<pre><code class="lang-auto"> a = torch.tensor([[-1., -1.],
                      [-1., -1.],
                      [-1.,  1.]])
mean = torch.ones((2))
var = torch.zeros(2)
</code></pre>
<p>Running <code>torch.nn.functional.batch_norm(a, mean, var, training = True                               )</code> updates the var to <code>tensor([0.0000, 0.1333])</code> which does not match the formula above.</p>
<p>According the the above formula the result should be equivalent to:</p>
<pre><code class="lang-auto">batch_var = torch.var(a, unbiased= False, axis = 0)
print(0.1 * batch_var + (1 - 0.1) * var)
</code></pre>
<p>which returns <code>tensor([0.0000, 0.0889])</code>.</p>
<p>EDIT: If we do torch.var(a, unbiased= True, axis = 0), we get the same result as applying batch_norm. However, according to the documentation “The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False).”</p>

ptrblck: <p>The <code>running_var</code> is updated via the unbiased variance as seen in my <a href="https://github.com/ptrblck/pytorch_misc/blob/31ac50c415f16cf7fec277dbdba72b9fb4d732d3/batch_norm_manual.py#L68">manual implementation</a> and this code snippet:</p>
<pre><code class="lang-python">a = torch.tensor([[-1., -1.],
                  [-1., -1.],
                  [-1.,  1.]])
mean = torch.ones((2))
var = torch.zeros(2)
out =  torch.nn.functional.batch_norm(a, mean, var, training = True ) 
print(var)
# tensor([0.0000, 0.1333])

n = a.numel() / a.size(1)
batch_var = torch.var(a, unbiased=False, axis=0)
print(0.1 * batch_var * n / (n-1) + (1 - 0.1) * torch.zeros(2) )
# tensor([0.0000, 0.1333])
</code></pre>

hello-fri-end: <p>Thanks for the information. <a class="mention" href="/u/ptrblck">@ptrblck</a> The documentation is a bit misleading though as it says “The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False).”. One thinks that the update is also done via biased variance. Maybe the documentation should explicitly mention that’s not the case.</p>

ptrblck: <p>Yes, this might be a good improvement. Would you be interested in submitting a PR with an improved documentation?</p>

hello-fri-end: <p>Sure, I’ll create a PR updating the documentation to mention this. Thanks.</p>
Unexpected error:TypeError: sub(): argument 'other' (position 1) must be Tensor, not NoneType

Takano: <p>Due to the limitations of the learning environment, it is necessary to write code to save the learning progress, load it, and resume learning. However, an error occurs and learning cannot be performed.</p>
<pre><code class="lang-auto">File "c:/Users/.../Desktop/PINN_ACSL/NN.py", line 531, in &lt;module&gt;
    optimizer.step(PINN.closure)
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\optim\optimizer.py", line 140, in wrapper
    out = func(*args, **kwargs)
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\optim\lbfgs.py", line 352, in step
    y = flat_grad.sub(prev_flat_grad)
TypeError: sub(): argument 'other' (position 1) must be Tensor, not NoneType
</code></pre>
<p>The optimizer used is <code>torch.optim.LBFGS</code> , and use <code>torch.save(optimizer.state_dict(), PATH)</code> for saving. <code>optimizer.load_state_dict(torch.load(PATH))</code> is used for loading.</p>
Why two multiplication produce different result?

Hyeonuk_Woo: <p>Hello, I think I did the same multiplication, but the result is different.</p>
<pre><code class="lang-auto">import torch

#Self attention with 100-tokens
n_token=100
c_hidden=128
query=torch.randn(n_token,c_hidden)
key  =torch.randn(n_token,c_hidden)
attn_matrix= torch.matmul( query, key.transpose(0,1)) #(query_token_index, key_token_index)

#Graph frame work : every 50-tokens are fully connected with each other
edge_index=torch.ones(n_token,n_token).nonzero(as_tuple=True)# all token is connected with all token 


#edge_index = ( source_token_index, destination_token_index )
query_graph=query[edge_index[0]] # query from source_token_index
key_graph=key[edge_index[1]] # key from destination_token_index

attn_graph_1= (query_graph*key_graph).sum(dim=-1)
attn_graph_2= torch.matmul(query_graph[:,None,:],key_graph[:,:,None]).squeeze()

diff_1=torch.abs(attn_matrix[edge_index] - attn_graph_1).sum()
diff_2=torch.abs(attn_matrix[edge_index] - attn_graph_2).sum()

print (diff_1) #0.0157
print (diff_2) #0.0089
</code></pre>
<p>I have some idea why ‘diff_1’ is not 0, but I don’t know why ‘diff_2’ is not 0. Does anybody have idea?</p>

eqy: <p>Do the differences become smaller when e.g., using a double precision data type? In general even if two expressions are mathematically equivalent, if they are dispatched to different (or even nondeterministic) implementations then there are no guarantees the results would be identical.</p>

Hyeonuk_Woo: <p>Using double precision data type, diff_1 become much smaller, and diff_2 become 0. Thanks!</p>
I want to use a optimizer to optimize a part of my input，how should i do？

Will97: <p>for example：<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fbbaeefb58cf02585a11d98eb4a4443ccb881ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/8fbbaeefb58cf02585a11d98eb4a4443ccb881ee" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fbbaeefb58cf02585a11d98eb4a4443ccb881ee_2_690x202.png" alt="image" data-base62-sha1="kvwo3q6mrfNydkNnR2oDqc1ClOu" width="690" height="202" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fbbaeefb58cf02585a11d98eb4a4443ccb881ee_2_690x202.png, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fbbaeefb58cf02585a11d98eb4a4443ccb881ee.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fbbaeefb58cf02585a11d98eb4a4443ccb881ee.png 2x" data-dominant-color="F5F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">931×273 25.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
its cause  Error：<br>
· raise ValueError(“can’t optimize a non-leaf Tensor”)<br>
ValueError: can’t optimize a non-leaf Tensor·<br>
Can somebody help me？</p>

Joschka: <p>Indexing your input is already registered as an operation by autograd, hence the tensor you pass to Adam is a “non-leaf” tensor in the internal computation graph.</p>
<p>As a workaround you should consider doing the indexing part at a later stage, e.g.,</p>
<pre><code class="lang-python">import torch
from torch.nn import Linear, Sequential, Flatten
from torch.optim import Adam


model = Sequential(
    Flatten(1, -1),
    Linear(32 * 32, 1),
)

model_input = torch.rand(1, 32, 32)
optimization_mask = torch.zeros(1, 32, 32, dtype=torch.bool)
optimization_mask[:, 27:32, 27:32] = True

num_input_params = int(optimization_mask.sum().item())
x_optimized = torch.rand(num_input_params, requires_grad=True)
optim = Adam([x_optimized], lr=1e-1)

for _ in range(1000):
    x = model_input.clone()
    x[optimization_mask] = x_optimized
    loss = (model(x) - 10)**2
    optim.zero_grad()
    loss.backward()
    optim.step()

# should equal approx. 10.0
print(model(x).detach())
</code></pre>

Will97: <p>Thank you！Thats help me a lot！</p>
RuntimeError: mean(): could not infer output dtype. Input dtype must be either a floating point or complex dtype. Got: Byte

Chao_Fan: <p>code: - mean and std</p>
<p>import torch<br>
from customDataset import CatsAndDogsDataset, ToTensor</p>
<p><span class="hashtag">#load</span> data<br>
dataset = CatsAndDogsDataset(csv_file =‘CatsAndDogsDataset.csv’,<br>
root_dir = ‘pictures_resized’,<br>
transform = ToTensor()<br>
)</p>
<p>train_loader = torch.utils.data.DataLoader(dataset = dataset,<br>
batch_size = 8,<br>
shuffle = False,<br>
)</p>
<p>def get_mean_and_std(loader):<br>
mean = 0,<br>
std = 0,<br>
total_images_count = 0<br>
for images, _ in loader:<br>
image_count_in_a_batch = images.size(0)<br>
images = images.view(image_count_in_a_batch, images.size(1), -1)<br>
<span class="hashtag">#print</span>(images.shape)<br>
mean += images.mean(2).sum(0)<br>
std += images.std(2).sum(0)<br>
total_images_count += image_count_in_a_batch</p>
<pre><code>    mean /= total_images_count
    std /= total_images_count

    return mean, std
</code></pre>
<p>mean, std = get_mean_and_std(train_loader)<br>
print(mean)<br>
print(std)</p>
<p>code - customDataset<br>
import os<br>
import pandas as pd<br>
import torch<br>
from torch.utils.data import Dataset<br>
from skimage import io</p>
<p>class CatsAndDogsDataset(Dataset):<br>
def <strong>init</strong>(self, csv_file, root_dir, transform = None):<br>
self.annotations = pd.read_csv(csv_file)<br>
self.root_dir = root_dir<br>
self.transform = transform</p>
<pre><code>def __len__(self):
    return len(self.annotations)

def __getitem__(self, index):
    img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])
    image = io.imread(img_path)
    y_label = torch.tensor(int(self.annotations.iloc[index, 1]))

    if self.transform:
        image = self.transform(image)

    return (image, y_label)
</code></pre>
<p>class ToTensor():<br>
def <strong>call</strong>(self, image):<br>
inputs = image<br>
return torch.from_numpy(inputs)</p>

ptrblck: <p>Based on the error message you would need to use floating point <code>dtype</code> to calculate the <code>mean</code> of a tensor as seen here:</p>
<pre><code class="lang-python">images = torch.randint(0, 256, (3, 224, 224)).byte()

# fails
images.mean(2)
# RuntimeError: mean(): could not infer output dtype. Input dtype must be either a floating point or complex dtype. Got: Byte

# works
images.float().mean(2)
</code></pre>

Chao_Fan: <p>i tried it , but it still has error:<br>
mean += images.float().mean(2).sum(0)<br>
TypeError: can only concatenate tuple (not “Tensor”) to tuple</p>

ptrblck: <p>Initialize the <code>mean</code> with <code>= 0</code> instead of a tuple <code>= 0,</code> (note the trailing comma).</p>

Chao_Fan: <p>thanks, it solved my issue, but the output is not in average form. any idea ?</p>
Why does torch.mp.spawn take longer with larger datasets?

Benjamin_Shwartzman: <p>I am currently training a model using DDP using 4 GPUs on 32 vcpus.</p>
<p>I have constructed my dataset to be a list of strings where each string represents a local path where the numerical data is stored.</p>
<p>When I run my code on a small dataset (e.g. 13 paths), torch.mp.spawn is extremely fast. The 4 processes are spawned extremely quickly and training beginnings. However, when I try to run my code on the full dataset (size of 2000), torch.mp.spawn takes multiple hours to complete. I’m confused as to why this is the case. I don’t understand why mp.spawn is dataset size dependent since the dataset is not passed as an argument for the spawn method. Any insight to this problem would be greatly appreciated.</p>
Indexing with tensors

gozdi: <p>Hi I came across an operation when someone was indexing a tensor with two other tensors and I would like to learn more about it, can anyone recommend any resources?</p>
<p>Here’s an example of what I’m talking about:</p>
<pre><code class="lang-auto">query_states # shape [10, 50, 128]
dim_for_slice = torch.arange(query_states.size(0)).unsqueeze(-1) #shape [10, 1]
M_top #shape [10, 8] with values in range 0, 49
Q_reduce = query_states[dim_for_slice, M_top] #shape [10, 8, 128]
</code></pre>
<p>what is the logic behind last operation?</p>

ptrblck: <p>PyTorch sticks to numpy’s indexing mechanism and <a href="https://numpy.org/doc/stable/user/basics.indexing.html#advanced-indexing">these docs</a> explaining advanced indexing might be helpful.</p>
Ensemble model training memory footprint

gessha: <p>I have 5 identical MLP models that I want to train in parallel on a single GPU and they are relatively small. Each has its own dataset so there’s no overlap in data or model parameters.</p>
<p>I load both the model and dataset in a for loop, assign them to an object and I append the object to an object list. During optimization, I iterate through these objects and I run a forward/backward pass+step through the model.</p>
<p>In nvidia-smi, I see only the memory footprint of a single model. Is this a bug where I’m training only one model or a Pytorch optimization of some sort?</p>
<p>I stepped through the optimization loop with pdb and all the model object hashes seem different, the datasets are different. What else can I check to confirm I’m optimizing through the right model/dataset?</p>
<pre><code class="lang-auto"># initialize data
dataset_array = []
for i in datasets:
    ds_obj = {}
    ds = Dataset(**kwargs)
    ds_obj['dataset'] = ds
    dataset_array.append(ds_obj)

# initialize models
model_array = []
for i in models:
    model_obj = {}
    model = MyModel(**kwargs)
    optimizer = Adam(**kwargs)
    model_obj['model'] = model
    model_obj['optimizer'] = optimizer
    model_array.append(model_obj)
    
# optimize
for i in range(max_iterations):
    for idx, model_obj in enumerate(model_array):
        dataset = dataset_array[idx]['dataset']
        model = model_array[idx]['model']
        optim = model_array[idx]['optimizer']
        
        model_input, GT = dataset[i]
        model_output = model(model_input)
        loss = loss_fn(model_output, GT)
        
        optim.zero_grad()
        loss.backward()
        optim.step()
</code></pre>

ptrblck: <p>Your code looks fine and I see an increase in memory usage:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        # should use 4GB
        self.fc = nn.Linear(1024*32, 1024*32, bias=False)
        
    def forward(self, x):
        x = self.fc(x)
        return x

# make sure no memory is allocated
print(torch.cuda.memory_allocated()/1024**3)
# 0.0

# initialize models
model_array = []
for i in range(3):
    model_obj = {}
    model = MyModel().cuda()
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
    model_obj['model'] = model
    model_obj['optimizer'] = optimizer
    model_array.append(model_obj)
    print("iter {}, memory allocated {}GB".format(i, torch.cuda.memory_allocated()/1024**3))
    
# iter 0, memory allocated 4.0GB
# iter 1, memory allocated 8.0GB
# iter 2, memory allocated 12.0GB
</code></pre>
What is the use of 32 and yield statements in `__iter__()` function of RandomSampler class?

pararthdave: <p>I’m trying to customise the <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/sampler.py" rel="noopener nofollow ugc">RandomSampler</a> subclass to prevent samples below a certain provided loss threshold from going into training. I’m confused about the <code>__iter__()</code> function’s utility and hence unable to make appropriate modifications to it. The <code>__iter__()</code> function from source code looks like this:</p>
<pre><code class="lang-auto">def __iter__(self) -&gt; Iterator[int]:
        n = len(self.data_source)
        if self.generator is None:
            seed = int(torch.empty((), dtype=torch.int64).random_().item())
            generator = torch.Generator()
            generator.manual_seed(seed)
        else:
            generator = self.generator

        if self.replacement:
            for _ in range(self.num_samples // 32):
                yield from torch.randint(high=n, size=(32,), dtype=torch.int64, generator=generator).tolist()
            yield from torch.randint(high=n, size=(self.num_samples % 32,), dtype=torch.int64, generator=generator).tolist()
        else:
            for _ in range(self.num_samples // n):
                yield from torch.randperm(n, generator=generator).tolist()
            yield from torch.randperm(n, generator=generator).tolist()[:self.num_samples % n]
</code></pre>
<p>I was wondering how the iterator function actually works in choosing the samples each time. Also, what’s the utility of having <strong>32</strong> when replacement is true. The use of <strong>yield</strong> statements and how <strong>num_samples</strong> is calculated in this function. What happens when value of num_samples and n is same and how the RandomSampler handles such cases. As in my case, the value of n and num_samples seems to be same which is making it loop over only a single sample one-by-one.</p>
<p>Additionally, I also want to add a validation mode where the forbidden list (samples prevented from going into training) are checked for loss after every epoch and the higher loss samples are sent to training again.</p>

nivek: <p>This seems be a duplicate of <a href="https://discuss.pytorch.org/t/random-sampler-iterator-function/175450" class="inline-onebox">Random Sampler iterator function</a> so I will respond in the separate thread</p>
Equivalent multiplication for VJP of matrix valued function

vityou: <p>If I have a function like</p>
<pre><code class="lang-auto">f = lambda x: 2 * x
</code></pre>
<p>and I want to calculate the VJP when it is applied to a matrix, I can do</p>
<pre><code class="lang-auto">point = torch.tensor([[1.,2.,3.], [4.,5.,6.]])
v =  torch.tensor([[1.,2.,3.], [4.,5.,6.]])
vjp(f, point, v)[1]
# tensor([[ 2.,  4.,  6.],
#         [ 8., 10., 12.]])
</code></pre>
<p>But what would be the equivalent product involving the size (2, 3, 2, 3) Jacobian calculated from</p>
<pre><code class="lang-auto">jac = jacobian(f, torch.tensor([[1.,2.,3.], [4.,5.,6.]]))
</code></pre>
<p>Because various ways of multiplying the Jacobian and vector do not give the same result as VJP</p>
<pre><code class="lang-auto"># all return rank-4 tensors
jac @ v.T
v @ jac.T
v.T @ jac
jac.T @ v
</code></pre>

soulitzer: <p>It’s v.T @ jac, but you may need to flatten your v and Jacobian first, e.g. <code>v.reshape(1, 6) @ jac.reshape(6, 6)</code></p>
Datapipe RandomSplit with Shuffle produce splits with duplicates

Moust_Holmes: <p>I was testing the new datapipes and I wanted to use the random splits to split my datapipe into training testing and validation.however i found that when I shuffle before I do the random splits i get duplicate values in my training and validation which is far from ideal.  I could just shuffle the datapipes after splitting them but I was just wondering why this is happening since it a scary mistake to make to have a overlap of training and validation data</p>
<pre><code class="lang-auto">train, test, valid = IterableWrapper(range(10)).shuffle().random_split(total_length = 10, weights={"train": 0.4, "test": 0.3, "valid": 0.3}, seed=0)

print(list(train))

print(list(test))

print(list(valid))
</code></pre>
<pre><code class="lang-auto">[4, 7, 6, 5]
[8, 7, 0]
[5, 9, 2]
</code></pre>

ptrblck: <p>CC <a class="mention" href="/u/nivek">@nivek</a> Is calling <code>shuffle()</code> before <code>random_split</code> valid as it seems the pipe will be reshuffled before each spit thus creating duplicates?</p>

nivek: <p>Thanks for the ping! <a class="mention" href="/u/ptrblck">@ptrblck</a>’s description is correct. It is indeed re-shuffling before each split because the <code>random_split</code> is generated lazily, such that when you read a different group, it reads the source DataPipe again. In this case, it re-reads from <code>Shuffler</code>, whose ordering changes when it is re-read.</p>
<p><a class="mention" href="/u/moust_holmes">@Moust_Holmes</a> My quick thought on this is:</p>
<ol>
<li>Shuffling before random split should not impact the randomness of the final results, because <code>random_split</code> is fully random anyway.</li>
<li>If you are concerned about the ordering of the final output (because <code>random_split</code> preserves the sequential order, you can perform <code>shuffle</code> after <code>random_split</code>.</li>
<li>If the above doesn’t work for you, you can consider caching the results after the initial shuffle (i.e. <code>dp.shuffle().in_memory_cache()</code>) prior to <code>random_split</code>, but this will use some memory and materialize partial result, rather than done lazily like in 2.</li>
</ol>
<p>If none of these works, please let me know why and I am happy to discuss further.</p>
Explicitly forcing torch's MHA to use Flash Attention

neel_g: <p>How can we force torch to use new SDPA implementation in <code>torch.nn.multiheadattention()</code>?</p>
<p>I know that it is supposed to use it automatically, but I’d still prefer an explicit version.<br>
There’s also a ctxmanager, but I’m unsure where its supposed to be wrapped. Using it in the <code>foward()</code> doesn’t change anything, nor does simply stating:</p>
<pre><code class="lang-py">torch.backends.cuda.enable_flash_sdp(enabled=True)
</code></pre>
<p>somewhere in the code.</p>

soulitzer: <p>Curious why you want to do this since I’d imagine that it would produce incorrect results?</p>

neel_g: <p>Why would that be? The new SPDA doesn’t really tradeoff precision AFAIK.</p>

soulitzer: <p>I don’t know the specifics, but I’d imagine that the speed ups are enabled by making specific assumptions about the inputs, and if those assumptions are violated, correctness may not be guaranteed. If you have a use case that you’d think should be supported, but is not currently supported, maybe you can file an issue?</p>

neel_g: <aside class="quote no-group" data-username="soulitzer" data-post="4" data-topic="175499">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<p>maybe you can file an issue</p>
</blockquote>
</aside>
<p>Doesn’t seem worth it tbh. maybe <a class="mention" href="/u/ptrblck">@ptrblck</a> has some idea what I should do?</p>
Can `torch.compile` work for model with dynamic graphs?

Pab_Peter: <p>So I guess the <code>torch.compile</code> won’t work for model with dynamic graphs?</p>
<p>Say for the first 100 epochs, I am training an RNN with 3-steps loss for the future prediction. Then in the next 100 epochs, I am training the same RNN but I changed the unrolling steps to 100-steps. Just work like a fine tunning.</p>
<p>Would that work?</p>

soulitzer: <p>When it runs for the first time, it should produces guards that tell us under what conditions the compiled graph is valid. If it is run again with different number of steps, the guard check would fail, and  torch.compile should recompile the graph.</p>
Loss.backward() logic update!

pnnaman: <p>I want to try communication primitives other than the Allreduce that PyTorch distributed uses currently. I am trying to update the communication logic that is overlapped with backward computation.  What file do I need to make the changes to implement this change? Thanks.</p>

Yanli_Zhao: <p>If you are talking about PyTorch DistributedDataParallel API, here is the entry point <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/distributed.py at master · pytorch/pytorch · GitHub</a></p>
<p>meanwhile DDP comm hook is a good way to overide DDP common methods <a href="https://pytorch.org/docs/stable/ddp_comm_hooks.html" class="inline-onebox" rel="noopener nofollow ugc">DDP Communication Hooks — PyTorch 2.0 documentation</a></p>

pnnaman: <p>Hi <a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a>! Thanks for the prompt reply. I want to implement a parameter server using the <strong>torch.distributed.send</strong> and <strong>torch.distributed.recv</strong> instead of the current decentralized approach using <strong>torch.distributed.all_reduce</strong>. Is this something that will be possible by updating the communication hooks and <strong>torch.nn.parallel.DistributedDataParallel.register_comm_hook()</strong>? If so, do you have any tutorials in mind that might be of any help? Thank you again.</p>
Model_heplers.py in is_overridden > raise ValueError("Expected a parent")

parzival: <p>I am running on a new remote server a code that used to work on another remote server. I think I setup things in the same way, but when I run my training script, I get this error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “train_model.py”, line 146, in <br>
seq_len=seq_len, mname=mname)<br>
File “/home/ds/AI-projects/hss_bearing_mk2/hss_bearing_mk2/models/pytorch_models.py”, line 322, in train_test<br>
trainer.fit(model, datamodule=dm)<br>
File “/home/ds/anaconda3/envs/hss_bearings/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 552, in fit<br>
self._run(model)<br>
File “/home/ds/anaconda3/envs/hss_bearings/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 849, in _run<br>
self.config_validator.verify_loop_configurations(model)<br>
File “/home/ds/anaconda3/envs/hss_bearings/lib/python3.7/site-packages/pytorch_lightning/trainer/configuration_validator.py”, line 34, in verify_loop_configurations<br>
self.__verify_train_loop_configuration(model)<br>
File “/home/ds/anaconda3/envs/hss_bearings/lib/python3.7/site-packages/pytorch_lightning/trainer/configuration_validator.py”, line 49, in __verify_train_loop_configuration<br>
has_training_step = is_overridden(“training_step”, model)<br>
File “/home/ds/anaconda3/envs/hss_bearings/lib/python3.7/site-packages/pytorch_lightning/utilities/model_helpers.py”, line 45, in <strong>is_overridden</strong><br>
<strong>&gt;     raise ValueError(“Expected a parent”)</strong></p>
</blockquote>
<p>Here is the part of code that looks buggy for some reason:</p>
<pre><code class="lang-auto">    model = get_model(mname=mname)

    dm = DataModule(
        X_train=X_train,
        y_train=y_train,
        X_val=X_val,
        y_val=y_val,
        X_test=X_test,
        y_test=y_test,
        keys_train=keys_train,
        keys_val=keys_val,
        keys_test=keys_test,
        seq_len=seq_len,
        batch_size=batch_size,
        num_workers=4
    )
    # trainer.logger_connector.callback_metrics
    trainer.fit(model, datamodule=dm)
</code></pre>
<p>Is it something related to environment setup? Something overridden by something something??</p>
<p>EDIT: I tried to run my project locally in a newly created environment and I have the same error.</p>
<p>EDIT 2: My DataModule inherits from LightningDataModule</p>
<pre><code class="lang-auto">class DataModule(pl.LightningDataModule):
</code></pre>

parzival: <p>The problem was that model was inheriting from <code>nn.Module</code> instead of from <code>pl.LightningModule</code></p>

Arthur_Masson: <p>With <code>lightning</code> versions 2.0.0, use <code> import lightning.pytorch as pl</code> instead of <code>import pytorch_lightning as pl</code>.</p>
Use of Global Average Pooling in the Model

Bibarys_Mussagaliyev: <p>Hi there,</p>
<p>I am wondering about how am I supposed to use GAP (global average pooling) in some CNN/ViT model?</p>
<p>As far as I understand the working principle of GAP - it is the <strong>per-channel feature map aggregator (by averaging)</strong> and thus avoids the use of FC layer at the end. Where each channel aggregation corresponds to the class category. For example, I have (16, 256, 64, 64):(Batch, Channel, Width, Height) tensor of information before transition to the GAP. The GAP layer will average each of 256 64-by-64 maps into (16, 256, 1, 1) tensor in the end.</p>
<p>So my question is:</p>
<p>For pre-trained ImageNet models: ResNet, ViT, etc. what is the correct way to put GAP layer into these models? I mean, all of them have final FC (1000, 1) layer in the end, there is no other channels and feature maps to be aggregated.</p>

Bibarys_Mussagaliyev: <p>I will answer my own question. I hope someone will find it useful.</p>
<p>In fact, I was right about last FC layers of the ImageNet pre-trained models. They do only have 1000 nodes in the output, thus making it impossible to use GAP, BN and other stuff involving multiple channels.</p>
<p>Assuming for this example we use PyTorch based <a href="https://github.com/huggingface/pytorch-image-models" rel="noopener nofollow ugc">timm</a> library, I should have done the following:</p>
<pre><code class="lang-auto">import timm
model = timm.create_model('eca_nfnet_l1', pretrained=True)
</code></pre>
<p>Code above initializes pre-trained “NFNet” model, this model has different main sections within:</p>
<ul>
<li>stem</li>
<li>stages</li>
<li>final_conv</li>
<li>head</li>
</ul>
<p>First three (stem, stages, final_conv) are considered as the “feature extractors” pipeline and last (head) is the final FC layer in the end. These are used to abstract the information flow for any particular purpose, in my case it was: <em>add GAP and BN layers to some custom model.</em></p>
<p>Thus, in order to customize your model with additional layers, you should interact with “feature extractors” which output the deep layered feature maps. These maps can be used further for in your custom layers (not-trained).</p>
<pre><code class="lang-auto">a = model.stem(imgTensor)
a = model.stages(a)
a = model.final_conv(a)
print(a.shape)
### prints: "torch.Size([1, 3072, 13, 13])"
</code></pre>
<p>This last tensor of shape <code>[1, 3072, 13, 13]</code> is one we can use in further processing in your custom layers.</p>
Atomic operations with PackedAccessor32 in cuda kernel?

Wesley_Neill: <p>I have a cuda kernel for raycasting where one of the dimensions of the grid involves comparing a computed minimum to the “current minimum” value, and replacing it if smaller.</p>
<p>The cuda kernel is blazing fast, and I’m really happy it works but I realized there is a race condition in there, and it is evident in the graphic output. I’m wondering if PyTorch has any atomic operations I could use on my PackedAccessor32 object?</p>

ptrblck: <p>I would assume you could directly use <code>atomicMin</code> as it’s also used e.g. in <a href="https://github.com/pytorch/pytorch/blob/c5a4844085ea4db27912f5174be2585aebf7079a/aten/src/ATen/native/transformers/cuda/mem_eff_attention/attention_scaling_coefs_updater.h#L17"><code>ATen/native/transformers/cuda/mem_eff_attention/attention_scaling_coefs_updater.h</code></a>.</p>

Wesley_Neill: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>, I will try that out soon and see how it goes!</p>
How to use normalize numbers of three channels for one channel?

nima_pw: <p>Hi,</p>
<p>I have a function (<code>extract_single</code>) that extracts features from train images:</p>
<pre><code class="lang-auto">prnu = transforms.Compose([
    transforms.Resize((resize , resize)),
    transforms.Lambda(extract_single),
    transforms.ToTensor(),
    transforms.Normalize([0.4786, 0.4728, 0.4528], [0.2425, 0.2327, 0.2564]),
])
</code></pre>
<p>The result of <code>transforms.Lambda(extract_single)</code> is numpy file with one channel, while images are three channels.<br>
I have already obtained the normalized numbers for the images. But it is not applicable to one channel. Now I don’t know which number of <code>[0.4786, 0.4728, 0.4528], [0.2425, 0.2327, 0.2564]</code> should be used or I need to calculate another mean &amp; std for the one channel?</p>

ptrblck: <aside class="quote no-group" data-username="nima_pw" data-post="1" data-topic="175540">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e19adc/48.png" class="avatar"> nima_pw:</div>
<blockquote>
<p>or I need to calculate another mean &amp; std for the one channel?</p>
</blockquote>
</aside>
<p>This might be a valid approach, but it would also be interesting to know what exactly <code>extract_single</code> does as you might be able to reuse the same approach for the stats.<br>
E.g. if <code>extract_single</code> slices the tensor in a single channel you could use the corresponding value from the <code>mean</code> and <code>std</code> stats, too.</p>

nima_pw: <p><code>extract_single</code> function extracts prnu noise from images:</p>
<pre><code class="lang-auto">def extract_single(im: np.ndarray,
                   levels: int = 4,
                   sigma: float = 5,
                   wdft_sigma: float = 0) -&gt; np.ndarray:
    """
    Extract noise residual from a single image
    :param im: grayscale or color image, np.uint8
    :param levels: number of wavelet decomposition levels
    :param sigma: estimated noise power
    :param wdft_sigma: estimated DFT noise power
    :return: noise residual
    """

    W = noise_extract(im, levels, sigma)
    W = rgb2gray(W)
    W = zero_mean_total(W)
    W_std = W.std(ddof=1) if wdft_sigma == 0 else wdft_sigma
    W = wiener_dft(W, W_std).astype(np.float32)

    return W
</code></pre>

nima_pw: <p>I want to do something like this (passing normalized image to the <code>single_extract</code> function to extract noises (features)):</p>
<pre><code class="lang-auto">my_transform = transforms.Compose([
    transforms.Resize((128 , 128)),
    transforms.ToTensor(),
    transforms.Normalize([0.4786, 0.4728, 0.4528], [0.2425, 0.2327, 0.2564]),
    transforms.ToPILImage(),
    transforms.Lambda(noise_extract),
    transforms.ToTensor()
])
</code></pre>
<p>It works, but I think this way causes increasing processing because of 2 times ToTensor() and because of the addition of ToPILImage(). Is there a better way?</p>
Pytorch.load() error:No module named 'model'

HaimianYu: <p>when i ues pytorch.load()  , it give an error : No module named ‘model’</p>
<p>import torch<br>
path = “D:\python\my_ML\model\resume.49.pkl”<br>
LM_model = torch.load(path)</p>

albanD: <p>Hi,</p>
<p>First, you should not serialize models but just their <code>state_dict()</code> to avoid such problem. Then you can recreate the model and <code>load_state_dict()</code> into it to get all the weights back.</p>
<p>This is a problem of python serialization, you should have exactly the same <code>import</code>s as when you saved the model when loading. You should import the <code>model</code> module the same way as it was done when you saved.</p>

HaimianYu: <p>Thank you very much. It has been done by the way you said. Since the model needs to be initialized by passing in a ‘’ data ‘’ object that contains parameters to build the model, I save the model directly. If only save the parameters, I need to create a ‘’ data ‘’ object when you build the model. It’s troublesome. Is there any good way?</p>

albanD: <p>I am not sure to understand in details what you do here.<br>
But maybe you can make this <code>data</code> object a state dict? That way you just need to save that, then load it and build your new model with it?</p>

meshghi: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, I’m using someone else’s repo as a module in my code, and thus I have to change the Python imports to relative imports, which gives me this model load error for the reason that you mentioned here. I was wondering if there is a way that I could still load the checkpoint given that I need to change the imports?</p>

Anandu_KB: <p>Hi <a class="mention" href="/u/meshghi">@meshghi</a> I’m facing similar issue, got any solutions?</p>
Strange oom when compiling aten/src/ATen/native/cuda/Sort.cu

10340505a: <p>When I’m compiling caffe2 from the pytorch source, it dies consistently at <code>aten/src/ATen/native/cuda/Sort.cu</code><br>
Error message/problem:</p>
<pre><code class="lang-auto">LLVM Error: Out of memory
nvcc error: 'cicc' died due to signal 6
</code></pre>
<p>But the system RAM (28GiB) was never filled, and there is a lot of swap (64GiB)<br>
What I have tried:<br>
Unsetting Architecture: no change<br>
Compile that single file: no change<br>
Add <code>-G</code> and compile that single file: success, but slower<br>
Add <code>-G -dopt=on</code> and compile that single file: same as first two<br>
Question:<br>
How do I let <code>-G</code> only be passed if <code>aten/src/ATen/native/cuda/Sort.cu</code> is the file being compiled? Or even better, is there a fix that doesn’t require removing all optimizations, which <code>-G</code> alone does?<br>
System:<br>
Gentoo Linux<br>
Build tool:<br>
Cmake (hooked by portage in non-single-file compiles)</p>

VincentLeeMax: <p>What gpu card and cuda version do you use?<br>
I solved this problem by changing cuda version from 12.0 to 11.8.</p>

10340505a: <p>I didn’t, and thanks for the advice</p>
Confusion with nn.CrossEntropyLoss

lkm9983: <p>I think I’m confused between the cross-entropy loss and the implementation of nn.CrossEntropyLoss. As far as I know, the general formula for cross-entropy loss looks like this.<br>
L(y,s) = - sigma(i=1 to c) {yi * log(si)}   (where si is the output of softmax)</p>
<p>However, when I see the documentation for nn.CrossEntropyLoss, it looks like there is no ‘yi’ value (the one-hot vector indicating the target class).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/9/39f516e133540cea3fb1679756e9d89993963293.png" data-download-href="https://discuss.pytorch.org/uploads/default/39f516e133540cea3fb1679756e9d89993963293" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39f516e133540cea3fb1679756e9d89993963293_2_690x73.png" alt="image" data-base62-sha1="8gIjFZS1Q2vGCCmmIpGY2GHSlZV" width="690" height="73" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39f516e133540cea3fb1679756e9d89993963293_2_690x73.png, https://discuss.pytorch.org/uploads/default/original/3X/3/9/39f516e133540cea3fb1679756e9d89993963293.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/9/39f516e133540cea3fb1679756e9d89993963293.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1018×108 7.93 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can anyone help me solve my confusion?"</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/cross-entropy-loss-clarification/103830/2">This example code</a> might be helpful which shows how the target is used to index the logits.</p>
Can we use the library kornia to learn data augmentations?

ArnolFokam: <p>Hi,</p>
<p>I have just discovered the Kornia library which is apparently a differentiable computer vision library for PyTorch. Since the computer vision operations in kornia are differentiable, Can we use Kornia to learn data augmentation during the training of a neural network or it’s just meant such that these operations can run on a GPU.</p>
<p>Example: If I use kornia’s augmentation module with randomly  initialized parameters. Can I train the network in such a way where these parameters can be updated during backprop?</p>

Raman1121: <p>Hi, did you come across any resource for this? I am also looking for something similar.</p>

Lelouch1: <p>You probably can.</p>
<p>An alternative way would also be to use a Spatial Transformer Network (STN),  which is also already implemented in Pytorch : <a href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Spatial Transformer Networks Tutorial — PyTorch Tutorials 2.0.0+cu117 documentation</a></p>
<p>This is a differentiable module which learns a geometrical transformation that will make the network more robust, if needed, to the encountered transformations (rotation, affine …).</p>
Can I use pytorch2.0 with cuda11.6?

Li_Rong: <p>because I use a high performance clustering and its highest cuda version is 11.6, and I cannot install cuda myself. So I want to ask if I can install pytorch2.0 under cuda version 11.6?</p>

ptrblck: <p>Yes, since the PyTorch binaries ship with their own CUDA libraries and you locally installed CUDA 11.6 toolkit will only be used if you build PyTorch from source or custom CUDA extensions.</p>

Li_Rong: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="175475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>build PyTorch from source or custom CUDA extensions.</p>
</blockquote>
</aside>
<p>thank you. could you give me some example that how I build pytorch from custom CUDA exension?</p>

ptrblck: <p>You can take a look at <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">this tutorial</a> but note that it’s not necessary to build and use custom extensions.<br>
What I meant in my previous post is that your locally installed CUDA toolkit will not be used to execute any plain PyTorch code, but only if you build PyTorch from source or a custom CUDA extension (which is optional).</p>
How to combine torch.compile with cudnn.benchmark, torch.utils.checkpoint and nn.DataParallel?

rjhan: <p>Does torch.compile already include these tricks (like cudnn.benchmark and torch.utils.checkpoint) ?</p>
<p>Should I first wrap the model with torch.compile or nn.DataParallel?</p>
<p>I can’t find it in the document.</p>
<p>I would be very grateful if someone could answer these questions or tell me where to find the answers.</p>
Confuse about transform PILToTensor and ToTensor

Anastasia_Berlianna: <p>hi, i’m working with rgb images of diabetic retinopathy dataset. since i want to train my unet model, i have to do transform first, but i confuse about which transform should i use. i’ve read the documentation of transform ToTensor and it says that it’ll return [0,1] image, so it’ll be binary input to the model? which one should i use? thanks!</p>

ptrblck: <aside class="quote no-group" data-username="Anastasia_Berlianna" data-post="1" data-topic="175482">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anastasia_berlianna/48/58506_2.png" class="avatar"> Anastasia_Berlianna:</div>
<blockquote>
<p>transform ToTensor and it says that it’ll return [0,1] image, so it’ll be binary input to the model?</p>
</blockquote>
</aside>
<p>No, <code>transforms.ToTensor()</code> will return a normalized tensors with values in the range <code>[0, 1]</code>.<br>
<code>transforms.PILToTensor()</code> will return a tensor using the same <code>dtype</code> and value range as the original input image.</p>

Anastasia_Berlianna: <p>ah get it! which one is better to use for my problem?</p>

ptrblck: <p>The more common approach is to use <code>ToTensor</code> to create normalized <code>FloatTensors</code> as you would need to transform the <code>dtype</code> to e.g. <code>float32</code> manually otherwise.</p>