akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Are Query, Key and Value always equal in TransformerEncoderLayer?

SalmonRon: <p>Hello,</p>
<p>I am looking at the <code>forward()</code> of <code>TransformerEncoderLayer</code> and it appears to me that in this implementation, Q=K=V=I, where Q, K, V and I denote Query, Key, Value and Input of the EncoderLayer, respectively (please correct me if I am wrong here).</p>
<pre><code class="lang-auto">def forward(
        self,
        src: Tensor,
        src_mask: Optional[Tensor] = None,
        src_key_padding_mask: Optional[Tensor] = None,
        is_causal: bool = False) -&gt; Tensor:
...
    x = src
    if self.norm_first:
        x = x + self._sa_block(self.norm1(x), src_mask, src_key_padding_mask, is_causal=is_causal)
        x = x + self._ff_block(self.norm2(x))
    else:
        x = self.norm1(x + self._sa_block(x, src_mask, src_key_padding_mask, is_causal=is_causal))
        x = self.norm2(x + self._ff_block(x))

return x

# self-attention block
def _sa_block(self, x: Tensor,
              attn_mask: Optional[Tensor], key_padding_mask: Optional[Tensor], is_causal: bool = False) -&gt; Tensor:
    x = self.self_attn(x, x, x,
                       attn_mask=attn_mask,
                       key_padding_mask=key_padding_mask,
                       need_weights=False, is_causal=is_causal)[0]
    return self.dropout1(x)
</code></pre>
<p>This seems to align with my understanding of the description in Vaswani et al (<a href="https://arxiv.org/abs/1706.03762" class="inline-onebox" rel="noopener nofollow ugc">[1706.03762] Attention Is All You Need</a>):</p>
<blockquote>
<p>The encoder contains self-attention layers. In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder.</p>
</blockquote>
<p>However, in many other uses of transformer encoders (e.g. <a href="https://arxiv.org/abs/2010.11929" class="inline-onebox" rel="noopener nofollow ugc">[2010.11929] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale</a>), Q, K and V are learnable linear transformations of I. As this is also quite common, am I missing a simple parameterization of <code>TransformerEncoderLayer</code>, which results in this behaviour, or do I need to code it myself?</p>

SalmonRon: <p>I might just have found the answer to my own question. If I understand it correctly, as <code>TransformerEncoderLayer</code> always uses <code>MultiheadAttention</code>, the linear transformations are implicitly learned for projecting Q, K and V into their multihead attention variants with dimensions q_k / num_heads and q_v / num_heads.</p>
Why does dynamo graph break on `or_` (`|`) operator between two dicts

Zacharias: <p>I noticed a graph break when using <code>{"a": torch.ones(42)} | {"b": torch.ones(43)}</code>, but none when using <code>{...}.update({...})</code>.</p>
<p>Example code</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

# works
@torch.compile(fullgraph=True)
def f():
    a = {"one": torch.ones(5)}
    a.update({"two": torch.ones(4)})
    return a

f()

# doesn't work, raises
# Unsupported: call_function BuiltinVariable(or_) [ConstDictVariable(), ConstDictVariable()] {}
@torch.compile(fullgraph=True)
def f():
    return {"one": torch.ones(5)} | {"two": torch.ones(4)}

f()

</code></pre>
<p>This leaves me with a number of questions,</p>
<ol>
<li>should this work, or what is problematic about this</li>
<li>Is there a spec that explains which features of python <em>should</em> be supported and which are expected to introduce breaks?</li>
</ol>
<p>I’ve done my best to search the github issues, docs and tutorials, but came up empty handed.</p>
Memory requirements for two similar computations

cruvy: <p>Something I always wondered about regarding how autograd works behind the scenes, and is relevant for me now. Assume I have a model that does</p>
<pre><code class="lang-auto"># targets is a [batch, 10], matrix with correct classes id, for 10 "codebooks".
logits = model(inputs) # logits of shape [batch, 10, n_classes] 
loss = 0
for i in range(10):
  loss += ce_loss(logits[:, i, :], targets[:, i], reduction="sum")
</code></pre>
<p>Is the memory requirement during forward / backward pass very different than doing:</p>
<pre><code class="lang-auto">logits = model(inputs)
loss = ce_less(logtis, targets, reduction="sum")
</code></pre>
<p>Note that in this case, obviously there is n reason not to use the latter version. In my actual situation it’s a little more involved to get the logits and targets each into one matrix that would allow computing the loss in one call, therefore the question.</p>

JuanFMontesinos: <p>Assume I may be mistaken, ofc.<br>
Mathematically it is equivalent, but computationally is not.</p>
<p>When you have a batched tensor and u backprop, you create a single graph and aggregate the gradients over a batch.</p>
<p>When you have a for loop you are creating N copies of the graph and then aggregate over them. Whereas the result is the same (gradients are sumed together), with the for loop you still need to do the exersize of backprop through the N graphs and store all the necessary info.</p>
<p>In practice it’s like calling a siamese network.</p>

cruvy: <p>However, d(logits)/d(params) is mutual to add those graphs, so I assume computed only once in both cases?</p>
Compatible versions for installing from source

mahmoodn: <p>Hi,<br>
I am running many difficulties while installing from source and most of them are about incompatible libraries.<br>
The question is what is the correct version of PyTorch, TorchVision and CUDA 12.2?</p>

ptrblck: <p>You could check the commits from successfully built nightly binaries as discussed in <a href="https://discuss.pytorch.org/t/what-is-commit-hash-of-pytorch-https-github-com-pytorch-pytorch-repository-for-a-particular-nightly/206060">this topic</a>. Generally, staying close to the same commit dates should also work. I.e. I usually <code>pull</code> all repositories after each other and haven’t seen a lot of issues in source building due to <code>torch</code> being incompatible with <code>vision</code>.<br>
CUDA 12.2 should work for any recent commit.</p>
Transformer categorization occasionally off by one

rand: <p>Hi all,</p>
<p>I have a strange problem with my transformer model that I am trying to train in PyTorch for self-educational purposes. I realize transformer models are a poor choice for time series, but nevertheless I downloaded daily S&amp;P 500 closing price data from Yahoo finance for 10 years back. Then I took ratios of adjacent prices and categorized the ratios into 20 bins according to an exponential function so that precision is higher at smaller price movement. I feed the model a window of the past, say 5, ratio bins and expect it to predict the next ratio bin. Problem is, the model seems to train OK about 75% of the time but in the remaining significant proportion of runs it trains off by one. That is, the categories that are output by the trained model are literally off by one versus when it trains “correctly” (or rather “optimally”). Sometimes it even goes off by two, but that is rarer.</p>
<p>For example, after a good training run validation looks like this:</p>
<pre><code class="lang-auto">(102, 315,
[0, 57, 49, 27, 4, 0, ...],
[0, 70, 6, 0, ...])
</code></pre>
<p>While after a bad one it looks like this:</p>
<pre><code class="lang-auto">(70, 315,
[0, 102, 57, 49, 27, 4, 0, ...],
[0, 6, 0, ...])
</code></pre>
<p>(The first line gives the number of accurate predictions and the total number of test vectors. The second line gives the frequency of being too high by i at index i, while the third line gives the frequency of being too low by i at index i – the first element of the frequency lists is expected to be 0 by definition since it is not “off” when it is off by zero).</p>
<p>I tried experimenting with different learning rates, different transformer model sizes, to no avail everything suffers from this problem. I even tried to adjust the training process to equalize the samples drawn from each class but it didn’t help. Anyone has run into something like this before and can offer advices?</p>
<p>(I realize I didn’t post code. The problem is my code is a hodgepodge of copy pasted code from other projects on the web and it is not clear what set of license terms apply to it that would allow me to post it.)</p>
How to avoid introducing `GLIBCXX_3.4.26` symbols when building with gcc 9.5.0

Rhett-Ying: <p>I built a module (so file, dynamic library) upon libtorch and I find even I utilize some calls like <code>std::string::data()</code> and build with gcc 9.5.0(the symbol version is GLIBCXX_3.4.26), no <code>GLIBCXX_3.4.26</code> symbol is found in the built so file. But if I build another so which calls the same functions, it contains symbols such as <code>GLIBCXX_3.4.26 _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE4dataEv</code>. The main difference may lie in the build options though they both uses gcc 9.5.0. The first one leverages torch cxx flags: <code>set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")</code>.</p>
<p>So I am wondering if this is the reason. and if yes, how to avoid introducing <code>GLIBCXX_3.4.26</code>?</p>
<p>Or a more general questions, how to make sure the library that is built with gcc 9.5.0 can run on ubuntu 18.04 or OS that has installed gcc 7(the glibcxx is 3.4.24)?</p>
C++ Traced model has issues on CUDA

Olek: <p>Hi there, Im working with a python model called transnetv2 - <a href="https://github.com/soCzech/TransNetV2" class="inline-onebox" rel="noopener nofollow ugc">GitHub - soCzech/TransNetV2: TransNet V2: Shot Boundary Detection Neural Network</a>.<br>
From I understand, it should be possible to trace this model on my cpu and then later load and do inference on the traced model in c++ using CUDA, but I am having alot of trouble with this.<br>
Ive been able to use the traced model completely fine in c++ when using my CPU, but if I attempt to use my GPU I get this runtime error:<br>
RuntimeError: Input type (CPUFloatType) and weight type (CUDAFloatType) should be the same or input should be a MKLDNN tensor and weight is a dense tensor</p>
<p>The thing is, if I trace this model using CUDA then I dont have this problem. I can use it in c++ no problem.<br>
Am I doing something wrong when I do my tracing? The original model uses TensorFlow but has  a pytorch implementation that Im using, thats the transnetv2_pytorch.py model in the github repo.</p>
<p>This is how Im doing my tracing:</p>
<pre><code class="lang-auto"># Initialize and load the model
model = TransNetV2()
state_dict = torch.load("inference-pytorch/transnetv2-pytorch-weights.pth")
model.load_state_dict(state_dict)
model.eval()

# Convert to TorchScript via tracing
example_input = torch.zeros(1, 100, 27, 48, 3, dtype=torch.uint8)  # An example input you would provide to your model

traced_script_module = torch.jit.trace(model, example_input)
traced_script_module.save("transnetv2_traced.pt")
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Olek" data-post="1" data-topic="206545">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/b2d939/48.png" class="avatar"> Olek:</div>
<blockquote>
<p>RuntimeError: Input type (CPUFloatType) and weight type (CUDAFloatType)</p>
</blockquote>
</aside>
<p>Move the input to the GPU and it should work.</p>

Olek: <p>Hi, thanks so much for responding<br>
Are you reffering to the input Im using when tracing the model? Will the traced model work on my CPU if I do this?</p>

ptrblck: <p>No, I’m referring to the quoted error message which seems to be raised in <code>libtorch</code> when you are trying to execute the model. The error message explains that the weight is on the GPU while the input is on the CPU and fails.</p>
How to reduce TPU RAM usage when calculating large tensors

Acerodancisto: <p>I am currently working on dealing with the task vector of the llama’s parameters, which is huge (over 13015864320). And I have a tensor shape (3,13015864320) in float16. Every time I try to get kthvalue or do element wise multiplication on this Tensor, RAM will always go unreasonably high (over 300GB). Is there any way to get this Tensor in batch and do calculations sequentially?<br>
Moreover, I found that Numpy with exact same computation (partition replace kth_value and np.multiply instead *)would use much less memory, can someone explain why this would happen?</p>

Sam_Kalaliya: <p>Batch Processing: Instead of processing the entire tensor at once, you can process it in smaller batches. This reduces the memory footprint as only a portion of the tensor is loaded into memory at any given time.</p>
<p>import torch</p>
<p>def process_in_batches(tensor, batch_size):<br>
num_batches = tensor.size(1) // batch_size<br>
results = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<pre><code>for i in range(num_batches):
    batch = tensor[:, i*batch_size:(i+1)*batch_size]
    # Perform your operations here
    # Example: kth_value = torch.kthvalue(batch, k)
    result = batch  # Placeholder for actual operation
    results.append(result)

# Handle the remainder
if tensor.size(1) % batch_size != 0:
    batch = tensor[:, num_batches*batch_size:]
    result = batch  # Placeholder for actual operation
    results.append(result)

return torch.cat(results, dim=1)
</code></pre>
<h1><a name="example-usage-1" class="anchor" href="#example-usage-1"></a>Example usage</h1>
<p>tensor = torch.randn(3, 13015864320, dtype=torch.float16)<br>
batch_size = 1000000  # Adjust this based on your memory constraints<br>
processed_tensor = process_in_batches(tensor, batch_size)</p>
TypeError: ToTensor.__init__() takes 1 positional argument but 2 were given

Pytorch_developer: <p><strong>I have this code:</strong></p>
<p>import torch<br>
import torchvision<br>
from torchvision import transforms, datasets</p>
<p>train = datasets.MNIST("", train = True, download = True, transform = transforms.Compose([transforms.ToTensor]))<br>
test = datasets.MNIST("", train = False, download = True, transform = transforms.Compose([transforms.ToTensor]))</p>
<p>trainset = torch.utils.data.DataLoader(train, batch_size = 10, shuffle = True)<br>
testset = torch.utils.data.DataLoader(test, batch_size = 10, shuffle = True)</p>
<p>for data in trainset:<br>
print(data)<br>
break</p>
<p><strong>But I am getting this error:</strong></p>
<p>TypeError: ToTensor.<strong>init</strong>() takes 1 positional argument but 2 were given</p>

Matias_Vasquez: <p>You are missing the parenthesis when passing the <code>ToTensor()</code> transform</p>
<pre><code class="lang-auto">transform = transforms.Compose([
              transforms.ToTensor()
            ])
</code></pre>
<p>This should work.</p>
<pre><code class="lang-python">import torch
import torchvision
from torchvision import transforms, datasets

train = datasets.MNIST("", train = True, download = True, transform = transforms.Compose([transforms.ToTensor()]))
test = datasets.MNIST("", train = False, download = True, transform = transforms.Compose([transforms.ToTensor()]))

trainset = torch.utils.data.DataLoader(train, batch_size = 10, shuffle = True)
testset = torch.utils.data.DataLoader(test, batch_size = 10, shuffle = True)

for data in trainset:
    print(data)
    break
</code></pre>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

Pytorch_developer: <p>Thank you this worked.</p>

SaturnTsen: <p>I’m interested in how you figure this out. It doesn’t appear to be evident…</p>
Using register forward hook from C++ API After torch.jit.load()

Gonen_Raveh: <p>Today (PY2.3), from C++ using libtorch we have a basic forward operation like:</p>
<pre data-code-wrap="model"><code class="lang-model">model.forward(some IValue Tensors)
</code></pre>
<p>For intermediate results, say from a deep neural network with K layers, actually, I need an API like the following that has support for:</p>
<pre><code class="lang-auto">module.forward(&lt;input&gt;, const char *layer_name); // or 
module.forward(&lt;input&gt;, const std::vector&lt;std::string&gt; &amp;layer_names);
</code></pre>
<p>Here is an example with request for “fc3” layer outputs:</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
mModule = torch::jit::load(fn_model);
mModule.eval();
c10::IValue pred = mModule.forward(inputs, "fc3");
</code></pre>
<p>So, I was trying to hack/propose a solution using the standard PyTorch C++ API. But, with no documentation how a python “def” function callback is being constructed as torch::jit::Function inside libtorch, this is hard. Here is my starting code. Any ideas?</p>
<pre><code class="lang-auto">mModule = torch::jit::load(fn_model);
mModule.eval();
// do we have any hooks to get model's intermediate layers as outputs?
std::shared_ptr&lt;torch::jit::CompilationUnit&gt; cu = mModule.type()-&gt;compilation_unit();
for (auto&amp; pre_hook : mModule.type()-&gt;getForwardPreHooks()) {            
}
// github pytorch/torch/csrc/jit/api/compilation_unit.h
c10::optional&lt;c10::QualifiedName&gt;    hookPrefix;        
std::vector&lt;torch::jit::Def&gt;         hookDefs;
std::vector&lt;torch::jit::ResolverPtr&gt; hookResolvers;
std::vector&lt;torch::jit::Def&gt;         preHookDefs;
std::vector&lt;torch::jit::ResolverPtr&gt; preHookResolvers;
std::vector&lt;std::string&gt; hooks_order;
std::map&lt;std::string, torch::jit::Def&gt; hook_def_map;
std::map&lt;std::string, torch::jit::ResolverPtr&gt; hook_resolver_map;
torch::jit::SourceRange range;
torch::jit::Ident name = torch::jit::Ident::create(range, "name");
std::vector&lt;torch::jit::Param&gt; subtrees;
torch::jit::List&lt;torch::jit::Param&gt; params = torch::jit::List&lt;torch::jit::Param&gt;::create(range, subtrees);
torch::jit::Maybe&lt;torch::jit::Expr&gt; return_type = torch::jit::Maybe&lt;torch::jit::Expr&gt;::create(range);
torch::jit::Decl decl = torch::jit::Decl::create(range, params, return_type);
std::vector&lt;torch::jit::Stmt&gt; subtrees_stmts;
torch::jit::List&lt;torch::jit::Stmt&gt; stmts = torch::jit::List&lt;torch::jit::Stmt&gt;::create(range, subtrees_stmts);
torch::jit::Def def = torch::jit::Def::create(range,name,decl,stmts);
const auto def_name = def.name().name();
hook_def_map[def_name] = def;
hook_resolver_map[def_name] = ;
for (const std::string&amp; hook_name : hooks_order) {
      hookDefs.emplace_back(hook_def_map.find(hook_name)-&gt;second);
      hookResolvers.push_back(hook_resolver_map.find(hook_name)-&gt;second);
}
(*cu).define_hooks(hookPrefix,hookDefs,hookResolvers,preHookDefs,preHookResolvers,NULL);
</code></pre>
Understanding custom loss function steps

stokecat: <p>Hi,</p>
<p>I’m very new to Pytorch (and ML in general), so I’m having difficulty understanding what is going on WRT a custom loss/cost function I’m looking at. I understand what’s going on in the function, but I need to understand how the gradient output of the last network layer is calculated.</p>
<p>NOTE: My task it to implement this custom loss function into our bespoke C++ ML lib. But to do this, I need to “manually” calculate the gradients for the last layer of the network.</p>
<p>So, if I run my network:</p>
<pre><code class="lang-auto">results = my_network(inputs)
</code></pre>
<p>And then my loss function:</p>
<pre><code class="lang-auto">loss = my_loss_fn(inputs, results, targets)
loss.backward()
</code></pre>
<p>Finally, if I print out the grad_fn chain on “results”, I see:</p>
<pre><code class="lang-auto">grad_fn chain of "results":
   SqueezeBackward1
   DivBackward0
   SliceBackward
   SqueezeBackward1
   Col2ImBackward
   TransposeBackward0
   MulBackward0
   FftC2RBackward
   ViewAsComplexBackward
   TransposeBackward0
   ViewBackward
   LeakyReluBackward0   &lt;=== This is the final layer of my_network
   NativeBatchNormBackward
   SlowConvTranspose2DBackward

</code></pre>
<p>So if I understand the autograd correctly, I would have to implement each of those grad_fn’s to arrive at the grad output for the LeakReluLayer?</p>

ptrblck: <aside class="quote no-group" data-username="stokecat" data-post="1" data-topic="206534">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stokecat/48/71513_2.png" class="avatar"> stokecat:</div>
<blockquote>
<p>NOTE: My task it to implement this custom loss function into our bespoke C++ ML lib. But to do this, I need to “manually” calculate the gradients for the last layer of the network.</p>
</blockquote>
</aside>
<p>If I understand your use case correctly, you thus want to implement the <code>backward</code>method of your custom function manually to port it to C++. If so, then yes, the <code>SqueezeBackward</code> to <code>ViewBackward</code> seem to correspond to the <code>backward</code> of your loss function.</p>
How to get the optimizer name from a state_dict or group_params

kirk86: <p>Suppose someone shared with you a pre-trained model and optimizer. The <code>optimizer.pt</code> has the ordered dict of <code>state</code> and <code>param_groups</code>, and, maybe you want to resume training but the issue is that we don’t know which optimizer has been used or scheduler for that matter.</p>
<p>How to resolve that? Maybe as a suggestion when saving the state <code>model.state_dict()</code> or <code>optimizer.state_dict()</code> the <code>state_dict()</code> function should automatically create an entry for <code>model.__name__</code>, <code>optimizer.__name__</code>, <code>scheduler.__name__</code> etc. making things easier to share and be reproducible.</p>

ptrblck: <p>You might be able to narrow down which optimizers might have been used by checking the stored keys in the <code>state_dict</code> as these should differ between optimizers.<br>
E.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">lin = nn.Linear(1, 1)
x = torch.randn(1, 1)

opt = torch.optim.SGD(lin.parameters(), lr=1.)
lin(x).mean().backward()
opt.step()
print(opt.state_dict())
# {'state': {}, 'param_groups': [{'lr': 1.0, 'momentum': 0, 'dampening': 0, 'weight_decay': 0, 'nesterov': False, 'maximize': False, 'foreach': None, 'differentiable': False, 'fused': None, 'params': [0, 1]}]}

opt = torch.optim.Adam(lin.parameters(), lr=1.)
lin(x).mean().backward()
opt.step()
print(opt.state_dict())
# {'state': {0: {'step': tensor(1.), 'exp_avg': tensor([[0.1863]]), 'exp_avg_sq': tensor([[0.0035]])}, 1: {'step': tensor(1.), 'exp_avg': tensor([0.2000]), 'exp_avg_sq': tensor([0.0040])}}, 'param_groups': [{'lr': 1.0, 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False, 'maximize': False, 'foreach': None, 'capturable': False, 'differentiable': False, 'fused': None, 'params': [0, 1]}]}
</code></pre>
<p>This would most likely not work to differentiate between all optimizers, but could give you a clue.</p>
Problem stacking two lists in a torch tensor

wilhelm: <p>Probably is a stupid question, but searching in the net leads to many similar problems, that do not solve this problem.</p>
<p>Basically I have two tensors from 2 different lists:</p>
<pre><code class="lang-auto">a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])

c = torch.stack([a, b])
</code></pre>
<p>Now… I <strong>want</strong> a tensor with this shape:</p>
<pre><code class="lang-auto">print(c)
tensor([[1, 4],
        [2, 5],
        [3, 6]])
</code></pre>
<p>but using <code>reshape</code>, <code>view</code> or whatever, I get only the following tensor:</p>
<pre><code class="lang-auto">tensor([[1, 2],
        [3, 4],
        [5, 6]])
</code></pre>
<p>which is definitely not what I want.</p>
<p>What should i do?</p>
<p>Thanks</p>

mikaylagawarecki: <p>Seems like the <code>dim</code> argument might be what you are looking for</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.tensor([1, 2, 3])
&gt;&gt;&gt; b = torch.tensor([4, 5, 6])
&gt;&gt;&gt; torch.stack((a, b), dim=1)
tensor([[1, 4],
        [2, 5],
        [3, 6]])
</code></pre>

wilhelm: <p>Thank you very much dude.</p>