akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Trouble with installing torchvision on Jetson Xavier NX

ViktorPavlovA: <p>Hello.</p>
<p>I have jetson xavier nx. I have already install version of torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl with using this <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048" rel="noopener nofollow ugc">guide</a> .  But i don’t found correct version of torchvision. it must be 0.15.1  but what’s the name of the branch ? I guess i try wrong branch…</p>
<p>Thank you!</p>

ptrblck: <p>This older PyTorch release shipped with <a href="https://github.com/pytorch/vision/commit/912c44fa">this <code>torchvision</code> tag</a> and your link shows how to build it from source.</p>

ViktorPavlovA: <p>Well. Thank you! But it’s not correct branch.</p>

ptrblck: <p>Yes, it is the right one, which was also used in the <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch">PyTorch 23.06 NGC container</a>.</p>
Performing torch.bmm with distinct batch sizes

Vinay_Joshi1: <p>I need to perform batched matrix multiplication between two matrices A and B where DIM(A) = [<strong>m1, n, o</strong>] and B = [<strong>m2, n, o</strong>]. Where the output C</p>
<blockquote>
<p>C = A x B</p>
</blockquote>
<p>has the same shape as A. Now, the batch dimension of A &amp; B is supposed to be different. Hence the resulting product C = A x B is performed like, for example,</p>
<blockquote>
<p>c[0:2] = A[0:2, :, :] x B[0, :, :]T</p>
</blockquote>
<blockquote>
<p>c[2:6] = A[2:6, :, :] x B[1, :, :]T</p>
</blockquote>
<blockquote>
<p>c[6:8] = A[6:8, :, :] x B[2, :, :]T</p>
</blockquote>
<p>The map between indices of <strong>m1</strong> dim to indices of <strong>m2</strong> dim is known upfront. Only one index in the <strong>m2</strong> dim is mapped to one or multiple indices in the <strong>m1</strong> dim. This is essentially running distinct matmul ops with different sizes in parallel. The naive way of implementing this in PyTorch would be to use a for-loop but that would not execute these matmuls in parallel. What is the best way to implement this in PyTorch to get the best out of GPU performance or is a custom Cuda kernel must?</p>

Vinay_Joshi1: <p>Can someone please help? Experts here <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Dot pt opening in ubuntu-has no answers even in previous questions

arinet1: <p>Requesting for an editor that I can use to open  a (.pt) file in Ubuntu so as to have a view of its contents in text format…</p>

ptrblck: <p>PyTorch checkpoints saved via <code>torch.save</code> are archives. Double click them in Ubuntu or extract with your favorite archive manager.</p>

arinet1: <p>thank you sir.<br>
I am going to try that and get back to you.</p>
How to reduce maximum cache size on GPU?

btjanaka: <p>I am running transformers from the HuggingFace transformers library and encountering issues with large GPU cache size. When running on a 3090, my max memory usage (<code>torch.cuda.max_memory_allocated()</code>) is 7GB, while the maximum allocated memory allocated (<code>torch.cuda.max_memory_reserved()</code>) is 12GB. This seems alright. However, when I run on an A6000, my max memory usage is still 7GB, but my allocated memory rises to 43GB.</p>
<p>I am wondering how the PyTorch caching mechanism works, i.e., how does PyTorch determine the maximum amount of memory to reserve, and can this be reduced? My motivation is that other folks may be running jobs on the machine I am using, so I do not want to hog the GPU memory. I saw there are some configuration variables for the cache / allocator on this page: <a href="https://pytorch.org/docs/stable/notes/cuda.html#environment-variables" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch 2.2 documentation</a> but they do not seem to accomplish what I am seeking.</p>
<p>What stands out to me about this issue is that the cache on the 3090 only goes to 12GB, so clearly it is possible to have a smaller cache on the A6000 as well.</p>
<p>Regarding code, this is part of a large project, so it’s difficult to make a minimal example, but I can try to make one if needed.</p>

ptrblck: <p>You can use <code>torch.cuda.set_per_process_memory_fraction(fraction, device=None)</code> to set the max allowed memory. The root cause for the increased cache usage is unclear since you are unable to share the model or any information.</p>
I have a question about RNN-T transducer loss

girlsending0: <p>When using the RNN-T transducer loss, there are ‘sum’ and ‘average’ in the reduction method, and I know that it means whether to update with the sum of the loss in the batch or with the average value, respectively.<br>
But I wonder why this is divided.<br>
In the case of CTC loss, the inf value is sometimes output, so zero infinity and the sum method may be helpful, but in the case of transducer loss, the inf value is not output. What are the differences in learning methods between the two methods?<br>
Anyway, the loss is -log(x)… but if I divide it by a constant, it doesn’t have any effect on the gradient.</p>
RuntimeError: tensor does not have a device - which tensor?

whatdhack: <p>Getting the following error.  How do I find out which tensor autograd is complaining  about ?</p>
<p>torch/autograd/<strong>init</strong>.py", line 251, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: tensor does not have a device</p>

srishti-git1110: <p>Hi,<br>
Please post a minimum executable snippet that reproduces this error.</p>

whatdhack: <p>That is lot of extra work and time.   PyTorch autograd should  have , and if not need to add additional logging. Is there any debug/trace  capability in autograd ? To me, not reporting the tensor looks like a bug in autograd.</p>

whatdhack: <p>Here  are few  tricks that helped  me figure out the issue.  It is often not possible to create a small reproducer, hence debugging/tracing logs are  invaluable tools.</p>
<ol>
<li>TORCH_SHOW_CPP_STACKTRACES=1</li>
<li>TORCH_CPP_LOG_LEVEL=INFO</li>
<li>with autograd.detect_anomaly():</li>
</ol>
Inquiry on GPU utilization of pytorch/pytorch:1.8.1-cuda11.1-cudnn8-runtime image

rurusungoa: <p>Inquiry on GPU utilization of pytorch/pytorch:1.8.1-cuda11.1-cudnn8-runtime image</p>
<p>When pytorch saw “cpu_mkl_py”, it only used the CPU.<br>
The runtime pytorch image provided by pytorch seems to have worked well because it contains the “py3.10_cuda11.8_cudnn8.7.0_0” version of the package.</p>
<p>However, I also checked the GPU device in the redhat software “openshfit”, but unlike in existing python’s jupyter-notebook,<br>
I did not configure the <a href="http://nvidia.com/gpu:1" rel="noopener nofollow ugc">nvidia.com/gpu:1</a> device by filling in the deployment, but I brought the image of pytorch as is and checked it,<br>
and the GPU was found. It comes with a card.</p>
<p>I have questions about how that part is different, but it was resolved well.</p>
<p>checking the device<br>
When running in python, the device was also confirmed well.</p>
<p>import torch<br>
print(torch.cuda.get_device_name(0))<br>
=&gt; NVIDIA GTX 6000<br>
print(torch.cuda.is_available())<br>
=&gt; True</p>
Bfloat16 CPU inference speed is too slow on AMD cpu

111114: <p>Hello, I’m trying to inference my model by torch.bfloat16 dtype in cpu.<br>
But when I set model and inputs to torch.bfloat16. inference speed is much slower than float32 inference on AMD Ryzen 9 5900X 12-Core Processor.</p>
<p>Inference speed is 9.1iter/s on float32, 9.1s/iter on bfloat16.</p>
<p>But bfloat16 inference was okay on Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz.</p>
<p>How can I know by python code if bfloat16 inference is okay on specific CPU device?</p>
<p>Thank you.</p>
<pre><code class="lang-auto">import torch
from tqdm import tqdm

from src.text_utils.multi_lingual_g2p import MultiLingualG2P
from src.models_vits.model_vits import VITSForInference
from src.base_utils import os, load_pth, load_config, load_df, load_json


DTYPE = torch.bfloat16

model_path = '{MODEL_PATH}'

configs = load_config(os.path.join(model_path, "config.yaml"))
model = VITSForInference(configs, "cpu").to(DTYPE)

model_state = {key : value.to(DTYPE) for key, value in load_pth(os.path.join(model_path, 'model.pth')).items()}
embeds = {key : value.to(DTYPE) for key, value in load_pth(os.path.join(model_path, 'embeds.pth')).items()}

model.load_state_dict(model_state)

model.eval()

model_info = load_json(os.path.join(model_path, "model_info.json"))

g2p = MultiLingualG2P()

text = "{INPUT_TEXT}"
language = model_info["language"][0]
phoneme = g2p.text_to_phoneme(text, language)[0]
text_sequence = g2p.phoneme_to_sequence(phoneme)
texts = torch.tensor(text_sequence, dtype=torch.long, device="cpu").unsqueeze(0)
style_embeds = embeds["embed_nat"].unsqueeze(0)

def inference(texts, style_embeds, d_control, p_control, noise_scale):
    with torch.no_grad():
        with torch.cpu.amp.autocast():
            model.forward(texts, style_embeds, d_control=1.0, p_control=1.0, noise_scale=0.1)


for i in tqdm(range(100)):
    inference(texts, style_embeds, d_control=1.0, p_control=1.0, noise_scale=0.1)
</code></pre>
Cannot build torchvision c++ on windows

ricardo_assaad: <p>Hi Everybody,</p>
<p>I am trying to build torchvision using cmake on windows. I have VS 2019 installed. libtorch 2.2.2 with cuda 11.8 is also installed and working. I need torchvision ops for my model to load.<br>
After I get torchvision 0.17.2 from the tags, I proceed to build it according to the readme file. cmake … gives me the following:</p>
<blockquote>
<p>CMake Error at CMakeLists.txt:18 (find_package):<br>
By not providing “FindTorch.cmake” in CMAKE_MODULE_PATH this project has<br>
asked CMake to find a package configuration file provided by “Torch”, but<br>
CMake did not find one.<br>
Could not find a package configuration file provided by “Torch” with any of<br>
the following names:</p>
<ul>
<li>TorchConfig.cmake</li>
<li>torch-config.cmake</li>
</ul>
<p>Add the installation prefix of “Torch” to CMAKE_PREFIX_PATH or set<br>
“Torch_DIR” to a directory containing one of the above files.  If “Torch”<br>
provides a separate development package or SDK, be sure it has been<br>
installed.<br>
– Configuring incomplete, errors occurred!</p>
</blockquote>
<p>I changed the name of TorchVisionConfig.cmake.in to TorchVisionConfig.cmake under the cmake folder<br>
now I get this error:</p>
<blockquote>
<p>Severity	Code	Description	Project	File	Line	Suppression State<br>
Error		CMake Error at C:/src/packages/torchvision-0.17.2/vision/cmake/TorchVisionConfig.cmake:7:<br>
Parse error.  Expected a command name, got unquoted argument with text<br>
“<span class="mention">@PACKAGE_INIT</span>@”.		C:/src/packages/torchvision-0.17.2/vision/cmake/TorchVisionConfig.cmake	7</p>
</blockquote>
<p>Any help on building or using torchvision is highly appreciated</p>
<p>Thanks</p>
Will modules passed to ModuleWrapPolicy each be in their own FSDP unit?

divinho: <p>Struggling to parse this from the docs</p>
<blockquote>
<p>For convenience, this accepts <code>ModuleWrapPolicy</code> directly, which allows users to specify the module classes to wrap (e.g. the transformer block).</p>
</blockquote>
<p>And what happens to all the other modules? E.g. the input and output embedding? Are they put in an “other” FSDP unit?</p>

divinho: <p>Looks to do that the title says: <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/fsdp/wrap.py#L153" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/distributed/fsdp/wrap.py at main · pytorch/pytorch · GitHub</a></p>
How to develop shared bottom tower serving different tasks

A_G1: <p>I have two model classes both pyramid architecture.</p>
<ul>
<li>Let’s say first task is predicting user will buy something with architecture [feature_embedding_128, dense_1048, dense_512, dense_128, dense_1]</li>
<li>Second task is predicting donating to charity at checkout with architecture [feature_embedding_64, dense_512, dense_256, dense_64, dense_1].</li>
</ul>
<p>Let’s say both these tasks are seperately optimized, with different learning rate, and learning rate scheduling. Now, let’s say I want to merge these tasks:</p>
<ul>
<li>We are adding much more feature embedding so we can not separate serve on both tasks, we will share these embeddings through a bottom tower to both and then serve tasks seperately in such an architecure:</li>
<li>bottom_embedding_1028, dense_512, dense_64 =&gt; output of these towers are concatanated with the bottom of two towers discussed above.</li>
</ul>
<p>Now what is my problem is that basically I have 3 towers to optimize, (1) buy?, (2) charity?, (3) bottom shared embedding.</p>
<p>I have been struggling to how to systematically set up the learning rate. My model is just too big and I cannot do random/grid search coming up with learning rate for each tower.</p>
<p>Is there any paper out there discussing this? Any previous experience? I do apprecaite this.</p>
What is the difference between nn.Threshold and torch.clamp

isalirezag: <p>can someone tell me wath is the difference between <a href="https://pytorch.org/docs/stable/nn.html?highlight=threshold#torch.nn.Threshold" rel="nofollow noopener">nn.threshold</a> and torch.clamp function if I just use the min value?</p>

kwm: <p>Sorry if this is a necro but this is the top result on Google right now when I search this question. I ran a little test to see which is faster, but your mileage will vary based on hardware.</p>
<p>Clamp seems to call a <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/vulkan/ops/Clamp.cpp" rel="noopener nofollow ugc">cpp script</a> where as threshold calls a <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/activation.html#Threshold" rel="noopener nofollow ugc">forward function</a>, which takes longer, but sometimes you want the functionality of a nn.Module function. Calling threshold in-place without the function call speeds it up. It’s also worth noting you kind of get two thresholds out of clamp, so the fact that it takes about twice as long as in-place threshold makes the results equal in my eyes.</p>
<pre><code class="lang-auto">import torch
from torch import nn
import time

input = torch.randn(1000)  # random inputs
m = nn.Threshold(0.0, 0.0)  # function that replaces values below 0 with 0
n_trials = int(1e5)

start = time.process_time()
result_threshold = [ m(input) for i in range(n_trials)]
print(f'Threshold time for {n_trials} loops: {time.process_time() - start} seconds')
&gt;&gt;&gt; Threshold time for 100000 loops: 1.5605827209999998 seconds
start = time.process_time()
result_inplace_threshold = [ torch.nn.functional.threshold(input, threshold=0.0, value=0.0, inplace=True) for i in range(n_trials)]
print(f'in-place threshold time for {n_trials} loops: {time.process_time() - start} seconds')
&gt;&gt;&gt; in-place threshold time for 100000 loops: 0.5645996109999993 seconds
start = time.process_time()
result_clamp = [ torch.clamp(input, min=0, max=100) for i in range(n_trials)]
print(f'Clamp time for {n_trials} loops: {time.process_time() - start} seconds')
&gt;&gt;&gt; Clamp time for 100000 loops: 1.0484716829999998 seconds
</code></pre>
GPU out of memory for simple RLHF

yehhhh: <p>Hello everyone,</p>
<p>I am trying to implement a simple policy gradient for fine-tuning an autoregressive transformer from scratch, but got CUDA memory error issue when I try to use batch size 64 and have a long sequence (&gt;100). The log_prob and log_prob take a lot of GPU memory, even though I have A100.</p>
<p>I was wondering if I did something wrong?</p>
<pre><code class="lang-auto">pg_total = []

for j in range(batch_size_rl):
    start_char = np.random.choice(list(aa_to_proba_first.keys()), p=list(aa_to_proba_first.values()))
    initial_context = torch.tensor([[stoi[start_char]]], dtype=torch.long, device=device)

    # Generating one sequence
    state, log_prob, log_prob_ref = model.generate_for_rlfh(idx=initial_context, max_new_tokens=100,
                                                            pad_token_idx=stoi["!"], ref_model=ref_model)

    reward = 1  # fake reward just for demonstrating the issue

    pg = (reward + ref_coef * log_prob_ref - e_coef * log_prob) * log_prob
    pg_total.append(pg)

pg_total = torch.stack(pg_total)
out = torch.mean(torch.sum(pg_total, dim=2))
actor_loss = -out

actor_optimizer.zero_grad(set_to_none=True)
actor_loss.backward()
actor_optimizer.step()
</code></pre>
<p>Thanks a lot!</p>
Return handle from compiled function from C++ to be used by backend

trirpi: <p>I’ve created a very simple Dynamo backend that gets executed and replaces some operations with <code>mock_compiled_handle</code>.</p>
<pre><code class="lang-auto">def mock_compiled_handle(inputs):
    print("Ran mock_compiled_handle")
    return inputs

def toy_backend(gm, sample_inputs):
    exec_graph = []  # list of compiled functions to run
    xx = sample_inputs[0]
    for n in gm.graph.nodes:
        if n.op == "call_module":
            exec_graph.append(mock_compiled_handle)
            a = gm.get_submodule(n.target)
            xx = a(xx)

    def exec_topt(*args):
        outs = None
        for fun in exec_graph:
            if outs is None:
                outs = fun(args[-1])
            else:
                outs = fun(outs)
        return [outs]

    return exec_topt


torch._dynamo.reset()
fn = torch.compile(Model(), backend=toy_backend, dynamic=True)

t = torch.zeros(5)
out = fn(t)
</code></pre>
<p>I now want to do the same thing, but using C++ to create the mocked handle. Using <a href="https://github.com/Andrechang/DL_cookbook/blob/main/bert_optim/topt/topt_compiler.cpp" rel="noopener nofollow ugc">this example</a>, I already managed to invoke a C++ function from Python. Currently, the compile function in C++ looks as follows:</p>
<pre><code class="lang-auto">void customCompile(torch::jit::Graph &amp;graph, std::vector&lt;torch::Tensor&gt; &amp;tensors)
{
    std::cout &lt;&lt; "Function customCompile executed." &lt;&lt; std::endl;
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m)
{
    m.def("custom_compile", &amp;customCompile, "Test compile function to be used by Dynamo backend.");
}
</code></pre>
<p>and can be invoked as follows:</p>
<pre><code class="lang-auto">xx = sample_inputs[0]
a = gm.get_submodule(n.target)
ts_trace = torch.jit.trace(a, xx)
ts_trace = torch.jit.freeze(ts_trace.eval())
test = dynamo_backend.custom_compile(ts_trace.graph, [xx])
</code></pre>
<p>Instead of returning <code>void</code>, it should return a handle that can be used (such as the <code>mock_compiled_handle</code>). How can this be achieved?</p>
<p>Any resources or examples regarding simple dynamo backend implementations would also be very useful!</p>

trirpi: <p>You can use the return type <code>std::function&lt;torch::Tensor &amp;(torch::Tensor &amp;)&gt; </code>, so something like this would work:</p>
<pre><code class="lang-auto">std::function&lt;torch::Tensor &amp;(torch::Tensor &amp;)&gt; customCompile(torch::jit::Graph &amp;graph, std::vector&lt;torch::Tensor&gt; &amp;tensors)
{
    return [](torch::Tensor &amp;a) -&gt; torch::Tensor &amp;
    {
        std::cout &lt;&lt; "Test handle called." &lt;&lt; std::endl;
        return a;
    };
}
</code></pre>
Training two models simultaniously: incorrect backpropagation

Jonas_VD: <p>When training two networks for online knowledge distillation on CUDA(2 GPU), the backpropagation for the teacher model fails. When executing, the code runs properly but the teacher’s loss is not changing more than within a range of 0.0005 which is too low. When training the teacher seperately, the loss drops with a factor 0.8 every minibatch. The student’s backward is going well.</p>
<p>The models are trained with nn.DataParallel on the same device. When trying training it on seperate CUDA devices the same problem occured.</p>
<pre><code class="lang-auto">    criterion = nn.BCELoss()
    optimizer_t = optim.AdamW(student.parameters(),lr=teacher.lr,weight_decay=teacher.wgt_decay)
    optimizer_s = optim.AdamW(student.parameters(),lr=student.lr,weight_decay=student.wgt_decay)
</code></pre>
<pre><code class="lang-auto"># TRAINING #
            if train_network:
                teacher.train()
                student.train()
            else:
                student.eval()
                teacher.eval()
            for i, data in enumerate(trainloader, 0):
                iTot += 1
                inputs, labels = data[0].to(device), data[1].to(device)  # assign device (X)
                inputs = (inputs - torch.from_numpy(m)[np.newaxis, np.newaxis, :].to(device)) / torch.sqrt(
                    torch.from_numpy(v)[np.newaxis, np.newaxis, :].to(device))

                sequences += inputs.shape[0]
                frames += inputs.shape[0] * inputs.shape[1]
                hours += (inputs.shape[0] * inputs.shape[1]) * (
                        trainset.datasets[0].frShift / trainset.datasets[0].fs) / (60 * 60)
                # teacher training
                outputs_teacher = teacher(inputs)
                outputs_teacher = outputs_teacher[0]
                optimizer_t.zero_grad()
                teacher_loss = criterion(outputs_teacher, labels)
                teacher_loss.backward()
                optimizer_t.step()
                torch.cuda.empty_cache()

                
                # student training
                optimizer_s.zero_grad()
                outputs, transmitted_feat = student(inputs.detach())  # ap = after pooling != amplitude phase
                student_loss = criterion(outputs, labels.detach())
                
                loss2 = student_loss + teacher_loss

                loss2.backward()
                optimizer_s.step()
                torch.cuda.empty_cache()

</code></pre>
<p>Help will be much appreciated.</p>

Jonas_VD: <p>The problem was solved by doing input.detach() and label.detach()</p>
Sparse matrix solver or wrapper

fakemade: <p>Hello guys. At first sorry for my english, because it isn’t my native language.<br>
I have question and don’t have answer. I have to solve big linear system with a lot of zero values in process of learning with using backward. I think sparse solver is the best decision for this. And my question is how i can solve sparse systems with autograd. Does PyTorch support this function or can i write a wrapper around maybe scipy.sparse.solve? I am junior in python programming and pytorch but i want to solve my problem and i hope the community of pytorch helps me.</p>
AttributeError: 'int' object has no attribute 'unflat_param_names' in FSDP.scatter_full_optim_state_dict

kleingeo: <p>I am loading the model weights and optimizer from a non FSDP training session and trying to apply it to an FSDP model, but I am getting the error.</p>
<pre><code class="lang-auto">  File "/lustre07/scratch/kleingeo/VertDetect/trainer/Trainer.py", line 566, in train_model
    sharded_optim_state_dict = FSDP.scatter_full_optim_state_dict(full_optim_state_dict, net)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kleingeo/vert_detect/lib/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1617, in scatter_full_optim_state_dict
    return FullyShardedDataParallel._optim_state_dict_to_load_impl(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kleingeo/vert_detect/lib/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1319, in _optim_state_dict_to_load_impl
    return _rekey_sharded_optim_state_dict(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kleingeo/vert_detect/lib/python3.11/site-packages/torch/distributed/fsdp/_optim_utils.py", line 930, in _rekey_sharded_optim_state_dict
    key.unflat_param_names, key.unflat_param_names
    ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'int' object has no attribute 'unflat_param_names'

[2024-04-04 10:35:35,613] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2024-04-04 10:35:35,613] torch._dynamo.utils: [INFO] Function, Runtimes (s)
</code></pre>
<p>The loading of the optimizer is as follows:</p>
<pre data-code-wrap="python"><code class="lang-python">
model_checkpoint = torch.load('/model/checkpoint/dict/path.pt/, map_location='cpu')

net = model_init(**model_parameters)
net = net.to(self.device)

mixed_precision = MixedPrecision(param_dtype=torch.bfloat16,
                                                     reduce_dtype=torch.bfloat16,
                                                     buffer_dtype=torch.float32)


net = FSDP(net,
                    sharding_strategy=ShardingStrategy.SHARD_GRAD_OP,
                    mixed_precision=mixed_precision,
                    device_id=gpu,
                    auto_wrap_policy=ModuleWrapPolicy(module_classes={model_init}),
                    backward_prefetch=BackwardPrefetch.BACKWARD_PRE,
                    )

optimizer = torch.optim.AdamW(net.parameters(), lr=learning_rate)

full_optim_state_dict = None
if gpu == 0:
    full_optim_state_dict = model_checkpoint['optimizer']

sharded_optim_state_dict = FSDP.scatter_full_optim_state_dict(full_optim_state_dict, net)

optimizer.load_state_dict(sharded_optim_state_dict)

</code></pre>
<p>Both the original model/optimizer and this are using the same conda environment, and both on <code>torch==2.2.1</code>.</p>
Shape '[92160, 1]' is invalid for input of size 188743680

AGM: <p>When training, in the step of introducing the inputs to the model…</p>
<pre><code class="lang-auto">outputs = model(inputs)
</code></pre>
<p>… in the forward process, I got an error of invalid shape for input of size in the following line:</p>
<pre><code class="lang-auto">x, _ = self.lstm(x)
</code></pre>
<p>My <strong>forward</strong> code is:</p>
<pre><code class="lang-auto">def forward(self, x):

        x = pool(F.relu(bn(conv(x))))
        x = x.view(x.size(0), -1, x.size(2)*x.size(3)*x.size(4))
        x, _ = self.lstm(x)

        for fc in self.fcs[:-1]: x = F.relu(fc(x))
        x = self.fcs[-1](x) 

        return x
</code></pre>
<p>My input data is 4-dimensional as it’s a RGB videoframe. My labels are 2-dimensional as it’s one set of features per frame of the video.</p>
<pre><code class="lang-auto">Inputs shape: torch.Size([1, 3, 10, 256, 360])
Labels shape: torch.Size([1, 10, 3])
</code></pre>
<p>If I print the shape of x in forward right before and after the x.view(), I obtain:</p>
<pre><code class="lang-auto">Before reshape: torch.Size([1, 32, 1, 32, 45])
After reshape: torch.Size([1, 32, 1440]) # 1*32*1440 = 1*32*1*32*45
</code></pre>
<p>However, in the step of x, _ = self.lstm(x) I got that now, the shape is 92160 and we are looking for a size of 188743680.</p>
<p>More info:<br>
batch_size = 1<br>
image_height = 256<br>
image_width = 360<br>
n_frames = 10</p>
<p>My designed network is made of 3 sets of…</p>
<ul>
<li>Conv3d: kernel_size=3, stride=1, padding=1</li>
<li>BatchNorm3d</li>
<li>MaxPool3d: kernel_size=2, stride=2</li>
</ul>
<p>… one LSTM layer, and 3 last Linear layers.</p>
Autograd derivatives of multioutput ANN

Konstantinos_Mexis: <p>I have a simple ANN with <code>input_dim=1</code> and <code>output_dim=2</code>.</p>
<pre><code class="lang-auto">class NN(nn.Module):
    def __init__(self):
        super(NN, self).__init__()
        self.fc1 = nn.Linear(1, 128)
        self.fc  = nn.Linear(128, 128)
        self.fc2 = nn.Linear(128, 2)
        
    def forward(self, x):
        x = torch.tanh(self.fc1(x))
        x = torch.tanh(self.fc(x))
        x = self.fc2(x)
        return x
</code></pre>
<p>I am studying the PINNs idea, and I want to calculate the ODE loss. Given that I have a system of 2 ODEs, how should I calculate the derivatives of the 2-outputs ANN? Currently, I am doing:</p>
<pre><code class="lang-auto">def physics_loss(model: nn.Module):
    t = np.linspace(0, 2, 250)
    t_ts = torch.tensor(t, dtype=torch.float32, requires_grad=True).reshape(-1, 1)
    y_pred_ts = model(t_ts)
    y_pred_1_ts = y_pred_ts[:, 0].view(-1, 1)
    y_pred_2_ts = y_pred_ts[:, 1].view(-1, 1)
    
    dy_pred_1_ts_dt = torch.autograd.grad(y_pred_1_ts, t_ts, torch.ones_like(y_pred_1_ts), create_graph=True)[0]
    dy_pred_2_ts_dt = torch.autograd.grad(y_pred_2_ts, t_ts, torch.ones_like(y_pred_2_ts), create_graph=True)[0]
    
    error_1 = dy_pred_1_ts_dt + 5 * y_pred_1_ts
    error_2 = dy_pred_2_ts_dt - 5 * y_pred_1_ts + y_pred_2_ts

    return torch.mean(error_1**2) + torch.mean(error_2**2)
</code></pre>
<p>Is this correct?</p>

srishti-git1110: <p>Your code isn’t buggy and runs fine:</p>
<pre data-code-wrap="python"><code class="lang-python">model = NN()
loss = physics_loss(model)
print(loss) # tensor(0.2380, grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>But I don’t quite see why you’d calculate the grads of <code>y_pred_1_ts</code> which is the model output wrt <code>t_ts</code> which is the model input (the data) rather than the model parameters. Assuming you wanted to show a minimum executable snippet and ask if <code>torch.autograd.grad</code> can be used the way you showed, I think you’re gtg. Also see <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html" rel="noopener nofollow ugc"><code>.backward()</code></a>.</p>

Konstantinos_Mexis: <p>I want to calculate the physics loss term of a PINN. This is why I calculate the derivatives of the model output wrt to the model input.</p>
Can PyTorch move a tensor along with its computational graph from GPU to CPU, and then move it back to GPU for backpropagation?

XZJIsme: <p>Can PyTorch move a tensor along with its computational graph from GPU to CPU, and then move it back to GPU for backpropagation? For instance, <code>a</code> is originally on GPU 0, and after computing with <code>b</code>, we get <code>c</code>. Then, I obtain <code>c.sum()</code> and move <code>c</code> to the CPU to free up memory. Next, I move <code>d</code> from the CPU to GPU 0, and continue computing on GPU 0 to get <code>e</code> by combining <code>c.sum()</code> with <code>d</code>. Starting backpropagation from <code>e.sum()</code>, when it propagates back to <code>c</code>, I move <code>d</code> back to the CPU to free up space for moving <code>c</code> back to GPU 0 for continued backpropagation. Can PyTorch do this? It is a workaround for memory constraints.</p>

ptrblck: <p>The <code>to()</code> operation is differentiable, but won’t move intermediates to the target device.<br>
To save memory via CPU offloading you might want to use <a href="https://discuss.pytorch.org/t/modifying-forward-backward-pass/169687/2">these hooks</a>.</p>

XZJIsme: <p>Thank you, it looks very useful, I will apply this to my code.</p>
How to get the torch-2.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl with source code compiling

PegX: <p>I wanna get a torch installation binary from source code. With the default configuration, I get the file name of “torch-2.1.0a0+gitunknown-cp39-cp39-linux_aarch64.whl”.</p>
<p>With this compiled binary, In my case, there is an error about the “undefined symbol: _ZNK5torch8autograd4Node4nameEv”.</p>
<p>Therefore, but with the downloaded binary of  "torch-2.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl ". In my case, that works normally. Therefore, I wanna to know how to configure to get this torch-2.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl binary from source code.</p>
Unique node name across graph re-compilations and graph breaks

kilianyp: <p>I am working a custom backend for torch dynamo. If a recompilation is re-triggered for some reason (for example input shape changes), the backend should process a fx.Node the same way like it was in the first pass (e.g. some sort of caching).</p>
<p>However, I struggle to match the nodes across recompilations:</p>
<ul>
<li>the node name is not unique across if there’s graph breaks in the model. For example, if the model is broken into two graphs, and both contain an <code>add</code>, there will be in model two nodes named <code>add</code>.</li>
<li>I tried using the number of graph breaks as an additional identifier, but noticed that in some models the re-compiled graph has a different number of graph breaks then the originally traced one.</li>
<li>the <code>id</code> of the graph or nodes are always different, so this seems to be newly created torch.fx.Node and torch.fx.Graph instances.</li>
</ul>
<p>Last but not least:</p>
<ul>
<li>I used the code snipped stored in the meta data to generate a hash and use that as an identifier together with the node name. However, I noticed that in the recompiled graph, the code can slightly differ. For example, one says “forward”, and during the recompilation it says <code>"&lt;resume in forward&gt;"</code></li>
</ul>
<p>I probably could work around this, but wanted to raise this here to see if a better solution can be found.</p>
<p>Thank you</p>
Extraction and modification of intermediate layer outputs from pre-trained models and layer by layer inference

Izan_C_G: <p>Hi everyone, I would like to know if it is possible to obtain and modify the outputs of several layers (features extraction), modify them and use them as new inputs for the following layers, continuing then with the inference.</p>
<p>The purpose is to inject faults into some features and check the behaviour of several pre-trained models analyzing their accuracy to some datasets like ImageNet (quantized int8 and fp32 models from pytorch vision repo (<a href="https://pytorch.org/vision/0.13/models.html" class="inline-onebox" rel="noopener nofollow ugc">Models and pre-trained weights — Torchvision 0.13 documentation</a>).</p>
<p>It occurs to me a possible solution is to make some kind of layer by layer inference if it is possible, but I am not sure if this is the proper way. Any help would be appreciated.</p>
<p>Thanks!</p>

ptrblck: <p>You could register forward hooks on the desired layers and manipulate the output.</p>

Izan_C_G: <p>Is there any easy example of how to reproduce that?</p>

ptrblck: <p>You could use <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">this code</a> as a template and add your manipulation instead of storing the activation in the <code>dict</code>.</p>

Izan_C_G: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I wonder, if there is also a way to use hooks in the following post that I submitted a few moments ago: <a href="https://discuss.pytorch.org/t/search-and-modify-layer-module-outputs-by-name/200196" class="inline-onebox">Search and modify layer/module outputs by name</a></p>

ptrblck: <p>I’m not familiar enough with the quantization util. so don’t know how these layers are registered and used.<br>
You might want to move your other post to the <code>Quantization</code> module so that the code owners are aware of it.</p>
Evaluating a pretrained model

juripf: <p>I’m trying to evaluate a pretrained model that i trained with the PPO example from the <a href="https://pytorch.org/tutorials/intermediate/reinforcement_ppo.html" rel="noopener nofollow ugc">pytorch tutorials website</a> but i’m failing to obtain the same results as during training.</p>
<p>I’ve saved the model (in the train script) with</p>
<pre data-code-wrap="Python"><code class="lang-Python">
model_weights_filename = "models/ppo_example_model_weights_500k.pth"
torch.save(policy_module.state_dict(), model_weights_filename)
</code></pre>
<p>and loaded them with</p>
<pre data-code-wrap="python"><code class="lang-python">model_weights_filename = "models/ppo_example_model_weights_500k.pth"
policy_module.load_state_dict(torch.load(model_weights_filename))
</code></pre>
<p>in my evaluation script.</p>
<p>Finally i’m using the same rollout function as in the tutorial (the <code>policy_module</code> is defined equal to the train script):</p>
<pre data-code-wrap="python"><code class="lang-python">""" Evaluate """
torch.manual_seed(37)

policy_module.eval()
with torch.no_grad():
    for _ in range(10):
        eval_rollout = env.rollout(1000, policy_module)
</code></pre>
<p>In some rare cases i’m getting good results, but most of the time the actor survives only a few steps instead of 999.</p>
<p>Am I missing something crucial here?</p>
<p>Thanks in advance<br>
Juri</p>
Pairwise absolute differences

mishooax: <p>I am trying to come up with an efficient implementation for the <em>variogram</em> score, equation (3) in <a href="https://journals.ametsoc.org/view/journals/mwre/143/4/mwr-d-14-00269.1.xml:" rel="noopener nofollow ugc">https://journals.ametsoc.org/view/journals/mwre/143/4/mwr-d-14-00269.1.xml:</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/6748d216554cef699618ccc3166024eb748077aa.png" data-download-href="https://discuss.pytorch.org/uploads/default/6748d216554cef699618ccc3166024eb748077aa" title="variogram-score"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/6748d216554cef699618ccc3166024eb748077aa_2_690x101.png" alt="variogram-score" data-base62-sha1="eJHcddmz0TyWSiZrjEtSyayT43U" width="690" height="101" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/6748d216554cef699618ccc3166024eb748077aa_2_690x101.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/6748d216554cef699618ccc3166024eb748077aa_2_1035x151.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/7/6748d216554cef699618ccc3166024eb748077aa.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">variogram-score</span><span class="informations">1208×178 21.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here y is the “true” value whereas X is an ensemble forecast. y.shape = (bs, 100000) and X.shape = (bs, 8, 100000) - each forecast is calculated at about 100k points. I’d like to calculate the score without explicitly building a pairwise differences matrix</p>
<pre data-code-wrap="python"><code class="lang-python">y_diff = torch.abs(y.unsqueeze(1) - y.unsqueeze(2))  # this will OOM
</code></pre>
<p>as that’d entail creating a matrix with 10^10 elements. Any suggestions?</p>
Float16 not faster than float32 with torch.matmul

hzzt: <p>Below is an example code and the benchmarking results. I am constantly dealing with tensors of sizes listed in the example code. I find that torch.float16 is barely faster than torch.float32 for batched matmul. Is torch.float16 casted to torch.float32 in the intermediate steps? How do I speed up matmul for torch.float16?</p>
<p>I am running on an A2 gpu, with torch version ‘2.2.2+cu121’.</p>
<pre><code class="lang-auto">import torch
import torch.utils.benchmark as benchmark

torch.backends.cuda.matmul.allow_tf32 = False

n = 1024
p = 8
m = 2048
b = 32
device = "cuda"
for dtype in [torch.float16, torch.float32]:
    a1 = torch.rand(b, n, p, device=device, dtype=dtype)
    a2 = torch.rand(b, p, m, device=device, dtype=dtype)

    t0 = benchmark.Timer(
            stmt='matmul(a1, a2)',
            setup='from torch import matmul',
            globals={'a1': a1, 'a2': a2})

    total = t0.timeit(100)
    print('\ndtype:',dtype)
    print(total)
</code></pre>
<p>Benchmarking results:</p>
<pre><code class="lang-auto">dtype: torch.float16
&lt;torch.utils.benchmark.utils.common.Measurement object at 0x7f8233925060&gt;
matmul(a1, a2)
setup: from torch import matmul
  1.61 ms
  1 measurement, 100 runs , 1 thread

dtype: torch.float32
&lt;torch.utils.benchmark.utils.common.Measurement object at 0x7f8233924df0&gt;
matmul(a1, a2)
setup: from torch import matmul
  1.64 ms
  1 measurement, 100 runs , 1 thread
</code></pre>
<p>Changing to <code>torch.backends.cuda.matmul.allow_tf32 = True</code> gives similar benchmarking results</p>

ptrblck: <p>It seems cuBLAS is not able to pick a faster kernel resulting in almost the same performance for both workloads. I do see a clear speedup in <code>float16</code> vs. <code>float32</code> on A100 so will check if it’s a heuristics issue on the A2.</p>
<p>What kind of model are these shapes used in?</p>

hzzt: <p>Hi,</p>
<p>Thanks for the reply, I am not training models, but rather using pytorch to do signal processing with linear algebra techniques involving SVDs.</p>
<p>How much speed up did you see on the A100? Is <code>float16</code> generally expected to be twice as fast as <code>float32</code>?</p>
Torch.jit.trace() memory leak workarounds

dshin: <p>It has been reported <a href="https://discuss.pytorch.org/t/possible-memory-leak-in-torch-jit-trace/74455">elsewhere</a> that <code>torch.jit.trace()</code> has a memory leak. This is a problem for me because my application repeatedly calls <code>torch.jit.trace().save()</code> in a loop (later loading the saved model with torchlib in c++).</p>
<p>One workaround suggested <a href="https://github.com/pytorch/pytorch/issues/35600#issuecomment-894807668" rel="noopener nofollow ugc">here</a> is to wrap the <code>torch.jit.trace()</code> call in a separate process by using <code>futures.ProcessPoolExecutor</code>.</p>
<p>This appears to work well in single-threaded settings. However, in multi-threaded settings, this occasionally leads to deadlock, due to a variant of this <a href="https://github.com/python/cpython/issues/50970" rel="noopener nofollow ugc">issue</a>. In general, forking multi-threaded processes in python is known to be quite fragile.</p>
<p>I am looking for a workaround. Is there a good way to <code>torch.jit.trace().save()</code> in a loop without deadlocking and without incurring a memory leak in a multi-threaded application?</p>
<p>If the answer is, “upgrade to pytorch 2.0 and use <code>torch.compile()</code>”, I’m ok with that. Anything that works.</p>

ptrblck: <aside class="quote no-group" data-username="dshin" data-post="1" data-topic="199184">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dshin/48/54230_2.png" class="avatar"> dshin:</div>
<blockquote>
<p>If the answer is, “upgrade to pytorch 2.0 and use <code>torch.compile()</code>”, I’m ok with that. Anything that works.</p>
</blockquote>
</aside>
<p>That might indeed be the recommendation here, since <code>TorchScript</code> is in maintenance mode and will thus not receive any major updates or big fixes anymore. I’m also hesitant to look for other workarounds as the risk of running into a new issue, that won’t be fixed, will still be there.</p>

dshin: <p>Thanks for the reply. Guess it’s time to upgrade!</p>

dshin: <p>I started to look into the torch 2.0 documentation a bit, and am a bit confused. The latest page on exporting models to c++ appears to be this <a href="https://pytorch.org/docs/stable/cpp_index.html" rel="noopener nofollow ugc">one</a>, which still talks about using TorchScript. But I understood your answer to mean that upgrading to torch 2.0 will make it so that TorchScript won’t be needed anymore to export a model to c++. What gives?</p>
<p>Is the recommended approach to export to c++ to export in ONNX format, now? And if so, on the c++ side, to use a non-torch library like ONNX Runtime?</p>

dshin: <p>There was in fact a simple fix to my original problem. Instead of doing the <code>torch.jit.trace()</code> call in a separate process by using <code>futures.ProcessPoolExecutor</code>, do it with <code>subprocess</code>, pickling the arguments to disk and then reloading in the subprocess. This gets around the multi-threaded + fork sensitivities.</p>
Play with large tabular data

Shang_Gao: <p>Hello,</p>
<p>I’m playing with a large tabular data with 20M samples and 25K features. The data is stored in 102 .mtx files(~250GB). I would like to train a score based generative model with this data. My memory size is 64GB and my GPU is 4060 16GB. What will be a good approach to prepare the dataset? My idea is to split the samples with batch size 1000 and save each batch to a file? Thanks in advance for any ideas, suggestions or discussion.</p>
How to run multiprocessing with cuda streams

Mamdouh_Aljoud: <p>I am trying to run two cuda streams in parallel, I initiate the streams then use them to run computations in the processes. The problem I have is that the processes are not firing. i.e., thecode is not executed inside the processes.<br>
Please refer to the code below.</p>
<pre><code class="lang-auto">

from torch.multiprocessing import Process, set_start_method
import torch
import time


stream1 = torch.cuda.Stream()
stream2 = torch.cuda.Stream()
torch.cuda.synchronize()
def process1():
    global stream1, stream2
    with torch.cuda.stream(stream1):
        print("IM HERE 1\n")
        print(time.time(),"time in process 1")
        time.sleep(5)
def process2():
    global stream1, stream2
    with torch.cuda.stream(stream2):
        print("IM HERE 2\n")
        print(time.time(),"time in process 2")
        time.sleep(5)
if __name__ == "__main__":
    set_start_method('spawn',force = True)
    start = time.time()
    p1 = Thread(target = process1)
    p2 = Thread(target = process2)
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    torch.cuda.synchronize()
print("Time for parallel implementation: {}".format(time.time() - start))


</code></pre>

pritamdamania87: <p>Was able to run your sample code successfully with a few minor tweaks:</p>
<pre><code class="lang-auto">from torch.multiprocessing import Process, set_start_method
import torch
import time


stream1 = torch.cuda.Stream()
stream2 = torch.cuda.Stream()
torch.cuda.synchronize()
def process1():
    global stream1, stream2
    with torch.cuda.stream(stream1):
        print("IM HERE 1\n")
        print(time.time(),"time in process 1")
        time.sleep(5)
def process2():
    global stream1, stream2
    with torch.cuda.stream(stream2):
        print("IM HERE 2\n")
        print(time.time(),"time in process 2")
        time.sleep(5)
if __name__ == "__main__":
    set_start_method('spawn',force = True)
    start = time.time()
    p1 = Process(target = process1)
    p2 = Process(target = process2)
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    torch.cuda.synchronize()
</code></pre>
<p>The output shows both processes are firing:</p>
<pre><code class="lang-auto">IM HERE 2

1610497442.7076824 time in process 2
IM HERE 1

1610497442.7735846 time in process 1
Time for parallel implementation: 8.420467615127563
</code></pre>

Tungdil: <p>Has anyone tried running this multi stream, multiprocessing architecture for running the same instance a multiple times? I thought the reason of building such architecture was to maintain high performance for multiple instances (ofc. within the constraints of the GPU memory), but running multiple instances decreases the FPS performance drastically.</p>

ptrblck: <p>Cross-post from <a href="https://discuss.pytorch.org/t/running-multiple-model-instances-on-a-single-gpu-performance-analysis/200048/2">here</a> with more information about compute resources besides the needed memory.</p>
The potential improved performance of custom configuration for cuda kernels

Jacob0226: <p>It appears to me that the torch source code, written in C++/CUDA, generalizes functions to suit various GPU specifications. Suppose I intend to train a large model, such as LLaMA. In that case, I would customize the configuration of kernel calls within torch based on my GPU specifications, possibly even altering algorithms. This approach aims to maximize throughput for my model on the specific GPU. I’m curious if such practices are common in practice. I’ve noticed that sometimes the CUDA kernels within Torch fail to fully utilize the hardware resources of the GPU. However, by modifying the kernel algorithms, I’ve achieved a 50% acceleration.</p>
<p>I acknowledge that my inquiry may be somewhat vague. Yet, considering the factors of GPU cost and training time, customizing Torch seems to offer significant acceleration. I wonder if this concept is feasible or unrealistic within AI technology companies.</p>

ptrblck: <p>Your description is indeed too vague and it’s unclear what exactly was changed to achieve what kind of speedup in which part of the code for what kind of model.<br>
With that being said, it’s unrealistic to try to change e.g. launch parameters of some kernels and rebuild PyTorch for one use case only. You should let DL compilers do these optimizations during the runtime.</p>