akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)

_lovepytorch: <p>I am running QAT yolov7. When I run with BN fusing before PTQ and QAT =&gt; it works on GPU. But when I comment out 2 lines (without BN fusing before PTQ and QAT) <a href="https://github.com/NVIDIA-AI-IOT/yolo_deepstream/blob/5af35bab7f6dfca7f1f32d44847b2a91786485f4/yolov7_qat/scripts/qat.py#L79" rel="noopener nofollow ugc">https://github.com/NVIDIA-AI-IOT/yolo_deepstream/blob/5af35bab7f6dfca7f1f32d44847b2a91786485f4/yolov7_qat/scripts/qat.py#L79</a> and i got an error when running on GPU (if I run on CPU it works).</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "scripts/qat_BN.py", line 338, in &lt;module&gt;
    cmd_quantize(
  File "scripts/qat_BN.py", line 179, in cmd_quantize
    quantize.apply_custom_rules_to_quantizer(model, export_onnx)
  File "/yolov7_custom_dataset/quantization/quantize.py", line 222, in apply_custom_rules_to_quantizer
    export_onnx(model, "quantization-custom-rules-temp.onnx")
  File "scripts/qat_BN.py", line 138, in export_onnx
    quantize.export_onnx(model, dummy, file, opset_version=13, 
  File "/yolov7_custom_dataset/quantization/quantize.py", line 394, in export_onnx
    torch.onnx.export(model, input, file, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 506, in export
    _export(
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 1548, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 1180, in _model_to_graph
    params_dict = _C._jit_pass_onnx_constant_fold(
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)
</code></pre>
<p>I seem that this error is related to onnx conversion. I used Pytorch 2.0.1. I tried this thread <a href="https://github.com/pytorch/pytorch/pull/101329/files" class="inline-onebox" rel="noopener nofollow ugc">[ONNX] Fix onnx constant folding by shingjan · Pull Request #101329 · pytorch/pytorch · GitHub</a> but I could not file <code>torch/csrc/jit/passes/onnx/constant_fold.cpp</code> to modify it.</p>
<p>How can I do to get rid of the error? Thanks.</p>
How to directly call CuBLAS using PyTorch?

DerekCresswell: <p>I’m working on an experiment and would like to measure the speedups I can get for using Cublas (specifically the 2:4 sparsity) over the usual PyTorch functions.</p>
<p>I’ve got all of the setup of what I need except for actually calling the Cublas library. Essentially, I have a forward function where I just want to perform a matmul using cublas.</p>
<p>Everything I see online only talks about enabling TF32 but this is not what I want.</p>
<p>There was also this issue <a href="https://github.com/pytorch/pytorch/issues/60854" class="inline-onebox" rel="noopener nofollow ugc">Sparse CSR layout GPU backend tracking issue · Issue #60854 · pytorch/pytorch · GitHub</a> which talks about almost what I want but it is difficult to test since it is not in any release yet.</p>
<p>To be clear, I am a-ok with a hacky method to do this, it’s just a timing experiment and not anything production like. If my forward function explicitly can call cublas that is good, I do not care about portability or anything.</p>
<p>Is anyone able to provide guidance in this respect?</p>

eqy: <p>Could you post more details on how you deduced your matmul <em>isn’t</em> calling into cuBLAS? Or is your question about circumventing Python in some way?</p>
Program stops randomly with segmentation fault

learner47: <pre data-code-wrap="Fatal"><code class="lang-plaintext">
Thread 0x00007fc9bc352700 (most recent call first):
&lt;no Python frame&gt;

Thread 0x00007fc99e9f5700 (most recent call first):
  File "/usr/lib/python3.8/threading.py", line 306 in wait
  File "/usr/lib/python3.8/threading.py", line 558 in wait
  File "/mnt/code/venv/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x00007fca571c8740 (most recent call first):
  File "/mnt/code/venv/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 196 in &lt;genexpr&gt;
  File "/mnt/code/venv/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 196 in __getitem__
  File "/mnt/code/venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51 in &lt;listcomp&gt;
  File "/mnt/code/venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51 in fetch
  File "/mnt/code/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 677 in _next_data
  File "/mnt/code/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 633 in __next__
  File "src/train.py", line 262 in &lt;module&gt;
Segmentation fault (core dumped)
</code></pre>
<p>I am getting this segmentation fault randomly once the training begins. This particular error was at 3223 epoch. The line number 262 in my code is as follows.</p>
<pre><code class="lang-auto">for x, y in data_loader:
</code></pre>
<p>where <code>data_loader = DataLoader(dataset, mini_batch_size)</code>. I have also used <code>tqdm</code> for displaying the training status and that also is shown as part of the fault. Can anyone please help me to fix this code?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you please have a look?</p>

ptrblck: <p>It’s hard to tell what might be causing the segfault, but you could try to execute your code via:</p>
<pre><code class="lang-auto">gdb --args python script.py
...
run
...
bt
</code></pre>
<p>to get the backtrace and post it here.</p>

learner47: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you for your reply. My code is based on python3 and I tried to run the following command</p>
<pre><code class="lang-auto">gdb --args python3 src/code.py
</code></pre>
<p>Pardon me if something is wrong, I have no experience using <code>gdb</code>. However, I get the following error</p>
<pre><code class="lang-auto">Reading symbols from python3...
(No debugging symbols found in python3)
</code></pre>
<p>Can you please let me know what is wrong?</p>

ptrblck: <p>Did you execute the <code>run</code> command after launching <code>gdb</code>? If so, did the code fail again? What did <code>bt</code> print?</p>

learner47: <p>Thank you for your reply. With <code>gdb</code> the program ran with no issues. I’ll keep running with <code>gdb</code> enabled. If the program stops with segmentation fault, I’ll get back to you.</p>
Zombie process after CUDA assert

hadarshavit: <p>When using pytorch on CUDA, sometimes after a CUDA assert (for example, size mismatch), the python process is stuck on kernel code (full red bar on htop) and can’t be killed (also with pkill -9, and all other signals). sudo also can’t kill the process and only restarting the compute node killed it.<br>
This happened with multiple pytorch versions (1.13 and 2) compiled with CUDA 11.7 and 6 from conda and pip. The OS is CentOS 7 and GPUs are 1080 TI. It should be noted that the same code on a different compute cluster, with 2080 Ti and A100 GPUs does not get stuck (there is a CUDA assert but the process is done afterwards).<br>
I’m not sure if this is the right place to report it so please let me know if there is a better place.</p>
NCCL error when using multi node configuration

Gabriele_Fatigati: <p>I’m using detectron2, a tool for object detection based on Pytorch, on a HPC machine having 4 gpu per node. When I use one node it works well, when I try to launch on multiple nodes, using fo example 2 nodes and 8 gpus in total (4 gpu per node) I get:</p>
<p><code>  File "*****/view/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3145, in barrier     work = default_pg.barrier(opts=opts) RuntimeError: NCCL error in: ***/spack-src/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1269, internal error, NCCL version 2.14.3</code></p>
<p>From what I found in pytorch source code, such line raise the following error:</p>
<p><code>"Tensor list mustn't be larger than the number of available GPUs"</code></p>
<p>The following is the output of NCCL_INFO:</p>
<p>`lrdn1629:236813:236813 [1] NCCL INFO cudaDriverVersion 11080<br>
lrdn1629:236813:236813 [1] NCCL INFO Bootstrap : Using ib0:10.128.31.149&lt;0&gt;<br>
lrdn1629:236813:236813 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
lrdn1629:236813:236852 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.31.149&lt;0&gt;<br>
lrdn1629:236813:236852 [1] NCCL INFO Using network IB<br>
lrdn1629:236813:236852 [1] NCCL INFO Call to connect returned Connection refused, retrying<br>
lrdn1629:236813:236852 [1] NCCL INFO Call to connect returned Connection refused, retrying<br>
lrdn1629:236813:236852 [1] NCCL INFO Call to connect returned Connection refused, retrying<br>
…<br>
lrdn1629:236813:236852 [1] NCCL INFO Call to connect returned Connection refused, retrying</p>
<p>lrdn1629:236813:236852 [1] misc/socket.cc:456 NCCL WARN Net : Connect to 10.128.31.153&lt;46829&gt; failed : Connection refused<br>
lrdn1629:236813:236852 [1] NCCL INFO bootstrap.cc:256 → 6<br>
lrdn1629:236813:236852 [1] NCCL INFO init.cc:516 → 6<br>
lrdn1629:236813:236852 [1] NCCL INFO init.cc:1089 → 6<br>
lrdn1629:236813:236852 [1] NCCL INFO group.cc:64 → 6 [Async thread]<br>
lrdn1629:236813:236813 [1] NCCL INFO group.cc:421 → 3<br>
lrdn1629:236813:236813 [1] NCCL INFO group.cc:106 → 3<br>
lrdn1629:236813:236813 [1] NCCL INFO comm 0x3843eb10 rank 1 nranks 8 cudaDev 1 busId 56000 - Abort COMPLETE`</p>
<p>When I run using 2 nodes and 2 gpus per node, the job seems stuck, no error but also no progress. The same code works well in other HPC machines also using multiple nodes.</p>
<p>Already ran Nccl-test and it works well. Any suggestion ? Thanks.</p>
Questions on dynamic world size

zhanghongyi: <p>Hi,<br>
I found a <a href="https://discuss.pytorch.org/t/rpc-dynamic-world-size/99702">discussion</a> on dynamic world size of torch rpc and I have some questions related to it. We are developing a distributed graph learning library for PyTorch <a href="https://github.com/alibaba/graphlearn-for-pytorch" rel="noopener nofollow ugc">graphlearn-for-pytorch</a>. Our current goal is to refactor the library’s RPC functionality and make it dynamic. Sampler processes sample mini-batches from a large graph and serve as background service. Trainer processes can join dynamically and request mini-batches from the sampling processes. This will allow us to run different experiments without reloading the graph.</p>
<p>I have the following questions:</p>
<ol>
<li>Is dynamic world size a stable and long-term supported feature? While I can identify relevant code changes, I couldn’t find any documentation or tutorials.</li>
<li>How are some RPC APIs expected to behave in this dynamic scenario? For example, I tested <code>rpc.api._all_gather</code>, and found that only one rank can receive the response.</li>
</ol>
<pre><code class="lang-auto">import torch.distributed.rpc as rpc
import time
import multiprocessing as mp
import torch
import os

os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '11111'

def worker(rank):
    rpc.init_rpc(
        name=f'worker_{rank}',
        rank=rank,
    )
    print(f'rank {rank} initialized')
    time.sleep(1)
    print(rpc.api._all_gather(torch.cuda.device_count()))
    if rank == 0:
        time.sleep(1)
    rpc.shutdown()
    print(f'rank {rank} exited')

if __name__ == '__main__':
    ranks = [0, 1, 2, 3]
    for rank in ranks:
        process = mp.Process(target=worker, args=(rank, ))
        process.start()
</code></pre>
Training on multiple dataloaders

Ali_Youssef: <p>Hello everyone!</p>
<p>I had a question regarding training on more than one dataloaders.</p>
<p>So I have multiple dataloaders each containing different datatsets, and I gathered them into a list</p>
<p>The training loop is a while loop, with a nested for loop looping over the dataloader. However, I’m using cycle from itertools, and when the nested for loop finishes, it cycle through the next dataloader in the lsit of dataloaders.</p>
<p>An example code is the following where train_loader is a list containing multipe dataloaders.</p>
<p>This seems to work, however I wanted to check if there is any drawbacks in doing so?</p>
<pre><code>training_dataloaders = cycle(train_loader)

while Train: 
    train_loader = next(training_dataloaders)    

    for batch in train_loader:
</code></pre>
<p>Thank you!</p>

mygithubid1: <p>I don’t see a problem with this approach.</p>
Which Nvidia GPUs does PyTorch support?

kmulka_bloomberg: <p>I’m trying to find a list of which Nvidia GPUs are supported by which PyTorch versions. Seems like this should be in the PyTorch documentation somewhere.</p>
<p>Specifically, I’m getting this error:<br>
“CUDA error: no kernel image is available for execution on the device”</p>
<p>Wondering if the version I’m using just doesn’t support my GPU any more.</p>

ptrblck: <p>You can check the supported compute capabilities via <code>torch.cuda.get_arch_list()</code> and check if your GPU is too old.</p>
Onnx export runtime error : outerNode->outputs().size() == node->inputs().size() INTERNAL ASSERT FAILED

moonimoo_13: <p>I converted the already created onnx model to phytorch, retrained it, but when I tried to export it back to onnx, I keep getting the following error. The same error appears when I export to onnx immediately without training after converting to pytorch so I don’t think there was a problem with training.</p>
<p>The same error appears every time I write ‘torch.onnx.export’. And I double checked the type and shape of the model input when writing ‘torch.onnx.export’.</p>
<p>Does anyone know why?</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/var/folders/49/9g9lxm9d3f3br8zlg2l2fmz80000gn/T/ipykernel_1349/2634282627.py in &lt;module&gt;
----&gt; 1 torch.onnx.export(model, x, "output.onnx")

/opt/anaconda3/lib/python3.9/site-packages/torch/onnx/utils.py in export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, opset_version, do_constant_folding, dynamic_axes, keep_initializers_as_inputs, custom_opsets, export_modules_as_functions)
    502     """
    503 
--&gt; 504     _export(
    505         model,
    506         args,

/opt/anaconda3/lib/python3.9/site-packages/torch/onnx/utils.py in _export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, export_type, opset_version, do_constant_folding, dynamic_axes, keep_initializers_as_inputs, fixed_batch_size, custom_opsets, add_node_names, onnx_shape_inference, export_modules_as_functions)
   1527             _validate_dynamic_axes(dynamic_axes, model, input_names, output_names)
   1528 
-&gt; 1529             graph, params_dict, torch_out = _model_to_graph(
   1530                 model,
   1531                 args,

/opt/anaconda3/lib/python3.9/site-packages/torch/onnx/utils.py in _model_to_graph(model, args, verbose, input_names, output_names, operator_export_type, do_constant_folding, _disable_torch_constant_prop, fixed_batch_size, training, dynamic_axes)
   1113 
   1114     try:
-&gt; 1115         graph = _optimize_graph(
   1116             graph,
   1117             operator_export_type,

/opt/anaconda3/lib/python3.9/site-packages/torch/onnx/utils.py in _optimize_graph(graph, operator_export_type, _disable_torch_constant_prop, fixed_batch_size, params_dict, dynamic_axes, input_names, module)
    580     _C._jit_pass_lint(graph)
    581     _C._jit_pass_onnx_autograd_function_process(graph)
--&gt; 582     **_C._jit_pass_lower_all_tuples(graph)**
    583 
    584     # we now record some ops like ones/zeros

**RuntimeError: outerNode-&gt;outputs().size() == node-&gt;inputs().size() INTERNAL ASSERT FAILED at "/Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/jit/passes/dead_code_elimination.cpp":140, please report a bug to PyTorch.**
</code></pre>

huzq85: <p>hi, has this issue been addressed?</p>
Why does backward() require 3x the memory of forward()

Zai: <p>I’m trying to understand the behavior of backward() in the following example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.utils.checkpoint import checkpoint

def status( label ):

    print( f"{label}: ( {torch.cuda.memory_allocated() // 1024 ** 2:,} MiB, {torch.cuda.max_memory_allocated() // 1024 ** 2:,} MiB )" )


x = torch.randn( 1, 1, 1024, 1024 ).cuda()

status( "x" )

ncs    = [ 1, 256, 128, 64, 32, 16, 8, 4, 2, 1 ]
layers = [ nn.Conv2d( ncs[ i - 1 ], ncs[ i ], 3, 1, 1 ).cuda() for i in range( 1, len( ncs ) ) ]

status( "x + layers" )

def doit( x, layers, start, end, dummy = None ):
    for i in range( start, end ):
        x = layers[ i ]( x )
    return x

# res  = x
# segs = [ 0, 2, 5, len( layers ) ]
#
# for i in range( 1, len( segs ) ):
#     res = checkpoint( doit, res, layers, segs[ i - 1 ], segs[ i ], layers[ 0 ].weight )
#
# res = res.mean()

res = doit( x, layers, 0, len( layers ) ).mean()

status( "x + layer + res" )

del x

res.backward()

status( "x + layer + res + backward" )
</code></pre>
<p>The result I get:</p>
<pre><code class="lang-auto">x: ( 4 MiB, 4 MiB )
x + layers: ( 5 MiB, 5 MiB )
x + layer + res: ( 2,049 MiB, 2,049 MiB )
x + layer + res + backward: ( 3 MiB, 6,027 MiB )
</code></pre>
<p>I have difficulties understanding why backward() triples the memory load. I’d expect a gradient of a layer to be as big its the input. So, it shouldn’ add more than 2 GiB. But this is already an overestimation, since not all gradients have to be stored, but only the most recent one while moving backward. So the additional memory should be roughly the size of two layer activations. And even that is an overestimation, since already processed activations can be freed. So all in all, I would expect ( for this example ) a backward memory load just slightly higher than the forward load. Can someone explain this unexpectedly high memory need?</p>

eqy: <p>Unfortunately just inspecting the peak memory usage via <code>torch.cuda.max_memory_allocated()</code> here isn’t a reliable way to deduce the memory used only by <em>tensors</em>. It looks like in this case there is a backward cuDNN kernel that allocates a temporary workspace which would increase what is reported by <code>torch.cuda.max_memory_allocated()</code>. In general, cuDNN will opportunistically allocate temporary workspaces in order to run faster kernels (that require these temporary workspaces) if there is enough memory available on the device.</p>
<p>It’s a bit crude, but you can force less memory to be used this way by setting <code>torch.cuda.set_per_process_memory_fraction(value)</code> in your script. On a 32GiB GPU, I set the value to 0.08 and saw <code>x + layer + res + backward: ( 3 MiB, 2,569 MiB )</code>.</p>
How should I calculate mean IoU if some classes are absent from either prediction or ground truth tensor?

JimW: <p>My use case is pretty much the same with this example:</p>
<pre><code class="lang-auto">from torchmetrics import JaccardIndex
import torch

pred = torch.tensor([1, 2, 19, 17, 17])
target = torch.tensor([1, 2, 3, 17, 4])

jaccard = JaccardIndex(num_classes=21)

jaccard(pred, pred)
Out: tensor([0., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.,
        0., 1., 0.])
</code></pre>
<p>How can I correctly calculate mIoU between pred and target when there are non-present classes? In other words, I don’t want it to simply assign zero to classes that were not even present in the test dataset. Also, we need to consider the zeros only if it is really a wrong prediction, instead of an absent class.</p>
Custom Nonlinear Activation Function that's not "1-to-1"

stevensagona: <p>The nonlinear activation functions typically used in pytorch that I am familiar with are 1-to-1 functions, like arctan, sigmoid, relu, etc.</p>
<p>Is it possible to have a custom nonlinear activation function that depends on multiple arguments?</p>
<p>So for example instead of feeding the result of a dot product into the activation function:<br>
$output = f_{nl}(x \cdot A)$</p>
<p>the output is just come custom nonlinear function $f_{nl}(x, A)$, say for example:<br>
$output = \sum_i^N (sum_j cos(A_i_j x_j - j/N) x_i)$</p>
<p>Is it posssible to use such a complex activation layer in a NN in pytorch?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/stevensagona">@stevensagona</a>,</p>
<p>You can always define your custom function as an <code>nn.Module</code> object, and just use it inside your model, e.g. pass an instance of the function class, and call it llike you would do with <code>nn.Tanh()(x)</code> for example.</p>

stevensagona: <p>The example you gave is a one-input to one-output nonlinear function.</p>
<p>I am interested in custom functions that are not one-input to one-output, but are many-inputs to one-output.</p>
<p>Is it straightforward to do a more complicated function that takes multiple weights and inputs and combines them to make one output, like in the example I gave.</p>
<p>And would it be straightfoward to train such a thing?</p>

AlphaBetaGamma96: <p>As I stated in my previous answer, you can define the custom function as an <code>nn.Module</code> object and it’ll  be fine. Use it <strong>like</strong> you would with any other activation function, e.g. <code>nn.Tanh()</code>.</p>
DDP and Gradient checkpointing

Thu_Nguyen: <p>Hi everyone,<br>
I tried to use torch.utils.checkpoint along with DDP. However, after the first iteration, the program hanged. I read one thread last year in the forum and a person said that DDP and checkpointing havent worked together yet. Is that true? Any suggestions for my case? Thank you.</p>

albanD: <p>Hi,</p>
<p>I am afraid this is true.<br>
We are working on a solution for in 1.10.</p>

rvarm1: <p>We currently have a prototype API <code>_set_static_graph</code> which can be applied to DDP if your training is static across all iterations (i.e. there is no conditional execution in the model). Documentation: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L1490" class="inline-onebox" rel="noopener nofollow ugc">pytorch/distributed.py at master · pytorch/pytorch · GitHub</a>.</p>
<p>With static graph training, DDP will record the # of times parameters expect to get gradient and memorize this, which solves the issue around activation checkpointing and should make it work.</p>

Brando_Miranda: <p>I don’t understand what the issue is. Why did your code hang - that is essential information to put in here. Did you try any of the following:</p>
<ul>
<li><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.10.1+cu102 documentation</a></li>
<li><a href="https://discuss.pytorch.org/t/checkpointing-ddp-module-instead-of-ddp-itself/115714/2" class="inline-onebox">Checkpointing DDP.module instead of DDP itself - #2 by mrshenli</a></li>
<li><a href="https://discuss.pytorch.org/t/checkpointing-ddp-module-instead-of-ddp-itself/115714/3" class="inline-onebox">Checkpointing DDP.module instead of DDP itself - #3 by Brando_Miranda</a></li>
</ul>
<p>if none of them worked can you provide more details? In particular Your original post does not describe enough to know what the problem is. Things can hang for many reasons - especially in complicated multip processing code.</p>

Bhavay_Malhotra: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, did you find a solution to this in 1.10.0.</p>
Efficient multiple function call

LLlearner: <p>I have a function to test some functionality of the training, and I want to repeat the testing for many times.<br>
How can I do that efficiently so that it runs in parallel and fast on the GPUs?<br>
For instance, how to replace this small block:</p>
<pre><code class="lang-auto">N = 100
for i in range(N):
    result = test_training()
</code></pre>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/lllearner">@LLlearner</a>,</p>
<p>Have a look at <code>torch.func.vmap</code> to efficiently vectorize your function call, the docs are here: <a href="https://pytorch.org/docs/stable/generated/torch.func.vmap.html#torch.func.vmap" class="inline-onebox" rel="noopener nofollow ugc">torch.func.vmap — PyTorch 2.0 documentation</a></p>
NVLink for 2x3090 necessary?

IoannisTselekoglou: <p>Hey,</p>
<p>I wanted to set up an ML workstation and dive deeper into CNN and Transformers. My question is, is it necessary to run them via NVLink or are the PCIe Lanes fast enough? I found multiple different sources on that and im just confused. Configuration so far: <a href="https://pcpartpicker.com/list/kLmfrv" rel="noopener nofollow ugc">https://pcpartpicker.com/list/kLmfrv</a><br>
From my understanding, it’s not possible to directly combine their VRAM, thus training larger models requires data/model parallelism. Therefore using NVLink provides higher bandwidth between both GPUs, right? I’m uncertain whether I should simply opt for 1 x4090 or 2x3090’s.</p>
<p>I would genuinely appreciate any recommendations and helpful insights you could provide.</p>
Voice Conversion

Nurkhan_Laiyk: <p>Hello,</p>
<p>I have  different audio files in .wav format. In each audio file there is a female/male voice saying  one word in Kazakh language. I would like to increase the samples, so that from 1 audio file I can generate 3-4 samples with clear pronunciation of the word but with different voice. I have tried different augmentations, but it didn’t help that much.</p>
<p>Is there any other options?</p>

ube: <p>Hi there,<br>
I was just trying to do the same for my own language.<br>
It is still under construction but check it out here: <a href="https://github.com/mube1/voice_clone/blob/main/data.py" rel="noopener nofollow ugc">https://github.com/mube1/voice_clone/blob/main/data.py</a><br>
Also this works : <a href="https://pytorch.org/audio/stable/tutorials/audio_data_augmentation_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Audio Data Augmentation — Torchaudio 2.0.1 documentation</a></p>
<p>Just wondering, what model are you considering?</p>
BPTT with CTC loss

Peter_Featherstone: <p>Is there a way to do back-propagation through time using CTC loss?<br>
I’ve seen implementations of BPTT where you partition your outputs and labels into chunks, calculate loss while retaining graphs. With CTC, partitioning labels into equal chunks and assigning each one to an output block is ill-defined since by design, the labels may be misaligned.<br>
Is there a way to make BPTT or MRBP (<a href="https://arxiv.org/pdf/2010.06891.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2010.06891.pdf</a>) work with CTC loss?<br>
Thank you</p>
Self-attention gives incorrect weight

happysyp000: <p>Hi, I was trying to use the pytorch implementation of multihead attention with masks (“key_padding_mask”) for the padded tokens. However, it seems the mask is doing weird things. Here is an example:<br>
I have three tokens, each with embedding dimension of 4. Assuming the last token is padding, I added the mask [False, False, True]. I would expect the attention weight for the masked token to be zero</p>
<pre><code class="lang-auto">import torch
from torch import optim, nn
self_attn = nn.MultiheadAttention(4,2, dropout=0, batch_first=True)
x=torch.rand((3,4)) #sequence length x embed_d
res, attn=self_attn(x,x,x,key_padding_mask=torch.Tensor([False,False,True]),need_weights=True)
</code></pre>
<p>Then attention matrix gives a result like following:</p>
<pre><code class="lang-auto">attn= tensor([[0.2125, 0.2203, 0.5673],
        [0.2041, 0.2077, 0.5881],
        [0.2086, 0.2112, 0.5803]], grad_fn=&lt;SqueezeBackward1&gt;)
</code></pre>
<p>The attention weight for the masked token (last column) is not zero, but a big value.<br>
What is going on?</p>
Model parameters not updating after optimizer.step()

Vishal-S-P: <p>I have written a custom neural net where there are 12 convolutional blocks each containing 7 layers. However, for the last 2 blocks, I am decomposing the CNN parameters in each layer into two sets among which one is updated for certain iterations.</p>
<p>An example of my model definition for single decomposed CNN layers looks like this defined inside a nn.Module -</p>
<pre><code class="lang-auto">self.conv_1_parameters_1 = nn.Parameter((torch.Tensor(3, 12, 3 , 3).normal_(0, 0.01)), requires_grad=True)
self.conv_1_parameters_2 = nn.Parameter(init.xavier_normal_(torch.Tensor(3, 32, 32*12 , 1, 1)), requires_grad=True)

</code></pre>
<p>So during the forward pass, I am selecting (based on the input to forward function) both self.conv_1_parameters_1 and self.conv_1_parameters_2 across dim 1 as follows (here the index of first dim is specified using k) :</p>
<pre><code class="lang-auto">coeffs_1 = self.conv_1_parameters_2[k].view(32,32, 12)
conv_w_1 = torch.einsum('cvm, mki-&gt; cvki', coeffs_1, self.conv_1_parameters_1[k]).reshape(32, -1, 3, 3).squeeze().contiguous().view((32,32,3,3)
x = F.conv2d(x_input, conv_w_1, padding=1)
</code></pre>
<p>I am using use two different optimizers for updating the parameters -</p>
<pre><code class="lang-auto">all_parameters = dict(self.model.named_parameters())
encoder_paras = [v for k, v in all_parameters.items() if ('encoder' in k)]
paramter_set_1 = [v for k, v in all_parameters.items() if ('coeff' in k)]  # These are just the parameters corresponfing to self.conv_1_parameters_1 in the last 2 conv blocks
paramter_set_2 = [v for k, v in all_parameters.items() if ('combiner' in k)] # These are just the parameters corresponfing to self.conv_1_parameters_2 in the last 2 conv blocks
self.optimizer_1 = torch.optim.Adam(encoder_paras+paramter_set_1, lr=0.0001)  # update all the parameters in convolution block 0 to 9 (which are encoder parameters) and only update parameter_set_1 in last 2 conv blocks
self.optimizer_2 = torch.optim.Adam(encoder_paras+paramter_set_2, lr=0.0001) # update all the parameters in convolution block 0 to 9 (which are encoder parameters) and only update parameter_set_2 in last 2 conv blocks
</code></pre>
<p>The training loop looks something like this -</p>
<pre><code class="lang-auto">for i in range(K): # this will select the subset of parameters for those last 2 conv blocks
  pass input through whole model with index i
  update the parameters corresponding to certain index using optimizer 1
  
</code></pre>
<p>So after the end of above loop I should be able to see only changes to the parameter_set_1 in last two conv blocks. But the these values don’t change at all. I printed the grad for parameter_set_1 and it is not None. I am not detaching any tensor in between. All the parameters in conv block 0 to 9 are updated by the optimizer. The problem is only with the conv block 10 and 11 which use the decomposed parameters.</p>
<p>What am I missing here?</p>
<p>Update : I figure that the gradients are very small like 10^-16.</p>
Federated Learning. SGD on server

lmaresdlc: <p>Hello! I am working on a Federated Learning scenario. I first train a global model on the server. I send copies of this model to several clients, and each of them trains its own data on the model. By means of SGD, each client locally updates the weights of the model and sends them back to the server. At the server, I compute a loss that is basically the difference between the initial global weights and the local weights sent back by the clients. I would like to use SGD on the server too to update the global weights based on this loss. However, I cannot do loss.backward(), since there is no previous forward() based on these global weights. Any idea on how to proceed with this scenario?</p>

Mohamed_Nabih: <p>Hi, I have the same problem did you manage to fix it or find a solution?</p>