akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to fix "RuntimeError: Trying to backward through the graph a second time" in SHOT code?

External_happy: <p>Hi, I am trying to run SHOT code on my dataset available at <a href="https://github.com/tim-learn/SHOT/blob/master/digit/uda_digit.py#L260" rel="noopener nofollow ugc">https://github.com/tim-learn/SHOT/blob/master/digit/uda_digit.py#L260</a>. However, I am getting the error “RuntimeError: Trying to backward through the graph a second time.” I have checked the code multiple times, but I do not understand which lines are making the gradient computed twice.???</p>
<pre><code>    image, _, idx = get_batch(batch)
    ### get the pseudo-labels
    if self.epoch % self.interval_iter == 0:
        model.eval()
        self.mem_label = self.obtain_labels()
        model.train()
    pred = self.mem_label[idx]
    logit, _ = model(image)
    softmax_out = F.softmax(logit, dim=1)
    entropy_loss = torch.mean(Entropy(softmax_out))
    msoftmax = softmax_out.mean(dim=0)
    entropy_loss -= torch.sum(-msoftmax * torch.log(msoftmax + 1e-5))
    loss = 0.1 * F.cross_entropy(logit, pred)
    loss += entropy_loss
</code></pre>

srishti-git1110: <p>Hey, please show the code where you’re making <code>.backward</code> calls on (loss) tensor.</p>
<p>This error precisely occurs when you try to make a second <code>.backward</code> call on a tensor whose graph has already lost references to the saved tensors required for grading computation (after the first <code>.backward</code> call) owing the autograd’s aggressive memory freeing mechanism.</p>

soulitzer: <p>Enabling anomaly mode can be helpful here <a href="https://pytorch.org/docs/stable/autograd.html#anomaly-detection" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 2.0 documentation</a><br>
See a similar thread here:<br>
<a href="https://discuss.pytorch.org/t/backward-twice-without-retain-graph-true-where-i-shouldnt/188787" class="inline-onebox">Backward twice without retain_graph=true where I shouldn't</a></p>
Error "RuntimeError: CUDA error: device-side assert triggered"

Flash31: <p>Hello! I have a problem with training a neural network on yolov8. Here is the error “RuntimeError: CUDA error: device-side assert triggered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.”<br>
The error appeared unexpectedly, I started training and everything was fine, but on the next launch this error appeared. I launch it in the console (Windows 11) using the command “yolo task=detect mode=train epochs=20 data=C:\AI\Realtime\data_custom.yaml model=yolov8m.pt imgsz=640”. Video card nvidia 4070, Cuda installed (cuda_12.2.2_537.13_windows), video card drivers are the newest, tried reinstalling torch, did not help. torch.<strong>version</strong> ‘2.0.1+cu118’.</p>

ptrblck: <p>Rerun your code with blocking launches as explained in the error message and narrow down the failing line of code.</p>

Flash31: <p>I run it not through code, but through the console.  How can I set the value here? If it is not possible through a command in the console, then how will it look in code?</p>

ptrblck: <p>You can <code>export</code> env variables in your terminal.</p>
Explicitly passing gradients to torch.optim

bgavran: <p>Hello,</p>
<p>I’m relatively new to PyTorch but not to deep learning and I have a question concerning implementation of gradient updates:</p>
<p><strong>Can I pass custom gradients to torch.optim.Adam and similar optimizers?</strong></p>
<hr>
<p>I’m trying to implement <a href="https://arxiv.org/abs/1608.05343" rel="nofollow noopener">DNI</a> which, in some parts, uses approximations of gradients (which are used in an optimization algorithm such as Adam) to update the parameters.</p>
<p>The way I understand how PyTorch works is that each optimizer contains a list of parameters it’s going to change and when it’s called and it uses torch.autograd.Variable’s .grad parameter as the inputs to its optimization procedure. It then assigns the transformed gradients to that same .grad parameter.</p>
<p>Now, my question is: how can sidestep this procedure where the torch.optim class has the “side effect” using the models parameters, instead of an explicit argument?</p>
<p>I’ve thought of two solutions, none which seem to work:</p>
<ul>
<li>Manually assigning the .grad parameter and then using the optimizer - but <a href="http://pytorch.org/docs/master/autograd.html" rel="nofollow noopener">documentation</a> says “This attribute is lazily allocated and can’t be reassigned.”</li>
<li>Manually changing the param.data value, but then I’d have to redefine the optimizer I want to use myself and that doesn’t seem like a genuine solution to this problem</li>
</ul>
<p>What is the preferred way of doing this?</p>
<hr>
<p>I’ve seen one <a href="https://discuss.pytorch.org/t/train-without-backprop/4300/2">post</a> that touches upon this question, and actually has several links to projects that are doing something similar.</p>
<p>But those don’t seem to be genuine solutions to this problem (the DFA project is using the 2nd method I described) and the DNI project doesn’t seem to have a clear explanation of what it is actually doing.</p>
<p>I figured I’d start a topic to discuss a principled way to solve problem of usage of custom gradients in torch.optim and provide a clear reference to people who are trying to solve the same problem as me.</p>
<p>I apologize if I’m breaking any rules or missing something obvious</p>
<p>Cheers!</p>

Fatma_Guney: <p>Have you found a solution for that?<br>
I think one solution is to explicitly assign the grad value of the variable as suggested here: <a href="https://discuss.pytorch.org/t/how-to-use-modified-gradient-to-update-parameters/13029/9">How to use modified gradient to update parameters</a></p>

depshad: <p>Did you find a solution? I’m implementing a method which needs gradient for a different variable to update the learnable parameters.</p>

albanD: <p>Hey!<br>
All the optimizer have a function that takes the gradient as input explicitly. For example <code>torch.optim.adam.adam()</code></p>

depshad: <p>Can you please share an example or the doc where I can learn more? Also how can I check what values of grad is the optimizer using?</p>
C++ Cmake can't find "libc10.so.1.8.0"

Brendan_Lydon: <p>Hi,</p>
<p>I have 2 separate cmake files, one for my development project and one for my unit testing w/ Boost.Test. I followed the instructions in the docs for the development cmake file and it works flawlessly. However, I am trying to build a separate Cmake file for my unit testing and I keep seeing the following error, I am pulling the TWS API in as well:</p>
<p>Cmake Error in CmdLine:</p>
<blockquote>
<p>CMake Error at /usr/lib/x86_64-linux-gnu/cmake/Caffe2/Caffe2Targets.cmake:116 (message):<br>
The imported target “c10” references the file</p>
<pre><code> "/usr/lib/lib/libc10.so.1.8.0"
</code></pre>
<p>but this file does not exist.  Possible reasons include:</p>
<ul>
<li>
<p>The file was deleted, renamed, or moved to another location.</p>
</li>
<li>
<p>An install or uninstall procedure did not complete successfully.</p>
</li>
<li>
<p>The installation package was faulty and contained</p>
<p>“/usr/lib/x86_64-linux-gnu/cmake/Caffe2/Caffe2Targets.cmake”</p>
</li>
</ul>
<p>but not all the files it references.</p>
<p>Call Stack (most recent call first):<br>
/usr/lib/x86_64-linux-gnu/cmake/Caffe2/Caffe2Config.cmake:116 (include)<br>
/usr/lib/x86_64-linux-gnu/cmake/Torch/TorchConfig.cmake:68 (find_package)<br>
CMakeLists.txt:4 (find_package)</p>
<p>– Configuring incomplete, errors occurred!<br>
Error: could not load cache</p>
</blockquote>
<p>My CMake File:</p>
<blockquote>
<p>cmake_minimum_required(VERSION 3.18 FATAL_ERROR)<br>
project(myProject/tests)</p>
<p>find_package(Torch REQUIRED)<br>
set(CMAKE_CXX_FLAGS “${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}”)<br>
find_package(Boost REQUIRED COMPONENTS unit_test_framework)</p>
<p>set(TWS_SOURCE “~/TWS_API/IBJts/source/cppclient/client”)</p>
<p>file(GLOB SOURCES <em>.cpp ${TWS_SOURCE}/</em>.cpp)<br>
file(GLOB TEST_SOURCES “Test_*.cpp”)</p>
<p>include_directories(${TWS_SOURCE})<br>
include_directories(“~/TWS_API/IBJts/source/cppclient/”)</p>
<p>foreach(TEST_SOURCE ${TEST_SOURCES})<br>
get_filename_component(TEST_NAME ${TEST_SOURCE} NAME_WE)</p>
<pre><code>add_executable(${TEST_NAME} ${TEST_SOURCE})

target_link_libraries(${TEST_NAME} PRIVATE Boost::unit_test_framework ${TORCH_LIBRARIES})
target_include_directories(${TEST_NAME} PRIVATE ${TORCH_INCLUDE_DIRS})

add_test(NAME ${TEST_NAME} COMMAND ${TEST_NAME})
</code></pre>
<p>endforeach()</p>
</blockquote>
<p>runtests.sh file to build and run the test_suite</p>
<blockquote>
<p>if [ ! -d “build” ]; then<br>
mkdir -p build<br>
fi<br>
cd build<br>
cmake -DCMAKE_PREFIX_PATH=~/libtorch …<br>
cmake --build . --config Release<br>
if [ $? -eq 0 ]; then<br>
clear<br>
ctest<br>
fi</p>
</blockquote>
<p>Any help would be appreciated!</p>
<p>Thanks,<br>
Brendan</p>

Mitchelll: <p>$ sudo ln -s /usr/lib/x86_64-linux-gnu/ /usr/lib/lib</p>
<p>Since libc10.so.1.8.0 is located in /usr/lib/x86…, making  a soft link solves the problem</p>
New dataloader process during training

p_q: <p>Hi Pytorch team,<br>
I’m using pytorch to traing a model, I see during the training, 3 new dataloader processes have been created, I would like to under what conditions these processes will be created?</p>
<p>the number of workers is 8, so there should be 8 dataloader processes,<br>
root      95860  71732 78 Sep29 ?        18:34:04 name<br>
root      95919  71732 78 Sep29 ?        18:25:29 name<br>
root      95980  71732 73 Sep29 ?        17:14:03 name<br>
root      95983  71732 76 Sep29 ?        18:02:14 name<br>
root      96029  71732 78 Sep29 ?        18:33:04 name<br>
root      96069  71732 78 Sep29 ?        18:31:12 name<br>
root      96102  71732 77 Sep29 ?        18:12:11name<br>
root      96112  71732 78 Sep29 ?        18:34:11 name<br>
root      99822  71732  1 Sep29 ?        00:09:25 name<br>
root      99831  71732  1 Sep29 ?        00:10:11 name<br>
root      99848  71732  1 Sep29 ?        00:09:39 name</p>
<p>As shown above, there are 3 new processes have been created 10 minutes before.</p>
Use dynamo and tensorRT backend for structured sparsity

se2: <p>I was trying to see how much structured sparsity can improve the huggingface Llama 2 model. I read this <a href="https://pytorch.org/TensorRT/user_guide/getting_started_with_fx_path.html#getting-started-with-fx" rel="noopener nofollow ugc">doc</a> and it seems I can somehow use the tensorRT backend with <code>sparse_weights</code> turned on. But it sounds like the <code>acc_tracer</code> and jit in general doesn’t play very nicely with control flows.</p>
<p>If dynamo can output fx graphs easily is it possible to use dynamo first on the model and then simply use the <code>TRTInterpreter</code> on the output fx graph for sparse weight acceleration? Any tips will be appreciated!</p>

marksaroufim: <p>So you should be able to use this with partial graphs <a href="https://github.com/pytorch/TensorRT/tree/main/examples/dynamo#id1">https://github.com/pytorch/TensorRT/tree/main/examples/dynamo#id1</a></p>

se2: <p>I see, thanks for the help! I checked the <a href="https://github.com/pytorch/TensorRT/blob/main/py/torch_tensorrt/dynamo/_settings.py" rel="noopener nofollow ugc">py/torch_tensorrt/dynamo/_settings.py</a> and it doesn’t seem to have options to explicitly enable/disable sparsity. Is structured sparsity used behind the scene somehow? I mainly wants to test the performance difference with &amp; without sparsity acceleration enabled.</p>
Streaming/chunking responses using TorchServe on Vertex AI

Eric_Sanchez: <p>I’m production-izing a transformer model deployed on a Vertex AI endpoint, but I’m running into issues that may be a limitation of the Vertex AI product.</p>
<p>The input is a single image and the output is a single image, however, the vertex ai endpoint fails due to the response size being too large, e.g.</p>
<pre><code class="lang-auto">ERROR: (gcloud.ai.endpoints.raw-predict) HTTP request failed. Response:
{
  "error": {
    "code": 400,
    "message": "Response size too large. Received at least 31517591 bytes; max is 31457280.",
    "status": "FAILED_PRECONDITION"
  }
}
</code></pre>
<p>This seems like a similar limit on Cloud Run reqs/responses, so I figured I need to chunk/stream responses back from this endpoint.</p>
<p>I’m using the <code>send_intermediate_predict_response</code> function to send chunked responses back, but it seems like this chunking only happens within torch serve, i.e. from its backend to its frontend and then the vertex ai endpoint attempts to send the entire response back.</p>
<p>Is there a way to get torch serve + vertex ai to return a chunked response?</p>

Eric_Sanchez: <p>This doesn’t seem possible. The solution for this was to write results to GCS and serve results from GCS.</p>

JamesTrick: <p>Hi Eric</p>
<p>Google cloud has a limit of 32MB as you’ve found. The best way to ‘increase’ this limit would be to use GCS and signed requests. Albeit in javascript, this <a href="https://medium.com/google-cloud/bypass-the-cloud-run-32mb-limit-with-cloud-storage-65c24156189" rel="noopener nofollow ugc">blog post is pretty helpful at explaining it</a>.</p>
<p>Ensure you’ve also increased the config within torchserve to handle the larger file size too, <a href="https://pytorch.org/serve/Troubleshooting.html#unable-to-send-big-files-for-inference-request" rel="noopener nofollow ugc">docs here</a>.</p>

Nedyali: <p>Perhaps consulting with a professional experienced with VertexAI will help resolve this issue <a href="https://tech-stack.com/services/big-data-and-analytics" rel="noopener nofollow ugc">https://tech-stack.com/services/big-data-and-analytics</a></p>
Epochs show large difference in running time

yang_chen: <p>Hi,<br>
When I ran my pytorch model in HPC. I used 4 CPU cores, batch size =1.<br>
The running time of each epoch ranges from 200-4000s. I tested 35 jobs. And only 5 jobs shows shuch results. In other jobs, each epoch only run in 80-90s. Would anyone to give some suggestions on such results?</p>
<p>Here is some logs:<br>
epoch:0; Training Loss:[0.07358009 0.         0.         0.         0.        ]<br>
474.0903596878052<br>
epoch:1; Training Loss:[0.06944189 0.         0.         0.         0.        ]<br>
603.3289322853088<br>
epoch:2; Training Loss:[0.06749273 0.         0.         0.         0.        ]<br>
4037.6397545337677<br>
epoch:3; Training Loss:[0.06656914 0.         0.         0.         0.        ]<br>
900.0050981044769<br>
epoch:4; Training Loss:[0.06594099 0.         0.         0.         0.        ]<br>
414.7940049171448<br>
epoch:5; Training Loss:[0.0652342 0.        0.        0.        0.       ]<br>
1492.0406074523926<br>
epoch:6; Training Loss:[0.06477792 0.         0.         0.         0.        ]<br>
1997.3091661930084<br>
epoch:7; Training Loss:[0.06439936 0.         0.         0.         0.        ]<br>
1552.9386072158813<br>
epoch:8; Training Loss:[0.06404987 0.         0.         0.         0.        ]<br>
1519.947157382965<br>
epoch:9; Training Loss:[0.06379405 0.         0.         0.         0.        ]<br>
1737.3557515144348<br>
epoch:10; Training Loss:[0.06337144 0.         0.         0.         0.        ]<br>
323.754665851593<br>
epoch:11; Training Loss:[0.0630567 0.        0.        0.        0.       ]<br>
220.557555437088</p>
<p>Thank you for help.</p>

albanD: <p>Hey!<br>
There can be many many reasons for this from cpu usage, to slow disk, or remote disks even depending on your cluster.<br>
I would suggest profiling your training loop to see which part causes this slow down and go from there!</p>

yang_chen: <p>Thank you. I will do and check which part of the code cost too many time.</p>
Issue with Variational encoder - while doing forward pass with Pytorch lightning

pankaja12: <ul>
<li>I followed <a class="mention" href="/u/kinyugo">@Kinyugo</a>  - notebook at<br>
<a href="https://stackoverflow.com/questions/64909658/what-could-cause-a-vaevariational-autoencoder-to-output-random-noise-even-afte/77202840#77202840" rel="noopener nofollow ugc">Stackoverflow - Kinyugo</a><br>
and the variational_encoder python script and even with the refactored LitVAE code, I am getting error at the forward pass.</li>
</ul>
<p>Attaching the screenshot <a href="https://i.stack.imgur.com/LePTF.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/LePTF.png" alt="enter image description here" width="690" height="345"></a></p>
<p><span class="mention">@ptrblck_de</span> - Any help is greatly appreciated. TIA.</p>
ArrayMemoryError - How should I organize input data for CNN?

mbblomquist: <p>Hello,</p>
<p>I am attempting to create a convolutional neural network for a regression task that takes in two frames of a video and outputs the relative distance moved between an object that is located in both frames. To start, I have 15 different videos (.mp4), so I am looping through each video, reading each frame with OpenCV, and appending the uint8 matrix of each frame to a numpy array. Then eventually I will convert this to a tensor and split into training and validation sets.</p>
<p>However, in the loop of the appending each frame to the numpy array, I get an ArrayMemoryError specifying that it is “Unable to allocate 30.7 GiB for an array with shape (61649, 498, 1072) and a type uint8”.</p>
<p>I understand that is a lot of data in one array (and I’ve read that CNNs perform better when data is stored as Float32 instead of uint8 which increases the memory also), so is there a better way to load/save the data so not all of it has to allocated at once? If so, can I still randomly split into training and validation sets for my CNN?</p>
<p>I apologize if this is rudimentary - this is my first time attempting to create a deep learning model, so still trying to figure out organization techniques!</p>
<p>Thank you for your help! Let me know if any other information is needed to better answer this question.</p>
Autograd.set_detect_anomaly clashes with CUDA Graphs

yifannie: <p>Hello community,<br>
We notice that if we set autograd.set_detect_anomaly(True), autograd.grad becomes not permitted in CUDA Graphs. We could not find any description on this topic on either CUDA Graphs and PyTorch. Could anyone please explain why it is the case, what is changed by enabling detect_anomaly, and is it safe for us to turn off detect_anomaly and safely accept the results calculated from CUDA Graphs?<br>
Thanks!<br>
Best wishes,<br>
Yifan</p>

albanD: <p>Hey!</p>
<p>detect_anomaly should only be enabled when you’re debugging (it’s pretty slow!) so if your code is working properly, it is ok to disable it before running with cudagraph yes.<br>
If you do have issues with NaN during the backward pass, I would suggest disabling cudagraph while debugging these (with anomaly mode) and then re-enable cudagraph once they are fixed.</p>
Difference ._grad and .grad

Gianluca_Maselli: <p>Hello,</p>
<p>I’m currently implementing A3C from scratch and I have some questions related to how the gradients are copied from the local model into the shared one. In particular, I’ve seen that after computing the loss and calling <code>.backward()</code> most of the versions already present on GitHub use the following slice of code:</p>
<pre><code class="lang-auto">for lp, sp in zip(local_net.parameters(), shared_net.parameters()):
     sp._grad = lp.grad
</code></pre>
<p>My questions are mainly 2:</p>
<ol>
<li>why did they not use simply <code>sp.grad = lp.grad.clone()</code> ?</li>
<li>what is the difference between <code>._grad</code> and <code>.grad</code> ?</li>
</ol>
<p>Thank you in advance for the help.</p>

srishti-git1110: <p>I looked into this a bit, and apparently <code>._grad</code> is an attribute of objects of type <code>nn.Parameter</code>. An underscore _ trails its name so it’s meant NOT to be modified according to PEP8 but that’s exactly what’s being done here and in various other implementations that I looked up and that you also mention.</p>
<p>I also couldn’t find a lot of info on the purpose of having <code>._grad</code>, while <code>grad</code> is already there to store, retrieve and possibly modify the gradients, apart from them being there as attributes in the stub file and also that their value is same as that of <code>.grad</code> after a <code>.backward</code> call.</p>
<p>BTW, I’m not into RL but since you mention using <code>.clone()</code>, that’d make sense if you don’t want the changes in <code>local_net</code>’s parameters’ <code>.grad</code> to reflect in <code>shared_net</code>’s parameters’ <code>.grad</code> and just want to explicitly copy the gradients from one model to another.</p>
<p>Couldn’t help much here sorry!</p>
<p>Edit: Also checked whether <code>.grad</code> and <code>._grad</code> refer/point to the same object in the memory by printing their reference address and by using <code>is</code> and they do!</p>

Gianluca_Maselli: <p>Thank you so much Srishti, I think you answered all my doubts.</p>

albanD: <p>The two are exactly the same and point to the same object. <code>._grad</code> is only here for backward compatibility reasons as some very old code still uses it. But you can replace it with no issue: <code>sp.grad = lp.grad</code> will do exactly the same thing</p>
yolov8 pytorch model Android vulkan usage

thegkhn: <p>When I integrated my model that I previously trained using yolov8 into Android, I saw that it always runs on cpu. What can I do to make PyTorch work on vulkan and nnapi. When I add backend=vulkan parameter to optimise_for_mobile function, Unknown custom class type vulkan.BatchNormPackedContext. Please ensure it is registered.: I get this error. I could not solve this in any way. How can I make a yolo model in PyTorch format to work with vulkan and nnapi on Android?</p>
Dataloader throws error when iterating

gaurabdhg: <p>I have a custom dataset where inputs and labels are both tensors.</p>
<pre><code class="lang-auto">class atDataset(Dataset):
    def __init__(self, file_paths, seed,transform=None):
        self.file_paths = file_paths
        self.transform = transform
        self.seed=seed

    def __len__(self):
        return len(self.file_paths)

    def __getitem__(self, idx):
        file_path = self.file_paths[idx]
        data_dict = th.load(file_path)
        th.manual_seed(self.seed[idx])
        _data=list(data_dict)[0]
        _noise=th.rand_like(_data)
        
        _roll_param=th.randint(-10,10,(2,))
        _data=th.roll(_data,shifts=tuple(_roll_param),dims=(0,1))
        _label=list(data_dict.values())[0]
        
        return _data+_noise,_label
</code></pre>
<p>The data is in “*.pt” files, and each file contains a single dictionary {data : label}.</p>
<p>The data is of shape 320x320 but labels are of shape (nx3) where n varies.<br>
Iterating on the dataloader throws:<br>
<code>RuntimeError: stack expects each tensor to be equal size, but got [4497, 3] at entry 0 and [3281, 3] at entry 1.</code><br>
How can I resolve this without resizing or padding the tensors.</p>

marksaroufim: <p>It sounds like you need to implement a custom <code>collate_fn()</code> - you can see this thread for how people deal with batches of variable sizes <a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/13" class="inline-onebox">How to create a dataloader with variable-size input - #13 by pinocchio</a></p>
Torch.nonzero breaks gradient?

leopard86: <p>I’m defining a custom loss that finds FFT peaks (didn’t find anything similar so far).<br>
I have found a peak picking method online that works well, but when using it in training I get a problem of grad_fn being none at some tensor.<br>
(this is the method from stackoverflow: <a href="https://stackoverflow.com/questions/54498775/pytorch-argrelmax-function-or-c" class="inline-onebox" rel="noopener nofollow ugc">python - Pytorch Argrelmax function (or C++) - Stack Overflow</a>)</p>
<p>I have been debugging the loss, and found that at this point the gradient breaks:<br>
peaksHgen = torch.nonzero(hgpk2, out=None) + 1</p>
<p>What I see is that hgpk2 has grad_fn correctly set to WhereBackward0, while peaksHgen has grad_fn=None</p>
<p>Why is that? Doesn’t nonzero have a gradient function?<br>
I’m not publishing my code here because it is identical to the one seen on stackoverflow besides changing variable names.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/leopard86">@leopard86</a>,</p>
<p>I’m pretty sure <code>torch.nonzero</code> doesn’t have a gradient function, because what would be the gradient function of a function that returns the index in an array whose value is not 0? In order for a function to have a derivative it needs to be locally continuous, which <code>torch.nonzero</code> is not.</p>
How to ensure each element in a tensor with random dimensions above the threshold?

new_py: <p>Before posting a query, check the <a href="https://discuss.pytorch.org/c/faq/30">FAQs</a> - it might already be answered!</p>
<p>Hi, I wanted to make sure each element inside the tensor is above the thredshold,</p>
<p>for example, <code>t = torch.rand(2, 3)</code>, if any each t[i][j] &lt;0.1, i want it be be 0.1.</p>
<p>How to do that?</p>

gaurabdhg: <p>Just set a threshold value and apply mask in the index.</p>
<pre><code class="lang-auto">t = th.rand(2, 3)
threshlold=# Define your threshold
t[t &lt; threshold] = #your value
</code></pre>
<p>This should do the job.</p>
I own www.pytorch.space

iqcw0rx: <p>Hey,</p>
<p>New here, have a portfolio with five thousand domains and zero working experience in tech, however major stake in qubit and aerospace software and defense sectors.</p>
<p>Suck at social, but want to contribute. Not looking to get paid, just want to contribute and learn.<br>
Thanks for considering me!</p>

ptrblck: <p>Could you explain what exactly you are trying to contribute? Are you interested in working on a specific feature?</p>

iqcw0rx: <p>i honestly am a n00b and my goal was to actually distinguish pytorch.space from other rootzones in that it only serves aerospace. I want to learn and discover what usecases we as a community can come up. the little i’ve seen about using pytorch for generative pentesting and the recent hackasat competition are a good start. anyways, let me know</p>
When will GitHub MPS issue #102436 be implemented?

KMT62: <p>Back in May '23 an issue was triaged, “<a href="https://github.com/pytorch/pytorch/issues/102436" rel="noopener nofollow ugc">multiple mps for base X86 Mac with multiple gpus</a>”, which has been largely ignored. Even in this forum, some still insist that all Macs only ever had/have one GPU. This issue is one of the many that is holding MPS back as a viable device for PyTorch.</p>
<p>Please accelerate resolution of this issue.</p>
A Novice question about slicing operation

Ashawojiangkele: <p>【】I have an index matrix<br>
A = torch.tensor([[1, 3], [2, 5], [4, 7]])<br>
【】I have a vector，too<br>
B = torch.tensor([10, 20, 30, 40, 50, 60])<br>
【】but how can i got a slice like this？    <img src="https://discuss.pytorch.org/images/emoji/apple/arrow_lower_left.png?v=12" title=":arrow_lower_left:" class="emoji" alt=":arrow_lower_left:" loading="lazy" width="20" height="20">(<span class="hashtag-raw">#_</span>#)↓<br>
C=B[A[:, 0]:A[:, 1]]</p>

KFrank: <p>Hi Ashawojiangkele!</p>
<aside class="quote no-group quote-modified" data-username="Ashawojiangkele" data-post="1" data-topic="189054" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e56c9b/48.png" class="avatar"> Ashawojiangkele:</div>
<blockquote>
<p>A = torch.tensor([[1, 3], [2, 5], [4, 7]])<br>
…<br>
C=B[A[:, 0]:A[:, 1]]</p>
</blockquote>
</aside>
<p>If I understand correctly what you want to do, you can’t do this with pytorch<br>
tensors.</p>
<p>It looks like you want the first row of <code>C</code> to be the 1 and 2 elements of <code>B</code>, that is,<br>
<code>B[1]</code> and <code>B[2]</code>, you want the second row of <code>C</code> to be the 2, 3, and 4 elements,<br>
and so on.  That is, you want the rows of <code>C</code> to have differing lengths.</p>
<p>The problem is that pytorch does not support such “ragged” tensors where<br>
rows (or columns, etc.) don’t all have the same length.</p>
<p>As an aside, <code>B</code> has length 6 (elements <code>B[0], ..., B[5]</code>), so if you did what<br>
you wanted, the third row of <code>A</code>, <code>[4, 7]</code>, would only slice out <em>two</em> elements<br>
(elements 4 and 5) because no element 6 exists.  (<code>B[5] = 60</code> and <code>B[6]</code> would<br>
give you an index-out-of-bounds error).</p>
<p>Best.</p>
<p>K. Frank</p>
Static link to a specific CUDA 12.1 libtorch version

sonjab: <p>I am integrating libtorch into my project and I need a static link to where I can download libtorch from. The only link that I’m able to find for CUDA 12.1 is <a href="https://download.pytorch.org/libtorch/nightly/cu121/libtorch-cxx11-abi-shared-with-deps-latest.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/nightly/cu121/libtorch-cxx11-abi-shared-with-deps-latest.zip</a>.<br>
The issue here is that this link is always pointing to the “latest” version which changes every now and then but I want to be in control of which version I have and when I update it. Where can I find a static link to this version or older versions for CUDA 12.1 instead of this link to the latest release?</p>

ptrblck: <p>The next 2.1.0 release will use CUDA 12.1 and you should be able to tag it then with a static URL.</p>

sonjab: <p>Thanks! That’s great news. Do you know when we can roughly expect that release?</p>

ptrblck: <p>My best guess is in the next few weeks.</p>
Hook function and list of losses with autograd

Martin_Barry: <p>Hello everyone ! =)</p>
<pre><code class="lang-auto">import torch

# Define hook functions
value = 0
def hook_fn1(grad):
    global value
    print("Hook 1 called")
    value += 1
def hook_fn2(grad):
    print("Hook 2 called")

# Create tensors with requires_grad=True
x = torch.randn(2, 2, requires_grad=True)
y = torch.randn(2, 2, requires_grad=True)

# Register hook_fn1 on the first tensor
x.register_hook(hook_fn1)

# Register hook_fn2 on the second tensor
y.register_hook(hook_fn2)

# Define loss1 as the sum of x.sum() and y.mean()
loss1 = x.sum()
loss2 = x.sum() + y.mean()
# Use torch.autograd to compute gradients for loss1
torch.autograd.backward([loss1,loss2])
# Both hooks registered on x and y will be called
print(value)
</code></pre>
<p>I ran this code expecting that it should go through twice the hook_fn1 but it only goes once. I am not sure to understand why.<br>
How can I adapt my code to make it go through for each? I don’t want to use a for loop because for big network it would take forever</p>

janeyx99: <p>Hi! Running this code snippet shouldn’t run any of the hooks, actually, since the hooks are called in the backward and there are no .backward() calls in the snippet.</p>
<p>If you are running the following, you should get that <code>hook_fn1</code> is called twice (<code>value=2</code>) while <code>hook_fn2</code> is called just once.</p>
<pre><code class="lang-auto">loss1.backward()
loss2.backward()
</code></pre>
<p>Is this the behavior you’re seeing?</p>

Martin_Barry: <p>Thanks for your answer but you missed the backward in my code</p>
<pre><code class="lang-auto">torch.autograd.backward([loss1,loss2])
</code></pre>
<p>this is where the bacward is. The point is I want to avoid using multiple backward and use the pytorch implementation</p>

janeyx99: <p>Welp LOL I just realized I could scroll down–sorry about that. The way the <code>torch.autograd.backward</code> API works is that it’ll sum <code>loss1</code> and <code>loss2</code> and then do a backward on that, which is why you observe your current behavior vs having <code>value</code> be 2.</p>
<p>I am curious what your use case is, as the closest API that comes to mind that would compute multiple forwards/backwards at a time would be per-sample gradients <a href="https://pytorch.org/tutorials/intermediate/per_sample_grads.html" class="inline-onebox" rel="noopener nofollow ugc">Per-sample-gradients — PyTorch Tutorials 2.0.1+cu117 documentation</a> but that API assumes a very particular structure where all the losses you are computing are pointwise. Here it looks like you have some dependency structure, so it may not be helpful.</p>

Martin_Barry: <p>Basically what backward does is compute per summed batch gradient</p>
<p>grad = sum_batch (dl_batch / d wik) but I would like to compute the squared version<br>
grad_squared = sum_batch (dl_batch / d wik) ^2.</p>
<p>Sorry could not find the equations</p>
Confusing nature of optim.LBFGS

Sentient07: <p>Hello!</p>
<p>I’d like to use LBGFS as an alternative to <code>scipy.optimize.fmin_l_bfgs_b</code>. I’m currently using <code>optimizer = torch.optim.LBFGS([U], lr=0.1, max_iter=50, line_search_fn='strong_wolfe')</code> as my optimizer. However, I don’t understand why we would need a parameter of learning rate when using line-search mechanism. I observe stark difference in results when using scipy’s function to torch’s. My objective is a polyconvex as explained in this paper(<a href="https://arxiv.org/pdf/2102.03069.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2102.03069.pdf</a>). Please also see the implementation in scipy in Listing 1 that I’m porting in pytorch. Reason to port this is that when I would use this on different object categories, I don’t want to compute analytical gradients. Thanks for the tips/suggestions</p>
Torch-mlir operation registry access

Stella_Laurenzo: <p>Hi folks – we’ve been doing a sprint to uplift <a href="https://github.com/llvm/torch-mlir" rel="noopener nofollow ugc">torch-mlir</a> to be fully based on <code>torch.compile</code> and <code>torch.export</code>, eliminating the need for the TorchScript era interop. The work is going well, and I’ve been really pleased to dig in and see the amazing progress the team has made on the PyTorch side in bringing all of these features together. (side note: I was elsewhere for a couple of years and it is rare to step away and come back to find so many things that were causing issues before to have been overhauled and vastly improved – kudos to everyone)</p>
<p>In any case, we’ve now got a path working without any C++ dependency on PyTorch, and it is doing pure Python bridging to the <code>torch</code> dialect in MLIR, using only PyTorch and MLIR’s Python APIs. It’s really nice to have the features (dynamic shapes being the big one I’m happy about) working this way for both eager and exported paths, and I’m looking at how to clean up the rest and sequester the old TorchScript paths to a legacy part of the project.</p>
<p>All that’s left that is load bearing is <a href="https://github.com/llvm/torch-mlir/blob/main/python/torch_mlir/dialects/torch/importer/jit_ir/csrc/get_registered_ops.cpp#L44" rel="noopener nofollow ugc">this <code>getRegisteredOps</code> API</a> that we added ages ago. It is used by project maintainers to introspect the PyTorch registry of ops and do code generation for a majority of the <a href="https://github.com/llvm/torch-mlir/blob/main/include/torch-mlir/Dialect/Torch/IR/GeneratedTorchOps.td" rel="noopener nofollow ugc">torch dialect ops</a>. It is only used by our <a href="https://github.com/llvm/torch-mlir/blob/main/python/torch_mlir/dialects/torch/importer/jit_ir/build_tools/torch_ods_gen.py" rel="noopener nofollow ugc"><code>torch_ods_gen.py</code></a> utility to generate that file. I remember when I wrote this many years ago, there wasn’t an easy way to get at this information from Python. I figured I’d ask now if anyone had a suggestion… if we eliminate this piece, then the core of the torch-mlir project just becomes dialects and compiler infra with no build time dependency on PyTorch, which would be really nice.</p>
<p>Let me know if anyone has suggestions, and it’ll save me some effort of poking around and seeing what the right way to do this now is.</p>
<p>Thanks.<br>
Stella</p>
<p>(and again: thanks so much for the great work, especially on the export path. it is such a vast improvement over what was there before)</p>
Inductor CPP codegen for WebAssembly target

vadimkantorov: <p>Can PyTorch generator AOT kernels for the wasm/browser target (e.g. generate AITemplate style a self-contained, minimal ggml/llama.cpp-style C++ program to benefit from wasm-simd and link it only to XNNPACK which also exists for wasm)?</p>
<p>Related to <a href="https://discuss.pytorch.org/t/small-depthwise-conv1d-maximum-perf-on-cpu/186408/4" class="inline-onebox">Small depthwise Conv1d: maximum perf on CPU? - #4 by smth</a></p>

marksaroufim: <aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/test/cpp/aot_inductor/test.py">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/test/cpp/aot_inductor/test.py" target="_blank" rel="noopener">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/test/cpp/aot_inductor/test.py" target="_blank" rel="noopener">pytorch/pytorch/blob/main/test/cpp/aot_inductor/test.py</a></h4>


      <pre><code class="lang-py">import shutil

import torch
from torch._export import aot_compile, dynamic_dim


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = torch.nn.Linear(64, 10)
        weights = torch.arange(640)
        weights = torch.reshape(weights, (10, 64))

        with torch.no_grad():
            self.fc.weight.copy_(weights)
            self.fc.bias.copy_(torch.zeros(10))

    def forward(self, x, y):
        return self.fc(torch.sin(x) + torch.cos(y))

</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/main/test/cpp/aot_inductor/test.py" target="_blank" rel="noopener">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


vadimkantorov: <p>I guess for this to work, we’d need to get raw self-contained C++ files as output + maybe Makefiles / compilation commands / CMake / etc, so that we can then build them using Emscripten</p>
<p>Maybe some libtorch.so-mobile version can be built for wasm as well, but super-aggressive tree-shaking is needed to reduce the size. Ideally we’d just need to have only used ops wasm-simd C++ code + XNNPACK.</p>
Optimizer not loading state dict to correct device

mmisiur: <p>Hey,<br>
I am saving state dict and then loading it for further training. I specify mapping location to cuda while doing torch.load, and it works for model weights, but not for the optimizer. Could you help me?</p>
<pre><code class="lang-auto">checkpoint = torch.load(net_path, map_location=torch.device("cuda:0"))
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])  # optimizer state tensors are loaded onto cpu
model.load_state_dict(checkpoint['model_state_dict'])  # weights are loaded into cuda
</code></pre>
<p>python 3.10<br>
pytorch 2.0.0</p>

tom: <p>Did you move the parameters to cuda before instantiating the optimizer?<br>
The optimizers will cast and transfer the weights to the type and device of the parameters.</p>
<p>Best regards</p>
<p>Thomas</p>

mmisiur: <p>Hi Tom, thank you for your response!<br>
You are right - instantiated the optimizer before loading model’s parameters to cuda. I didn’t know it works that way. Thank you!</p>
Backward() with manipulation on weight tensor in training loop

Lukas_Hebing: <p>I am trying to implement a training loop, in which I only want to update a subset of weights of a weight matrix. According to <a href="https://discuss.pytorch.org/t/only-train-certain-rows-of-a-linear-layer/118940">this discussion</a>, or <a href="https://discuss.pytorch.org/t/prevent-updating-weight-for-specific-element-of-weight-matrix/135661">this</a>, it is possible to let the optimizer do the step and then manually set some parts of the weights to their initial values.</p>
<p>A toy example is this: Here I try to use ADAM on the parameter <code>x</code>, but want to keep its last column. Therefore, I save the initial values as <code>xsave</code> and set the values by <code>torch.where(...)</code>.</p>
<pre><code class="lang-auto">x = torch.nn.Parameter(torch.randn(2, 3))
  
xsave = x.clone().detach()
mask_prevent_update = torch.full(x.shape, False)
mask_prevent_update[:, 2] = True
  
def loss_func():
    return torch.sum(x**2)
print(loss_func())
  
optimizer = torch.optim.Adam([x], lr=.1)
  
for i in range(100):
    optimizer.zero_grad()
    loss = loss_func()
    loss.backward()
    optimizer.step()
    x = torch.where(mask_prevent_update, xsave, x)
</code></pre>
<p>However, this operation causes the error “RuntimeError: Trying to backward through the graph a second time”</p>
<p>How can I avoid the error and still do the reset-operation?</p>

KFrank: <p>Hi Lukas!</p>
<aside class="quote no-group quote-modified" data-username="Lukas_Hebing" data-post="1" data-topic="189046" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lukas_hebing/48/64242_2.png" class="avatar"> Lukas_Hebing:</div>
<blockquote>
<p>I am trying to implement a training loop, in which I only want to update a subset of weights of a weight matrix.<br>
…</p>
<pre><code class="lang-auto">x = torch.nn.Parameter(torch.randn(2, 3))
...
    x = torch.where(mask_prevent_update, xsave, x)
</code></pre>
</blockquote>
</aside>
<p>This last line sets the python variable <code>x</code> to refer to the new tensor created by<br>
<code>torch.where()</code>.  It doesn’t modify the <code>Parameter</code> to which <code>x</code> originally referred.</p>
<p>Try:</p>
<pre data-code-wrap="python"><code class="lang-python">    with torch.no_grad():
        x.copy_ (torch.where (mask_prevent_update, xsave, x))
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
How to include transforms.functional.equalize in a compose code block

Schroter: <p>Hi All,</p>
<p>I have a code block which I took from <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">here</a>. I am trying to customise it and want to include<code> transforms.functional.equalize</code> in the compose section as below:</p>
<pre><code class="lang-auto"># Data augmentation and normalization for training
# Just normalization for validation
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224,scale=(0.85, 0.95)),
        transforms.functional.equalize() #Not sure how to include it here
        transforms.RandomAdjustSharpness(1.2),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

</code></pre>
<p>Would anyone be able help me to include the transforms.functional.equalize function in the compose pipline.</p>
<p>Thanks &amp; Best Regards<br>
Michael</p>

ptrblck: <p>You can create a new transformation module and add it to the <code>Compose</code> object. Take a look at e.g. <a href="https://github.com/pytorch/vision/blob/1a9ff0d728787df66a372283328158879805dcad/torchvision/transforms/transforms.py#L107"><code>ToTensor</code></a> to see how the object is implemented and how the functional API is used.</p>
R3D Expected channels error

thomas_dumont: <p>Hello,</p>
<p>i’ve been trying to use the r3d_18 pretrained model from pytorch :</p>
<p><a href="https://pytorch.org/vision/main/models/generated/torchvision.models.video.r3d_18.html#torchvision.models.video.r3d_18" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/models/generated/torchvision.models.video.r3d_18.html#torchvision.models.video.r3d_18</a></p>
<p>but i keep having this error :</p>
<p>RuntimeError: Given groups=1, weight of size [64, 3, 3, 7, 7], expected input[1, 1, 3, 112, 112] to have 3 channels, but got 1 channels instead</p>
<p>Yet, from what i understand from the error, my input which is BTCHW format has 3 channels.</p>
<p>Here is the code that leads to this error :</p>
<p>import cv2<br>
import torch<br>
from torchvision.models.video import r3d_18<br>
import torchvision.transforms as transforms</p>
<h1><a name="load-the-pretrained-r3d-18-model-1" class="anchor" href="#load-the-pretrained-r3d-18-model-1"></a>Load the pretrained R3D-18 model</h1>
<p>model = r3d_18(pretrained=True)</p>
<h1><a name="set-the-model-to-evaluation-mode-2" class="anchor" href="#set-the-model-to-evaluation-mode-2"></a>Set the model to evaluation mode</h1>
<p>model.eval()</p>
<h1><a name="open-the-video-file-3" class="anchor" href="#open-the-video-file-3"></a>Open the video file</h1>
<p>video_capture = cv2.VideoCapture(‘pytorch_resnet/videos/cam_int_grab_2.mp4’)</p>
<p>while True:<br>
# Read a frame from the video<br>
ret, frame = video_capture.read()<br>
if not ret:<br>
break</p>
<pre><code># Ensure the frame has 3 color channels (RGB)
if frame.shape[-1] != 3:
    raise ValueError("Input frame should have 3 color channels (RGB)")

    # Resize and central crop
resize = transforms.Compose([
    transforms.ToPILImage(),
    transforms.Resize((128, 171)),
    transforms.CenterCrop((112, 112)),
    transforms.ToTensor(),
])

# Normalize
normalize = transforms.Compose([
    transforms.Normalize(mean=[0.43216, 0.394666, 0.37645], std=[0.22803, 0.22145, 0.216989]),
])

# Preprocess the frame
frame = cv2.resize(frame, (171, 128))  # Resize to 171x128
frame = resize(frame)  # Resize and central crop
frame = normalize(frame)  # Normalize

# Add batch dimension
frame = frame.unsqueeze(0)  # Shape will be [1, 3, 112, 112]

# Check input tensor shape and data type
print(f"Input tensor shape: {frame.shape}, data type: {frame.dtype}")

# Perform inference
with torch.no_grad():
    output = model(frame)   #ERROR HERE
</code></pre>
<p>Note : i’ve followed r3d documentation for the preprocessing of the frames.</p>