akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Parameter Server with RPC and NCCL

Osama_Abu_Hamdan: <p>Greetings,<br>
I am trying to implement ParameterServer following this <a href="https://pytorch.org/tutorials/intermediate/rpc_async_execution.html" rel="noopener nofollow ugc">tutorial</a><br>
I have two observations<br>
1- The execution is too slow. To train one epoch it takes 2-3 minutes, while in DDP with NCCL it takes 30 seconds.<br>
2- I noticed that the communication backend is GLOO. How to switch it to NCCL?</p>
<p>Also, any other recommendations to speedup the training?<br>
Best</p>

H-Huang: <p>RPC does not use GLOO or NCCL backends and uses <a href="https://github.com/pytorch/tensorpipe" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/tensorpipe: A tensor-aware point-to-point communication primitive for machine learning</a> for its backend. RPC is currently in maintenance mode but there is limited support for doing RPC using cuda tensors should be an order of magnitudes (up to x10) faster <a href="https://pytorch.org/tutorials/recipes/cuda_rpc.html" class="inline-onebox" rel="noopener nofollow ugc">Direct Device-to-Device Communication with TensorPipe CUDA RPC — PyTorch Tutorials 2.4.0+cu124 documentation</a></p>
<p>If there are features requests feel free to create an issue on github or perhaps look into trying to implement parameter server architecture with just collective communication.</p>
Question about the CrossEntroyLoss and BCELoss？

h_jie: <p>During the process of attempting to extract table lines (which I divided into horizontal and vertical lines), I found that when I used CrossEntropyLoss, the loss value remained at a constant level without changing. However, when I used BCELoss, the loss gradually decreased with each iteration.<br>
For example:<br>
Input:    (batch_size, 3, 1024, 1024)<br>
Output: (batch_size, 2, 1024, 1024)<br>
Target:  (batch_size, 2, 1024, 1024)</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> can be used for multi-class classification use cases (where each sample belongs to a single class) while <code>nn.BCEWithLogitsLoss</code> (don’t use <code>nn.BCELoss</code> as it has a lower numerical stability compared to <code>nn.BCEWithLogitsLoss</code>) can be used for multi-label classification use cases (where each sample belongs to 0, 1, or multiple classes).<br>
I don’t know which use case you are working on.</p>

h_jie: <p>For example, in an image, I would first mark the lines of the table (that is, all parts that belong to the table are marked as 1, and those that do not belong to the table are marked as 0). Then I use a model to determine whether each pixel belongs to a line.</p>

ptrblck: <p>In this case it sounds like a multi-class segmentation use case and <code>nn.CrossEntropyLoss</code> would be the commonly used criterion.<br>
The target would then contain class indices in <code>[0, 1]</code> and have the shape <code>[batch_size, height, width]</code> (note the missing class dimension) or would contain “soft” targets (i.e. floating point values between 0 - 1) in the same shape as the model output.</p>
Maintaining class distribution in dataloader after batching

Atia: <p>I have a dataset with two classes say <strong>class a</strong> and <strong>class b,</strong> which is highly imbalanced -  <strong>class a</strong> is about 3000x <strong>class b</strong>.  I am creating a dataloader using a stratified kfold to maintain class distribution. However I noticed that although this distribution is maintained when you take the dataloader as a whole, when I batch them into mini-batches for training, some mini-batches sometimes contain only one class and the distribution is broken. How do I ensure that this distribution is maintained in the mini-batches for training. My first idea was to pass a sampler to the <code>sampler</code> argument in <code>DataLoader</code> but I am facing challenges on how to properly write my own sampler for that purpose mainly because I do not understand properly what the sampler or batch_sampler argument really does.</p>
<p>Could someone kindly point me to the right resoure if available. Thanks in advance.</p>
Weight manipulation

FIavio: <p>Hi everyone!<br>
I’m writing a program to add a watermark to a very simple FFNN model. To do this I have to manually change almost every weight in the model; the process is: I take the value, transform it into binary, change the LSB, transform it back into float.<br>
Now, I managed to do everything, but I’m having a problem that I can’t solve and above all understand.<br>
I take the value, modify it, check that the modification was successful and insert it into the specific weight, the problem is that when I go to verify that the value has been inserted, the value returns to the initial weight, and not the modified one, here I leave log data to make it better understood:</p>
<p><em>param item        -1.2060359716415405</em><br>
<em>rounded param  -1.2060359716415405</em><br>
<em>changed param -1.2060359716415407</em><br>
<em>-----</em><br>
<em>bin param     1011111111110011010010111110110001100000000000000000000000000000</em><br>
<em>bin changed  1011111111110011010010111110110001100000000000000000000000000001</em><br>
<em>-----</em><br>
<em>param final         -1.2060359716415405</em><br>
<em>-----</em></p>
<p>and the code that creates it:</p>
<p>print(“param” + str(param[2,9]))<br>
print("param item " + str(param[2,9].item()))<br>
work_value = round(param[2,9].item(),16)<br>
print("rounded param " + str(work_value))<br>
lsb_changed = operation(work_value, 1)<br>
print("changed param " + str(lsb_changed))</p>
<p>print(“-----”)<br>
print("bin param " + str(float_to_bin(work_value)))<br>
print("bin changed " + str(float_to_bin(lsb_changed)))</p>
<p>print(“-----”)</p>
<p>param[2,9] = lsb_changed</p>
<p>print("param final " + str(param[2,9].item()))</p>
<p>print(“-----”)</p>
<p>By working on it a bit I think, more or less, I have understood that it is a problem of precision of the representation of the value, when I go to insert the new value Pytorch changes it in some way that I don’t understand; but this is just a guess.</p>
<p>Please help me I’m almost going crazy.</p>

AlphaBetaGamma96: <p>Could this be a shallow copying issue? Could you try cloning the tensor before you modify it?</p>

FIavio: <p>After I cloned it, what should I do to modify the copy or the clone?</p>

AlphaBetaGamma96: <p>You should modify the clone of the Tensor, as python defaults to shallow copying.</p>

FIavio: <p>After many hours I solved the problem, the problem was not the tensor, but the value entered. I would enter a decimal number with many digits after the decimal point, up to 20; I discovered (unfortunately after a long time) that pytorch (or the tensor) cannot handle more than 6 decimal places after the decimal point, so if I enter a number with x &gt; 6 decimal places after the decimal point, it is not copied and therefore ignored.<br>
I don’t know if the 6 is due to the fact that the tensor value is float32 or something else, so if this sensitivity can be increased.</p>

AlphaBetaGamma96: <p>Perhaps when you convert to a string it is bound by the print options, if that’s the case you could have a look at the <code>torch.set_printoptions</code>, and set that to a higher number?</p>
OpenNMT beam.py beam search class for seq2seq language models

MLangner: <p>I am trying to reuse code from the slightly outdated openNMT library for beam search in a custom language model (<a href="https://github.com/OpenNMT/OpenNMT-py/blob/49185121e46c4f65d68101590bad231a8dd73e4f/onmt/translate/beam.py" class="inline-onebox" rel="noopener nofollow ugc">OpenNMT-py/onmt/translate/beam.py at 49185121e46c4f65d68101590bad231a8dd73e4f · OpenNMT/OpenNMT-py · GitHub</a>). So basically, what the code does in the beam class (line 125 -140) is as follows: For each beam, there is a probability distribution over n words. With beam size 4, this would mean we have 4 distributions. These are flattened into a tensor of size [4n]. Then, topk(4) selects the 4 most probable items and their ids. For reconstructing from which beam the most probable items come from, the ids are divided by the size of the vocabulary n. When I run the code, this produces float values, which then can’t be used as indeces, such that I get an exception. I googled a bit and it seems that openNMT was implemented when pytorch still had a different division semantics, namely flooring, such that division of an id by n results in a rounded integer. This behaviour would make sense here. Given n is 100 and the topk ids are [50, 90, 130, 230], then divided by 100, the floored values would be [0,0,1,2], which would be the indices of the respective beams the best probs come from. Since I use the latest torch version, and  according to (<a href="https://github.com/pytorch/pytorch/issues/5411" class="inline-onebox" rel="noopener nofollow ugc">Integer division behavior is different from Python and NumPy · Issue #5411 · pytorch/pytorch · GitHub</a>) this issue, the behaviour has changed in order to conform with python and numpy, I assume I only have to add flooring to the division in order to fix that error, right?</p>
Questions about torch.onnx.export

Ardeal: <p>Hi,</p>
<p>I remember that previously when <code>train=_C_onnx.TrainingMode.EVAL</code>, <code>batchnormalizaiton.running_mean</code> and <code>running_var</code> will be changed when we call <code>torch.onnx.export</code>.</p>
<p>For the latest <code>torch.onnx.export</code>, when <code>training=_C_onnx.TrainingMode.EVAL</code>, <code>batchnormalizaton.running_mean</code> and <code>running_var</code> will not be changed when we call <code>torch.onnx.export</code>.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff4eb8a46242e4f58b33cb8592a33c1b0281354e.png" alt="image" data-base62-sha1="AqypHgNClCAZOVGYoeqI5pzxAaa" width="511" height="123"></p>
<p>My questions are:</p>
<ol>
<li>is the logic of <code>torch.onnx.export</code> changed?</li>
<li>where is the code about whether <code>batchnormalizaton.running_mean</code> and <code>running_var</code> will be updated when we call <code>torch.onnx.export</code>?</li>
<li>according to the upper picture, PRESERVE means doing nothing, what is done when EVEL? what is the difference between EVAL and PRESERVE?</li>
</ol>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Could help to answer those upper questions?</p>

ptrblck: <p>I’m not an expert in the ONNX backend, but will try to answer your questions.</p>
<ol>
<li>I don’t see any recent changes to the <code>mode</code> part in <a href="https://github.com/pytorch/pytorch/blame/eb5883f8aab28ae7439fcd37057b304858cc0c31/torch/onnx/utils.py#L98"><code>torch/onnx/utils.py</code></a> and the last changes landed ~2 years ago.</li>
<li>The <code>model.train(True/False)</code> call will set the model to training or validation mode and will thus change the behavior of all registered batchnorm layers. Internally the <code>self.training</code> flag will be changed accordingly and used in the <code>forward</code> pass to change the execution path. I don’t know how this is propagated to ONNX though.</li>
<li><code>PRESERVE</code> keeps the training or validation mode of the model as it was set before without changing it. I.e. if you’ve called <code>model.eval()</code> before, the validation mode will ke kept.</li>
</ol>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for your answer!</p>
<p>I still cannot understand the difference of EVAL and PRESERVE.</p>
<p>Could you please help to at the ONNX expert who can see the post and answer it?</p>
Getting "RuntimeError: Numpy is not available" whenever trying with "torch.from_numpy"

jay154: <p>When I try to run a simple code, I receive an error saying that Numpy is not available.</p>
<p>import torch<br>
import numpy as np<br>
data = np.array([1, 2, 3], dtype=np.uint8)<br>
tensor = torch.from_numpy(data)</p>
<p>Traceback (most recent call last):  File “E:\test.py”, line 74, in <br>
tensor = torch.from_numpy(data)<br>
E:\test.py:74: UserWarning: Failed to initialize NumPy: _ARRAY_API not found (Triggered internally at …\torch\csrc\utils\tensor_numpy.cpp:84.)<br>
tensor = torch.from_numpy(data)<br>
Traceback (most recent call last):<br>
File “E:\test.py”, line 74, in <br>
tensor = torch.from_numpy(data)<br>
^^^^^^^^^^^^^^^^^^^^^^<br>
RuntimeError: Numpy is not available</p>
<p>Currently i am using<br>
python : 3.11<br>
torch    : 2.3.1<br>
numpy : 2.0.1</p>
<p>why is this error happening in PyTorch? When i am using the same version of pytorch with the older version numpy(1.26.4) i am not getting any errors.</p>

ptrblck: <aside class="quote no-group" data-username="jay154" data-post="1" data-topic="206871">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/47e85d/48.png" class="avatar"> jay154:</div>
<blockquote>
<p>When i am using the same version of pytorch with the older version numpy(1.26.4) i am not getting any errors.</p>
</blockquote>
</aside>
<p>I guess <code>numpy&gt;=2</code> might not be supported in your PyTorch build yet. You might want to update to <code>torch==2.4.0</code> or a nightly release as it seems Linux binaries work while Windows has issues according to <a href="https://github.com/pytorch/pytorch/issues/131668#issue-2428032675">this issue</a>.</p>
Reading Images from 4D numpy array

DeraDeru: <p>I am learning PyTorch and I want to write a custom data loader for my training. My training data is saved in to two numpy arrays. Input_images.npy  and Output_images.npy. Each of them has a shape of (N,W,H,C), where N is number of samples, e.g. 2000. I have wrote the following:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import numpy as np

from torch.utils.data import Dataset
from torch.utils.data import DataLoader

class CustomNumpyDataset(Dataset):
    def __init__(self, input_list, output_list, transform):
        self.input_list = input_list
        self.output_list = output_list
        self.transform = transform

    def __len__(self):
        list_in = np.load(self.input_list)
        return list_in.shape[0]
    def __getitem__(self, idx):
        sample_in = np.load(self.input_list)
        sample_in = sample_in[idx,:,:,:]
        sample_in = torch.from_numpy(sample_in)
        

        sample_out = np.load(self.output_list)
        sample_out = sample_out[idx,:,:,:]
        sample_out = torch.from_numpy(sample_out)
        
        return sample_in, sample_out

dataset = CustomNumpyDataset("./Input_images.npy", "./Output_images.npy")

dataloader = DataLoader(dataset, batch_size=8, shuffle=True)
</code></pre>
<p>This works fine, the problem is it is slow. Can you help me understand why it is slower, and what can I do to speed it up? I tried using <code>num_workers=2</code> in the DataLoader call and it speeds it up a bit. will there be another solution without increasing the number of workers.</p>

ptrblck: <p>Repeatedly loading arrays in <code>__getitem__</code> before indexing it is expensive and you won’t have any benefit from it, as you are already loading the entire array.<br>
Move it to the <code>__init__</code> and just index the array in the <code>__getitem__</code>.</p>
<p>Alternatively, revisit your approach and check if you can split the data into separate files, index the file list, and load the actual sample only in the <code>__getitem__</code>.</p>
How to use numpy for other general library to rewrite nn.KLDiv?

JimW: <p>In my project, I used nn.KLDiv(A.log(), B)  to calculate the KL divergence between A and B.<br>
But now I have to rewrite this part as numpy or other general python libraries in order to some model conversion.</p>
<p>I did a lot of search on google, and tried the following implementations:</p>
<pre><code class="lang-auto">(1) First method
def KL(a, b):
    a = np.asarray(a, dtype=np.float)
    b = np.asarray(b, dtype=np.float)
    return np.sum(np.where(a != 0, a * np.log(a / b), 0))

(2) Second method
scipy.special.kldiv
Neither of them show the similar output with the nn.KLDivLoss method in pytorch.
</code></pre>
<p>Could someone provide some guidance how to implement KL divergence with numpy to provide similar output with pytorch?  Thanks.</p>

ptrblck: <p>Using the formula from the <a href="https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html">docs</a>, I get the same results (up to floating point precision):</p>
<pre data-code-wrap="python"><code class="lang-python">def KL(a, b):
    return b * (np.log(b) - a)

kl_loss = nn.KLDivLoss(reduction="batchmean")
# input should be a distribution in the log space
input = F.log_softmax(torch.randn(3, 5, requires_grad=True), dim=1)
# Sample a batch of distributions. Usually this would come from the dataset
target = F.softmax(torch.rand(3, 5), dim=1)
ref = kl_loss(input, target)

# numpy reference
out = KL(input.detach().numpy(), target.numpy())
out = np.sum(out) / out.shape[0]

print((ref - out).abs().max())
# tensor(2.9802e-08, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
Global (not per param) optimizer state

qq-me: <p>So state_dict, to add something to it, you pick a parameter, and you say <code>state = self.state(parameter)</code>, and then you put something in that parameters state. But I have a variable that it would be more convenient to store globally, not in every parameter state, for now I just say self.variable = …, but that wouldn’t get saved and loaded in state_dict. So I was wondering, is there any way to save such variable to optimizer state_dict?</p>

ptrblck: <aside class="quote no-group" data-username="qq-me" data-post="1" data-topic="206769">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/qq-me/48/70055_2.png" class="avatar"> qq-me:</div>
<blockquote>
<p>But I have a variable that it would be more convenient to store globally, not in every parameter state, for now I just say self.variable = …, but that wouldn’t get saved and loaded in state_dict.</p>
</blockquote>
</aside>
<p>I’m unsure if I understand your use case correctly, so please let me know if I misunderstand the “global” meaning.<br>
<code>self.variable = ...</code> will register an <code>nn.Parameter</code> inside an <code>nn.Module</code> and will add it to the <code>state_dict</code>. If you want to register a non-trainable tensor inside this module and also want to add it to the <code>state_dict</code>, you could use <code>self.register_buffer</code> (with <code>persistent=True</code>, which is the default).</p>
<aside class="quote no-group" data-username="qq-me" data-post="1" data-topic="206769">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/qq-me/48/70055_2.png" class="avatar"> qq-me:</div>
<blockquote>
<p>So I was wondering, is there any way to save such variable to optimizer state_dict?</p>
</blockquote>
</aside>
<p>The optimizer receives references to parameters in its initialization from the model. Unsure, why you want to add new parameters manually.</p>

qq-me: <p>Sorry I wasn’t too clear on this, I am creating an optimizer, and optimizers also has their own state dicts where they have stuff like momentum, but it is stored in a dictionary, where each parameter is a key to the dictionary, like <code>state = self.state(parameter)</code>, and that way you can easily store momentum for each parameter. But I have a something where it would be more convenient if it was global and not linked to a particular parameter. So  I was wondering if I could have that but still be able to save that variable to the optimizer’s state_dict.</p>

ptrblck: <p>Would it be possible to add your custom parameter to the parameter <code>group</code> instead of the <code>state</code>?<br>
The <code>state_dict</code> would contain both as seen <a href="https://github.com/pytorch/pytorch/blob/a9e63562713c8628771a571439285fed4c21b3a7/torch/optim/optimizer.py#L706-L709">here</a>.</p>
Storing large datasets on disk in a format that allows indexing without loading?

syomantak: <p>I want to store a large dataset to the extent that I won’t be able to load it to memory later when I want to use the <code>Dataset</code> class. I was wondering what are the possible ways of storing it in a convenient form. Here are the possibilities I have come up with:</p>
<ol>
<li>
<p>Save each example, or a small batch of examples, in a separate file so that <code>__getitem__</code> in the <code>Dataset</code> class can load the relevant file.</p>
</li>
<li>
<p>Use HDF5 file format and create a single file with all the examples/data. My understanding is that loading HDF5 file does not actually load it to memory and we can load the relevant examples from it by indexing (based on this question on SO - <a href="https://stackoverflow.com/questions/62666729/how-do-i-efficiently-load-data-from-disk-during-training-of-deep-learning-models" rel="noopener nofollow ugc">python - How do I efficiently load data from disk during training of deep learning models in pytorch? - Stack Overflow</a> ).</p>
</li>
</ol>
<p>What are some other possible options? Are there other file formats that will allow me to index to the relevant example without loading the entire file?</p>

saandeep_aathreya: <p>You could use <a href="https://github.com/webdataset/webdataset" rel="noopener nofollow ugc">webdataset</a> to store your dataset in .tar format. Then you can load them as DataLoader just like PyTorch. It also supports multiprocessing during training.</p>
Delete the .ipynb_checkpoints in my dataset folder

Ganga: <p>i have resulted .ipynb_checkpoints in my dataset folder after removing the samples manually.how can i remove this permanently and why this happens</p>

tom: <p>These come from Jupyter. I’m <a href="https://stackoverflow.com/questions/51887758/is-there-a-way-to-disable-saving-to-checkpoints-for-jupyter-notebooks" rel="nofollow noopener">not sure there is a way to disable them</a>, but the linked answer might be out of date, too.</p>

Ganga: <p>it creates because of deleting img samples manually?</p>

tom: <p>This is something that comes from the notebook interface and has nothing to do with PyTorch or what you are doing with PyTorch.</p>

Ganga: <p>Thanks for your concern, Main intention was to do inference for multiple images and to check the accuracy of my model</p>
<ul>
<li>
<p>I was trying to find the accuracy of my pretrained model by evaluating</p>
</li>
<li>
<p>so, i was manipulating with the img samples to check the accuracy of my model,in order to that i was removing the img samples manually</p>
</li>
<li>
<p>Its there any method or blog or something related to remove this checkpoint notebook thing?</p>
</li>
</ul>

Kapil_Rana: <p>You can use this command in the terminal.</p>
<p>rm -rf <code>find -type d -name .ipynb_checkpoints</code></p>
<p>use " before find and at the end</p>

NguyenTanTrung: <p>In Jupyter Notebook,<br>
choose Setting, click turn off Autosave Documents</p>
Does DDP Require Additional GPU Memory for Model Maintenance?

xiuguangLi: <p>Does the Distributed Data Parallel (DDP) require additional GPU memory for maintaining a model?</p>
<p>Here is the code:</p>
<p>python</p>
<pre><code class="lang-auto">print(f"before 1 {torch.cuda.memory_reserved()/(1024**2)}")  ---&gt; 0
model = model.to(self.device)
print(f"before 2 {torch.cuda.memory_reserved()/(1024**2)}") ---&gt; 172
model = torch.nn.parallel.DistributedDataParallel(
                        model, 
                        device_ids=device_ids, output_device=output_device, **kwargs
                    )
print(f"after ddp {torch.cuda.memory_reserved()/(1024**2)}") ---&gt; 324
model.to("cpu")
torch.cuda.empty_cache()
print(f"after cpu {torch.cuda.memory_reserved()/(1024**2)}")  ---&gt; 158
</code></pre>
<p>In this case, it can be observed that the model size occupies 166, and the CUDA overhead occupies 6. The remaining 158-6=152 is the additional cost generated by DDP. Is there a way to remove or move this additional overhead to the CPU? Because I have many models to train, every time I wrap them with DDP, it will generate additional overhead, causing my code to run out of GPU memory.</p>
Single-GPU Multiprocessing

Matthew_Rajan: <p>I am currently running 2 models sequentially with each taking around ~30min to complete. My GPU utilization is only around ~25% when each model is running so I want to run parallel processes.</p>
<p>When I try using multiprocess I get the following error:<br>
[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]<br>
RuntimeError: Attempted to send CUDA tensor received from another process; this is not currently supported. Consider cloning before sending.</p>
<p>Does anyone have a workaround for parrallel computing on a SINGLE GPU?</p>
<p>Below is a test script that demonstrates my point:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp


def test_thread(a, b, queue):
    queue.put(a + b)


if __name__ == "__main__":
    mp.set_start_method('spawn')

    """WORKS WHEN DEVICE IS CPU NEED TO WORK WHEN DEVICE IS CUDA"""
    device = 'cuda'
    # device = 'cpu'

    manager = mp.Manager()
    queue1 = manager.Queue()
    queue2 = manager.Queue()

    p1 = mp.Process(target=test_thread, args=(torch.tensor([1, 1]).to(device),
                                              torch.tensor([1, 1]).to(device), queue1))
    p2 = mp.Process(target=test_thread, args=(torch.tensor([2, 2]).to(device),
                                              torch.tensor([1, 1]).to(device), queue2))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    print(queue1.get())
    print(queue2.get())
</code></pre>
Customizing PyTorch memory management on CPUs

Sathvik_Swaminathan: <p>The PyTorch docs only have documentation on how to tweak its memory management for CUDA allocations. It uses the caching allocator for it’s memory management which maintains the allocated memory blocks and reuses them instead of freeing them immediately.</p>
<p>On CPUs however, memory gets freed immediately after it has been used. How do I get PyTorch to reproduce the caching allocator behavior on CPUs? I want my program to retain all of its memory in the first epoch and reuse it in the subsequent epochs so that the memory allocation / de-allocation overhead is eliminated. Is this a standard practice?</p>
Pytorch 2.3.1 CUDA compatibility

gragris: <p>Hello people. Im fairly new at anything related to python. Im trying to install CUDA for my GTX 1660. I installed Cuda Toolkit 12.5 first but then i downgraded it to 12.1. I still can’t get it to work.</p>
<p>Here is the error. You can also see the versions.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/661125b58c0332dec00d353cab078854b4e35a92.png" data-download-href="https://discuss.pytorch.org/uploads/default/661125b58c0332dec00d353cab078854b4e35a92" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/661125b58c0332dec00d353cab078854b4e35a92_2_690x440.png" alt="image" data-base62-sha1="eyVrh0Nux574iMnkdlztrpozYpc" width="690" height="440" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/661125b58c0332dec00d353cab078854b4e35a92_2_690x440.png, https://discuss.pytorch.org/uploads/default/original/3X/6/6/661125b58c0332dec00d353cab078854b4e35a92.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/6/661125b58c0332dec00d353cab078854b4e35a92.png 2x" data-dominant-color="F9E2E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">892×570 59.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Your don’t need to install a CUDA toolkit as the PyTorch binaries ship with their own CUDA dependencies.<br>
As shown in your screenshot you’ve installed a CPU-only binary: <code>2.3.1+cpu</code>. Install a PyTorch binary with CUDA support and it should work.</p>