akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Running data loader workers on GPU

Priyanka_Perera: <p>Hi all !</p>
<p>I use below data loaders. How can ensure that workers of the data loaders run on GPU ?</p>
<p>I’m on below package versions.</p>
<p>torch 2.0.1<br>
pytorch-forecasting 1.0.0</p>
<pre><code class="lang-auto"># Training data set timeseries
training_data_timeseries = TimeSeriesDataSet(
    train_data,
    time_idx="Timestamp",
    target="Target Variable",
    group_ids=["Timeseries"],
    min_encoder_length=max_encoder_length // 2,
    max_encoder_length=max_encoder_length,
    min_prediction_length=max_prediction_length // 2,
    max_prediction_length=max_prediction_length,
    time_varying_unknown_reals=['Target Variable'],
    add_relative_time_idx=True,
    add_target_scales=True,
    add_encoder_length=True,
    target_normalizer=None
)

# Validation data set timeseries
validation_data_timeseries = TimeSeriesDataSet.from_dataset(training_data_timeseries, train_data, predict=True, stop_randomization=True)


# Create dataloaders for our model
train_dataloader = training_data_timeseries.to_dataloader(train=True, batch_size=batch_size, num_workers=workers, pin_memory=True)
val_dataloader = validation_data_timeseries.to_dataloader(train=False, batch_size=batch_size, num_workers=workers, pin_memory=True)
</code></pre>
<p>Thanks<br>
Priyanka</p>
OutOfMemoryError: CUDA out of memory despite available GPU memory

Humza_Sami: <p>Hello PyTorch community,</p>
<p>I’m encountering an issue with GPU memory allocation while training a GPT-2 model on a GPU with 24 GB of VRAM. Despite having a substantial amount of available memory, I’m receiving the following error:</p>
<blockquote>
<p>OutOfMemoryError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 23.68 GiB total capacity; 18.17 GiB already allocated; 64.62 MiB free; 18.60 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF.</p>
</blockquote>
<p>Here are the specifications of my setup and the model training:</p>
<ul>
<li>GPU: NVIDIA GPU with 24 GB VRAM</li>
<li>Model: GPT-2 with approximately 3 GB in size and 800 parameters of 32-bit each</li>
<li>Training Data: 36,000 training examples with vector length of 600</li>
<li>Training Configuration: 5 epochs, batch size of 16, and fp16 enabled</li>
</ul>
<p>These are my calculations:</p>
<ol>
<li>Model Size:</li>
</ol>
<ul>
<li>GPT-2 model: ~3 GB</li>
<li>Parameters: 800 parameters of 32 bits each</li>
</ul>
<ol start="2">
<li>Gradients:</li>
</ol>
<ul>
<li>Gradients are typically of the same size as the model’s parameters.</li>
</ul>
<ol start="3">
<li>Batch Size and Training Examples:</li>
</ol>
<ul>
<li>Batch Size: 16</li>
<li>Training Examples: 36,000</li>
<li>Vector Length: 600</li>
</ul>
<ol start="4">
<li>Memory Allocation per Batch:</li>
</ol>
<ul>
<li>Model: 3 GB (unchanged per batch)</li>
<li>Gradients: 3 GB (unchanged per batch)</li>
<li>Input Data: 16 x 600 (vector length) x 4 bytes (assuming each value is a 32-bit float) = 37.5 KB per batch</li>
<li>Output Data: 16 x 600 (vector length) x 4 bytes (assuming each value is a 32-bit float) = 37.5 KB per batch</li>
</ul>
<p>Based on the above calculations, the memory allocation per batch for my scenario would be approximately:</p>
<ul>
<li>Model: 3 GB</li>
<li>Gradients: 3 GB</li>
<li>Input and Output Data: 75 KB</li>
</ul>
<p>I would appreciate any insights or suggestions on how to resolve this issue. Thank you in advance for your assistance!<br>
<a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <aside class="quote no-group" data-username="Humza_Sami" data-post="1" data-topic="182881">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/humza_sami/48/54169_2.png" class="avatar"> Humza_Sami:</div>
<blockquote>
<p>Based on the above calculations, the memory allocation per batch for my scenario would be approximately:</p>
</blockquote>
</aside>
<p>In your calculation you are ignoring the intermediate forward activations, which need to be stored in order to compute the gradients. Based on the model architecture these activations could larger than the model parameters.</p>
Remove hooks after deepcopying model

Chame_call: <p>hello. From what I’ve seen the only possible way to remove registered hook it to keep its handle up to some moment and then call <code>remove()</code> on it but what to do if we registered hooks on model and then did deepcopy of it? Cause our kept handles are no longer valid for our copied model and there’s no way to obtain new handles from it while the old hooks are still working on “new” model.<br>
Any ideas please?</p>

Chame_call: <pre><code class="lang-auto">def remove_forward_pre_hooks(model):
    for module in model.modules():
        module._forward_pre_hooks = None
    return model
        
remove_forward_pre_hooks(net)
</code></pre>
LibTorch redistributable for Arm?

Danijel: <p>Arm 64 platform:</p>
<p>I’ve built my C++ app on Arm 64. It depends on LibTorch, which I built myself on Arm from source.</p>
<p>I’d like to distribute this app to 3rd parties. I’d like to make the demo process for the 3rd parties as simple as possible, the demo should basically work out-of-the-box.</p>
<p>How do I distribute the LibTorch runtime with my app demo?</p>
<ul>
<li>Which dirs/files are necessary?</li>
<li>Where do I copy those?</li>
<li>What’s the dir structure?</li>
<li>Etc.</li>
</ul>
<p>I understand this is the dir structure:</p>
<pre><code class="lang-auto">libtorch/
  bin/
  include/
  lib/
  share/
</code></pre>
<p>But it doesn’t feel right to hand pick the files and do the whole packaging manually. Is there a better way which would pack the runtime during the build process or similar?</p>
What is the correct way of using cycle, shuffle, shading and seed for datapipes and dataloader2?

Nabarun_Goswami: <p>I have a few questions. First the scenario:</p>
<p>Let’s say I have a bunch of files that I want to cycle infinitely and also use shuffling+sharding for distributed/multiprocessing reading services.</p>
<p>Is this the proper flow?</p>
<pre><code class="lang-auto">dp = dp.cycle()
dp = dp.shuffle()
dp = dp.sharding_filter()
</code></pre>
<p>Regarding the iterator of Dataloader2 and setting seed, is this flow correct?</p>
<pre><code class="lang-auto">dist_rs = DistributedReadingService()
mp_rs = MultiProcessingReadingService(num_workers=num_workers)
rs = SequentialReadingService(dist_rs, mp_rs)
dl = DataLoader2(dp, reading_service=rs)
dl_iter = iter(dl)

for epoch in range(10):
    dl.seed(epoch)
    for i in range(100):
        batch = next(dl_iter)
</code></pre>
<p>Finally, does setting the <code>dl.seed(epoch)</code> cause the datapipe to be re-shuffled and re-sharded? And does it recreate all multiprocessing workers, or are the workers persistent?</p>
<p>Thanks in advance!</p>

Benjamin_Rhodes: <p>Bump. I also would like to have clarification on this.</p>
Cant succeed in using backward or autograd

tototo: <p>class call_option_MC(torch.nn.Module):<br>
def <strong>init</strong>(self, nb_steps):<br>
super(call_option_MC, self).<strong>init</strong>()<br>
“”"<br>
torch.set_num_threads(2)<br>
“”"<br>
self.nb_steps = nb_steps<br>
self.sqrt_dt = torch.sqrt(torch.tensor(1 / torch.randn(self.nb_steps).shape[0]))</p>
<pre><code>#Return train set X, y, Z
def train_set(self, states):
    initial_price = states[:, 0]
    strike = states[:, 1]
    time_to_expiry = states[:, 2]
    implied_vol = states[:, 3]
    risk_free_rate = states[:, 4]

    dW = self.sqrt_dt * torch.randn(self.nb_steps * states.shape[0]).reshape(states.shape[0], self.nb_steps)
    implied_vol = states[:, 3]
    path = states[:, 0].clone()
    path_list = []
    for step in range(self.nb_steps):
        path += path * dW[:, step] * implied_vol
        path_list.append(path.clone())

    final_price = path_list[-1]
    discount_factor = torch.exp(-risk_free_rate * time_to_expiry)

    option_payoff = torch.max(final_price - strike, torch.tensor(0.0))
    option_price = discount_factor * option_payoff


    return states, option_price
</code></pre>
<p>Hi everyone, I am new here and would need some help. I have this code above and would like to get a variable derivatives of dimension (states[0], states[1]) where the element (i,j) is the derivatives of option_price[i] with respect to states[i,j]</p>
<p>I tried using backward or autograd but keep getting errors …</p>

srishti-git1110: <p>Please post a minimum executable code snippet enclosed within ``` that reproduces your error along with the complete error message.</p>
A more efficient way to create independent parallel MLP layers in Pytorch

ttamg: <p><strong>I have a use case where we need independent MLPs acting on a features in parallel.  This code works but it seems that there may be a more efficient way to do this in Pytorch.</strong></p>
<p>Note that in this case we have input which has dimensions <code>batch x feature x latent</code>.</p>
<p>In normal use the one MLP would be specified.  This would be used on all the features.</p>
<p>Another normal use is to flatten the <code>feature x latent</code> dimensions and run that through one MLP.  This performs mixing across the features.</p>
<p>But my use case is unusual and different.  It calls for a separate MLP to be trained for each of the 4 features and for them to run in parallel.  The following code works and does what it should. It uses the <code>ModuleList</code> and <code>Sequential</code> to define the independent MLPs.  But in the forward pass it requires a bit of looping which feels inefficient to me.</p>
<p>Is there a better and more efficient way of doing this in Pytorch?  I’ve had a dig around and can’t find any helpful pre-built layers or discussions on this.   Much appreciated.  Thanks.  Matt</p>
<pre><code class="lang-auto">class IndependentMLPs(torch.nn.Module):
    """
    A set of parallel MLPs working independently on each feature to transform 
    each feature in a three layer FF network.
    
    Input dimensions = batch x feature x latent

    Separate MLP of Linear(latent x latent) layers operating on each of the features INDEPENDENTLY
    """

    def __init__(self, feature_dim=4, latent_dim=32):
        super().__init__()

        self.feature_layers = torch.nn.ModuleList([
            torch.nn.Sequential(
                # Three layer FF network
                torch.nn.Linear(latent_dim, latent_dim),
                torch.nn.ReLU(),
                torch.nn.Linear(latent_dim, latent_dim),
                torch.nn.ReLU(),
                torch.nn.Linear(latent_dim, latent_dim),
                torch.nn.ReLU(),
            ) for i in range(feature_dim)]
        )


    def forward(self, x):
        """ Input X has the dimensions batch x feature (4) x latent (32) """

        outputs = []
        for i, layer in enumerate(self.feature_layers):
            in_ = x[:, i, :]  # Input from feature i
            out = layer(in_)
            out = out.reshape(out.shape[0], 1, out.shape[1])
            outputs.append(out)
          
        x = torch.cat(outputs, axis=1)  # Rebuild transformed x

        return x
</code></pre>

BetterWorld-Liuser: <p>I have the same question… do you have any solutions?</p>
Classification of 2d time series data

ATYslh: <p>Hello everyone,<br>
I know this is a rather general question, but maybe someone can point me in the right direction.<br>
I have a dataset containing pictures from two different cameras.<br>
In total I have 30 different labels (classes).<br>
In each “timestep” (input I want to give to my network) contains 24 frames from each camera and has one of those 30 labels.<br>
In total there are around 30000 of those 24-frame-“timesteps”.</p>
<p>The classic approach for image classification is usually some CNN, but this one is maybe closer to some sort of video classification. Due to the limited number of classes I don’t need any huge architecture, but I am not sure how to approach this. I saw that there is something called “LSTM” that takes temporal dependence into account, but I am not sure how this would work with input from two different cameras.<br>
All sorts of help is appreciated.</p>

KFrank: <p>Hi At!</p>
<aside class="quote no-group quote-modified" data-username="ATYslh" data-post="1" data-topic="182559" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atyslh/48/61504_2.png" class="avatar"> ATYslh:</div>
<blockquote>
<p>In each “timestep” (input I want to give to my network) contains 24 frames from each camera and has one of those 30 labels.<br>
…<br>
The classic approach for image classification is usually some CNN</p>
</blockquote>
</aside>
<p>I would still recommend starting with a convolutional neural network, but<br>
with three-dimensional convolutions.</p>
<p>Think of each timestep as being a 3d image with shape <code>[24, height, width]</code><br>
(and two channels – one for each of your two cameras).  Therefore the<br>
input to your model – including the batch dimension – will have shape<br>
<code>[nBatch, 2, 24, height, width]</code>.</p>
<p>So look for an off-the-shelf 3d CNN-based classification model, start with<br>
an off-the-shelf 2d classification model but convert it to a 3d model by<br>
replacing its <code>Conv2d</code>s with <code>Conv3d</code>s, or build your own 3d CNN-based<br>
classification model by using a series of <code>Conv3d</code> layers followed by a<br>
couple of fully-connected <code>Linear</code> layers.</p>
<p>Note that that third dimension – even if only of size <code>24</code> – will require more<br>
memory than an otherwise similar 2d model.</p>
<p>Best.</p>
<p>K. Frank</p>

ATYslh: <p>Thank you for the answer. I will give it a shot!</p>
Extreme slowdowns due to dataloading when using compile

ASpeiser: <p>When trying to speed up my training Loop I’m seeing some weird behaviour. I’m using the new L4 GPUs on the google cloud platform, CUDA 12.1 and pytorch nigthly from today.<br>
Training with mixed precision.</p>
<p>Dataloader uses 16 workers (which is needed to max out the GPU without compilation), with pin_memory = True.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3bfe7a81b32b5f52bdd22efc3214fef5d4c99dfb.png" alt="image" data-base62-sha1="8yJo5sxO8o13ChZlhilgn3y8wSD" width="335" height="497"></p>
<p>Graph shows training iteration over time.</p>
<p>Purple: No compilation, default dataloading.</p>
<p>Blue: No compilation, no dataloading (instead just computing on the same preloaded batch).</p>
<p>Barely any speedup, so dataloading does not seem to be the limiting factor.</p>
<p>Green: Compiled with mode=‘default’, no dataloading. Large speedup.</p>
<p>Pink: Compiled with mode=‘default’, default dataloading. Short bursts with the same speedup, but then large stretches where training slows down to a crawl and GPU usage breaks down.</p>
<p>Going from 16 to 32 CPUs does not solve the issue. So I would understand if dataloading is not able to serve at a high enough speed to max out the compile operation. But why does is slow down by a factor of 4 compared to no compilation?</p>
<p>I see the same behaviour with the nvidia T4 (though compile doesn’t give a speedup).</p>

marksaroufim: <p>The jagged pattern in the pink graph looks like a recompilation to me, do you see an error if you run <code>torch.compile(..., fullgraph=True)</code> if so then it might best to fix the highlighted issues or make sure your inputs are static shape</p>
<p>Alternatively try <code>torch.compile(..., dynamic=True)</code></p>

ASpeiser: <p>That was indeed the problem. I didn’t have drop_last enabled in my dataloader so in between training smaller batches would appear. Getting a nice speed up now, thank you!</p>
Is it fine to dump/overlap the tensor name that is already existing?

daegartwo: <p>I have no idea how pytorch’s autograd and constructing computational graph work,<br>
thus I’m not sure whether it is fine to write a code like below:</p>
<p>def forward(self, x):<br>
x = self.conv(x)<br>
return x</p>
<p>I think in order to track the gradient of each tensor, the name of output of convolution layer should be different  from “x”, but I found many code actually dump their tensor name.</p>
<p>Is it still possible to backpropagate the gradient properly till the front part of the model?<br>
I mean, what if the input ‘x’ of above forward funciton actually came from another trainable network or something</p>
<p>If it is, how is that possible?</p>

ptrblck: <p>Yes, you can reassign the output of a differentiable operation to a tensor with the same name as its input. Autograd will track these operations and will reference the input (thus prevent its deletion) if it’s needed for the gradient computation.</p>
Memory footprint in the action of the matrix exponential algorithm

kawahi: <p>Hello,</p>
<p>While PyTorch supports the computation of the matrix exponential (mexp) since version 1.7, it does not seem to feature the action of the matrix exponential on a few specific vectors. This feature is implemented in scipy for instance, <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.expm_multiply.html" rel="noopener nofollow ugc">link</a>. In practice, computing the whole mexp may be unnecessarily costly. This led me to try and implement the algorithm proposed in “<em>Computing the Action of the Matrix Exponential</em>” by Al-Mohy and Higham <a href="https://eprints.maths.manchester.ac.uk/1591/1/alhi11.pdf" rel="noopener nofollow ugc">link</a>. In my first attempt, I implemented a function and thus exploited the underlying forward and backward inherent support of PyTorch. Unfortunately, I suspect that the memory footprint of my implementation is not ideal.</p>
<p>The main question is what can be done to improve the memory footprint? Initially, I am tempted to implement it as a new Function and extending torch.autograd either in Python or C++. However, it is unclear whether I will gain any significant memory reduction. The algorithm above is a “scaling-and-squaring” algorithm, in which the action of mexp is computed iteratively. At each iteration the vectors are multiplied by the scaled matrix, followed by a sum of the results. Please see the code below. Now, while auto-differentiating (AD) this computation in forward-mode is somewhat efficient as you can perform it greedily, this is unfortunately not the case in reverse-mode. Basically, one needs to store all the intermediate scaled versions of the matrix, although, I might be missing something here.</p>
<p>Any tips on how to proceed would be highly appreciated.</p>
<p>Below, is a <em>redacted</em> version of the algorithm, I tried to keep the “memory-wise important” parts of the algorithm:</p>
<pre><code class="lang-auto">def expmv(A, B, t, s, m):
    # A is the matrix for mexp, B is a few vectors
    F = B

    for ii in range(s):
        for jj in range(m):
            B = (t / (s * (jj+1))) * (A @ B.T).T
            F = F + B
        B = F
   return F
</code></pre>
Torch.jit.trace device

firminshao25: <p>I know inputs and model must be the same type, cpu or cuda.That is to say they must be on the same device.However,what’s the difference?<br>
<strong>ON CPU:</strong></p>
<pre><code class="lang-auto">    best_model = torch.load('checkpoints/test.pth')
    device = torch.device("cpu")
    best_model.to(device)
    example = torch.rand(1, 3, 256, 256)
    traced_script_module = torch.jit.trace(best_model, example)
    traced_script_module.save("test_cpu.pt")
</code></pre>
<p><strong>ON CUDA:</strong></p>
<pre><code class="lang-auto">    best_model = torch.load('checkpoints/test.pth')
    best_model.cuda()
    example = torch.rand(1, 3, 256, 256).cuda()
    traced_script_module = torch.jit.trace(best_model, example)
    traced_script_module.save("test_gpu.pt")
</code></pre>
<p>Both the above codes can transfer a model to a torch script.I am wondering what’s difference between the two methods.<br>
Thank you for your attention.</p>
Lexicographic max operator

orrivlin: <p>Hi,</p>
<p>I’m trying to implement a lexicographic max operator that does the following:<br>
given a tensor of N dimensional elements, return the first non-zero value according to the given order.<br>
so for example:<br>
input = [[0, 1, 2],<br>
[1, 4, 2],<br>
[0, 0, 3],<br>
[0, 3, 1]]<br>
output = [[1],<br>
[1],<br>
[3],<br>
[3]]<br>
is there any existing method to do so? if not, could you suggest a way to efficiently implement such a thing (that would also work for “batch” inputs).</p>
<p>Thanks!</p>

ptrblck: <p>I’m unsure if any built-in method can be easily used. <code>tensor.scatter_reduce_</code> could be a candidate, but would also use the <code>reduce</code> argument to select the right values thus creating an error in the last row since you want to return the <code>3</code> while the <code>1</code> would be the min. value:</p>
<pre><code class="lang-python">x = torch.tensor([[0, 1, 2],
                  [1, 4, 2],
                  [0, 0, 3],
                  [0, 3, 1]])

idx = x.nonzero()
out = torch.zeros(4).long()
out.scatter_reduce_(dim=0, index=idx[:, 0], src=x[idx[:, 0], idx[:, 1]], reduce="amin", include_self=False)
print(out)
# tensor([1, 1, 3, 1])
</code></pre>
<p>EDIT:<br>
OK, this should work:</p>
<pre><code class="lang-python">idx = x.nonzero()
index = torch.zeros(x.size(0)).long()
index.scatter_reduce_(dim=0, index=idx[:, 0], src=idx[:,1], reduce="amin", include_self=False)
print(index)
# tensor([1, 0, 2, 1])

x[torch.arange(x.size(0)), index]
# tensor([1, 1, 3, 3])
</code></pre>
Id(untyped_storage()) returns the same for different input tensor, why?

Fei_Liu: <p>I accidentally found this:</p>
<pre><code class="lang-auto">import torch
x = torch.rand(10)
y = torch.rand(10)
print(id(x.untyped_storage()) == id(y.untyped_storage()))  # output (why?): True
</code></pre>
<p>Further more, if I do this repetitively:</p>
<pre><code class="lang-auto">print(id(x.untyped_storage()), id(y.untyped_storage()))
</code></pre>
<p>I find that it will print two same numbers every time, but the number keeps changing per each run. What is happening behind the scene?</p>

KFrank: <p>Hi Fei!</p>
<aside class="quote no-group" data-username="Fei_Liu" data-post="1" data-topic="182847" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fei_liu/48/36219_2.png" class="avatar"> Fei_Liu:</div>
<blockquote>
<pre><code class="lang-auto">print(id(x.untyped_storage()) == id(y.untyped_storage()))  # output (why?): True
</code></pre>
</blockquote>
</aside>
<p><code>id()</code>, at least in practice, gives you the location in memory of the python object.</p>
<p>When you run <code>id(x.untyped_storage())</code>, a temporary <code>UntypedStorage</code><br>
object is created.  It lives long enough for its <code>id()</code> to be taken, but when it<br>
goes out of scope (which is after the call to <code>id()</code>), it is destroyed and its<br>
memory is reclaimed.  Then when you run <code>id(y.untyped_storage())</code>, a<br>
new <code>UntypedStorage</code> object is created that <em>happens to</em> occupy the same<br>
memory that was occupied by the just-destroyed <code>x.untyped_storage()</code>.</p>
<p>To see the memory where the actual data in tensors <code>x</code> and <code>y</code> reside, look<br>
at <code>x.untyped_storage().data_ptr()</code> and <code>y.untyped_storage().data_ptr()</code>.<br>
These will be different.</p>
<p>Best.</p>
<p>K. Frank</p>

Fei_Liu: <p>Thank you for your thorough explanation!</p>
Integrating Distributed Matrix Multiplication Algorithm into PyTorch

kabicm: <p>Would it be possible to integrate a Distributed Matrix-Matrix Multiplication Algorithm like <a href="https://github.com/eth-cscs/COSMA" rel="noopener nofollow ugc">COSMA</a> into PyTorch. COSMA is communication-optimal, gpu-accelerated algorithm for matmul, that is already used in some HPC applications with great performance results!</p>
<p>I am one of the main developers of COSMA and would be happy to integrate it into PyTorch.</p>
<p>Let me know your thoughts on that!</p>

pbelevich: <p>CC <a class="mention" href="/u/wanchaol">@wanchaol</a> <a class="mention" href="/u/pritamdamania87">@pritamdamania87</a> <a class="mention" href="/u/yaroslav_bulatov">@Yaroslav_Bulatov</a></p>

amitabe: <p>I’m also interested to see this feature merged into PyTorch!</p>
Why isn't the memory being released after inference?

J_B_28: <p>Hello,<br>
I am wondering why the memory is not being released after inference, the following is my code (transformer is imported from huggingface):</p>
<pre><code class="lang-auto">@torch.no_grad()
    def inference(self, hidden_state, mask, id):
        self.eval()
        print("begin_max", torch.cuda.max_memory_allocated("cuda:4")/ 1024 / 1024)
        print("begin", torch.cuda.memory_allocated("cuda:4")/ 1024 / 1024)
        distilbert_output = self.bert(inputs_embeds=hidden_state, attention_mask=mask, return_dict=False)
        print("middle_max", torch.cuda.max_memory_allocated("cuda:4")/ 1024 / 1024)
        print("middle", torch.cuda.memory_allocated("cuda:4")/ 1024 / 1024)

        gc.collect()
        
        torch.cuda.empty_cache()

        gc.collect()

        hidden_state = distilbert_output[0]                    
        pooled_output = hidden_state[:, 0] 
        x = pooled_output
        x = F.dropout(x, p=args.dropout, training=self.training)

        del hidden_state, pooled_output
        
        print("final_max", torch.cuda.max_memory_allocated("cuda:4")/ 1024 / 1024)
        print("final", torch.cuda.memory_allocated("cuda:4")/ 1024 / 1024)
        for i, lin in enumerate(self.lins[:-1]):
            x = lin(x)
            #x = self.bns[i](x)
            x = F.relu(x)
            x = F.dropout(x, p=args.dropout, training=self.training)
        x = self.lins[-1](x)
        self.z_mlp = self.z_mlp.to(device)
        self.z_mlp[id] = x.clone().detach()
        print("final2", torch.cuda.max_memory_allocated("cuda:4")/ 1024 / 1024)
        torch.cuda.empty_cache()
        return x
</code></pre>
<p>Here are the results:</p>
<p>begin_max 4217.28662109375<br>
begin 4217.28662109375<br>
middle_max 39844.28662109375<br>
middle 7967.28662109375<br>
final_max 39844.28662109375<br>
final 7996.58349609375</p>
<p>It seems after either <code>distilbert_output = self.bert(inputs_embeds=hidden_state, attention_mask=mask, return_dict=False)</code> or MLP, the memory still increases, do I miss something? Is this a memory leakage?</p>
<p>Any help would be appreciated!</p>

ptrblck: <p>The increase in memory usage is expected since you are performing differentiable operations on the output of the model via <code>x</code>. Deleting <code>hidden_state</code> and <code>pooled_output</code> won’t change it since <code>x</code> still refers to the computation graph and holds references to these tensors.<br>
Also neither <code>gc.collect</code> nor <code>empty_cache</code> will change this behavior.</p>
When not to use OneCycleLR

carbocation: <p>The <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.OneCycleLR.html" rel="noopener nofollow ugc">OneCycleLR</a> learning rate scheduler is designed around Leslie Smith’s concept of super-convergence. But presumably super-convergence is not always possible.</p>
<p>For example, in the LR scheduler plot below, there is not a classic example of descending training loss with greater learning rates:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e82e39dc301b4e28bda489b176aa2e65eea551f.png" alt="image" data-base62-sha1="du5jh1JhjxfqKGaKhEETVxFP3uT" width="567" height="437"></p>
<p>Is this an example where the OneCycleLR approach should be avoided, favoring a more classical learning rate schedule instead? Or is there no real connection between a descending loss in an LR Finder slope and the OneCycleLR aside from the fact that they were both described by Leslie Smith?</p>
Mac Mini M2 Pro: import torch error, Library not loaded: @rpath/libffi.8.dylib

E11: <p>Hi Friends,</p>
<p>I just got my Mac Mini with M2 Pro Chip today, and so excited to install and run pytorch on it. I followed the instruction <a href="https://developer.apple.com/metal/pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Accelerated PyTorch training on Mac - Metal - Apple Developer</a></p>
<pre><code class="lang-auto">curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
sh Miniconda3-latest-MacOSX-arm64.sh
</code></pre>
<pre><code class="lang-auto">conda create -n torch
conda activate torch
conda install pytorch torchvision torchaudio -c pytorch-nightly
</code></pre>
<p>without any error.</p>
<p>However when I actually import the package, I saw below error. Sincerely appreciate your advice about how to solve it or let me know it is not solvable … (I tried hard to google search it, yet no luck <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> )</p>
<pre><code class="lang-auto">(torch) Evas-Mac-mini:Projects evah$ python
Python 3.11.3 (main, Apr 19 2023, 18:49:55) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/Users/evah/miniconda3/envs/torch/lib/python3.11/site-packages/torch/__init__.py", line 17, in &lt;module&gt;
    import ctypes
  File "/Users/evah/miniconda3/envs/torch/lib/python3.11/ctypes/__init__.py", line 8, in &lt;module&gt;
    from _ctypes import Union, Structure, Array
ImportError: dlopen(/Users/evah/miniconda3/envs/torch/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so, 0x0002): Library not loaded: @rpath/libffi.8.dylib
  Referenced from: &lt;984D5B9C-4A47-3C59-BA2F-19CA5FA76707&gt; /Users/evah/miniconda3/envs/torch/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so
  Reason: tried: '/Users/evah/miniconda3/envs/torch/lib/python3.11/lib-dynload/../../libffi.8.dylib' (no such file), '/Users/evah/miniconda3/envs/torch/lib/python3.11/lib-dynload/../../libffi.8.dylib' (no such file), '/Users/evah/miniconda3/envs/torch/bin/../lib/libffi.8.dylib' (no such file), '/Users/evah/miniconda3/envs/torch/bin/../lib/libffi.8.dylib' (no such file), '/usr/local/lib/libffi.8.dylib' (no such file), '/usr/lib/libffi.8.dylib' (no such file, not in dyld cache)
&gt;&gt;&gt; 
</code></pre>

E11: <p>I am checking out this old thread <a href="https://github.com/platformio/platform-lattice_ice40/issues/7" class="inline-onebox" rel="noopener nofollow ugc">dyld: Library not loaded: /usr/local/opt/libffi/lib/libffi.6.dylib · Issue #7 · platformio/platform-lattice_ice40 · GitHub</a>, trying out things they suggest to see if the problem could be solved (not yet)</p>
Vmapping over dict of categories and respective tensors - Simultaneously loading multiple categories

nwn: <p>Hello PyTorch community,<br>
I am trying to build some kind of ensemble learner which includes the loading of different categories from a large category dataset. The ensemble consists of multiple networks, each should be trained on different categories. For that, it is really hard for me to implement some kind of Dataloader for loading multiple categories simultaneously and stacking those to be used by a functional model ensemble (created with torch.func).<br>
I have already transformed the dataset into a dictionary containg the categories as keys and the training data as respective values. I’d like to sample a certain number of samples for given categories that are stored in a separate tensor and stack those. Does anyone have any idea how to do so?<br>
That’s my code so far which is obviously not working <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto"># Note that every category holds a multidimensional tensor containing the training data, ie. multiple samples
# The dict keys are used as labels, respectively
# All samples have the same dimensionality
# '&lt;&gt;' used to indicate it is a variable that should be defined

data_dict = {667: torch.tensor([[1.,1,...,1], ...]), 887: torch.tensor([[1.,1,...,1], ...])}
categories = tensor([[667, 887, 157, 437, 493], 
                     [ 90, 870, 509, 378, 132],
                     ...])

def get_data_from_category(category, data):
    # This throws an error because .item() is not allowed within vmap
    return data[category.item()] [&lt;number_of_samples&gt;]

sampler = torch.func.vmap(get_data_from_label, in_dims=(0, None))
sampled_data = sampler(categories.flatten(), data_dict)
</code></pre>
<p>Desired output e.g.: Tensor([[values_of_category_667, values_of_category_887,…], …]</p>
<p>Besides, to the question above, I am open for alternative approaches that allow me to sample multiple, but certain, categories simultaneously.<br>
Thank you all <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Will running Pytorch CUDA 11.8 on CUDA 12.0 GPUs negatively affect performance?

AlphaBetaGamma96: <p>Hi All,</p>
<p>This is a probably stupid question, but if I’ve installed the latest stable pytorch 2.0 with CUDA 11.8, but I’m running them on GPUs that have CUDA 12.0, will this negatively affect performance at all? I’ve installed via conda for reference, but I’m just wondering how this will affect performance, if at all?</p>
<p>Thanks!</p>

ptrblck: <p>Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension, since the PyTorch binaries (pip wheels and conda binaries) ship with their own CUDA dependencies.</p>
Train mse doesn't converge in 0 when overfitting single batch

milan_kalkenings: <p>i debug my model by training many iterations on just one training batch and monitoring the loss achieved on the very batch.<br>
the loss should converge in 0, but clearly doesn’t.<br>
the loss decreases very smooth and just plateaus.<br>
any idea what i am doing wrong?</p>
<p>batch size: 32<br>
task: image super resolution<br>
loss (of interest): mse (mean reduced)<br>
probabilistic modules in network: no (no dropout, noise adding, …)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26a2b4f7dbc9a9a06172fabf2f4db2c27e6e31dc.png" alt="2, 8, 16, 32, 64, 128, 256, 512_combined_2_1lr_0.0001_losses" data-base62-sha1="5vMJZxUHW5IhWLxjHxZptCKBjsg" width="640" height="480"></p>

srishti-git1110: <p>I don’t see this as totally unexpected or necessarily wrong.<br>
Did you try changing the network architecture? Like adding more layers to it to parameterize it even more – the intuition is maybe the current architecture simply doesn’t have enough parameters to completely fit (all the examples in) this one batch.</p>
<p>Another way to get out of loss stagnation is to fiddle with the learning rate. Maybe try schedulers, lr decay etc.</p>

milan_kalkenings: <p>interestingly it plateaus for all learning rates and network widths/depths i tried in the same value (like 2% variance)</p>
<p>lrs i tried : [0.1, 0.01, 0.001, 0.0001, 0.00001, 0.000001]</p>

srishti-git1110: <p>I see.<br>
A constant loss value implies the network has stopped learning (or is moving at a pace that’s virtually constant). One way to get out of it is to increase the learning rate after a constant loss is observed for a certain no. of epochs (lr scheduling).</p>
<p>Anyway, I would still think that it’s not necessarily incorrect to not be able to obtain exact zero loss on a batch.</p>
Where does torch::tensor create the object? stack or heap?

velenos14: <p>When I write:</p>
<p><code>torch::Tensor y = ...</code> ;</p>
<p>where does the object <code>y</code> live? On the stack or on the heap?</p>
<p>If it is on the stack, how can I delete this allocated tensor <code>y</code> after I finished working with it?</p>
<p>Thank you!</p>

bdhirsh: <p>Hi!</p>
<p>There’s some nice info here: <a href="https://github.com/pytorch/pytorch/wiki/Writing-Python-in-cpp-(a-manifesto)" class="inline-onebox" rel="noopener nofollow ugc">Writing Python in cpp (a manifesto) · pytorch/pytorch Wiki · GitHub</a>.</p>
<p>The short answer is that at::Tensor is a pointer type (like a shared pointer), with the pointer on the stack, and data on the heap. The pointer is freed when it goes out of scope. If you have multiple references to a tensor in c++, the data will be freed when the reference count drops to 0.</p>

velenos14: <p>Thank you very much! It answers all my doubts.</p>
Help with my first PyTorch deep learning model! Classification

puscu: <p>Hi, I am starting working with PyTorch and learning about deep learning and neural networks.</p>
<p>After different lessons and tutorials, I built my very first simple model. I get a datasets of credit risk customers classification and tried to work on a classification problem. The idea is to classify into “good” and “bad” creditors based on 18 features.</p>
<p>I got 70.5% of accuracy, but this occurs since the first epoch and every iteration never improves. If I change some parameter like hidden neurons, learning rate, etc. the only variation is changing from 70.5% to 29.5% (just the opposite to 100%). The same if I had layers or activation such as ReLU.</p>
<p>Could you help me to understand how to work on it? What approach should I start to improve the model?</p>
<p>Here the Colab link to see the notebook: <a href="https://colab.research.google.com/drive/1fPqEo8AxQ_qn0awSZlr2W7Uo22lbwT26?usp=sharing" rel="noopener nofollow ugc">notebook link</a></p>
<p>Thanks!</p>

David_Gomez: <p>Hi Pasquale,</p>
<p>First, I see you are using loss_fn = nn.BCEWithLogitsLoss(). BCEWithLogitLoss already implements the sigmoid function so there is no need to apply it on the result of your model before passing it into the loss function. Also, and I’m not sure here, rounding the results of the sigmoid function will make the predicted labels either 0 or 1.<br>
For a richer training process, you might want your predicted labels to be the raw result of the sigmoid function. This makes a difference between a predicted label of 0.9 and another of 0.56. Assuming the correct label is 1 in both cases, the first predicted label should have a lower error than the second one. Rounding up both values would make the error be the same for every non correct predicted label.</p>
<p>I sugest changing a few things. Instead of<br>
y_pred = torch.round(torch.sigmoid(y_logits)).float() # turn logits → pred probs → pred labls<br>
try doing directly<br>
loss = loss_fn(y_logits.float(), y_train.float()) # Remember BCEWithLogitsLoss already implements sigmoid.</p>
<p>I hope this helps a bit. Good luck!</p>
<p>About BCEWithLogitLoss: <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" class="inline-onebox" rel="noopener nofollow ugc">BCEWithLogitsLoss — PyTorch 2.0 documentation</a></p>

puscu: <p>Thanks David, it helped me a lot! First of all for the theorical explanation and then because increased by 8% accuracy (78%). 12% of wrong predictions is still bad, but in my credit risk model, what I really want to avoid are false positive for the negative impact they could have on business. So I will evaluate the percentage of false positive on the test dataset.</p>
Torch.save shows out of memory

Debasmit_Das: <p>I am using torch.save to store some tensors as .pt files. I am storing a lot of such .pt files for offline computation. After a while it throws the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "save_optical_flow.py", line 249, in &lt;module&gt;
    save_flow(image_path, n_frames)
  File "save_optical_flow.py", line 149, in save_flow
    torch.save(flow, out_path)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 381, in save
    _legacy_save(obj, opened_file, pickle_module, pickle_protocol)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 214, in __exit__
    self.file_like.close()
OSError: [Errno 28] No space left on device

</code></pre>
<p>I checked that it has still lot of storage left. Not sure why its happening. Can anybody help ?</p>
Loading a model multiple times

Debodeep_Banerjee: <p>I am using the following format in a training procedure.<br>
output1= model(input, mode=‘a’)<br>
loss1= loss_fun1(output1)<br>
output2=model(input, mode=‘b’)<br>
loss2=loss_fun2(output2)<br>
total_loss = loss1+loss2<br>
optimizer.zero_grad()<br>
total_loss.backward()<br>
optimizer.step()</p>
<p>Now, the problem is I am getting a memory error when I am calling the model twice. Again, if use torch.no_grad() for the mode ‘b’, I am breaking the computational graph and,  I cannot do that. Is there any solution to the problem?</p>

ptrblck: <p>Your current approach keeps the intermediate forward activations for both forward passes, which are needed to compute the gradients. You could try to lower the batch size, if possible, or trade compute for memory via <code>torch.utils.checkpoint</code>.</p>
Segment Anything Model as pre-processing

TeDataPro: <p>For pre-processing my  images, I use the Segment Anything Model ( <a href="https://github.com/facebookresearch/segment-anything" rel="noopener nofollow ugc">GitHub - facebookresearch/segment-anything: The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.</a> ) to work with masks from the original image.</p>
<p>In the <em>getitem</em> of my CustomDataset class, the original image is read then sent to the SAM model, then pre-processed from these masks before being returned.</p>
<p>However, when I iterate over loaders I have the following issues</p>
<pre><code class="lang-auto">RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
</code></pre>
<p>I solved the problem by adding the line</p>
<pre><code class="lang-auto">torch.multiprocessing.set_start_method('spawn')
</code></pre>
<p>However after this, iteration over my DataLoader only works with num_workers=0. When it is strictly positive, the DataLoader iteration does not converge and does not even show an error.</p>
<p>I have absolutly no idea where it can come from. Please, do you have an idea ?</p>
<p>Thank you</p>

ptrblck: <p>Based on the error message you are trying to initialize a CUDA context in worker processes, which would be the case when data is moved to the GPU inside the <code>__getitem__</code>.<br>
Using the <code>spawn</code> method should work or you could keep the data on the CPU and move the entire batch to the GPU inside the <code>DataLoader</code> loop.</p>
Indicating parameters as static to `torch.compile`

marcelroed: <p>In Jax, when I JIT-compile my functions, I indicate which parameters I want to be static with the <a href="https://jax.readthedocs.io/en/latest/_autosummary/jax.jit.html" rel="noopener nofollow ugc"><code>static_argnums</code> argument to <code>jax.jit</code></a>. When I do this, Jax knows to compile one version of this function for each value of these parameters, and to create compiled functions that accept variables for each of the other parameters.</p>
<p>In my case I’m working on fluid simulations, where the values of my fields will change, but for instance I won’t ever have any changes in my boundary conditions. In this case I have classes containing field data, and domain data, where the domain is completely static between runs.</p>
<pre><code class="lang-auto">@torch.compile
def simulate(fluid:​ Field, domain:​ Domain):
    ...
</code></pre>
<p>Here I want to recompile when any of the data in <code>domain</code> changes, but not when the data in Field changes. For instance, I have used Equinox in Jax to define data structures with static parameters anywhere in the tree, like the <code>static=True</code> flag in <code>eqx.field</code> <a href="https://docs.kidger.site/equinox/api/module/advanced_fields/#equinox.field" rel="noopener nofollow ugc">here</a>. In Jax this is easy, and I can control in what cases I need to guard and recompile vs when I can use a function I’ve previously compiled.</p>
<p>Is there a way accomplish this in PyTorch 2 with <code>torch.compile</code>? If there are any resources or if anyone has any knowledge on how these compile guards are decided, I would appreciate to hear about them. Thank you!</p>
How to make compatible my custom env in torchrl

Deep_Gravity: <p>Hi all,</p>
<p>I recently started to use <code>torchrl</code> to train an RL agent for my <strong>custom</strong> environment. I used to train agents with <code>Rllib</code>, but not I would like to test <code>torchrl</code>. My custom env is already compatible with <code>gym</code> and <code>gymnasium</code> and of course with <code>rrlib</code>. I tried to use <code>torchrl</code> for the same env, but it does not work. I wonder if anyone knows how to make a custom env in <code>torchrl</code>?</p>
<p>Thanks!</p>

vmoens: <p>Can you post an issue on TorchRL with that?<br>
Happy to make a patch if needed!<br>
Please provide code snippet + error if possible</p>
The python module i am importing in C++ with the Python/C API can't import torch

Edoar: <ul>
<li>OS: Windows 10</li>
<li>C++ environment was created in Visual Studio Community 2017 with CMake.</li>
</ul>
<p>I have a C++ file importing a Python module and calling a function from it, everything seems to properly work until i do an “import torch” in the Python file.</p>
<p>** C++ CallPython function: **</p>
<blockquote>
<p>void CallPython(string PythonModuleName, string PythonFunctionName) {<br>
char* funcname = new char[PythonFunctionName.length() + 1];<br>
strcpy_s(funcname, PythonFunctionName.length() + 1, PythonFunctionName.c_str());<br>
char* modname = new char[PythonModuleName.length() + 1];<br>
strcpy_s(modname, PythonModuleName.length() + 1, PythonModuleName.c_str());<br>
DBG(“initialize the Python interpreter”);<br>
Py_Initialize();<br>
TCHAR cwd[2048];<br>
GetCurrentDirectory(sizeof(cwd), cwd);<br>
DBG(“Load the Python module”);<br>
PyObject* my_module = PyImport_ImportModule(modname);<br>
PyErr_Print();<br>
DBG(“Module found”);<br>
DBG(" find function from Python module");<br>
DBG(“Getting address of the funct in Python module”);<br>
PyObject* my_function = PyObject_GetAttrString(my_module, funcname);<br>
PyErr_Print();<br>
DBG(“Function found”);<br>
DBG(“call function from Python module”);<br>
PyObject* my_result = PyObject_CallObject(my_function, NULL);<br>
PyErr_Print();<br>
DBG(“Your function has been called”);<br>
Py_Finalize();<br>
delete<span class="chcklst-box fa fa-square-o fa-fw"></span> funcname;<br>
delete<span class="chcklst-box fa fa-square-o fa-fw"></span> modname;}</p>
</blockquote>
<p>**C++ main: **</p>
<blockquote>
<p>CallPython(“PythonFile2”, “helloworld”);</p>
</blockquote>
<hr>
<p>**Python file named PythonFile2.py: **</p>
<blockquote>
<p>import re<br>
import string<br>
import yaml<br>
<span class="hashtag-raw">#import</span> torch<br>
def helloworld():<br>
fh = open(‘myfile.txt’, ‘w’)<br>
fh.write(‘testing’)<br>
fh.close()</p>
</blockquote>
<hr>
<p>de-commenting the “import torch” line results in a “access violation while reading” exception generated by the “PyObject_GetAttrString” function. I think the problem is that i am not working in the properly python environment and the one that i am in doesn’t have torch installed, but i have no idea how to set the python environment from the C++ application.</p>
<p>Thanks for your attention!</p>

tempaccountforissue: <p>Were you ever able to solve this? I’m running into the same issue.</p>