akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Loading Big Data using DataLoader

nasim_nouri: <p>I use DataLoader to load a big dataset (16 GB). However, I ran out of memory. Is there any suggestion how can we use DataLoader in cases that we don’t have enough memory?</p>
<p>Here’s my code:</p>
<pre><code class="lang-auto">
class StockDataset(Dataset):
    def __init__(self, file_path):
        self.data = []
        with jsonlines.open(file_path, 'r') as reader:
            for item in tqdm(reader, total=2564721, desc="Loading Data"):
                self.data.append(item)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        sample = self.data[idx]
        input_ids = torch.tensor([item['Close'] for item in sample['sample']])
        label = torch.tensor(sample['label'])
        return input_ids, label

class StockDataModule(pl.LightningDataModule):
    def __init__(self, train_file, val_file, test_file, batch_size=32):
        super().__init__()
        self.train_file = train_file
        self.val_file = val_file
        self.test_file = test_file
        self.batch_size = batch_size

    def setup(self, stage=None):
        self.train_dataset = StockDataset(self.train_file)
        self.val_dataset = StockDataset(self.val_file)
        self.test_dataset = StockDataset(self.test_file)

    def train_dataloader(self):
        return DataLoader(self.train_dataset, batch_size=self.batch_size, shuffle=True)

    def val_dataloader(self):
        return DataLoader(self.val_dataset, batch_size=self.batch_size)

    def test_dataloader(self):
        return DataLoader(self.test_dataset, batch_size=self.batch_size)
</code></pre>
How to backpropagate combined loss over generator and only classification loss over discriminator?

ivana_janickova: <p>Hi. I have an architecture based on an autoencoder (AE) and a discriminator (D) (see image below). I have an MSE reconstruction loss and CE classification loss. I want to train the AE to minimize the reconstruction (MSE) loss and maximise the classification (CE) loss while training the discriminator to get better in the classification - hence minimizing the CE over discriminator (D).</p>
<p>Hence I want to backpropagate CEloss over discriminator and (MSEloss - CEloss) over autoencoder.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a59e2bba5b4a0db5909509b9301af4a8e980e73.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5a59e2bba5b4a0db5909509b9301af4a8e980e73" title="IMG_0006"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a59e2bba5b4a0db5909509b9301af4a8e980e73_2_690x244.jpeg" alt="IMG_0006" data-base62-sha1="cThyc0TinkDcbW9rqsoBHJga1FN" width="690" height="244" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a59e2bba5b4a0db5909509b9301af4a8e980e73_2_690x244.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a59e2bba5b4a0db5909509b9301af4a8e980e73_2_1035x366.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a59e2bba5b4a0db5909509b9301af4a8e980e73_2_1380x488.jpeg 2x" data-dominant-color="F3F1E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_0006</span><span class="informations">1667×590 97 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>What is the correct way to do this?</strong></p>
<p>Currently, I have this  (code below) but at <code>loss_ae.backward()</code> I have the following error: <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64]] is at version 2; expected version 1 instead.</code></p>
<pre><code class="lang-auto">input, true_class = data
optim_ae = AdamW(AEmodel.parameters(), lr1)
optim_d = AdamW(Dmodel.parameters(), lr2)

reconstruction, latent = AEmodel(input)
loss_mse = nn.MSELoss(reconstruction, input)

pred = DModel(latent)
loss_ce = nn.CrossEntropyLoss(pred, true_class)

loss_ae = loss_mse - loss_ce

optim_d.zero_grad()
loss_ce.bacward(retain_graph=True)
optim_d.step()

optim_ae.zero_grad()
loss_ae.backward()
optim_ae.step()
</code></pre>

ptrblck: <p>You are likely running into <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this issue</a> causing the second <code>backward</code> call to fail as the parameters were updated resulting in stale forward activations.</p>
Count how many elements of one tensor exists in another tensor

rockstar: <p>I have two tensors:</p>
<pre><code class="lang-auto">A = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
B = torch.tensor([2, 5, 6, 8, 12, 15, 16])
</code></pre>
<p>The tensors are different in size, random, and not in sequence. I want to get the count of the number of elements of <code>B</code> that (i) exists in <code>A</code> (ii) do not exist in <code>A</code>. So the output will be:</p>
<pre><code class="lang-auto">Exists: 4
Do not exist: 3
</code></pre>
<p>How to do this efficiently?</p>

ptrblck: <p>If your tensors are small and thus memory is not a concern this should work:</p>
<pre><code class="lang-python">A = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
B = torch.tensor([2, 5, 6, 8, 12, 15, 16])

exist = len((A.unsqueeze(1) == B).any(1).nonzero())
not_exist = len(B) - exist

print("exist {}, not exist {}".format(exist, not_exist))
# exist 4, not exist 3
</code></pre>

rockstar: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your help. Unfortunately, the tensors are extremely large, and I have to do this inside a loop. That’s why I need a fast and efficient approach.</p>

Mah_Neh: <p>Not sure how inefficient is this, but seems to work in this example.</p>
<pre><code class="lang-auto">import torch

A = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

B = torch.tensor([2, 5, 6, 8, 12, 15, 16])

# Check if elements of B exist in A

mask = torch.isin(B, A)

# Count the number of elements of B that exist in A

exists = mask.sum().item()

# Count the number of elements of B that do not exist in A

not_exists = (~mask).sum().item()

print(f"Exists: {exists}")

print(f"Do not exist: {not_exists}")
</code></pre>
<p>probably <a class="mention" href="/u/ptrblck">@ptrblck</a> easily realises if it’s too bad, or just test.</p>

Finch23: <p>We use Indexing and Slicing to access the values of a tensor. Indexing is used to access the value of a single element of the tensor, whereasSlicing is used to access the values of a sequence of elements. <a href="https://mybalancenow.fun/" rel="noopener nofollow ugc">My Balance Now</a> We use the assignment operator to modify the values of a tensor.</p>
Keras/TF model file size vs PyTorch

Weijun_Tan: <p>I implemented a standard Inception-V3 for custom image classification in both Keras/TF2 and PyTorch. Both give similar accuracy and loss value.  I notice that the model file sizes are very different.  In TF2 saved checkpoint files cp.ckpt-exx.data-00000-of-00001, the file size is 25M.  After saving to a h5 file, it becomes 9M.  However, in PyTorch, the model file is 85M. I calculate the number of parameters,</p>
<p>The TF/Keras model has number of parameters 21,780,646.<br>
The PyTorch model has number of parameters 21,797,286.<br>
If every parameter is 4 byte, then PyTorch file size is  21,797,286*4/1024**2 = 83.8 MB.</p>
<p>I am wondering how the Keras/TF makes the saved model file size as small as 9M?  I look up everywhere online but cannot find an answer.  Is the Keras/TF using variable length for the parameters?</p>
<p>Thanks a lot.</p>

ptrblck: <p>Maybe TF/Keras is compressing the data. The TF discussion board might be a better place to ask this.</p>

Mah_Neh: <p>are you saving as state dic? that takes more space I think ? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Weijun_Tan: <p>That only changes the file size a little bit. I am talking about Keras’ 9M vs. PyTorch’s 87M.</p>
UserWarning in torch.nn.MultiheadAttention

Rodrigo_Sanchez: <p>Hey! I am using torch==‘2.0.1+cu117’ on Linux and I am receiving a warning that claims that converting mask without torch.bool dtype to bool will negatively affect performance. It suggests to use a boolean mask directly…</p>
<p>I am calling the module like this:</p>
<p>self._attention_1(key=x, value=x, query=x, key_padding_mask=padding_mask) and ti have tested the padding mask to be a torch.tensor of type torch.bool or torch.float32.</p>
<p>I guess the warning is caused inside that function but i do not know if could be achieving more performance.</p>
The difference between torch.jit.script and torch.compile?

mirksonius: <p>Can anybody explain the difference between torch.jit.script decorator and torch.compile decorator? It seems to me that they both do the same thing. Is torch.compile just a new version of torch.jit.script?<br>
Thanks a lot!</p>

marksaroufim: <p>They differ in their graph capture mechanism, see this for more details <a href="https://dev-discuss.pytorch.org/t/the-nuances-of-pytorch-graph-capture/501" class="inline-onebox">The nuances of PyTorch Graph Capture - compiler - PyTorch Dev Discussions</a></p>
Accessing Internal Modules in DistributedDataParallel for Train/Eval Mode Switching

Miguel_Campos: <p>Hello everyone,</p>
<p>I’m currently working with a model that follows an encoder-decoder architecture. I need the encoder to be in evaluation mode while the decoder is the component being trained. Running the model on a single GPU is clear to me. However, I face some doubts when I try to use DistributedDataParallel (DDP). Here’s the relevant portion of my code:</p>
<pre><code class="lang-auto"># Set up for DDP
args.gpu = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(args.gpu)  # set default device
dist_backend = "nccl"
# ... (additional setup) ...

# Initialize the model
model = create_model().to(args.device)  # args.device = 'cuda'
model_without_ddp = model
if distributed:
    model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])  # args.gpu = 0
    model_without_ddp = model.module

# Training loop
for epoch in range(start_epoch, args.epochs):
    train_one_epoch(model, model_without_ddp, optimizer, dataloader, args)
    evaluate(model, dataloader_test, args)
</code></pre>
<p>You’ll notice I’m using both <code>model</code> and <code>model_without_ddp</code>. This is based on PyTorch’s reference scripts, which I’ve used to ensure consistent code and minimize condition checks for DDP usage.</p>
<pre><code class="lang-auto">def train_one_epoch(model, model_without_ddp, optimizer, dataloader, args):
    model.train()
    model_without_ddp.encoder.eval()  # This replaces: if distributed: model.module.encoder.eval() else model.encoder.eval()
    train_loss = 0.0
    for x, _ in dataloader:
        x = x.to(args.device)

        loss = model(x)
        optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model_without_ddp.decoder.parameters(), 1.0)  # Replaces: if distributed: model.module.decoder.parameters() else model.decoder.parameters.
        optimizer.step()

        train_loss += loss.item()
    return train_loss

@torch.inference_mode()
def evaluate(model, dataloader_test, args):
    model.eval()  # entire model is in eval mode
    # ... (rest of the evaluation code) ...
    return metrics
</code></pre>
<p>My first question is: Is this the correct approach for accessing and modifying the internal modules of a model wrapped in DDP, especially when switching between the <code>train()</code> and <code>eval()</code> modes?</p>
<p>Additionally, I have another question. In the training loop, I use <code>model(x)</code> for the forward pass, which implicitly includes both the encoder and decoder. Still, I explicitly use <code>model_without_ddp.decoder.parameters()</code> (the component I train). I wonder if I should be using <code>model.parameters()</code> because of some weird stuff inside the DPP class that I don’t know about (although the coder parameters are unnecessary) or if, on the contrary, my methodology is a fine approach.</p>
<p>Thanks</p>

Miguel_Campos: <p>Hi there! just in case</p>
Faster RCNN With Varying Number of Bounding Boxes

David_Valensi: <p>Hi,<br>
I’m finetuning a Faster RCNN model. When it comes to the labels, the number of bounding boxes in the images varies between 0 to 3. Most images have only one bounding box. i.e. one appearance of an object. I saw that during training the input to the model should be a tuple of images and labels, and the labels should be in the form if a dictionary with five keys (boxes, labels, image_id, area and is_crowd). All values of the dictionary are tensors. The value of area, for example, is a tensor of shape batch size X maximal number of objects in an image. Because the maximal number of objects in an image in my dataset is 3, does it mean that I need to input the model fake objects for images that have fewer than the maximal number of objects? I mean that if an image has only one corresponding labeled bounding box I can’t input it to the network as is so I need to fill some fake information. In this case what area values do I need to input for the second and third bounding boxes of an image that in reality has only one bounding box? More generally, what should I do of each image has a different number of bounding boxes?<br>
The same dilemma applies to the other values of the dictionary.</p>
<p>I should mention that when I tried to input the model with a different number of bounding boxes per image I got the following error: “Expected target boxes to be a tensor of shape [N, 4]…”. The error was solved when I began inputting the model with three bounding boxes per each image.</p>
How to use Inductor to generate Triton code on GPU?

ybai62868: <p>I build the pytorch from source with the commit id :f9a250c35bd061e2e6f4c2d92e2b1b16390e8636</p>
<p>I want to use the inductor to help me generate the triton code on the GPU? How to do it?<br>
When I write the softmax, matmul, or bmm kernels and compile it with “TORCH_COMPILE_DEBUG=1” flag, I find all of them are executed with the aten:: implementation in the torch_compile_debug folder.<br>
How to assign a kernel to execute with the triton template directly? Is there any flag I should set?<br>
Thanks a lot!</p>
PyTorch Chatbot Tutorial: Loss not decreasing as expected in MPS device (Apple Silicon)

codingGoose: <p>After following the code in the <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" rel="noopener nofollow ugc">PyTorch Chatbot Tutorial</a>, I was not able to obtain the relatively low loss from the tutorial using a Mac with Apple Silicon. To make sure I didn’t introduce a bug, I downloaded the ipynb file from the tutorial page and only changed the device to MPS:</p>
<pre><code class="lang-auto"># USE_CUDA = torch.cuda.is_available()
# device = torch.device("cuda" if USE_CUDA else "cpu")
device = torch.device("mps")
</code></pre>
<p>I also set <code>conda env config vars set PYTORCH_ENABLE_MPS_FALLBACK=1</code> in my conda environment.</p>
<p>Despite running the same code as the tutorial with the same input data, the loss I obtain is around ~5.1049, while the tutorial’s is around ~2.9412. I am aware that in this setting I would not be able to reproduce the tutorial’s result, but I am surprised by the noticeable difference. Running the same ipynb file in Google colab (using CPU) gives a loss of ~2.5723.</p>
<p>Is the difference in loss between MPS and the tutorial/Google colab in this case due to stochasticity in PyTorch’s backend or is there something else going on?</p>
Elementwise_kernel low efficiency when input size is small?

tianyma: <p>Hi, I am profiling my torch code using NVIDIA nsight-cli tool, and the experiment is on GPU V100. I found that <strong>elementwise_kernel</strong> is the most time-consuming. I found the actual reason is that the <strong>elementwise_kernel</strong> uses few cuda threads, for example, a two vector’s multiplication calls the <strong>elementwise_kernel</strong>, and the size of each vector is 500 in FP32, but the profiling result is that <strong>elementwise_kernel</strong> actually uses 128 threads, rather than 500 threads.</p>
<p>So my question is why does the <strong>elementwise_kernel</strong> use such few CUDA threads? Is it because the input size is too small?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/8/9805476ac8828cd8a134ba0e100d64574598afe2.png" data-download-href="https://discuss.pytorch.org/uploads/default/9805476ac8828cd8a134ba0e100d64574598afe2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/8/9805476ac8828cd8a134ba0e100d64574598afe2.png" alt="image" data-base62-sha1="lGPSUsD6nFR70VeR3rjK6zZZVgm" width="555" height="500" data-dominant-color="282626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">946×852 40 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Batch sparse matrix multiplication

C_Kim: <p>Hi guys.</p>
<p>I’m studying the FEM in neural network with pytorch.</p>
<p>My question is existence of the ‘batch’ + ‘sparse’ + ‘matrix multiplication’ function in a single code.<br>
There are several method for this:</p>
<ul>
<li>torch.bmm</li>
<li>torch.sparse.mm</li>
</ul>
<p>However, I cannot find the ‘batch’ + ‘sparse’ matrix multiplication in a single function.</p>
<p>Here is my data:</p>
<ul>
<li>batch sparse matrix size: (batch, 126, 126)</li>
<li>batch dense vector size: (batch, 126, 1)</li>
</ul>
<p>I think that the matrix would be extended 120,000 by 120,000 in size so that the memory issue can have to be dealt with.<br>
Is there any approach to address ‘batch’ + ‘sparse’ matrix multiplication without memory issue?</p>
<p>Thank you!</p>
Libtorch inference on CPU about 5 times slower than python version

Source82: <p>Hello,<br>
So am using pytorch to make inference (using Resnet50), I have been constrained to run it on CPU - so I have no option.</p>
<p>To my surprise and confusion the python version makes inference in ~0.25 secs. However the libtorch version takes ~0.7 secs.</p>
<p>Please suggestions will be appreciated as am really new to this, and please GPU is a no no in this scenario we are specifically asked to do it on CPU.</p>

tom: <p>There are three typical causes of such a thing:</p>
<ul>
<li>you made a debug build, or one without avx,</li>
<li>you are missing some back end library in libtorch, or</li>
<li>the threading setup is missing.</li>
</ul>
<p>The first two won’t happen when you use libtorch from the PyTorch installation.</p>
<p>For the third, you can debug this by looking at <code>top</code> while running your programming to see if CPU goes over 100%.</p>
<p>Best regards</p>
<p>Thomas</p>

Source82: <p>You are right my build had an issue.<br>
Thanks</p>

FantasyJXF: <p>In my case, I tested the pretrained ResNet34 with both PyTorch and libtorch, I use 400 images, the the preprocess is the same.<br>
On my MAC:<br>
PyTorch used 34s, CPU usage &gt; 90%;<br>
libtorch used 73s, CPU usage &lt; 40%.</p>
<p>Why did this happen?.<br>
The libtorch is download from the official website</p>

gemfield: <p>You may have a look at <a href="https://zhuanlan.zhihu.com/p/363319763" class="inline-onebox" rel="noopener nofollow ugc">PyTorch vs LibTorch：网络推理速度谁更快？ - 知乎</a> and <a href="https://github.com/DeepVAC/libdeepvac/issues/30" class="inline-onebox" rel="noopener nofollow ugc">libtorch性能问题调研 · Issue #30 · DeepVAC/libdeepvac · GitHub</a></p>

WolfKingAlex: <p>would you bother describe what kind of issue it might be, I met the same performance loss on libtorch inference.</p>
Torch.round() gradient

hafez_farazi: <p>Hi,</p>
<p>As far as I understand round function is not differentiable. So what is it doing in backward pass of torch.round()?</p>

tumble-weed: <p>It’ll give you 0, as this snippet shows:</p>
<pre><code class="lang-auto">a = torch.tensor(1.5)
a.requires_grad_()
b = torch.round(a)
b.backward()
a.grad
</code></pre>

mattinjersey: <p>I tried this…it looks like torch.round is not differentiable. But torch.clamp is differentiable. Is this clear from the documentation or how do we know which functions are differentiable?</p>

colesbury: <p>Both functions are differentiable almost everywhere. Think about what the functions look like.</p>
<p>round() is a <a href="https://en.wikipedia.org/wiki/Step_function" rel="nofollow noopener">step function</a> so it has derivative zero almost everywhere. Although it’s differentiable (almost everywhere), it’s not useful for learning because of the zero gradient.</p>
<p>                    <a href="https://i.stack.imgur.com/lIlGw.jpg" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.stack.imgur.com/lIlGw.jpg" width="666" height="500">
          </a>

</p>
<p>clamp() is linear, with slope 1, inside (min, max) and flat outside of the range. This means the derivative is 1 inside (min, max) and zero outside. It can be useful for learning as long as enough of the input is inside the range.</p>
<p>                    <a href="http://www.yaldex.com/open-gl/images/05fig10.jpg" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="http://www.yaldex.com/open-gl/images/05fig10.jpg" width="318" height="252">
          </a>

</p>

Lucy_Jackson: <p>Hi,</p>
<p>I was just wondering if you know a way that you can round the output of a network and still allow for successful learning?</p>
<p>Thanks</p>

Zhongzhi_Yu: <p>One possible approach is to implement your own round function with customized backpropagation.</p>
<p>E.g.</p>
<pre><code class="lang-auto">class my_round_func(InplaceFunction):
    @staticmethod
    def forward(ctx, input):
        ctx.input = input
        return torch.round(input)

    @staticmethod
    def backward(ctx, grad_output):
        grad_input = grad_output.clone()
        return grad_input
</code></pre>

zhang1: <p>how do i use it?replace <code>InplaceFunction</code> to <code>nn.Module</code>?</p>

bog: <p>Should be torch.autograd.function.InplaceFunction</p>

gabrieldernbach: <p>If you need a gradient to be passed, you can make use of a <code>straight through estimator</code></p>
<pre><code class="lang-auto">o = x + x.round().detach() - x.detach()
</code></pre>
<p>During forward pass, the x and -x will cancel.<br>
During backward pass, x.round() and -x will be ignored, gradient flow will continue from x, instead of x.round().</p>
Doing builds on m1 macs

Matt_Davis: <p>Does anyone have a guide for this? I need to do a rebuild of pytorch and libtorch to use the la pack for a model I’m trying to move to mobile. Does anyone have any guides or experience that would help me with this?</p>

dhruvbird: <p>cc <a class="mention" href="/u/xta0">@xta0</a> - do you have some pointers how to make this work?</p>

Linbin: <p><a class="mention" href="/u/matt_davis">@Matt_Davis</a> so far I don’t have experience in building PyTorch on M1 Mac. I did some web search and find some folks successfully built it on M1.</p>

Lukasz_Wojciak: <p>Hi Matt,</p>
<p>What I run is:<br>
OPENSSL_ROOT_DIR="/opt/homebrew/opt/openssl/" BLAS=“Generic” MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install</p>
<p>Possible BLAS option: “Generic” (which uses cblas), “OpenBLAS”, “vecLib”</p>
<p>You may need to add other env variables depending on what you have in your env - generally a sane homebrew python install should get you there eventually</p>

davidradl: <p>An update MACOSX_DEPLOYMENT_TARGET=10.9 does not work any more as you now get errors related to this target being too low:</p>
<pre><code class="lang-auto">In file included from /Users/davidradley/pytorch/aten/src/ATen/native/mkl/SpectralOps.cpp:206:

**/Users/davidradley/pytorch/third_party/pocketfft/pocketfft_hdronly.h:156:15:** **error:** **'aligned_alloc' is only available on macOS 10.15 or newer [-Werror,-Wunguarded-availability-new]**

void *ptr = ::aligned_alloc(align,(size+align-1)&amp;(~(align-1)));

**^~~~~~~~~~~~~~~**

**/Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/usr/include/malloc/_malloc.h:50:10: note:** 'aligned_alloc' has been marked as being introduced in macOS 10.15 here, but the deployment target is macOS 10.9.0

void *aligned_alloc(size_t __alignment, size_t __size) __result_use_check __alloc_size(2) __OSX_AVAILABLE(10.15) __IOS_AVAILABLE(13.0) __TVOS_AVAILABLE(13.0) __WATCHOS_AVAILABLE(6.0);

**^**

**/Users/davidradley/pytorch/third_party/pocketfft/pocketfft_hdronly.h:156:15: note:** enclose 'aligned_alloc' in a __builtin_available check to silence this warning

void *ptr = ::aligned_alloc(align,(size+align-1)&amp;(~(align-1)));
</code></pre>

digantdesai: <p>Perhaps try disabling PocketFFT if possible, via <code>AT_POCKETFFT_ENABLED</code> CMake option?</p>
Propagate gradients through torch.kron?

jgscott: <p>Hi!  I’m wondering if someone has insight on the following issue.  My problem is more complicated but I’ve tried to boil it down to the essential.  Suppose that $\theta$ is an $n \times n$ matrix and I have a loss function that looks like:</p>
<p>L(theta) = trace{ V(theta) (X - theta) (X - theta)^T } - (n/2) log det(V(theta))</p>
<p>However, V(theta) is itself a function of theta.  Specifically, V is defined <em>implicitly</em> as solution to the discrete-time Lyapunov equation V(\theta) = \theta V(\theta) \theta^T + Q for fixed Q.</p>
<p>I know several methods, of varying degrees of efficiency, for finding the V(\theta) that solves the discrete time Lyapunov equation for fixed \theta.  The question is: what’s a way to do it so that gradients will propagate appropriately through this solution?  The only way I can see is to use the fact that one can use properties of the vec operator and Kronecker product to write:<br>
$$<br>
(I - \theta \oprod \theta^T) \text{vec}(V(\theta) = \text{vec}(Q)<br>
$$</p>
<p>So using torch.kron, torch.solve, and torch.reshape, I can express V(\theta) using torch functions and (presumably) get gradient propagation.</p>
<p>My questions are two fold:</p>
<ol>
<li>Concerning the API of torch.kron: it’s not clear to me, but <a href="https://pytorch.org/docs/stable/generated/torch.kron.html" rel="noopener nofollow ugc">the way it’s phrased</a> makes me worry that gradients only propagate through the first argument.  I’m hopeful this is wrong!  In other words, if I call torch.kron(theta, torch.transpose(theta)), will gradients propagate correctly?</li>
<li>This method is O(n^6) to solve for $V(\theta)$.  Ouch.  Does anyone know how to get one of the more efficient ways of solving the DT Lyapunov equation to play nicely with torch autograd?  Basically an autograd-friendly version of scipy.linalg.solve_discrete_lyapunov?</li>
</ol>
<p>Thanks for your time and thought!</p>
How to invert stacking a set of tensors?

AlphaBetaGamma96: <p>Hi All,</p>
<p><strong>TL;DR</strong> How can I de-stack a large tensor into a dict of Tensors in PyTorch?</p>
<p>Let’s say I compute the gradients of a model for a set of samples and concatenate them into a single tensor, is there a way to invert this process?</p>
<p>I have a minimal reproducible example below, is there an efficient way to convert <code>grad_params_stack</code> back to <code>grads_params</code>?</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.func import jacrev, vmap, functional_call

class network(nn.Module):

  def __init__(self, num_input, num_hidden, num_output):
    super(network, self).__init__()
    self.num_input = num_input
    self.num_hidden = num_hidden
    self.num_output = num_output
    
    self.func = nn.Tanh()
    
    self.fc1=nn.Linear(self.num_input, self.num_hidden)
    self.fc2=nn.Linear(self.num_hidden, self.num_hidden)
    self.fc3=nn.Linear(self.num_hidden, self.num_hidden)
    self.fc4=nn.Linear(self.num_hidden, self.num_output)
    
  def forward(self, x):
    x=self.fc1(x)
    x=self.func(x)
    x=self.fc2(x)
    x=self.func(x)
    x=self.fc3(x)
    x=self.func(x)
    x=self.fc4(x)
    return x.squeeze(0)
    
batch_size=100
num_input=4
num_hidden=32
num_output=1

net = network(num_input=num_input, num_hidden=num_hidden, num_output=num_output)
num_params = sum([p.numel() for p in net.parameters()])

#inputs
params = dict(net.named_parameters())
x = torch.randn(batch_size, num_input)

def fcall(params, x):
  return functional_call(net, params, x)

#per-sample gradients of the network w.r.t model params
grads_params = vmap(jacrev(fcall, argnums=(0)), in_dims=(None,0))(params, x) #compute gradients via torch.func

grad_params_stack = torch.cat([v.flatten(start_dim=1) for v in grads_params.values()], dim=-1)
print(grad_params_stack.shape) # batch_size by num_params 

#How to invert grad_params_stack back to grads_params?
</code></pre>

AlphaBetaGamma96: <p>Figured it out,</p>
<pre><code class="lang-auto">#How to invert params_stack back to params?
import numpy as np
param_loc = np.cumsum([0]+[p.numel() for p in net.parameters()])
layer_idx = [[param_loc[i], param_loc[i+1]] for i in range(len(param_loc)-1)]
grad_params_destacked = {key: grad_params_stack[:,indices[0]:indices[1]].reshape(batch_size, *value.shape) for (key, value), indices in zip(params.items(), layer_idx)}

for (k1,v1), (k2,v2) in zip(grads_params.items(), grad_params_destacked.items()):
  assert k1==k2
  print(k1, torch.allclose(v1,v2)) #returns True
</code></pre>
Install pytorch using system cuda and cudnn

Kinbote: <p>I have an nvidia docker container with its own system wide cuda and cudnn. I’m trying to create several python environments, each with its own torch install. I was wondering if it’s possible to link these torch installs with the system cuda and cudnn so that I don’t have to download and install cuda and cudnn in each of these environments.</p>

Mah_Neh: <p>Can’t you just install those in a conda base environment ?</p>

ptrblck: <p>Yes, you can build PyTorch from source which would use your locally installed CUDA toolkit and cuDNN in a new virtual environment.</p>
Confused about stored activation memory usage

infiniteperplexity: <p>Could someone explain the memory usage for this block of code?</p>
<pre><code class="lang-auto">import torch
from torch import nn
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
def cuda_memory(msg):
    print("usage after", msg, torch.cuda.memory_allocated(device)/1024**2)

#with torch.no_grad():
with torch.enable_grad():
    dim, rank, outer_product_layers = 768, 3, 4
    vocab_size, seq_len = 10, 20
    inputs = torch.randint(0, vocab_size, (seq_len,))
    cuda_memory("initial") # 0.0
    acts = nn.Embedding(vocab_size, dim)(inputs).to(device)
    cuda_memory("inputs on device") # 0.029
    linear = torch.randn(dim, dim, requires_grad=True).to(device)
    cuda_memory("linear on device") # 2.279
    acts = torch.matmul(acts, linear)
    cuda_memory("linear activations") # 10.404
    for layer in range(outer_product_layers):
        u = torch.randn(dim, rank, requires_grad=True).to(device)
        v = torch.randn(rank, dim, requires_grad=True).to(device)
        cuda_memory(f"u and v on device layer {layer}") # increases ~0.02 each time
        acts = torch.matmul(acts, linear+torch.matmul(u, v))
        #acts = torch.matmul(acts, linear) # memory doesn't increase much
        #acts = torch.matmul(acts, torch.matmul(u, v)) # memory increases about the same amount
        cuda_memory(f"layer {layer} activations") # increases ~2.25 each time
</code></pre>
<p>I was attempting a weight-sharing scheme wherein each layer’s weights are a low-rank update added to the previous layer’s weights. Naively, I thought this would save a lot of GPU memory by re-using weight values from the initial linear layer. But it looks like some intermediate values are being saved as well - either the activations or the product of u and v? Is that required in order to calculate the gradients? The memory bump doesn’t happen if I change enable_grad() to no_grad().  It also doesn’t happen if I simply re-use the linear layer at each step; it <strong>does</strong> happen if I use the outer product at each step without adding linear.</p>
<p>Thanks in advance for any insights.</p>
<p>Edit: Thanks for the formatting advice!</p>

ptrblck: <p>Could you format your code by wrapping it into three backticks ```, please?</p>

infiniteperplexity: <p>Done, thanks for the tip.</p>
ToTesnor() vs Normalize()

Abdelkareem_Elkhateb: <p>I have was reading that ToTensor() turns the data into a 32-bit floating point per channel,scaling the values form 0.0 to 1.0<br>
so what is the purpose of transfomers.Normalize(std,mean)!</p>

ptrblck: <p><code>Normalize</code> subtracts the passed <code>mean</code> from the input and divides by the <code>std</code> to create a zero-mean and unit-variance output. This normalization is also known as standardizing, z-scoring, or whitening in signal processing and was shown to help ML training.</p>
Normalizing with imagenet mean and std vs normalizing with my own dataset's mean and std

shrutishrestha: <p>I am using a pre-trained network with imagenet data .Tthe mean and std of imagenet are<br>
mean = [0.485, 0.456, 0.406]<br>
std = [0.229, 0.224, 0.225]<br>
whereas, I have 1000 images in my dataset.<br>
Mean and std of my dataset are :<br>
mean = [0.5589, 0.3216, 0.2356]<br>
std =[ 0.3056, 0.21442, 0.1775]<br>
I am confused if I should use imagenet MEAN AND STD or use my own mean and std to normalize my images?</p>
<p>Is there any way to find if fine-tuning is good or feature extraction i.e. updating the last layer in good for this case?</p>

ptrblck: <aside class="quote no-group" data-username="shrutishrestha" data-post="1" data-topic="94953">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shrutishrestha/48/51287_2.png" class="avatar"> shrutishrestha:</div>
<blockquote>
<p>Is there any way to find if fine-tuning is good or feature extraction i.e. updating the last layer in good for this case?</p>
</blockquote>
</aside>
<p>You could try both approaches and check the training as well as validation losses.<br>
Let us know, which stats worked better for your use case. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

Abdelkareem_Elkhateb: <p>is not this a strange answer! or this how deep learning works! <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I would claim it’s unfortunately how the majority of things work and the answer is often: “it depends”.<br>
E.g. commonly it’s believed fine-tuning the last layer only while freezing the feature extractor works if the data comes from a similar domain (e.g. ImageNet used for pretraining while your current dataset also contains “natural” images). Training more layers (including the feature extractor) could be helpful if the data domain changes (e.g. if you are now using medical CT scans). While this makes sense I wouldn’t know when the data domain changes significantly enough to swap to a full model training.<br>
E.g. would another JPEG encoding already force you to finetune the full model? What happens if your “natural” images were transformed via a filter and their colormap changed?<br>
So, I would claim you would still have to run experiments as “it depends” <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
Backpropagating Error Through Neural Network to Train Another Neural Network

SmartAutonomy: <p>Hello! I am reading Nguyen and Widrow’s 1990 paper “Neural networks for self-learning control systems.” I have a general understanding of the paper: a neural network (NN) is first trained to emulate an unknown plant. Then a second NN is trained to control the NN emulator.</p>
<p>I understand how to train the NN emulator. However, I don’t understand how to train the NN controller. In the subsection “Training the Neural Network Controller,” the paper states that we can backpropagate the plant error through the (trained) NN emulator to calculate the control error to train the NN controller. This sounds like we are reversing the NN emulator and inputting the plant error and outputting the control error.</p>
<p>Is there a PyTorch function for doing this? Or is there a GitHub repository that I can look at to see how this is done? Any help is greatly appreciated! Thank you so much for your time!</p>
`torch.compile` a vmapped ensemble

zhixuan-lin: <p>I’m trying to use <code>torch.compile</code> around a vmapped ensemble but it fails with <code>Batching rule not implemented for aten::t</code>. A minimal example:</p>
<pre><code class="lang-python">import torch
from torch import nn
from torch._dynamo import allow_in_graph
from functools import wraps
from torch.func import stack_module_state, functional_call
import functorch
import copy

def traceable(f):
    f = allow_in_graph(f)

    @wraps(f)
    def wrapper(*args, **kwargs):
        return f(*args, **kwargs)

    return wrapper

x = torch.randn(3)

class Net(torch.nn.Module):
    def __init__(self, *args, **kwargs) -&gt; None:
        super().__init__(*args, **kwargs)
        self.fc = nn.Linear(3, 5)

    def forward(self, x):
        return self.fc(x)


models = [Net() for _ in range(5)]
params, buffers = stack_module_state(models)

# Construct a "stateless" version of one of the models. It is "stateless" in
# the sense that the parameters are meta Tensors and do not have storage.
base_model = copy.deepcopy(models[0])
base_model = base_model.to('meta')

def fmodel(params, buffers, x):
    return functional_call(base_model, (params, buffers), (x,))
vmapped_fmodel = functorch.vmap(fmodel, in_dims=(0, 0, None))

# Works
vmapped_fmodel(params, buffers, x)

# Doesn't work
torch.compile(traceable(vmapped_fmodel))(params, buffers, x)
</code></pre>
<p>I already opened a <a href="https://github.com/pytorch/pytorch/issues/109025" rel="noopener nofollow ugc">Github issue</a> regarding <code>aten::t</code>. Here I’m just wondering if there is any simple trick to get around this issue. Thanks!</p>
How to handle the value outside the fp16 range when casting?

111357: <p>I know the fp32 and fp16 have different ranges.</p>
<p>How does the PyTorch handle the tensor whose value is outside the fp16 range when casting?</p>
<p>For example, x =  torch.Tensor([66666])</p>
<p>If it cast x into inf, does this mean the gradient is Nan, and the training will fail?</p>

ptrblck: <p>Yes, a direct cast to <code>float16</code> will overflow and create invalid values. During mixed-precision training with <code>flaot16</code> this could happen if the loss scaling factor is too large and the gradients thus overflow.<br>
The <code>scaler.step(optimizer)</code> call skips the <code>optimizer.step()</code> call if invalid gradients are detected and will decrease the scaling factor until the gradients contain valid values again.</p>

111357: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,  thanks for your reply. Is possible to paste the source code link of the <code>skip</code> behavior? I want to explore the details. <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Yes, you can take a look at <a href="https://github.com/pytorch/pytorch/blob/878f4f09d211f4c24f210999cf64b24506732915/torch/cuda/amp/grad_scaler.py#L291-L345"><code>GradScaler.step</code></a> and <a href="https://github.com/pytorch/pytorch/blob/878f4f09d211f4c24f210999cf64b24506732915/torch/cuda/amp/grad_scaler.py#L285-L289"><code>_maybe_opt_step</code></a> to see the implementation.</p>

111357: <p>Got it, thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>

hei_Z: <p>Hello, I would like to ask if I use fp16 for the convolution operation during the inference process, so is the result after multiplication and addition fp32 or fp16? If it is fp32, how does pytorch intercept fp16? Is there any specific code for this part that I can refer to? looking forward to your reply.</p>

ptrblck: <p>Take a look at the <a href="https://pytorch.org/docs/stable/amp.html">Automatic Mixed Precision Package</a> to see how PyTorch applies <code>amp</code>.</p>
How to compute gradient for specific parameters to implement a penalisation term?

JB_Guimbaud: <p>I would like to compute a gradient penalty w.r.t. to model parameters that are related to specific input variables in the first layer. To do that I implemented a custom loss function such as:</p>
<pre><code class="lang-auto">def custom_loss(model, output, target, penalization_coef, mode:str):
    assert(mode=="train" or mode=="eval")
        
    main_loss = torch.mean((output - target)**2) # some loss function for the predictive task

    # compute penalisation norm
    if mode == "eval":
        penalisation_norm = 0
    else:
        grad_params = main_loss.backward(retain_graph=True)
        param = next(model.parameters()) # get first layer parameters
        penalisation_norm = penalization_coef * param.grad[:, [input_indexes]].abs().sum() # compute a L1 norm of the gradient associated with specific input variable listed in input_indexes
        for param in model.parameters(): # Reset gradients
            param.grad = None
    
    aggregated_loss = main_loss + penalisation_norm
    
    ...
    
    return main_loss, aggregated_loss
</code></pre>
<p>Problem is that it doesn’t seem to make any difference with the training without penalisation… What i’m doing wrong here?</p>
Reset a model's certain parameters using .data.clone() or .data.detach().clone()?

JaneChen: <p>Hello, may I consult about the difference in the effects of <code>tensor.data.clone()</code> and <code>tensor.data.detach().clone()</code>?</p>
<p>My goal is to interpolate between two parameters (of two models, respectively) and then assign the result to a new parameter (of a new model). My attempt is</p>
<p><code>new_param.data = 0.1 * old_param_1.data.clone() + 0.9 * old_param_2.data.clone()</code></p>
<p>I expect that <code>new_param</code> has no history of operations or gradients of the old parameters, and the modification of the new parameter or old parameters does not affect each other. Is my attempt correct?</p>
<p>Additionally, what if I do<br>
<code>new_param.data = 0.1 * old_param_1.data.detach().clone() + 0.9 * old_param_2.data.detach().clone()</code>?</p>
<p>Is it necessary to add <code>.detach()</code>?</p>
<p>Thank you in advance!</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/janechen">@JaneChen</a>,</p>
<p>If you want the <code>new_params</code> to have no history of the previous operation you will need to detach those values in order to get that behavior.</p>
<p>Also, don’t call the <code>.data</code> attribute of a tensor as you’ll ignore some of the metadata of the tensor, you should call <code>.detach().clone()</code> directly on the tensor itself rather than the <code>.data</code> attribute.</p>
How to sample values within a range from pytorch distributions?

ADONAI_TZEVAOT: <blockquote>
<blockquote>
<blockquote>
<p>m = Normal(torch.tensor([0.0]), torch.tensor([1.0]))<br>
m.sample()<br>
How do I sample values only within -1 and 1.</p>
</blockquote>
</blockquote>
</blockquote>
<p>Please help!</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/adonai_tzevaot">@ADONAI_TZEVAOT</a>,</p>
<p>If you want sample in [-1, 1), you can simply called <code>2*torch.rand(num_samples) - 1</code>, where <code>num_samples</code> is the number of samples you want</p>

ADONAI_TZEVAOT: <p>But i want the values to be sampled from the normal distribution</p>

AlphaBetaGamma96: <p>A normal distribution isn’t bound between -1 and +1, it’s bound between -infinity and +infinity, why do you want only the values in [-1, +1]?</p>

Ayush_Singhal: <p>One way is to pass the values through <code>tanh()</code> function</p>
<pre><code class="lang-auto">vals = torch.tanh(torch.rand(num_of_samples))
</code></pre>
<p>Tanh function will simply kind of scale your values in a range of <code>[-1 : 1]</code>, which satisfies your question</p>
<pre><code class="lang-auto">How do I sample values only within -1 and 1.
</code></pre>

AlphaBetaGamma96: <p>Yes, that would scale between -1 and +1, but the probability values associated with those points would be incorrect. You’d need to multiply those values by the determinant of the Jacobian of the transformation in order to correct the probability values as well.</p>
Scaling data parallel training on a single-machine with multiple CPUs (no GPUs)

mcft: <p>What is the best way to scale <strong>data parallel</strong> training on a <strong>single machine with multiple CPUs</strong> (<em>no GPUs</em>)?</p>
<ul>
<li>We need to speed up training for a customer, because the training dataset grew substantially recently</li>
<li>We <strong>can’t use GPUs</strong>, but we can increase CPU-cores and memory on a dedicated machine</li>
</ul>
<p>I researched the usual options for accelerating PyTorch, but I can’t figure out what the “right” approach is for a single-machine multiple-CPUs scenario:</p>
<p><strong>1 <code>DataParallel</code> and <code>DistributedDataParallel</code></strong><br>
From reading the documentation, I got the impression that <code>DataParallel</code> and <code>DistributedDataParallel</code> are designed to work only with GPUs.</p>
<p>Is that assumption correct?<br>
If not, could you point me to a code sample for correctly setting those up for CPUs?</p>
<p><strong>2 Vanilla PyTorch on CPUs</strong><br>
We tested our vanilla PyTorch training loop on a single 8-core CPU machine.<br>
All cores were used during training, which implies</p>
<ul>
<li>that PyTorch is somehow parallelizing across CPUs already</li>
<li>and that we could add cores to speed things up.</li>
</ul>
<p>But what is happening under the hood exactly?<br>
We are assumingly not using data parallelization (or model parallelization for that matter), because we are using our unaltered training code.</p>
<p>Is naively adding cores and memory in that setup the right approach?</p>
<p><strong>3. Ray Train</strong><br>
We set up Ray Train (based on <a href="https://docs.ray.io/en/latest/train/dl_guide.html" rel="noopener nofollow ugc">this</a>) and it worked fine on a single machine with multiple CPUs. It even has an explicit <code>use_gpu</code> flag to disable GPU usage.</p>
<p>However according to the documentation, Ray Train uses <code>DistributedDataParallel</code> under the hood. Referring to the questions raised in points 1) and 2) above, is Ray Train the recommended way to scale on a single CPU-only machine?</p>
<p><strong>4 did we miss anything?</strong></p>
<p>Thank you!</p>
BCEWithLogitsLoss inherits_Loss and not _WeightedLoss

tomerg11e: <p>hey, while working on my own loss function I have notice that<br>
CrossEntropyLoss and BCELoss  inherits _WeightedLoss (which inherits _Loss)<br>
while BCEWithLogitsLoss inherits the _Loss class.<br>
The only difference between _WeightedLoss  and _Loss is the weight param and registering it in self’s buffer.<br>
in BCEWithLogitsLoss initiallization, we initialize a wight param and pos_weight param and use those exactly in the same way as in _WeightedLoss.</p>
<p>Is there a reason for this? should BCEWithLogitsLoss also inherits _WeightedLoss ?</p>
<p>I am using torch 2.0.1<br>
Thanks!</p>