akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Levy stochastic jump path using tensors instead of numpy

Mohammed_Alruqimi: <p>I have this function to generate a Levy stochastic jump path implemented by numpy.</p>
<pre><code class="lang-auto">np.random.seed(4)
#generate the merton jump

def merton_jump_paths_numpy( T, r, sigma,  lam, m, v, steps, Npaths):
    size=(steps,Npaths)
    dt = T/steps
    poi_rv = np.multiply(np.random.poisson( lam*dt, size=size),
                     np.random.normal(m,v, size=size)).cumsum(axis=0)
    geo = np.cumsum(((r -  sigma**2/2 -lam*(m  + v**2*0.5))*dt +\
                          sigma*np.sqrt(dt) * \
                          np.random.normal(size=size)), axis=0)

    return np.exp(geo+poi_rv)
</code></pre>
<p>I have a torch.nn network to calibrate <code>r,sigma,  lam, m, and v </code> variables. So they are tensor paramaters with <code>requires_grad=True</code>.<br>
For this reason, I need help to rewrite this function using tensors instead of numpy.<br>
I have tried some code like this:</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(4)

def merton_jump_paths_tensors( T, r, sigma,  lam, m, v, steps, Npaths):
   lam = torch.tensor(lam) 
   m = torch.tensor(m) 
   v = torch.tensor(v)  
   r = torch.tensor(r) 
   sigma = torch.tensor(sigma)
   size=(steps,Npaths)
   size = torch.tensor(size)
   dt = T/steps
   dt_tensor = torch.tensor(dt)
   poi_rv = torch.poisson(lam * dt_tensor, size=size)
   normal_rv = torch.normal(m, v, size=size)
   poi_rv = poi_rv * normal_rv
   poi_rv = torch.cumsum(poi_rv, dim=0)
   drift_term = r - sigma **2 / 2.0 - lam * (m + v**2 / 2.0) * dt_tensor
   diffusion_term = sigma * torch.sqrt(dt_tensor) * torch.normal(size=size)  
   geo = torch.cumsum(drift_term + diffusion_term, dim=0)
   return np.exp(geo+poi_rv)
</code></pre>
<p>But not worked.<br>
any one can help please?</p>

ptrblck: <p>Recreating new tensors will detach them and the original parameters won’t be trained. Use the parameters directly to create the computation graph properly.</p>
When training 2 models simultaneously, how to stop training on one and continue on the other?

raulc: <p>i am training two models m1 and m2 by passing their parameters together to the optimizer:</p>
<blockquote>
<p>params = list(m1.parameters()) + list(m2.parameters())</p>
<p>torch.optim.SGD(params, lr=0.01)</p>
</blockquote>
<p>at one point i want to continue training only on m1.<br>
how to remove m2 from the training?<br>
should i set requires grad to false? or change the params property of the optimizer directly?<br>
or recreate the optimizer?</p>
<p>thanks.</p>

ptrblck: <p>Freezing the parameters should work, if the optimizer does not use any running stats (your simple <code>SGD</code> optimizer does not use running stats unlike e.g. <code>Adam</code>) and if the gradients are set to <code>None</code>, which is the default behavior in newer PyTorch releases (this would also allow you to use optimizers with running stats since updates for parameters with <code>None</code> gradients will be skipped).<br>
Recreating the stateless optimizer should also work, but you can already see that the best approach depends on your setup and if the optimizer uses running stats.</p>

raulc: <p>thanks will try them both.<br>
any way to check that it worked? like check somehow the numbers of parameters that have been updated on a step, to see the before freezing and after freezing count?</p>

ptrblck: <p>You could create copies of (some of) the parameters and compare them after calling the <code>step</code> operation.</p>
Using output layer vector as weights to the next layer

Roi_Mashiah: <p>Im looking to force some mathematical model on the neural network architecture. I want to train two neural networks such that one estimates a vector and the second estimates another vector, but the final output is the convolution between the outputs of the two networks where the target vector would be compared to the output of the convolution. shortly, im trying to estimate the FIR filter coefficients that change dynamically as a function of the input data. how can i achieve this in Pytorch? im having trouble understanding how and if the backprop will work in this scenario…<br>
for more clarification, one can say that the two networks operate in parallel and at the final layer they merge and the loss is calculated at the merged layer.</p>

ptrblck: <p>You can directly implement your idea in passing the outputs of both models to an additional conv layer or functional call since Autograd will track all differentiable operations.</p>

Roi_Mashiah: <p>so suppose signal_k, signal_m are the outputs from both networks, using for example np.convolve(signal_K,signal_M) in the final stage of the forward pass would be valid? (or an equivalent function in torch)</p>

ptrblck: <p>Yes, a convolution is valid but you should stick to PyTorch operations to avoid detaching the computation graph.<br>
Something like this should work:</p>
<pre data-code-wrap="python"><code class="lang-python">modelA = Model()
modelB = Model()

outA = modelA(input)
outB = modelB(input)

out = F.conv2d(outA, outB)
</code></pre>
Need help reducing the number of bounding boxes in Mask R-CNN inference

agn3ya: <p>I am currently working on an object detection project using Mask R-CNN in PyTorch. Also, I want to export the trained model to ONNX format for deployment in production. However, I am encountering challenges during the inference process as I am getting bounding boxes all over my video stream. I used CVAT for annotation of my videos and I am using only 1 class the other is the background.</p>
<p>Training Process:<br>
I am using a custom class to load video data and annotations. The dataset is then fed into a Mask R-CNN model with a ResNet-50 backbone and I am training it for 15 epochs.</p>
<p>Inference Process:<br>
During inference, I am generating a large number of bounding boxes for each frame of the video input leading to cluttered visualisation and inaccurate detections. I have attempted to filter out redundant detections using IoU thresholding and NMS but the result were not satisfactory.</p>
<p>Additional Information:<br>
PyTorch Version: 1.10.1+cu111<br>
CUDA Version: 11.2<br>
OS: Ubuntu 20.04</p>
Generating Merton jump using torch

Mohammed_Alruqimi: <p>I have a numpy code to generate Merton jump.</p>
<pre><code>poi_rv = np.multiply(np.random.poisson( lam*dt, size=size),
                     np.random.normal(m,v, size=size)).cumsum(axis=0)

geo = np.cumsum(((r -  sigma**2/2 -lam*(m  + v**2*0.5))*dt +\
                          sigma*np.sqrt(dt) * \
                          np.random.normal(size=size)), axis=0)
numpy_out = np.exp(geo+poi_rv)
</code></pre>
<p>Now, I want to do the smae function but with torch.tensor instead of numpy.<br>
I wrote this code:</p>
<pre><code>rates = torch.rand(steps,Npaths)  
poisson = torch.poisson(rates)


poi_rv_torch = torch.mul(poisson, (torch.randn(size)*v).cumsum(dim=0))
geo_torch = torch.cumsum(((r - sigma_t ** 2 / 2 - lam_t * (m_t + v_t ** 2 * 0.5)) * \
            dt_t +sigma_t * torch.sqrt(dt_t) * torch.randn(size)), dim=0)
torch_out = torch.exp(geo_torch + poi_rv_torch)
</code></pre>
<p>where:</p>
<pre><code>S = 100 # current stock price
T = 1 # time to maturity
r = 0.02 # risk free rate
m = 0 # meean of jump size
v = 0.3 # standard deviation of jump
lam =1.0 # intensity of jump i.e. number of jumps per annum
steps =10000 # time steps
Npaths = 1 # number of paths to simulate
sigma = 0.2 # annaul standard deviation , for weiner process
r_t,m_t,v_t,lam_t,sigma_t = torch.tensor(r),torch.tensor(m),torch.tensor(v),torch.tensor(lam),torch.tensor(sigma)
</code></pre>
<p>But when I plot the outputs, it show big differences between both numpy_out and torch_out code as shown in the figures.</p>
<p>I use torch.manual_seed(4) and np.random.seed(4).</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aadcf70e4f86fd98b280c6e49f76bde7af6c96fd.png" alt="stck" data-base62-sha1="onwAiqoGSpUBOL73t1dV3w09wiF" width="501" height="187"></p>
Compute gradients around variable assegnation

LDN: <p>I know that p.copy(p - H) breaks the computational graph, so I need a workaround to retrieve the gradient of the loss wrt to the parameters of the optimizer</p>
<pre><code class="lang-auto">class SCALTRA(nn.Module):
    def __init__(self, model, epochs, 
                 loss_criterion,
                 beta = 0.9, tau = 1, gamma = 0.9):
        super().__init__()

        # Initialization of the learnable parameters
        #self.lambda_ = nn.Parameter(torch.ones(1)*lambda_, requires_grad = True)

        self.tau = nn.Parameter(torch.ones(1)*tau, requires_grad = True)
        self.gamma = gamma
        self.beta = beta

        self.B = []
        self.momentum = []

        for _, p in model.named_parameters():
            self.B.append(nn.Parameter(torch.randn(p.shape[0]), requires_grad = True))
            self.momentum.append(torch.zeros_like(p))

        self.params = nn.ParameterList(self.B + [self.tau])

        # Some more initialization

        self.loss_criterion = loss_criterion
        self.meta_optimizer = torch.optim.Adam(params=self.parameters(), lr=1e-1, weight_decay = 5e-4)
        self.epochs = epochs


    def forward(self, model, X, y):
        train_losses = []

        for _ in tqdm(range(self.epochs)):
            for i, (_, p) in enumerate(model.named_parameters()):

                # Local optimization step
                if len(p.shape) == 1:
                    H = 1/(self.B[i]**2 + self.tau*torch.ones(self.B[i].shape[0])) * self.momentum[i]
                    
                else:
                    H = torch.linalg.inv(torch.outer(self.B[i],self.B[i]) + self.tau*torch.eye(self.B[i].shape[0])) @ self.momentum[i]

            with torch.no_grad():

                # Convex smoothing
                p.copy_(p - H)

            self.gamma *= 0.99

            loss = self.loss_criterion(model(X), y)

            # Zero gradients for optimizer parameters
            self.zero_grad()

            # Compute gradients of loss with respect to optimizer parameters
            loss.backward(retain_graph=True)

            # Update optimizer parameters
            self.meta_optimizer.step()  

            # Update momentum
            for i, (_, p) in enumerate(model.named_parameters()):

                self.momentum[i] = self.beta * self.momentum[i] + (1 - self.beta) * p.grad

            train_losses.append(loss.detach().numpy())
            
        return train_losses
</code></pre>
<p>How can I do so?</p>
Model loading causes access violation (-1073741819) on Windows

masterLazy: <p>Hi,<br>
I’m new to posting here and also to LibTorch. I’m facing an issue when trying to load a PyTorch model in Debug mode, and I need your help to understand what might be going wrong.<br>
Here’s my code snippet:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;
int main()
{
	try
	{
		model = torch::jit::load("model.pt");
	}
	catch (const c10::Error&amp; e)
	{
		std::cerr &lt;&lt; "Error loading model: " &lt;&lt; e.what() &lt;&lt; std::endl;
		exit(-1);
	}
	return 0;
}
</code></pre>
<p>When I run this code in <strong>Debug</strong> mode, the process exits (without printing any errors) with the code <code>-1073741819</code>, which indicates an access violation. However, in <strong>Release</strong> mode, the model loads successfully, and I’m able to perform predictions without any issues.</p>
<p>I’m using LibTorch on Windows and Visual Studio 2022, and I’m not sure why this behavior is occurring. Can someone tell me what happened? Thanks.</p>

QFuser: <p>Just randomly browsing . Had that problem too . You need to use the debug version for debugging and the release version for the release . Which completely blows you project setup up in size</p>
Libtorch 2.3.0 binaries for X86 Darwin is missing

npatsakula: <p>Hello everyone!</p>
<p>C++ libtorch X86 Darwin binary available for 2.2.3 but not for 2.3.0. Should I create an issue for this?</p>
<p><a href="https://download.pytorch.org/libtorch/cpu/libtorch-macos-x86_64-2.3.0.zip" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cpu/libtorch-macos-x86_64-2.3.0.zip</a></p>

ptrblck: <p>I believe Mac + x86 is deprecated but <a class="mention" href="/u/malfet">@malfet</a> can confirm it.</p>

npatsakula: <p>Looks like you’re right (pypi torch binaries missing as well), thank you!</p>
How does allgather handle memory allocation?

f_g: <p>Hi,</p>
<p>I am noticing that doing <code>torch.distributed.all_gather_object</code> on cpu tensors does not fully release the RAM after the call. As a result my training job’s RAM would keep going higher as larger tensors are gathered, since some frames could have bigger data than the other frames.</p>
<p>I used this snippet to test out just this function itself:</p>
<pre><code class="lang-auto">    def f(i):
        scores = torch.ones(i * 10**7, dtype=torch.float32)
        fn = torch.zeros(i * 10**7, dtype=torch.bool)
        tp = torch.zeros(i * 10**7, dtype=torch.bool)
        fp = torch.zeros(i * 10**7, dtype=torch.bool)
        gathered_scores = [None] * torch.distributed.get_world_size()()
        torch.distributed.all_gather_object(gathered_scores, scores.to("cpu"))
        gathered_fn = [None] * torch.distributed.get_world_size()()
        torch.distributed.all_gather_object(gathered_fn, fn.to("cpu"))
        gathered_tp = [None] * torch.distributed.get_world_size()()
        torch.distributed.all_gather_object(gathered_tp, tp.to("cpu"))
        gathered_fp = [None] * torch.distributed.get_world_size()()
        torch.distributed.all_gather_object(gathered_fp, fp.to("cpu"))
f(2)
time.sleep(60)
f(3)
time.sleep(60)
</code></pre>
<p>and the RAM graph looks like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f19ebc781d834e3e028743e592e4adfc491610e8.png" data-download-href="https://discuss.pytorch.org/uploads/default/f19ebc781d834e3e028743e592e4adfc491610e8" title="Screen Shot 2024-05-03 at 12.40.11 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f19ebc781d834e3e028743e592e4adfc491610e8_2_690x152.png" alt="Screen Shot 2024-05-03 at 12.40.11 PM" data-base62-sha1="ytt9ZoYCwMQC510PPGK9zV9BLDO" width="690" height="152" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f19ebc781d834e3e028743e592e4adfc491610e8_2_690x152.png, https://discuss.pytorch.org/uploads/default/original/3X/f/1/f19ebc781d834e3e028743e592e4adfc491610e8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/1/f19ebc781d834e3e028743e592e4adfc491610e8.png 2x" data-dominant-color="FBFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2024-05-03 at 12.40.11 PM</span><span class="informations">814×180 10.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m not a memory expert, I wonder if it’s because, all gather uses pin_memory or something under the hood, that is holding reference to the Python objects? Or is it because I’m doing all gather wrong?</p>
<p>Thanks!</p>
Should we consider the preprocessing of Faster-RCNN while selecting anchor sizes

Manu_Johny: <p>Hello guys!</p>
<p>I have a small question. My images have a size of 224x224 pixels. But when I looked into the code of pytorch’s Faster R-CNN, it was mentioned that all images will be rescaled in the code. I think, they used GeneralizedRCNNTransform for this purpose. So when I select the anchor sizes, should I only consider the sizes of my annotations on the 224x224 original image? Or should I adapt the anchor sizes to the sizes of my annotation in the transformed image?</p>
<p>Would be happy to get an answer.</p>
Predicted values come out biased and clipped

Anton_Neznamov: <p>Hi! i’ve been training a model to predict a continuous value based on input videos.<br>
My train and eval losses seem to converge at a good value of 0.02, which is ~98% accurate for my dataset and value range.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/418d79a0fac41b81cd7d2b773c8fb1e0275589a3.png" data-download-href="https://discuss.pytorch.org/uploads/default/418d79a0fac41b81cd7d2b773c8fb1e0275589a3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418d79a0fac41b81cd7d2b773c8fb1e0275589a3_2_410x500.png" alt="image" data-base62-sha1="9lU8XSCc6wxO7Cpg5thQVTwiYAH" width="410" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418d79a0fac41b81cd7d2b773c8fb1e0275589a3_2_410x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418d79a0fac41b81cd7d2b773c8fb1e0275589a3_2_615x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/1/418d79a0fac41b81cd7d2b773c8fb1e0275589a3.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">783×954 59.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But when I predict the values and plot it against ground truth, I get that predictions are heavily biased and clipped.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9d9e589a6ac157c12025840970f1b20113d42b8.png" data-download-href="https://discuss.pytorch.org/uploads/default/f9d9e589a6ac157c12025840970f1b20113d42b8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9d9e589a6ac157c12025840970f1b20113d42b8_2_690x330.png" alt="image" data-base62-sha1="zEhJlYbBHaZydXeDueUPVu3Sgkw" width="690" height="330" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9d9e589a6ac157c12025840970f1b20113d42b8_2_690x330.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9d9e589a6ac157c12025840970f1b20113d42b8_2_1035x495.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9d9e589a6ac157c12025840970f1b20113d42b8_2_1380x660.png 2x" data-dominant-color="FCFAF8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2432×1166 163 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m literally stuck at this point. I’ve ensured my eval dataset is properly stratified and there are no outliers. I would appreciate any advice or point of view on my case.</p>
Import pytorch failed

Elisaling: <blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “D:\Anconda_pytorch\envs\pytorch\Lib\site-packages\torch_<em>init</em>_.py”, line 1593, in <br>
from torch import hub as hub<br>
File “D:\Anconda_pytorch\envs\pytorch\Lib\site-packages\torch\hub.py”, line 13, in <br>
import zipfile<br>
ImportError: bad magic number in ‘zipfile’: b’a\r\r\n’</p>
</blockquote>
</blockquote>
</blockquote>

ptrblck: <p><a href="https://stackoverflow.com/a/514395">This post</a> might be helpful explaining what the magic number is.</p>
Extreme GPU memory usage

luuklelifeld: <p>I am getting some wild memory usage using the code below.</p>
<pre data-code-wrap="python"><code class="lang-python">class MyDataSet(Dataset):
    def __init__(self, root_dir, dataframe):
        self.root_dir = root_dir
        self.dataframe = dataframe
        self.images = [img[:-4] for img in os.listdir(root_dir) if img[-4:] == ".jpg"]

    def __len__(self):
        return len(self.images)

    def __getitem__(self, index):
        image_path = os.path.join(self.root_dir, f"{self.images[index]}.jpg")
        image = v2.functional.convert_image_dtype(read_image(image_path)).to(device)
        boxes = self.dataframe.loc[self.dataframe.page_id == self.images[index]]
        
        target = {}
        target["boxes"] = tv_tensors.BoundingBoxes(torch.tensor(boxes[["x", "y", "x1", "y1"]].values), format="XYXY", canvas_size=v2.functional.get_size(image), device=device)
        target["labels"] = torch.ones(len(boxes), dtype=torch.int64, device=device)
        
        return image, target

model = fasterrcnn_resnet50_fpn_v2(num_classes=classes)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, classes)
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.0001)

def collate_fn(batch):
    return list(zip(*batch))

training_data, test_data = random_split(dataset, [0.75, 0.25])
train_loader = DataLoader(training_data, batch_size=2, collate_fn=collate_fn, shuffle=True)
test_loader = DataLoader(test_data, batch_size=2, collate_fn=collate_fn, shuffle=True)

def train(num_epochs):
    best_accuracy = 0.0

    for epoch in range(num_epochs):  # one epoch = the entire dataset one time
        running_loss = 0.0
        running_acc = 0.0

        for i, (images, targets) in enumerate(train_loader, 0):  
            optimizer.zero_grad()
            outputs = model(images, targets)
            loss = sum(loss for loss in outputs.values())
            #loss = criterion(outputs["loss_classifier"], targets)

            loss.backward()
            optimizer.step()
            running_loss += loss.item()
            if i % 1000 == 999:    
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 1000))
                running_loss = 0.0

train(1)
</code></pre>
<p>The root folder directory contains 90MB worth of images. The instant <code>outputs = model(images, targets)</code> is called, gpu memory usage spikes to almost 100% (16gb available). What am I doing wrong here?</p>

ptrblck: <p>Besides the inputs and model parameters the intermediate activations will also be saved during the training so you might want to check <a href="https://discuss.pytorch.org/t/resnet-50-takes-10-13gb-to-run-with-batch-size-of-96/117402/2">this post</a>.</p>
Missing torch/csrc

hshreeshail: <p>I wanted to look at the source code of <code>torch.cuda.caching_allocator_alloc()</code>. I followed the link which led me to the function declaration<br>
<code>def _cuda_cudaCachingAllocator_raw_alloc(size: _int, cuda_stream: _int)</code><br>
in the file:<br>
<code>python3.9/site-packages/torch/_C/__init__.pyi</code></p>
<p>This file consists has a comment above the function declaration<br>
<code># Defined in torch/csrc/cuda/Module.cpp</code></p>
<p>However, I don’t see a folder named <code>csrc</code> in <code>torch</code>. How can I view the source code?</p>

ptrblck: <p>You can check the code <a href="https://github.com/pytorch/pytorch">on GitHub</a> as not the entire (compiled) code is shipped in the binaries.</p>
Executorch editable install

Fredrik_Knutsson: <p>When doing <code>pip install -e .</code> as opposed to <code>pip install .</code>, submodules don’t seem to exist. That is:</p>
<p><code>pip install -e .</code><br>
<code> python -c "import executorch"</code> ← works<br>
<code> python -c "import executorch.exir"</code> ← don’t work</p>
<p>on the other hand</p>
<p><code>pip install .</code><br>
<code> python -c "import executorch"</code> ← works<br>
<code> python -c "import executorch.exir"</code> ← works</p>
<p>Anyone see the same issue? I’m on Ubuntu 20.04 and Python 3.10, but I heard from colleagues I doesn’t work on other OS’s either…</p>

dbort: <p>We’ve never tested with <code>pip install -e</code>, so I’m not too surprised that it doesn’t work. If you would find this useful, please create an issue in the <a href="https://github.com/pytorch/executorch/issues" rel="noopener nofollow ugc">github issue tracker</a> describing what you’re looking for, and what the benefits would be. Or, if you’re able to figure out why <code>-e</code> doesn’t work, you could create a PR and we could take a look.</p>

oscarandersson8218: <p>It works with:</p>
<pre><code class="lang-auto">pip install -e . --config-settings editable_mode=strict
</code></pre>
<p>Is this something that we want to add as part of install_requirements.sh or as part of documentation?</p>

Fredrik_Knutsson: <p>Nice! That works for me. It would be nice to have this documented in a README…</p>

Fredrik_Knutsson: <p>Editable installation support is not working again since <a href="https://github.com/pytorch/executorch/pull/2499/files" class="inline-onebox" rel="noopener nofollow ugc">New pip build system by dbort · Pull Request #2499 · pytorch/executorch · GitHub</a>.</p>

Fredrik_Knutsson: <p>Btw, there’s a ticket for this: <a href="https://github.com/pytorch/executorch/issues/2871" class="inline-onebox" rel="noopener nofollow ugc">Error in Pip Install Editable Mode · Issue #2871 · pytorch/executorch · GitHub</a></p>
Is there a memory leak happening?

songsong0425: <p>Greetings. I have a question about the GPU memory usage in the PyTorch.<br>
When I ran the <code>GAT</code> model using pytorch and torch_geometric, I countered the cuda out-of-memory problem every time.<br>
Although I use the large knowledge graph (105.43MB) and the 9 types of datasets for the training, validation, and test, I think it is weird.<br>
Is it normal for memory usage to increase with each epoch?<br>
Please check the memory usage in the screenshot.</p>
<p><strong>After the first epoch</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f23925ce8b4ced1ab264cf582247b52a40c70e9.png" alt="image" data-base62-sha1="kqguoVeRXPBrMiruvH6QqTSvnHr" width="635" height="273"></p>
<p><strong>After the second epoch</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/592c6a4b24fc962959fdcdcdd8aad3c8066d1aef.png" alt="image" data-base62-sha1="cIREy2tmGAW9vUomMtBH73vPG0f" width="632" height="169"></p>
<p><strong>After the third epoch</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/8/983a431ecb100867e28466c33b4f70921fe1bd87.png" alt="image" data-base62-sha1="lIFoU1BEeNj2IQARhMEqtgrZOfl" width="632" height="271"></p>
<p><strong>After the fifth epoch</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0ec60e609636c8aa200da9c0eb8c21a5c688748.png" alt="image" data-base62-sha1="pf8tlPwTOqo3Qd3XP9zMhbj3Lvi" width="630" height="168"></p>
<p>If it is a memory leak, what part of my code should I look at?<br>
Thank you for reading the question.</p>

ptrblck: <p>The memory usage goes down in the third epoch which does not point towards a memory leak.</p>

songsong0425: <p>Thank you for your reply! I’ll try other approaches to reduce the memory usage.</p>
Transform graph module of the exported program

mhs4670go: <p>Hello.</p>
<p>I’m trying to transform the graph module in the exported program after importing a pt2 file.</p>
<p>So, I’ve written the transformer class according to <a href="https://pytorch.org/docs/stable/torch.compiler_transformations.html#one-to-x-pass" rel="noopener nofollow ugc">this</a>. But, after the transformation, there is no way to update the exported program. I tried to create a new exported program with the transformed graph module. But, I couldn’t call the <code>export</code> method because it seems that imported exported program doesn’t have example inputs.</p>
<pre data-code-wrap="python"><code class="lang-python">class DecomposeFakeQuant(torch.fx.Transformer):
    """
    Original:
        def f(x):
            return torch.fake_quantize_per_tensor_affine(x, s, zp, q_min, q_max)

    After pass:
        def f(x):
            x = qd.quantize_per_tensor(x, s, zp, q_min, q_max, dtype)
            x = qd.dequantize_per_tensor(x, s, zp, q_min, q_max, dtype)
            return x
    """
    def call_function(self, target, args, kwargs):
        if target != torch.ops.aten.fake_quantize_per_tensor_affine_cachemask.default:
            return super().call_function(target, args, kwargs)

        q_x = super().call_function(torch.ops.quantized_decomposed.quantize_per_tensor.default, args, kwargs)
        args = (q_x, *args[1:])
        return super().call_function(torch.ops.quantized_decomposed.dequantize_per_tensor.default, args, kwargs)

exported_program = torch.export.load(input_pt2)
transformed_gm = DecomposeFakeQuant(exported_program.graph_module).transform()
# exported_program._graph_module = transformed_gm -&gt; ERROR because graph_module is immutable
# transformed_ep = export(transformed_gm, exported_program.example_inputs) -&gt; ERROR because exported_program doesn't have example_inputs
</code></pre>
<p>Is there any proper method for this case?</p>
How to compute batch divergence?

iamctR: <p>Hello all!</p>
<p>I just want to ask a silly question…</p>
<p>how can we compute the divergence of a vector field f(x,v,t) only with respect to v by using the jacrev, trace and vmap?</p>
<p>My current implementation is</p>
<pre><code class="lang-auto">vecfield= lambda v: f(x,v,t)
div = vmap(torch.trace(jacrev(vecfield)))(x)
</code></pre>
<p>and it reports the error XD</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/iamctr">@iamctR</a>,</p>
<p>Do you have a minimal reproducible example?</p>

iamctR: <p>Hello, here is the example code:</p>
<pre><code class="lang-auto">from torch.func import vjp, jvp, vmap, jacrev
import torch

def div_fn(u):
    J = jacrev(u)
    return lambda x: torch.trace(J(x))

def fnc(x,y,z):
    """
    x: [bs, 4]
    y: [b,1]
    z: [bs,1]

    output: [bs,1]
    """
    x= x.reshape(bs,-1)
    y= y.reshape(bs,-1)
    z= z.reshape(bs,-1)
    return z+y+x.sum(-1)[...,None]


bs=512
x=torch.randn(bs,4)
y=torch.randn(bs,1)
z=torch.randn(bs,1)


vecfield=fnc(x,y,z)
vecfield = lambda yy: fnc(x,yy,z)
div = vmap(div_fn(vecfield))(y)
print(div)
</code></pre>
<p>Thank you!</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/iamctr">@iamctR</a>,</p>
<p>Here’s a working example, but some parts of your example weren’t 100% clear to me. So this might not be what you want, but it should help you understand how <code>torch.func</code> works.</p>
<p>When working with <code>torch.func</code>, you need to create a function which is independent of your batch-dim, i.e. the function should work fine with 1 sample, so you can vectorize over all samples via <code>torch.func.vmap</code>.</p>
<p>So, remove <code>.reshape(bs,-1)</code> for example, as within the <code>vmap</code> call the batch size is technically 1.</p>
<pre><code class="lang-auto">from torch.func import vjp, jvp, vmap, jacrev
import torch

def fnc(x,y,z):
	inp = torch.cat([x,y,z],dim=-1)
	return torch.sum(inp,dim=-1)
	
def div_fn(x,y,z):
	J = jacrev(fnc,argnums=1)(x,y,z) #grad w.r.t y
	return J #torch.trace(J)

bs=512
x=torch.randn(bs,4)
y=torch.randn(bs,1)
z=torch.randn(bs,1)

div=vmap(div_fn, in_dims=(0,0,0))(x,y,z)
print(div.shape) #returns shape [512,1]
</code></pre>

iamctR: <p>Thank you sooo much!</p>
GPU vs. CPU Performance: Unusual Attention Head Effects

Harsh_Mittal: <p>Hi there!</p>
<p>I tried out a little experiment to see how does a GPU and CPU actually impacts the training time of a transformer based language model by changing the number of attention heads to see how it affects the performance.</p>
<p>Here’s what I found:<br>
When attention head count increases from 1 to 128 (n_embed: 128), I observed that GPU performance decreases from 12x to 1.23x relative to the CPU.</p>
<p>I mean in the ideal situation my brain is expecting a linear trend which is not the case because there are underlying stuff that I don’t understand, but when GPU gain went from 12x to 1.23x - I’m CONFUSED!</p>
<p>I’m curious to find a model configuration where the GPU gain is close to 1x w.r.t CPU - case where the compute performances are equal (which doesn’t make any sense to me).</p>
<p>Here’s a google collaboration notebook link: <a href="https://colab.research.google.com/drive/14oIN1bwSS95VckiUItBtvp5cYXIFCW15?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>If anyone would like to share their thought process behind this trend, I would appreciate it.<br>
Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a049245389f73995b2b7962a9b687ed5497d2286.png" data-download-href="https://discuss.pytorch.org/uploads/default/a049245389f73995b2b7962a9b687ed5497d2286" title="Screenshot 2024-05-02 at 6.12.48 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a049245389f73995b2b7962a9b687ed5497d2286_2_690x408.png" alt="Screenshot 2024-05-02 at 6.12.48 PM" data-base62-sha1="mRX6gvD5njXbYaVyt2cuLHHaC0K" width="690" height="408" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a049245389f73995b2b7962a9b687ed5497d2286_2_690x408.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a049245389f73995b2b7962a9b687ed5497d2286_2_1035x612.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a049245389f73995b2b7962a9b687ed5497d2286_2_1380x816.png 2x" data-dominant-color="F3F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-05-02 at 6.12.48 PM</span><span class="informations">2498×1480 373 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Implementing a custom unsupervised loss function for Graph Convolution

Arun_Kumar: <p>I have a graph convolution model built and wanted to implement a custom unsupervised loss function like the one shown in below picture:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/59af748f961995153fbafd2acca4adf6815378be.png" alt="Screenshot 2024-05-02 at 2.01.52 PM" data-base62-sha1="cNopbf5nQKtSz7L4rmARwydou3Y" width="630" height="156"></p>
<p>Where <code>yv</code> is the learned embedding of a node v and rand denotes random sampling operation over all the nodes</p>
<p>I am new to Pytorch so not sure if this is already implemented. Any help will be sincerely appreciated.</p>
nn.Parameter is set to zero

girlsending0: <p>Currently, the weight value is set with nn.parameter(randn). However, during the actual learning stage, the self.weight value continues to be 0.<br>
I set param.requires_grad=True in train.py, but I don’t know why this error keeps happening.</p>
<p>below is model code.</p>
<p>‘’’</p>
<p>class SubjectLayers(nn.Module):</p>
<pre><code> def __init__(self, in_channels: int, out_channels: int, n_subjects: int, init_id: bool = False):
     super().__init__()
     self.weights = nn.Parameter(torch.FloatTensor(n_subjects, in_channels, out_channels), requires_grad=True)
     if init_id:
         assert in_channels == out_channels
         self.weights.data[:] = torch.eye(in_channels)[None]
 def forward(self, x, subjects):
     _, C, D = self.weights.shape
     #print("before", self.weights.data)
     weights = self.weights.gather(0, subjects.long().view(-1, 1, 1).expand(-1, C, D))
     #print("after", self.weights.data)
     return torch.einsum("bct,bcd-&gt;bdt", x, weights)
 def __repr__(self):
     S, C, D = self.weights.shape
     return f"SubjectLayers({C}, {D}, {S})"
</code></pre>
<p>‘’’</p>
<p>It is declared in the parent module as follows:</p>
<p>meg_dim = in_channels[“meg”]<br>
dim = {“hidden”: hidden[“meg”], “input”: meg_dim}[subject_layers_dim]<br>
self.subject_layers = SubjectLayers(meg_dim, dim, n_subjects, subject_layers_id)<br>
in_channels[“meg”] = dim</p>

ptrblck: <aside class="quote no-group" data-username="girlsending0" data-post="1" data-topic="201961">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/girlsending0/48/65447_2.png" class="avatar"> girlsending0:</div>
<blockquote>
<p>However, during the actual learning stage, the self.weight value continues to be 0.</p>
</blockquote>
</aside>
<p>I don’t understand this claim as you describe it as “continues to be 0” which implies it was already zero before or is the parameter update moving <code>self.weight</code> towards zero?</p>

girlsending0: <p>It is already zero before. When I print it in the forward statement, it is 0…</p>
<p>I set requires_grad to True in the code, but I don’t know why only nn.parameter becomes 0. When I declared nn.Embedding and output the weight, the weight value other than 0 was output correctly.</p>
<p>Is it because the subjectlayer class is a submodule? I don’t understand.</p>
Access denied downloading static-linked libtorch

hsnyder: <p>Though the website doesn’t directly link it, I’ve seen mention of the fact that static libraries are in fact generated during official libtorch builds, and that they’re supposed to be accessible by replacing ‘-shared-’ with ‘-static-’ in the URL. I tried that and got the following message:</p>
<pre><code class="lang-auto">&lt;Error&gt;
&lt;Code&gt;AccessDenied&lt;/Code&gt;
&lt;Message&gt;Access Denied&lt;/Message&gt;
&lt;RequestId&gt;FWDR4Q179HMAPQB6&lt;/RequestId&gt;
&lt;HostId&gt;
dDXioT1Od1uTkliU5J+WS8SWNofWzJlxBvOI0tb8wvS0kTNn9QujGbqFHfPvClGGbMP4raSWFQE=
&lt;/HostId&gt;
&lt;/Error&gt;
</code></pre>
<p>I’d consider this a bug, but I’m curious if it’s user error on my part?</p>
<p>Here’s the URL:</p>
<p><a href="https://download.pytorch.org/libtorch/cu121/libtorch-cxx11-abi-static-with-deps-2.3.0%2Bcu121.zip" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu121/libtorch-cxx11-abi-static-with-deps-2.3.0%2Bcu121.zip</a></p>

ptrblck: <p>Where did you see this claim as it seems to be wrong?</p>

hsnyder: <p>Here: <a href="https://github.com/pytorch/pytorch/issues/25699#issuecomment-528389300" class="inline-onebox" rel="noopener nofollow ugc">PyTorch C++ API as a static lib: how to compile ? · Issue #25699 · pytorch/pytorch · GitHub</a></p>

ptrblck: <p>This post is ~4 years old so I’m not sure if the response is still valid, but <a class="mention" href="/u/malfet">@malfet</a> would know.</p>
How to freeze alternate filter/channels within a model?

pytorchuser101: <p>I have a layer named “conv2” and want to freeze all filter’s weights which are at even index positions.</p>
<p>I am trying this:</p>
<pre><code class="lang-auto">for i, filter_val in enumerate(model_eval.conv2.weight):
    if i % 2 == 0:
        filter_val.requires_grad = False
</code></pre>
<p>I get this error:</p>
<pre><code class="lang-auto">RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().
</code></pre>
<p>Since each filter is created from the layer, which requires grad to be set to True of False, I cannot set individual filter’s which are a subset of the layer to True of False.</p>
<p>Is there any work around for this ?</p>

soulitzer: <p>It is not possible to assign requires_grad at the subtensor level. A way around this is to take a view and .detach(), filter_val instead. But you need to make sure you don’t use conv2.weight directly later during forward.</p>
Custom gradients only for inputs that require grad

haj: <p>I have to implement a custom backpropagation bc. I need to do some calculations with numpy. The problem is I will be doing the gradient calculation in the <code>for</code> loop in the forward pass since I don’t want to go through another <code>for</code> loop in the backward pass. However, the gradient calculation involves some complex indexing that takes a lot of time, so I don’t want to calculate gradient for inputs that don’t require grad. Thus,  <code>func</code> returns the gradient in <code>grad</code> when <code>backprop</code> is <code>True</code> and returns <code>None</code> in <code>grad</code> when <code>backprop</code> is <code>False</code> by skipping gradient calculation. Likewise, I will save <code>grad</code> for backward using <code>ctx.save_for_backward</code> method only when <code>backprop</code> is <code>True</code>. My question is, is this the right method to implement custom gradients for only inputs that require gradient calculation?</p>
<pre><code class="lang-auto">class CustomGrad(Function):
    @staticmethod
    def forward(ctx, x, func):
        backprop = x.requires_grad
        result_collec = []
        grad_collec = []
        for b, data in enumerate(x):              # iterate over batch
            result, grad = func(data, backprop)
            result_collec.append(result)
            if grad is not None:
                grad_collec.append(grad)
        if backprop:
            ctx.save_for_backward(torch.from_numpy(np.stack(grad)).to(torch.float32))
        return torch.from_numpy(np.stack(result_collec)).to(torch.float32)
    
    @staticmethod
    @once_differentiable
    def backward(ctx, grad_output):
        local_grad = ctx.saved_tensors
        grad_input = # calculate downstream gradient using local grad and grad_output
        return grad_input, None
</code></pre>

soulitzer: <blockquote>
<p>ctx has an attribute :attr:<code>ctx.needs_input_grad</code> as a tuple<br>
of booleans representing whether each input needs gradient. E.g.,<br>
:func:<code>backward</code> will have <code>ctx.needs_input_grad[0] = True</code> if the<br>
first input to :func:<code>forward</code> needs gradient computed w.r.t. the<br>
output.</p>
</blockquote>
<p>If you need to know during backward whether particular inputs requires_grad or not during forward,  you could use <code>ctx.need_input_grad</code>.</p>

haj: <p>Thanks for the reply! Since I only call <code>ctx.save_for_backward</code> in the forward pass only when <code>backprop=True</code>, would a backward like this be fine?</p>
<pre><code class="lang-auto">    @staticmethod
    @once_differentiable
    def backward(ctx, grad_output):
        grad_input = None
        if ctx.needs_input_grad[0]:
            grad_local = ctx.saved_tensors
            grad_input = # calculate downstream gradient using local grad and grad_output
        return grad_input, None
</code></pre>

soulitzer: <p>Yes, that should be fine</p>
Pytorch Int2 training and export

khn4hi: <p>Hello, I am aware that during the training we can make model aware of the low precision kernels and activations. However, once training is over how to export the model in int2 or int1 using torch. Is there any  tool from torch itself available? if not, is there any equivalent of larq in tensorflow in torch?</p>

jerryzh168: <aside class="quote no-group" data-username="khn4hi" data-post="1" data-topic="201793">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/5e9695/48.png" class="avatar"> khn4hi:</div>
<blockquote>
<p>larq</p>
</blockquote>
</aside>
<p>these are not supported natively right now, but we’ll support them through tensor subclasses like: <a href="https://github.com/pytorch/ao/blob/main/torchao/dtypes/uint4.py" class="inline-onebox" rel="noopener nofollow ugc">ao/torchao/dtypes/uint4.py at main · pytorch/ao · GitHub</a>, for now I think you could simulate these with torch.uint8 + quant_min/quant_max restrictions to the range of the desired dtype during quantization and then lower to the correct quantized ops in the end.</p>
Skip gradient propagation if gradients less than a threshold

Aradhye_Agarwal: <p>Hi,</p>
<p>I was thinking of an optimization for gradient computation where we don’t propagate the gradients to the children, ie. detach the current node from the computation graph if the node’s gradient is less than a given threshold.</p>
<p>I tested this by having a tensor v and u defined as v**2 and calling u.sum().backward(). I registered a hook on u which made its gradients as None but I found that gradient computation was still being done on v.</p>
<p>Hence I was wondering how to do this. Of course the naive way would be to register a hook and zero out the gradients if it is less than a threshold, but while it would have the same effect, I wouldn’t get the desired speedup, defeating the purpose.</p>

GuWei007: <p>What is the purpose of doing this and what problem is solved?</p>

ptrblck: <p>I don’t know if your idea to “stop” the backward pass is currently possible, but <a class="mention" href="/u/alband">@albanD</a> might know if any utils could be helpful.</p>

albanD: <p>Returning “None” is not working as we consider that as “not changing the gradient”. This is because in python, any function that doesn’t return anything returns None.<br>
You can <code>return torch.zeros_like(grad)</code> to have gradients full of 0s.</p>

Aradhye_Agarwal: <p>But I’m guessing I won’t get the speedup right?</p>

albanD: <p>I’m afraid there is no way to stop the backprop in the middle of the execution no <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"><br>
You can open an issue on github if you want to discuss what we could be doing there!</p>
Saving tensor with torch.save uses too much memory

Ben_Marinberg: <p>Hi all!<br>
Im trying to better manage the training set for my CNN.<br>
The training data is a tensor with shape [54K,2,8,1320,14] (targets are with same shape),and i use batch of 50, mini-batch shape [50,2,8,1320,14], as it enters conv3D layer (2 in channels). since the whole set is about 250 GB (125 GB for each data and targets), which is too big for the RAM to hold, so currently its 5 data-target pair with 10800 samples each, which is 50GB per pair (25GB for data and 25 for inference).<br>
I want to save it in separated files each contains 25 samples each so i’ll be able to load 2 random files for constructing a mini-batch, without overloading my memory.<br>
When i try to save 25 samples with 'torch.save(training_set.data[:25],‘training_data_00.pt’) it gives me a file ‘training_data_00.pt’ of size 25GB (same as 10800 samples) but has the correct shape [25,8,2,1320].<br>
what am I doing wrong? do you have any solution to it?<br>
Thanks</p>

albanD: <p>Hi,</p>
<p>selecting a subset of a tensor does not actually create a new tensor in most cases but just looks at a subset of the original one.<br>
When saving, the original tensor is saved.<br>
You can save <code>training_set.data[:25].clone()</code> to save only the part you want as the clone operation will force the creation of a new smaller tensor containing your data.</p>

Ben_Marinberg: <p>Seems to work perfectly!<br>
It is still weird that i have loaded the data and still got only a subset of the dataset even though it stored the whole set .<br>
Thanks!!</p>

Crazyai: <p>Thanks for the answer!</p>
<p>I have known that the slice op does not create new tensor, but It is still a weird behavior for me. Could you explain this in more details why PyTorch has the design that saving the original whole tensor? For memory efficiency?</p>

albanD: <p>Hey!</p>
<p>This is a side effect of the fact that we support serializing Tensor that share memory. And that memory sharing will be preserved by serialization.<br>
That way, if your model has tied weights or weights that look at the same data (even if partially), then serialization will preserve that property of your model. This is one of the major different between pt serialization and safetensors for example.</p>