akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to increase batch size with limited GPU memory

username: Sravan_nani
cooked: <p>I’m facing challenge working on NLP application, where I can provide batch size at max 2 due to memory issue (I’m using 8 gb GPU).</p>
<p>I want to increase my batch size because model is not converging well with small batch size.<br>
My question is instead of using the gradient accumulation, can i use the following procedure ?</p>
<p>“”"<br>
“batch_size” is the required batch size, say 16<br>
“max_batch_size” is the max batches can allocate on limited memory, say 2</p>
<pre><code>1)calculate the loss for batch size 16 in forward pass without grads
     2) the pass the batch size of 2 and calculate the gradients
        3) update the gradients by averaging  on each iteration
</code></pre>
<p>“”"</p>
<p>for epoch in range(epochs):</p>
<pre><code>for idx,sample in enumerate(dataloader):
    # dataloader with batch_size =16
    # here calculating the loss for batch size of 16
    with torch.no_grads():
        outputs = model(inputs)
    loss = loss_fn(outputs, labels)    

    # Pass the batch size of 2 (becuase of memory constraint) and calculate teh gradients
    for idx in range(0,batch_size,max_batch_size): 
        inputs, labels = sample[idx:idx+max_batch_size]

        # Forward Pass
        outputs = model(inputs)
        
        loss.backward()
        
        model.grads  /= 2


    
    optimizer.step()        
    optimizer.zero_grad()
</code></pre>

username: ptrblck
cooked: <p>No, your approach won’t work since <code>loss</code> was created from an <code>output</code> tensor which is not attached to any computation graph. Calling <code>loss.backward()</code> will fail. Besides that your approach looks indeed like gradient accumulation since you are executing the forward pass multiple times with smaller batch sizes so you could directly call <code>backward</code> on these output tensors.</p>

username: Sravan_nani
cooked: <p>Thank you very much for your quick response.</p>
<p>One more doubt, is there any difference in output if i use gradient accumulation NUM_ACCUMULATION_STEPS = 4 with batch size of 4 instead of directly defining batch size of 16 ?</p>
<p>Because in gradient accumulation gradients are added and loss will be divided by NUM_ACCUMULATION_STEPS.</p>
<p>Where in defining batch size 16, calculated gradients on loss for sample size of 16 will be averaged. How these 2 things same ?</p>
<p>What I tried to do in the above code was to replicate the latter one.<br>
Is my thought process correct? Is it possible to do ?</p>

username: ptrblck
cooked: <p>Since you are scaling the losses during gradient accumulation the gradients should be the same at the end. However, some layers use the batch size for their computation, such as batchnorm layers which calculate the stats from the incoming activation tensor, which would show different behavior in the gradient accumulation case vs. the large batch size case.</p>
How to install pytorch v2.0.0 cuda 17 on my pc?

username: Erfan2001
cooked: <p>Hi there,<br>
I have wanted to install Pytorch on my pc, but I couldn’t.<br>
First, by this command, I have got lots of errors related to my connection.</p>
<p><code>pip3 install torch --index-url https://download.pytorch.org/whl/cu117</code></p>
<p>So, I decided to install it directly to download. Now, I download a file which is named “torch-2.0.0+cu117-cp310-cp310-win_amd64.whl” and I have a venv (virtual environment) for my project, but I don’t know how to connect this file to the project to import and use it.</p>
<p>Before that, I handled all the configs related to Cuda of my system, so I don’t have any problems with them.<br>
<strong>Cuda:</strong><br>
This is some information about my PC:<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2023 NVIDIA Corporation<br>
Built on Wed_Feb__8_05:53:42_Coordinated_Universal_Time_2023<br>
Cuda compilation tools, release 12.1, V12.1.66<br>
Build cuda_12.1.r12.1/compiler.32415258_0</p>
<p>Pip version: 21.2.3<br>
Python version: Python 3.10.0 &amp; 3.11.2<br>
System type: 64-bit operating system<br>
GPU: NVIDIA GeForce GTX 1650 Ti</p>
<p>Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Your locally installed CUDA toolkit won’t be used by the PyTorch binaries as they ship with their own CUDA library dependencies. You would only need to install a proper NVIDIA driver.<br>
Could you post the error message you were getting from the <code>pip install command</code>?</p>

username: Erfan2001
cooked: <p>Hi,<br>
Did I install a proper version of NVIDIA?<br>
I put some information related to my laptop and this version of the toolkit was recommended to me.</p>
<p>I got this error:</p>
<pre><code class="lang-auto">pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='download.pytorch.org', port=443): Read timed out. 
</code></pre>

username: ptrblck
cooked: <p>I don’t know what exactly you’ve installed but if applications can communicate with your GPU the driver installation might be alright.<br>
The posted error message indicates a timeout so I would recommend either rerunning the same command or checking if e.g. a firewall is preventing the download.</p>

username: Erfan2001
cooked: <p>How should I check that GPU can communicate with applications?<br>
I think that’s ok because I have a driver in the device manager of my PC and also I can check the version of my NVIDIA toolkit. Thus, I installed all of them right.</p>
<p>For that error, I’ve tried many times but when the downloaded file reaches 1.7GB, it suddenly stops and gives me that error. I asked about this in some groups in my country, they said that was because of our connection.</p>
<p>So, if I want to install that downloaded file I mentioned earlier, I mean “torch-2.0.0+cu117-cp310-cp310-win_amd64.whl”, how can I do that?</p>

username: ptrblck
cooked: <p>Try to download it again from a better connection and save it for future installations.</p>
100 Training Samples Per Class, 80/20 Train-Val, Rest for Evaluation

username: bongbonglemon
cooked: <p>My dataset is structured such that every sub-directory is a class (and all images in the subdirectory are images labeled that corresponding class). I want to make the split as described in the title. And I also want to make 10 random partitions using the same split.</p>
<p>To accomplish this, I shuffled all the filenames in every class sub-directory, took 80 and added to the training subset, took 20 and added to the validation subset, and added the rest to the evaluation subset. So I ended up with 10x3 sets of filenames. With that, I made a Custom Dataset that will use those filenames. And it looks as follows:</p>
<pre data-code-wrap="class"><code class="lang-plaintext">    def __init__(self, root_dir, filenames, label2id, transform=None):
        self.root_dir = root_dir
        self.filenames = filenames
        self.label2id = label2id
        self.transform = transform

    def __len__(self):
        return len(self.filenames)

    def __getitem__(self, index):
        filename = self.filenames[index]
        classname = os.path.dirname(filename)
        image_path = os.path.join(self.root_dir, filename)

        # Load the image in parallel
        with open(image_path, 'rb') as f:
            img = Image.open(f).convert('RGB')
            if self.transform is not None:
                img = self.transform(img)

        return img, self.label2id[classname], filename
</code></pre>
<p>Previously, when I was only doing a 60/20/20 split on the whole dataset, I just used ImageFolder. And that was about 4x faster per epoch than what I am having here.</p>
<p>So my question is: How should I optimize my Dataset? Is my idea of splitting up the filenames not a good idea to begin with?</p>
Torch Tensor to Plot to Torch Tensor again

username: bornexmachina
cooked: <p>I try to follow an approach of a paper, where guys</p>
<ol>
<li>use <code>librosa</code> to create a melspectrogram plot</li>
<li>use this picture as input for their pytorch pipeline.</li>
</ol>
<p>Now in the meantime torch audio can create a basis for the plot. And <code>matplotlib.imshow()</code> generates the picture I would like to use as the input (<a href="https://stackoverflow.com/questions/76128423/librosa-specshow-what-data-processing-is-done-under-the-hood-compared-to-plt-im/76128949#76128949" rel="noopener nofollow ugc">here</a> you can see my related question with a plot)</p>
<p>I can generate a grey scale image from the data using</p>
<pre><code class="lang-auto">def scale_minmax(X, XMIN, XMAX, min=0.0, max=1.0):
    X_std = (X - XMIN) / (XMAX - XMIN)
    X_scaled = X_std * (max - min) + min
    return X_scaled
</code></pre>
<p>But I have the impression, that I am losing some details here, as I cannot recreate the results of the paper.<br>
Thus I would follow the footsteps and process the colorful image and let <code>pytorch</code> do its magic.<br>
<strong>By this I mean, that I would like to get a 3D array representing RGB channels, which later will be used as input for pretrained ResNet model (which needs 3D arrays)</strong></p>
<p>One very tedious way is something like</p>
<pre><code class="lang-auto">import torch
from PIL import Image
import matplotlib.pyplot as plt

# Generate a sample plot
x = [1, 2, 3, 4]
y = [1, 4, 9, 16]
plt.plot(x, y)

# Save the plot as an image
plt.savefig('plot.png')

# Open the image using PIL
img = Image.open('plot.png')

# Convert the image to a PyTorch tensor
tensor_img = torch.from_numpy(numpy.array(img))
</code></pre>
<p>but this requires the intermediate save of a picture I don’t actually need.</p>
<p>Thus, my question: Is there any way to bypass this save?</p>
<p>Either some RGB extraction from my initial array (instead the grey scale I got) or at least passing the <code>matplotlib.imshow()</code> directly to <code>torch</code>.</p>

username: eqy
cooked: <p>You may want to take a look at Agg buffers: <a href="https://matplotlib.org/3.1.1/gallery/misc/agg_buffer.html" class="inline-onebox" rel="noopener nofollow ugc">Agg Buffer — Matplotlib 3.1.2 documentation</a></p>
Different Results using Transforms and Albumentations

username: Ali_Youssef
cooked: <p>Hello everyone!</p>
<p>I noticed I was getting different results during training when using transforms once and albumentations on the other.</p>
<p>So I decided to remove all kind of transformations I’m doing on the images, and just resize and normalize using Albumentations once and Transforms  as the other print the tensors.</p>
<p>Surpsingly they were different, however I do not understand why I’m using same interpolation and normalization values?</p>
<p>For example here is one row from the first channel,</p>
<ul>
<li>
<p>Albumentations= [-1.4843, -1.4843, -1.5357, -1.5528, -1.5357, -1.4672, -1.5357, -1.5014,<br>
-1.4843, -1.5185]</p>
</li>
<li>
<p>torchvision transforms: [-1.4843, -1.5014, -1.5185, -1.5528, -1.5357, -1.4843, -1.5185, -1.5014,<br>
-1.4843, -1.5185]</p>
</li>
</ul>
<p><strong>EDIT:</strong> So I tested with resizing only, with same interpolation for each, and it appears that image and mask tensors using Albumentations are not exactly the same as when using Transforms. Secondly, I removed resizing and kept normalization, since the masks are not normalized, they are the same using both methods, but the images are still slightly off using both methods. Could the difference be because transforms uses PIL and Albumentations uses cv2?</p>
<p>Here is the code I used to test.</p>
<pre><code class="lang-auto">import albumentations as A
import cv2
from PIL import Image
import numpy as np
from torchvision import transforms

img_path = "./data/oxford-iiit-pet/images/Abyssinian_2.jpg"
seg_path = "./data/oxford-iiit-pet/annotations/trimaps/Abyssinian_2.png"

tf = transforms.ToTensor()

O_img = Image.open(img_path).convert("RGB")

temp_img = np.array(O_img)
album_t = A.Compose([A.Resize(256, 256,interpolation=cv2.INTER_LINEAR,always_apply=True),
                     A.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225],always_apply=True)])
preprocess_A = album_t(image=temp_img)
Album_img = preprocess_A["image"]
A_img = tf(Album_img )


img_transform = transforms.Compose([ transforms.Resize((256, 256)),
                                    transforms.ToTensor(),
                                    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                                            std=[0.229, 0.224, 0.225])])


tf_img = img_transform(O_img)

print(A_img[0,1,0:10])
print(tf_img[0,1,0:10])
</code></pre>
Torch compile change the module name

username: malioboro
cooked: <p>When I tried <code>torch.compile()</code> for my project, it raised some errors. One of the error is that my model has a hook that checks the module name, and <code>torch.compile()</code> change the module name with an additional prefix <code>_orig_mod</code>.</p>
<p>The way I debug my model:</p>
<pre><code class="lang-auto">...
model = torch.compile(model)
for p in model.modules():
    if len(list(p.children()))==0:
        print(p.__name__)  # I store module name to `module.__name__`
...
</code></pre>
<p>How I store my module name, before registering the hook:</p>
<pre><code class="lang-auto">...
for name, module in model.named_modules():
    if len(list(module.children()))==0:
        module.__name__ = name
        module.register_forward_pre_hook(self.myhook())
...
</code></pre>
<p>In this situation, do I need to change the way my hook work, or is there any better workaround to avoid this problem?</p>
<p><strong>UPDATE</strong><br>
Somehow I missed that module and hooks are not fully supported yet: <a href="https://pytorch.org/get-started/pytorch-2.0/#hooks" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 2.0 | PyTorch</a></p>

username: marksaroufim
cooked: <p>Hook support is a bit better in nightlies, are you using the official 2.0 release?</p>

username: malioboro
cooked: <p>Yes, currently I am using 2.0, I will check the nightly build. Thanks</p>

username: marksaroufim
cooked: <p>So actually just reread your issue, this is likely not to get fixed since it was deliberate - in the nightlies there’s another api for <code>module.compile()</code> which doesnt modify the nn module name which you can use</p>
Using Kineto profiler from C++ in pytorch

username: Benjamin_Sergeant
cooked: <p>Hi there,</p>
<p>I’d like to know if someone ever tried to use the kineto/builtin profiler from C++ (TorchScript).<br>
Essentially I’d like to see what’s happening when executing a method on a torchscript module (loaded in C++).</p>
<p>We have a model whose runtime keeps on increasing, and I’d like to see if maybe there’s a memory leak or something related.</p>
<p>Thanks !</p>
Forward pass hook to get weights multiplied by input?

username: Goldname
cooked: <p>Given an arbitrary model, I’d like to get a matrix of weights*inputs. I do not want the activations/outputs. I want a matrix of values where an entry is something like the ith weight multiplied by the jth input: w_i * input_j</p>
<p>How can I make a forward hook to do this?</p>

username: ptrblck
cooked: <p>Could you post a code snippet (with nested for loops if needed) showing the desired output using a plain <code>nn.Linear</code> module and a random input?</p>

username: Goldname
cooked: <p>I’d like something like this:</p>
<pre><code class="lang-auto">activation = {}
def get_activation(name, current_linear_module):
    def hook(model, input, output):
        activation[name] = current_linear_module.weight * input
    return hook
</code></pre>
<p>It should work for a linear layer:</p>
<pre><code class="lang-auto">current_linear_module= nn.Linear(5, 2)
input = torch.randn(1, 5)
print(input*current_linear_module.weight)
</code></pre>
<p>But how do I get this to work with an arbitrary module? Like a CNN or Transformer? They’re still implemented using matrix multiplication, so how do I grab this intermediate result of an individual neuron connection * input scalar?</p>
<p>For the CNN, worst case scenario I’d need to do something like this: <a href="https://stackoverflow.com/questions/56702873/is-there-an-function-in-pytorch-for-converting-convolutions-to-fully-connected-n" class="inline-onebox" rel="noopener nofollow ugc">Is there an function in PyTorch for converting convolutions to fully-connected networks form? - Stack Overflow</a> However, it would be impossible for me to implement this for a bunch of different pytorch modules. Pytorch should’ve already implemented all of this into matrix multiplications</p>

username: ptrblck
cooked: <p>I still don’t fully understand the use case as it seems you want to recompute the output value without the bias. Wouldn’t subtracting the <code>bias</code> tensor be easier in this case?<br>
If you want to manually recompute the output with a matrix multiplication for some reason, you might indeed need to reimplement this approach for all modules since e.g. conv layers would not always use matmuls depending on the availability of 3rd party libs and kernels.</p>

username: Goldname
cooked: <p>I need the intermediate elementwise multiplication: weight * value. I don’t want the output, which is the dot product between all rows of the weight matrix and the vector input: W*v.</p>
<p>To clarify, we would gather the output by elementwise multiplying first and then accumulate into a single scalar. Is it possible to get the result before the accumulate?</p>
Model restart with Learning rate decay

username: PastelBelem8
cooked: <p>Hey everyone,</p>
<p>I’m training an LSTM model, and I am struggling to make it converge. After the first update, the validation loss just kept going up, and never improved in the following 100 epochs, even though I used <code>torch.optim.lr_scheduler.ReduceLROnPlateau</code>.</p>
<p>In my mind, it makes sense that whenever you’d reduce the learning rate you’d go back to the model version that achieved the best validation previously. But I’ve barely seen anyone mentioning this strategy, so I was wondering whether there’s any particular reason for why we shouldn’t do that.</p>
CPU oneDNN optimization issue

username: rubende
cooked: <p>Hello,</p>
<p>I am trying to optimize a CPU model using oneDNN, following this tutorial: <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html#use-onednn-graph-with-torchscript-for-inference" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html#use-onednn-graph-with-torchscript-for-inference</a></p>
<p>The situation is that I observe a huge difference in performance by inference between the model generated for CPU with PyTorch and the model generated directly with the oneDNN library version 1.6 (300 microseconds in the first case and less than 20 microseconds in the second).</p>
<p>The procedure I am following with the model in Python to prepare it for CPU is this:</p>
<p>model = model.to(“cpu”)<br>
torch.jit.enable_onednn_fusion(True)<br>
model.eval()<br>
model = torch.jit.trace(model, torch.rand(shape))<br>
model = torch.jit.freeze(model _scripted)<br>
torch.jit.save(model, path)</p>
<p>After that, in C++, I use it as follows:</p>
<p>module = torch::jit::load(path);<br>
output = module.forward(tensor_inputs).toTensor();</p>
<p>Not getting the expected results, I have also tried with the library intel-extension-for-pytorch from Intel, including in the previous code this line before “torch.jit.trace”:</p>
<p>model = ipex.optimize(model, dtype=torch.float32)</p>
<p>The inference time is, also, around 300 microseconds. Also, performing the command " ldd" on the compiled executable file, and comparing with <a href="https://github.com/intel/intel-extension-for-pytorch/blob/master/docs/tutorials/examples.md" rel="noopener nofollow ugc">this tutorial</a>, I notice that I am missing the file “libdnnl_graph.so”, which I am not clear where it should come from because I cannot find it in any of the aforementioned libraries.</p>
<p>Any idea or suggestion of what could be happening? Does the optimization process for CPU seem correct to you? Is this huge difference in performance normal?</p>
<p>Thank you.</p>
Confused on an example on pytorch official documentation

username: NoExcept_Const
cooked: <p>Hi there, please check the <a href="https://pytorch.org/docs/stable/generated/torch.asarray.html#torch.asarray" rel="noopener nofollow ugc">torch.asarray — PyTorch 2.0 documentation</a> example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.tensor([1, 2, 3], requires_grad=True).float()
&gt;&gt;&gt; b = a + 2
&gt;&gt;&gt; b
tensor([1., 2., 3.], grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>I tried this script and it shows:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.tensor([1, 2, 3], requires_grad=True).float()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: Only Tensors of floating point and complex dtype can require gradients
&gt;&gt;&gt; a = torch.tensor([1.0, 2, 3], requires_grad=True).float()
&gt;&gt;&gt; a
tensor([1., 2., 3.], requires_grad=True)
&gt;&gt;&gt; b = a+ 2
&gt;&gt;&gt; b
tensor([3., 4., 5.], grad_fn=&lt;AddBackward0&gt;)
&gt;&gt;&gt; 
</code></pre>
<p>Do I miss something? Looking forward to your advice. Thanks.</p>

username: ptrblck
cooked: <p>I also don’t understand how <code>a + 2</code> should return the same values, but also the code will already fail before in the creation of <code>a</code>:</p>
<pre><code class="lang-python">    a = torch.tensor([1, 2, 3], requires_grad=True).float()

RuntimeError: Only Tensors of floating point and complex dtype can require gradients
</code></pre>
<p>CC <a class="mention" href="/u/ysiraichi">@ysiraichi</a> as the author of this PR.<br>
Could you check if the example is missing a few things?</p>

username: ysiraichi
cooked: <p>Ah, those examples are definitely wrong!<br>
I have created a <a href="https://github.com/pytorch/pytorch/pull/100971" rel="noopener nofollow ugc">PR</a> fixing that. Thank you for spotting that.</p>
Conda trying to install torchtext 0.6 instead of latest build

username: vlacerda
cooked: <p>I am trying to use torchtext in my conda environment but it is not working.</p>
<p>My steps are the following:</p>
<p>Create conda environment<br>
Update python to the latest version (currently 3.11.3)<br>
Install the latest stable version of Pytorch (2.0.1)<br>
% conda install -c pytorch torchtext</p>
<p>No matter how many times I’ve done this procedure, it always returns:<br>
The following NEW packages will be INSTALLED:</p>
<p>colorama           conda-forge/noarch::colorama-0.4.6-pyhd8ed1ab_0<br>
six                conda-forge/noarch::six-1.16.0-pyh6c4a22f_0<br>
torchtext          pytorch/noarch::<em><strong>torchtext-0.6.0</strong></em>-py_1<br>
tqdm               conda-forge/noarch::tqdm-4.65.0-pyhd8ed1ab_1</p>
<p>Edit: I’ve also tried installing this old version and updating it, to no avail. Conda just says “All requested packages already installed.”</p>
<p>This version is so old it doesn’t even have .vocab objects, for example. I am using a MacBook Pro 13-inch, M1, 2020. Currently on OS Monterey 12.5 (I’ll try to update it and try again, but that seems very arbitrary). Has anyone heard/had a similar problem?</p>
nn.ReLU6 vs. nn.functional.relu6

username: gokul_uf
cooked: <p>Hi folks,<br>
I am looking at the implementations of nn.ReLU6 and nn.functional.relu6.</p>
<p>It seems that nn.ReLU6 is implemented as a subclass of HardTanh while nn.functional.relu6 calls to the C API directly.</p>
<p>Is there any reason why there’s a difference? Why can’t nn.ReLU6 directly call nn.functional.relu6?</p>
<p>Thanks!</p>
Summarywriter.add_scalar() logs issue

username: viduzz84
cooked: <p>Hi, I experienced a problem in the behavior of tensorboard, when recording a scalar value with <code>summarywriter.add_scalar()</code></p>
<p>In my experiment I use a set of loss criteria, whose values I record at each epoch, with <code>add_scalar('Lossname', loss.item(), epoch)</code>.</p>
<p>Two values are recorded correctly, so in the tensorboard viewer the scalar graphs are displayed correctly.<br>
One value produces a graph with all infinite values, but if I print <code>loss3.item()</code> at runtime, it is a simple float variable with values neither too large nor too small (between 0 and 10).<br>
Why is it not logged correctly?</p>
<p>Also, since the total training loss is a weighted sum of the previous functions, I know that loss3 never goes to infinity because the training is successful.</p>
<p>The three loss functions are as follows:</p>
<ul>
<li>loss1: <em>MSELoss()</em> from PyTorch</li>
<li>loss2: <em>LogSTFTMagnitudeLoss()</em> from <a href="https://github.com/kan-bayashi/ParallelWaveGAN/blob/4615144d75bcb519ff1d2df7699ddd626787b5a4/parallel_wavegan/losses/stft_loss.py#LL64C7-L64C27" rel="noopener nofollow ugc">ParallelWaveGAN</a>
</li>
<li>loss3: <em>SpectralConvergenceLoss()</em> from <a href="https://github.com/kan-bayashi/ParallelWaveGAN/blob/4615144d75bcb519ff1d2df7699ddd626787b5a4/parallel_wavegan/losses/stft_loss.py#LL43C7-L43C7" rel="noopener nofollow ugc">ParallelWaveGAN</a>
</li>
</ul>
<p>Why are the values returned by <code>loss3.item()</code> not logged correctly?<br>
I suspect that it is because loss1() and loss2() use pytorch’s default functions: loss1() is the simple MSELoss while loss2() contains L1Loss. In contrast, loss3() is completely hand-defined and perhaps its class is missing some method…</p>
Torchserve docker - not accessible outside

username: seanbenhur
cooked: <p>I have created a docker image for the torchserve model used for sentiment analysis. I am able to run the torchserve model and get right responses but after creating the docker image, when I send curl request, I get <code>curl: (56) Recv failure: Connection reset by peer</code>.</p>
<p>This is my Dockerfile</p>
<pre><code class="lang-auto">FROM pytorch/torchserve

#set the working directory

WORKDIR /app

COPY model_store app/model_store

COPY requirements.txt app/

RUN pip install -r app/requirements.txt

COPY start_torchserve.sh app/

EXPOSE 8080/tcp
EXPOSE 8080/udp
EXPOSE 8081/tcp
EXPOSE 8081/udp

CMD ["torchserve", "--start", "--model-store","app/model_store","--models", "FirstModel.mar"]
</code></pre>
<p>I ran the docker image with <code>docker run -it -p 8080:8080 --gpus all torchserve</code>.<br>
I am able to receive responses inside the docker container but when I hit the api outside the container I get <code>curl: (56) Recv failure: Connection reset by peer</code>.</p>
How to do the anomaly detection via VAE?

username: Jeff.TSAI
cooked: <p>If I have some conditions and assumptions described below:</p>
<ol>
<li>Dataset (training set &amp; testing set) are both color images</li>
<li>The input of VAE is [batch_size, 3, 256, 256]<br>
VAE has been trained, including an encoder and decoder<br>
The output of the encoder is mu and the log_var, dimension is [batch_size, 256]<br>
The input of the decoder is [batch_size, 3, 256, 256]</li>
<li>The data x to be tested today is [batch_size, 3, 256, 256]</li>
</ol>
<p>I want to use the algorithm 4. in the VAE paper “Variational Autoencoder based Anomaly Detection<br>
using Reconstruction Probability” to do the anomaly detection, but I don’t know how to define the reconstruction probability function and how can I get mu and log_var via decoder??? (decoder output is a color image)</p>

username: tom
cooked: <p>Note that in the paper the output of the decoder is not just a “point estimator”  as in your description (if that is the output of the decoder rather than the input) but a distribution p(x | z) which is assumed to be Gaussian (when the inputs are continuous).</p>
<p>So you should have (for example)</p>
<ul>
<li>encoder input [batch_size, 3, 256, 256]</li>
<li>encoder output mu, log_var as [batch_size, 256]</li>
<li>decoder input [batch_size, 256]</li>
<li>decoder output mean, log_variance of [batch_size, 3, 256, 256]</li>
</ul>
<p>The reconstruction probability is defined as the likelihood of the input given by the encoder-decoder. As that requires integrating the latents, the latents are sampled from the mu and log_var output by the encoder. Then the likelihood of the input given the decoder with each given latent input is computed and then averaged over the samples.</p>
<p>If you cannot modify your setup to give mean and var as decoder outputs, you might assume a fixed log_var (maybe estimated from “known good” data) and use the output of your decoder as means.</p>
<p>Best regards</p>
<p>Thomas</p>
Data-loader time is higher than network-time

username: whatdhack
cooked: <p>I am puzzling over the timing info from the following simple  loop with data loader. The observation is that <em>tdata</em> is more than half of <em>titer</em>.  Is there any kind of synchronization happening  in the data loader  that waits for  the backward pass to finish ?</p>
<blockquote>
<p>print("iter: ", “batch_idx”, “tdata”, “tfwd”, “tloss”, “tbwd”, “titer”, “titercheck”, “args.batch_size/titer”, “args.batch_size/(titer-tdata)”, “time.time()” )<br>
extralong = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
time00 = time.time()<br>
for batch_idx, (data, target) in enumerate(train_loader):<br>
data = data.to(args.device)<br>
target = target.to(args.device)<br>
if args.nhwc:<br>
data = data.to(memory_format=torch.channels_last)<br>
tdata = time.time() - time00<br>
optimizer.zero_grad()<br>
output = model(data)<br>
tfwd = time.time() - tdata - time00<br>
loss = criterion(output, target)<br>
tloss = time.time() - tdata - tfwd - time00<br>
loss.backward()<br>
optimizer.step()<br>
tend = time.time()<br>
tbwd = tend - tdata - tfwd - tloss - time00<br>
titer = tdata + tfwd + tloss + tbwd<br>
titercheck = tend - time00<br>
time00 = time.time()<br>
if True:<br>
print("iter: ", batch_idx, tdata, tfwd, tloss, tbwd, titer, args.batch_size/titer, args.batch_size/(titer-tdata), len(extralong), time.time() )<br>
print("iter: ", batch_idx, tdata, tfwd, tloss, tbwd, titer, args.batch_size/titer, args.batch_size/(titer-tdata), len(extralong), time.time() )</p>
</blockquote>

username: ptrblck
cooked: <p>I assume you are using the GPU for the actual model training. If so then note that synchronizations are missing which need to be added before starting and stopping timers since CUDA kernels are executes asynchronously. Without synchronizing the code the timers could accumulate any synchronizing operation.</p>
Knowledge distillation with DDP

username: Jiaji_Huang
cooked: <p>This post is basically asking this <a href="https://discuss.pytorch.org/t/question-about-the-usage-of-distributeddataparallel/145501/1">question</a> again.</p>
<p>In a distributed data parallel setup, we use a frozen teacher model to regularize the training of a student. The student has to be wrapped by DDP wrapper, but how about the teacher?</p>
<p>Could anyone give an explanation why or why not the teacher needs to be DDP wrapped?</p>

username: eqy
cooked: <p>Note that the purpose of wrapping the model with DDP is to ensure that each replica (e.g., one per GPU) of the model stays in sync during training after e.g., each replica computes its gradients (and if normalization layers like batchnorm are used, their statistics also need to be synchronized). Since there would be no gradient updates or normalization statistics needed for the teacher model (because it is frozen), it wouldn’t be necessary to wrap it with DDP.</p>

username: Jiaji_Huang
cooked: <p>I see. Thank you for your detailed reply! <a class="mention" href="/u/eqy">@eqy</a></p>
<p>As a follow-up question, how about DataParallel? Would it be necessary to wrap the teacher with <code>torch.nn.DataParallel</code>?</p>

username: eqy
cooked: <p>I don’t think so, as <code>DataParallel</code> is effectively a more limited version of DDP and would not be necessary for the same reasons DDP wouldn’t be necessary.</p>
Counterintutive on matrix-vector mulitplication running time

username: linye-boli
cooked: <p>I have a counterintutive result on measuring the running time of following matrix-vector multiplication</p>
<p>first case we do normal matrix-vector multiplication</p>
<pre><code class="lang-auto">Khh = torch.rand((5,1,10001,10001)).cuda()
uh = torch.rand((5,1,10001)).cuda()

niter = 1000
times = []

for i in range(niter):
    torch.cuda.synchronize()
    start_time = time.time()
    wh = torch.einsum('bcmn,bcn-&gt;bcm', Khh, uh) * h 
    torch.cuda.synchronize()
    end_time = time.time()
    elapsed = end_time - start_time 
    times.append(elapsed)

print(sum(times) / niter)
</code></pre>
<p>the average time on my machine is 0.003968036651611328</p>
<p>the second case, I down sampled Khh along one axis</p>
<pre><code class="lang-auto">niter = 1000
times = []

for i in range(niter):
    start_time = time.time()
    KhH = Khh[...,::2]
    uH = uh[...,::2]
    wh = torch.einsum('bcmn,bcn-&gt;bcm', KhH, uH) * H
    end_time = time.time()
    elapsed = end_time - start_time 
    times.append(elapsed)

print(sum(times) / niter)
</code></pre>
<p>the average time of the second case is  0.0053208649158477785</p>
<p>The result is confusing, since we do less multiplication and addition.<br>
I don’t know why the second case costs more time compare to the first one</p>

username: ptrblck
cooked: <p>Could you describe why you removed the needed synchronizations in the second example? Also, it would be interesting to profile the actual matrix multiplication without the slicing kernel.</p>

username: linye-boli
cooked: <p>Sorry, I forgot that line, but after adding  synchronizations the result is still same.</p>
<pre><code class="lang-auto">Khh = torch.rand((5,1,10001,10001)).cuda()
uh = torch.rand((5,1,10001)).cuda()

niter = 1000
times = []

for i in range(niter):
    torch.cuda.synchronize()
    start_time = time.time()
    wh = torch.einsum('bcmn,bcn-&gt;bcm', Khh, uh) * h 
    torch.cuda.synchronize()
    end_time = time.time()
    elapsed = end_time - start_time 
    times.append(elapsed)

print(sum(times) / niter)

niter = 1000
times = []

for i in range(niter):
    torch.cuda.synchronize()
    start_time = time.time()
    KhH = Khh[...,::2]
    uH = uh[...,::2]
    wh = torch.einsum('bcmn,bcn-&gt;bcm', KhH, uH) * H
    torch.cuda.synchronize()
    end_time = time.time()
    elapsed = end_time - start_time 
    times.append(elapsed)

print(sum(times) / niter)
</code></pre>
<p>the result is as following</p>
<pre><code class="lang-auto">0.0036851165294647216
0.0076900506019592285
</code></pre>
<p>Thank you</p>
Trying to convert a .pt file into a .onnx file

username: Hellman_x
cooked: <p>I think I’ve spent about 6 hours trying to do this.</p>
<p>So is the closest I’ve gotten to getting the file to convert.<br>
<code>torch.onnx.export(model, dummy_input, "ipcam-dark.onnx", verbose=True)</code><br>
============== Diagnostic Run torch.onnx.export version 2.0.1+cpu ==============<br>
verbose: False, log level: Level.ERROR<br>
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================</p>
Elementary architecture implementation problem

username: niki_mino
cooked: <p>I am new to pytorch and I was wondering how I can create the neural network described in figure 2 of this paper <a href="https://arxiv.org/abs/1909.07481" class="inline-onebox" rel="noopener nofollow ugc">[1909.07481] Deep Neural Networks for Choice Analysis: Architectural Design with Alternative-Specific Utility Functions</a>. I can create figure 1 but I am confused on how to go about creating figure 2. I tried structural pruning by setting weights and biases to zero but I believe it should be much simpler than that.</p>
<p>A simple example would be highly appreciated.</p>
<p>Thank you.</p>
Seeding everything to get the same masked words

username: Surya_Narayanan
cooked: <p>Hi,</p>
<p>I am using a few different libraries (torch, hf transformers, lightning) so please feel free to let me know if this is outside the main torch forum’s wheelhouse.</p>
<p>I have two datasets, for which I want to create two dataloaders that and want to mask the same tokens in each.</p>
<p>I tried L.seed_everything() and was unable to get what I want.</p>
<p>Does anyone have any suggestions? My code is here: <a href="https://colab.research.google.com/drive/13aJSCQTj5X0KIHrkhoxwAkbr5RYtPhE4?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
Unexpected Behavior in Inference Transforms of Pretrained Models

username: torchwood
cooked: <p>Hi, I have encountered an unexpected behavior in the inference transforms of a pretrained model. The issue occurs when loading images from a dataset with and without the <code>transform</code> function applied.</p>
<p>When the images are loaded with <code>transform</code> applied, everything seems to work as intended, and the model accuracy is reasonably high. However, when loading images without applying <code>transform</code> and then applying it later (after converting the image to a tensor using <code>to_tensor</code>), the resulting images differ from those loaded with <code>transform</code>.</p>
<p>I have tried adjusting the <code>antialias</code> attribute of the <code>transform</code> function, but it did not lead to any improvements.</p>
<p>Here is the code:</p>
<pre><code class="lang-auto"># torchvision
import torchvision.transforms as transforms
from torchvision import models
from torchvision.datasets import ImageNet

# torch
import torch
import torch.nn.functional as F

weights = models.ResNet50_Weights.IMAGENET1K_V2
model = models.resnet50(weights=weights).eval().to('cuda')
transform = weights.transforms()

# Setting `antialias` to either `True` or `False` didn't fix things.
# transform.antialias = ...

# Dataset with transform.
dataset = ImageNet('./data/', split='val', transform=transform)
img_1, _ = dataset[111]

# Dataset without transform.
dataset = ImageNet('./data/', split='val')
img_2, _ = dataset[111]

# 1. Convert PIL image to tensor.
# 2. Trnasform it.
img_2 = transforms.functional.to_tensor(img_2)
img_2 = transform(img_2)

# Expected to be same but it's not.
print(img_1 == img_2)
</code></pre>
<p>Is there anything I’m missing here? I would appreciate any insights or suggestions to resolve this issue.</p>

username: torchwood
cooked: <p>Upon reviewing the documentation (<a href="https://pytorch.org/vision/main/generated/torchvision.transforms.Resize.html" class="inline-onebox" rel="noopener nofollow ugc">Resize — Torchvision main documentation</a>), I discovered that the issue arises from the differences in how PyTorch and PIL handle downsampling.</p>
<p>The warning says:<br>
The output image might be different depending on its type: when downsampling, the interpolation of PIL images and tensors is slightly different, because PIL applies antialiasing. This may lead to significant differences in the performance of a network. Therefore, it is preferable to train and serve a model with the same input types. See also below the antialias parameter, which can help making the output of PIL images and tensors closer.</p>
Check the norm of gradients

username: marcman411
cooked: <p>I have a network that is dealing with some exploding gradients. I want to employ gradient clipping using <a href="https://pytorch.org/docs/stable/_modules/torch/nn/utils/clip_grad.html" rel="nofollow noopener"><code>torch.nn.utils. clip_grad_norm_</code></a> but I would like to have an idea of what the gradient norms are before I randomly guess where to clip.</p>
<p>How can I view the norms that are to be clipped?</p>

username: marcman411
cooked: <p>Actually it seems the answer is in the code I linked to:</p>
<p>For a 2-norm:</p>
<pre><code class="lang-auto">        for p in model.parameters():
            param_norm = p.grad.data.norm(2)
            total_norm += param_norm.item() ** 2
        total_norm = total_norm ** (1. / 2)
</code></pre>

username: yijiang
cooked: <p>It is safer to detach the grad:</p>
<pre><code class="lang-auto">for p in model.parameters():
    param_norm = p.grad.detach().data.norm(2)
    total_norm += param_norm.item() ** 2
total_norm = total_norm ** 0.5
</code></pre>
<p>Another tensor-style way is:</p>
<pre><code class="lang-auto">parameters = [p for p in model.parameters() if p.grad is not None and p.requires_grad]
if len(parameters) == 0:
    total_norm = 0.0
else:
    device = parameters[0].grad.device
    total_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), norm_type).to(device) for p in parameters]), 2.0).item()
</code></pre>

username: milan_kalkenings
cooked: <p>in the tensor-style way: why do you use the norm of the parameter norms?</p>

username: milan_kalkenings
cooked: <p>doesn’t work in my case, cause the grad can be none,<br>
I use:</p>
<pre><code class="lang-auto">    total_norm = 0
    parameters = [p for p in model.parameters() if p.grad is not None and p.requires_grad]
    for p in parameters:
        param_norm = p.grad.detach().data.norm(2)
        total_norm += param_norm.item() ** 2
    total_norm = total_norm ** 0.5
    return total_norm
</code></pre>
<p>This works, I printed out the gradnorm and then clipped it using a restrictive clipping threshold.</p>

username: yijiang
cooked: <p>Hi, milan, I don’t fully understand your question. The loop-way is also to compute the norm of parameter gradient norms, right?</p>

username: milan_kalkenings
cooked: <p>yes exactly. I didn’t have any question, it’s just that your solution doesn’t work in my case</p>
<ul>
<li>in case any further people run into the same problem =)</li>
</ul>

username: David_Gilbertson
cooked: <p>I might be missing something, but calculating the norm twice isn’t necessary. It’s about 3x faster to concat all the grads into a single tensor then calculate the norm once:</p>
<pre><code class="lang-python">    grads = [
        param.grad.detach().flatten()
        for param in model.parameters()
        if param.grad is not None
    ]
    norm = torch.cat(grads).norm()
</code></pre>
Difference between torch 2.0.0.post200 and torch 2.0.0

username: Abdelkareem_Elkhateb
cooked: <p>I have torch 2.0.0.post200 when i  type torch.<strong>version</strong> but when in try to install dinov2  or any any models they say that i don’t have torch 2.0.0 and start to download one !</p>

username: ptrblck
cooked: <p><code>2.0.0.post200</code> sounds like a custom build from e.g. <code>conda-forge</code> which is not built by us. Could you explain how you’ve installed the binary and what <code>pip list</code> or <code>conda list</code> shows?</p>

username: Abdelkareem_Elkhateb
cooked: <p>I get it from the main website from the cuda version but this was after the announce of it from a long time …and now i tried to donwload new models i get the above error . i will create a new env thanks.</p>
Torchvision.ops.roi_align cruch session in colab

username: sbht_sbht
cooked: <p>I’m trying to use ops.roi_align in my faster r-cnn. When I run training loop all goes fine in first epoch, but when the second one starts  and training reaches roi align colab crashes due to lack of RAM.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/6748b29bb5de347306b5f77662f1b22a173695ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/6748b29bb5de347306b5f77662f1b22a173695ce" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/6748b29bb5de347306b5f77662f1b22a173695ce.png" alt="image" data-base62-sha1="eJGVSvDcTe6gKNRsC9Vej6D5lSu" width="690" height="92" data-dominant-color="3C3D39"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">940×126 10.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Ones it raise me this error but I cant get it again, cause colab crashes first:</p>
<blockquote>
<p> → 206 roi_out = ops.roi_align(feature_map, proposals_list, self.roi_size)<br>
207 print(“CM stop point 1”)<br>
208 print(roi_out)</p>
<p>/usr/local/lib/python3.10/dist-packages/torchvision/ops/roi_align.py in roi_align(input, boxes, output_size, spatial_scale, sampling_ratio, aligned)<br>
61 if not isinstance(rois, torch.Tensor):<br>
62 rois = convert_boxes_to_roi_format(rois)<br>
—&gt; 63 return torch.ops.torchvision.roi_align(<br>
64 input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned<br>
65 )</p>
<p>/usr/local/lib/python3.10/dist-packages/torch/_ops.py in <strong>call</strong>(self, *args, **kwargs)<br>
500 # We save the function ptr as the <code>op</code> attribute on<br>
501 # OpOverloadPacket to access it here.<br>
 → 502 return self._op(*args, **kwargs or {})<br>
503<br>
504 # TODO: use this to make a <strong>dir</strong></p>
<p>ValueError: cannot create std::vector larger than max_size()</p>
</blockquote>
<p>When I run the same code in Jupyter lab nothing crashes and no error appears. But instead, the execution of the program stucks at this step and hangs on it indefinitely.</p>
<p>It turned out quite a lot of code, so I don’t know what exactly needs to be attached to the problem description <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Edit 1: I forgot to mention that earlier use roi pool. Faster r-cnn train without any problem but trained network return output bboxes the size of one pew image was always larger than that of the image.</p>
Multivariate auto grad

username: magnusderrote
cooked: <p>I am implementing PINN on Pytorch (<a href="https://github.com/maziarraissi/PINNs" class="inline-onebox" rel="noopener nofollow ugc">GitHub - maziarraissi/PINNs: Physics Informed Deep Learning: Data-driven Solutions and Discovery of Nonlinear Partial Differential Equations</a>, written in TF). The gist is that I have a prediction loss as well as the loss from gradients. Suppose my input is <code>x_1, x_2, x_3</code> and my output is <code>y_1, y_2</code>, then <code>dy_1/d_x1, dy1/dx_2, ..., dy_2/dx_3</code> is needed to compute a different kind of loss.</p>
<p>Right now, my options are</p>
<ol>
<li>
<code>autograd.grad(y[1], x, allowed_unused=True, retain_graph=True)</code> <code>retain_graph</code> because I am going to need  <code>autograd.grad(y[2], x, ...)</code> too. This is univariate derivation and doesn’t seem efficient, but it works.</li>
<li>To make it a vector derivation, I need to make <code>autograd.grad(y, x)</code> works, which I don’t know how, and still retain the graph for <code>loss.backward()</code> later to actually change the weights of the network.</li>
</ol>
<p>How do I implement 2, or I have no other choice than 1. Most PINN Pytorch implementation I found use 1, but for performance concerns, I’d like to explore 2.</p>
<p>Thank you.</p>

username: KFrank
cooked: <p>Hi Magnus!</p>
<aside class="quote no-group" data-username="magnusderrote" data-post="1" data-topic="179407" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/5daacb/48.png" class="avatar"> magnusderrote:</div>
<blockquote>
<p>Suppose my input is <code>x_1, x_2, x_3</code> and my output is <code>y_1, y_2</code>, then <code>dy_1/d_x1, dy1/dx_2, ..., dy_2/dx_3</code> is needed to compute a different kind of loss.</p>
</blockquote>
</aside>
<p>What you’ve asked for is the <em>Jacobian</em> of the function that maps <code>x</code> to <code>y</code>.<br>
Consider using pytorch’s <a href="https://pytorch.org/docs/2.0/generated/torch.autograd.functional.jacobian.html" rel="noopener nofollow ugc">jacobian()</a> (older and more established) or <a href="https://pytorch.org/docs/2.0/generated/torch.func.jacrev.html" rel="noopener nofollow ugc">jacrev()</a><br>
(newer and maybe faster, but I’ve never really used it).</p>
<p>Best.</p>
<p>K. Frank</p>