akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Vectorizing Log Probability Calculations

armal: <p>I currently have a 2 tensors of size ([N, D]) and ([N, D, D]). The first one represents a N means and the second represents N covariance matrices. Currently I am having to create N different Multivariate Gaussians in a for loop and then evaluating the log probability of some data which has size ([X, D]) on each MV Gaussian. Is there a way to vectorize this process at all by passing all the means and all the covariances at once into the Multivariate function and then having a single log_prob evaluation which output a vector with size ([X])?</p>

ptrblck: <p>Wouldn’t this directly work using <code>MultivariateNormal</code>:</p>
<pre><code class="lang-python">loc = torch.stack((torch.zeros(2), torch.ones(2)+100., torch.ones(2)+1000.))
m = torch.distributions.multivariate_normal.MultivariateNormal(loc, torch.eye(2)[None])
m.sample()
# tensor([[-8.6749e-02, -1.2250e-01],
#         [ 1.0310e+02,  1.0090e+02],
#         [ 1.0028e+03,  1.0015e+03]])
</code></pre>

armal: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> .</p>
<p>Yes you are absolutely correct, I realised there were some mistakes in my code last night when I was trying to do this as I was summing across the wrong axis to produce the quantity I needed. This caused me to have an output tensor which is not the shape I expected it to be. Apologies, I should have come back and noted this after I found out.</p>
Change Dataset values on each epoch with persistent workers

javier_selva: <p>Hi!</p>
<p>I am working on a video-loading pipeline. Instead of loading all possible sequences from each video in the dataset, I want to randomly sample N sequences from each video (so all videos are seen at each epoch, but the epoch is a bit shorter). In order to use the full dataset, I would like that at each epoch these N sequences are <em>potentially</em> different. My current approach is holding a “mapping” variable that holds a randomly selected set of indices to be used in the current epoch. What I need is that, at the end of the epoch, some function is called so this mapping is recomputed (and that all workers have the same mapping).</p>
<p>I wrote the following simplified version of what I want to achieve:</p>
<pre><code class="lang-auto">import torch 
class Dummy(torch.utils.data.Dataset):
    def __init__(self,num_sequences, dim):
        self.data = torch.stack([torch.ones(dim)*x for x in range(num_sequences*2)])
        self.num_sequences = num_sequences
        self.mapping = list()
        self.make_mapping()
    
    def __len__(self):
        return self.num_sequences
    
    def __getitem__(self,idx):
        return self.data[self.mapping[idx]]
    
    def make_mapping(self):
        self.mapping = sorted(torch.randperm(len(self.data))[:self.num_sequences].numpy())
        
dset = Dummy(10,1)
dload = torch.utils.data.DataLoader(dset,batch_size=5, num_workers=2,persistent_workers=True)
for e in range(5):
    print(f"Epoch {e}")
    for b in dload:
        print(b)
    dload.dataset.make_mapping()
</code></pre>
<p>The problem is when persistent_workers=True, as each worker holds its own copy of the Dataset and this call to make_mapping() seems to have no effect.</p>
<p>I have been searching and while I find people with similar problems (generally related to cropping or different resolution inputs), I do not seem to find a way to fit their solutions (workarounds to actually changing Dataset variables). I also found <a href="https://discuss.pytorch.org/t/communicating-with-dataloader-workers/11473/9">this</a>, but seems like too much hassle to just shuffle the data.</p>
<p>I would think there should be a way to handle some event at the end of the epoch such that the make_mapping function is called. I tried to do something in these lines by keeping count of the number of times <strong>getitem</strong> is called, so each worker calls make_mapping if the counter reaches num_sequences // num_workers. This kinda works but as each worker makes its own shuffle, then the same sequence might be seen multiple times in a single epoch.</p>
<p>Having persistent_workers=False definitely works, but I was wondering if there could be a different way to achieve this that I am missing. I took a look at Samplers, but I have never used them and I am not sure if I could achieve this with them.</p>
<p>Thanks a lot in advance!</p>
Generating Visual Studio Solution for Libtorch, using CMake

yoman: <p>Hi,<br>
I’m trying to generate a visual studio solution for the libtorch sources, using CMake.<br>
Having lots of issues, and I can’t find any example of someone doing that.<br>
If there is some step-by-step guide - which programs to install, the cmake command. etc - it would be VERY helpful.</p>
<p>Appreciate your help,<br>
Thanks.</p>
Difference between _Loss and _WeightedLoss classes

Sayan_Chatterjee: <p>In the implementation of pytorch, the following two classes are defined in torch.nn.modules.loss:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6414e0e4c642a60127499ccbe4af9dcac487145.png" data-download-href="https://discuss.pytorch.org/uploads/default/c6414e0e4c642a60127499ccbe4af9dcac487145" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6414e0e4c642a60127499ccbe4af9dcac487145_2_690x260.png" alt="image" data-base62-sha1="shQsabQ7DBMO4cPaXiubWocYPiZ" width="690" height="260" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6414e0e4c642a60127499ccbe4af9dcac487145_2_690x260.png, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6414e0e4c642a60127499ccbe4af9dcac487145.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6414e0e4c642a60127499ccbe4af9dcac487145.png 2x" data-dominant-color="F9F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">904×341 35.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I don’t understand the code here, but few losses like L1Loss inherits the class _Loss while some inherit the _WeightedLoss class.<br>
I want to know how does a loss function that inherits _WeightedLoss different from the one that inherits _Loss?<br>
Should I really care about it or as a user of the library, it is completely irrelevant to me?</p>

Sayan_Chatterjee: <p>okay. I think I figured it out.<br>
I just found this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf982dfc0ae252cf67c33e4d7d73964d589df55a.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf982dfc0ae252cf67c33e4d7d73964d589df55a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf982dfc0ae252cf67c33e4d7d73964d589df55a.png" alt="image" data-base62-sha1="tCsSxGne2AEB7xVRFwsDPUvZ402" width="690" height="129" data-dominant-color="F2DBDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">740×139 19.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
so this _WeightedLoss class is for the losses like Negative Log Likelihood loss and Binary Cross Entropy loss which provide a measure for probabilities and so, need to obtain values from all the classes of the input, to generate the corresponding probabilities for each class. The additional weight parameter would provide a weight to a certain class.<br>
Whereas _Loss is for simple loss metrics between input and target values, like an L1 or L2 loss which is calculated only between an input value and the ground truth.</p>
Is advanced indexing without data copy possible to implement in C++?

Espritdelescalier: <p>I heard arguments that it betrays the philosophy of pytorch, that it’s hazardous, etc … My use case is niche but absolutely valid, I need to be able to manipulate a tensor in more complex ways that are possible with the python front end. In particular I would like to be able to do what index_select does (uses a LongTensor of indices to slice a tensor)  without copying the data into a new tensor. Is such a thing feasible without delving into the Aten code or elsewhere in the C++ backend?</p>

ptrblck: <p>I don’t know what kind of PyTorch philosophy would be betrayed, but the <code>libtorch</code> indexing can be found <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html">here</a> with their Python counterparts.</p>

Espritdelescalier: <p>Thanks for the heads up, I could then create an extension accomplishing what I’ve set out to do basing my new indexing function on those you provided?<br>
I’ve been looking at the C++ backend and as one should expect for such a library things are complex and interconnected, I wasn’t sure where I could get the low level access to tensors required to do advanced indexing without copying data into a new tensor.<br>
What I want is a customized view of a tensor but the very structure of pytorch tensors (of the little I have actually read from the C++ code) with the stride looked to me as if they presupposed creating views with a regular stepping through the tensor. If I’m wrong I’m glad I am. I’ll look into the resources you provided.<br>
Thanks.</p>
Does pytorch 2 exploit parallelism in a computational graph during inference?

aarmbruster: <p>I have a pytorch model, the forward pass looks roughly like the following</p>
<pre><code class="lang-python">def forward(self, x):

    # following two encoders could be run in parallel
    lidar_features = self.lidar_encoder(x['pointcloud'])
    camera_features = self.camera_encoder(x['images'])
    
    # need to sync here
    combined_features = torch.stack((lidar_features, camera_features))
    predictions = self.prediction_head(combined_features)
    return predictions
</code></pre>
<p>If the model is in eval mode, is pytorch 2 smart enough to know that the lidar encoder and camera encoder can be run at the same time on the GPU, but then a sync needs to be inserted before the <code>torch.stack</code>? or will kernels be run in the serial order of the python code?</p>
<p>What about pytorch 1.X?</p>

ptrblck: <p>Both modules will be scheduled on the same CUDA stream and will thus run sequentially. You could use custom CUDA streams and synchronize the code manually and check if any performance improvements would be seen. Note that you might not see any overlaps if a) your CPU is too slow to schedule the kernels, b) if your code synchronizes and thus blocks the CPU, c) if the compute resources are already allocated by one module.</p>
Transform normalization

richardatpytorch: <p>Forgive me if I misunderstand this operator. Here is the transform that I am applying:</p>
<pre><code class="lang-auto">output_size = 256
color_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Resize((output_size, output_size), antialias=True),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
depth_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Resize((output_size, output_size), antialias=True),
    transforms.Normalize(mean=[0.5], std=[0.5])
])
</code></pre>
<p>I suppose that the output value of tensor will be in the range [mean-std, mean+std], but the value that I print out is much larger/smaller. What did I do wrong?<br>
Some context about my case. I am trying to read RGB and depth images with cv2.imread(filepath, -1) then applying the above transformation directly.</p>

ptrblck: <p><code>transforms.Normalize</code> subtracts the provided <code>mean</code> value from the tensor and divides by <code>std</code> to create a normalized sample. The values are calculated in a way to create a zero-mean and unit-variance output. The actual values are not bound to <code>[mean-std, mean+std]</code> as seen in this example:</p>
<pre><code class="lang-python">x = torch.randn(10000) * 123. + 5467
print(x.mean(), x.std())
# tensor(5469.2778) tensor(122.1908)

mean = x.mean()
std = x.std()

y = (x - mean) / std
print(y.mean(), y.std())
# tensor(-2.7153e-06) tensor(1.)
print(y.min(), y.max())
# tensor(-3.7724) tensor(3.4398)
</code></pre>

richardatpytorch: <p>I suppose to get the mean and std as in the formation, I should rescale the tensor value in the range of [0,1] first. Is is right?</p>

ptrblck: <p>Yes, the posted <code>mean</code> and <code>std</code> values from the <code>color_transform</code> are the ImageNet stats using the normalized inputs in the range <code>[0, 1]</code> while the second example seems like placeholder stats also for normalized inputs in <code>[0, 1]</code>.</p>
Deploy CNN model from python to c++ with torch script

uint15_t: <p>is it possible to deploy a python created CNN model with torch script to C++ Code ?</p>

ptrblck: <p>Yes, you can script the model and load in in a <code>libtorch</code> C++ application as described in <a href="https://pytorch.org/tutorials/advanced/cpp_export.html">this tutorial</a>.</p>
Freezing and training subcomponents of the model

anupsingh15: <p>Hey folks!</p>
<p>I have a model which consists of 3 submodules( A, B, and C) stacked sequentially. The pipeline of the model looks like A–&gt;B–&gt;C. For the first few epochs, I train the A–&gt;B and compute the loss; thereafter, I freeze the A and B submodules and train only C. The forward pass looks like this:</p>
<pre><code class="lang-auto">def forward(self, batch):
   z = B(A(batch))
   if self.current_epoch &gt; 30:
     z = C(B(A(batch))
   .....

def callback(self):
  if self.current_epoch &gt; 30 and self.freeze:
    for param in A.parameters():
      param.requires_grad = False
    for param in B.parameters():
      param.requires_grad = False
   
   self.optimizer[0] = torch.optim.Adam(self.C.parameters(), lr=pl_module.lr)
   self.freeze = False
</code></pre>
<p>Note that I also reset the Adam optimizer to train the C component.</p>
<p>I did not get the desired results with this pipeline. However, if I train a model which consists of only A–&gt;B, and store its outputs <code> (z = B(A(batch)))</code> externally as a database after the training, and then use this database to train a model which contains only C, then it works for me. Basically, I have to break the pipeline into two to get results rather than just having a single pipeline. Any suggestions where I might be doing something wrong?</p>

ptrblck: <p>I would recommend running a few sanity check and making sure the frozen parameters are indeed not updated anymore after 30 epochs, that all <code>C</code> parameters get valid gradients and are updated, etc.<br>
If all of this looks OK I would then double check the outputs of <code>B(A(batch))</code> using both pipelines and a static input (e.g. just <code>torch.ones</code> or one specific sample) and compare these features to see if something still differs in the forward passes.</p>
Torchvision.transforms failed to transform a 16-bit grayscale tif

proshm: <p>I use PIL to open a 16-bit grayscale tif image, then use torchvision.transforms to transform the image. an error “ValueError: image has wrong mode” comes out from transform operation. But when I open a RGB image of bmp or jpg format, no error comes out. The code is follow:<br>
import os<br>
from PIL import Image<br>
import torchvision.transforms as transforms<br>
rrc_trans = transforms.RandomResizedCrop(size=224, scale=(0.2, 1.0))<br>
rhf_trans = transforms.RandomHorizontalFlip()<br>
to_sensor = transforms.ToTensor()<br>
norm_trans = transforms.Normalize(mean=0.45, std=0.225)<br>
img = Image.open(‘aa.tif’)<br>
img1 = rrc_trans(img)<br>
img2 = rhf_trans(img1)<br>
img3 = to_sensor(img2)<br>
img4 = norm_trans(img3)</p>

ptrblck: <p>Based on <a href="https://github.com/python-pillow/Pillow/issues/4402">this issue</a> the default <code>BILINEAR</code> interpolation doesn’t seem to be supported for <code>uint16</code> images and <code>NEAREST</code> should work:</p>
<pre><code class="lang-python">x = np.random.randint(0, 65535, (224, 224), dtype=np.uint16)
img = PIL.Image.fromarray(x)

rrc_trans = transforms.RandomResizedCrop(size=224, scale=(0.2, 1.0), interpolation=PIL.Image.Resampling.NEAREST)
img1 = rrc_trans(img)
</code></pre>
Dataloader with two sets of augmentation on same image

vikash0837: <p>Hi,<br>
I am trying to do weak and strong augmentation of the same set of images by maintaining the actual correspondence. I tried with <strong>ConcatDataset</strong>. Though images come from two sets of augmentations, it doesn’t maintain the one-to-one correspondence in the first half and 2nd half of the batch.<br>
Please let me know if you have any idea.<br>
Thank You</p>

ptrblck: <p><code>ConcatDataset</code> will concatenate multiple <code>Dataset</code>s sequentially, i.e. the <code>len</code> of the <code>ConcatDataset</code> will be the sum of all passed <code>Dataset</code>s.<br>
If you want to augment the same sample in a different way, you could use different transformations in the <code>__getitem__</code> of the <code>Dataset</code> and return both transformed samples.<br>
This would double your batch size, so that you could lower it in the <code>DataLoader</code>, if needed.</p>

Carl1: <p>Hi, I’m trying to do something similar where each batch of bs samples should contain (bs/n) images each augmented n times (where n is fixed for a model run), i.e. each batch should be:<br>
(t_1(x_1), … t_n(x_1))<br>
(t_1(x_2), … t_n(x_2))<br>
… .  .  … .  .  … .  .<br>
the transformations can all be defined by the same (randomised) function, but should be called n times tp  give different results.<br>
I have tried constructing a Transformation where <strong>call</strong>(self, x) outputs:<br>
torch.cat([self.train_transform(x) for _ in range(n)], 0)       [ or can use torch.stack ]<br>
so a pre-defined “transform” is applied independently n times<br>
(Note: I previously used [transform(x)] * n, but I think that may give n copies of the same transformed x?)<br>
…  but this seems v slow (1 batch of bs samples with n=1 transform is much faster than 1 batch of bs/n samples with n&gt;1 transforms even though they both output the same number of transformed samples).</p>
<p>Is there a “best” way to do this? e.g.</p>
<ul>
<li>using ConcatDataset to pull together n Datasets (but this seems heavy handed as the x’s are the same and the transform function is/can be the same?</li>
<li>or through some smarter way of calling the same transform and combining</li>
<li>or something else…</li>
</ul>
<p>Many thanks</p>

ptrblck: <p>I don’t know how many workers in the <code>DataLoader</code> you are using (specified via <code>num_workers</code>), but note that you are increasing the work of each worker by a factor of <code>n</code> since each one loads and transforms <code>n</code> samples.<br>
You could increase the number of workers to check if loading the data in the background could help.</p>
Random forest classification criteria

John4: <p>Hi, I need to implement the inference of the RandomForestClassifier of sklearn in C language following the equation illustred in <a href="https://scikit-learn.org/stable/modules/tree.html#mathematical-formulation" class="inline-onebox" rel="noopener nofollow ugc">1.10. Decision Trees — scikit-learn 1.2.2 documentation</a>.<br>
Could you explain me how to get the y and the nm in python? Is the equation of the link the right equation to be considered?</p>

ptrblck: <p>Your question seems to be unrelated to PyTorch so you might want to post it in a scikit-learn - specific discussion board.</p>
Libtorch does not (really) support multithreading

slaweks17: <p>Hi,<br>
One of the use-cases for Libtorch was supposed to be multithreading. OK, I can read data this way, I can do forward(), but when it comes to learning, the autograd machinery appears to be serialized per device, backward(), which in my case takes a lot of time, is serialized, see <a href="https://github.com/pytorch/pytorch/issues/17566" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/17566</a></p>
<p>Here is a comment in engine.cpp:<br>
// XXX: Changes to the way multithreading works in execute should be done with<br>
// great care. Right now the implementation guarantees that a single function’s<br>
// apply will never be entered concurrently (even if multiple graphs are<br>
// executed at the same time). Adding multiple threads per-device or removing<br>
// engine thread affinity to the device can break this invariant, and we depend<br>
// on it in a few places (e.g. AccumulateGrad function).</p>
<p>So, any hope this will get fixed and Libtorch can really support multithreaded learning?<br>
My use case is a concurrent training of several independent NNs, with own trainers etc, so the synchronization is  not a problem.</p>
<p>Regards,<br>
Slawek</p>

JamesDickens: <p>Any updates to this?</p>
Missing child_counter from StreamWrapper?

jcg: <p>Hello!</p>
<p>While using datapipes (torchdata 0.5.1) I ran into the following:</p>
<p>S3FileLoader feeds into a ZipArchiveLoader. There is an exception in line 71 of ZipArchiveLoader because children_counter is missing from the parent stream (the one that the S3 loader provides) If I add (before try/except)</p>
<pre><code class="lang-auto"> data_stream.child_counter = 0
</code></pre>
<p>then everything runs as expected. Is this a bug?</p>
<p>Thanks!</p>

nivek: <ol>
<li>Can you try your code with TorchData 0.6.0?</li>
<li>What does your DataPipe set up looks like? Is there a minimal, reproducible code snippet that you can provide?</li>
</ol>

jcg: <p>Hello!</p>
<p>Yes → I will update to 0.6.0 after this experiment completes.<br>
Yes → I will put together the snippet tomorrow.</p>
<p>Thanks!</p>

jcg: <p>Hello, this is the code that reproduces the problem</p>
<pre><code class="lang-auto">            for url, stream in S3FileLoader(IterableWrapper([f's3://{self.__bucket}/{self.__dataset}/zip/{archive}'])):
                if archive.lower().endswith('.zip'):
                    for name, doc in ZipArchiveLoader(IterableWrapper([('XXXX', stream)])):
                        yield name, doc
</code></pre>
<p>I hope this helps</p>

nivek: <p>I don’t think your code snippet is using TorchData in the intended pattern, can you try something like this?</p>
<pre><code class="lang-python">dp = IterableWrapper([f's3://{self.__bucket}/{self.__dataset}/zip/{archive}'])
dp = dp.filter(_insert_filter_fn_here)  # Filter for ".zip"
dp = dp.load_files_by_s3()
dp = dp.load_from_zip()
</code></pre>

jcg: <p>This works. The attribute is in place now. Thanks!</p>
Pytorch-GPU% utilization 'CUDA'

axnet: <p>Hello,</p>
<p>I’m using NVIDIA gpu device and i’m using pytoch-GPU. Is there a way to check the GPU ‘CUDA’ utilization while running a code.<br>
I tried torch.cuda.utilization(device=‘cuda’) but this  generated the below error<br>
pynvml.nvml.NVMLError_LibraryNotFound: NVML Shared Library Not Found.</p>

eqy: <p><code>torch.cuda.utilization</code> is documented as using <code>nvidia-smi</code> underneath, could you check if running <code>nvidia-smi</code> in a separate terminal works instead?</p>

axnet: <p>I’m using jetson AGX orin and it doesn’t support nvidia-smi</p>
ONNX export of Faster RCNN from PyTorch

KiaraEsguerra: <p>Hello, I am trying to export the Faster RCNN model from PyTorch after performing quantization on the backbone:</p>
<pre><code class="lang-auto">import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor


model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False)
example_input = torch.randn(1, 3, 224, 224)
model.eval()
model.backbone = quantize_fx.fuse_fx(model.backbone)  # The modules of the model are fused
model.train()  # The model needs to be set to train before preparation for QAT
qconfig_mapping = get_default_qconfig_mapping("fbgemm")
model.backbone = quantize_fx.prepare_qat_fx(model.backbone, qconfig_mapping, example_input)
model.backbone = quantize_fx.convert_fx(model.backbone)
model.eval()
torch.onnx.export(model,
                 example_input,
                 "fast_rcnn.onnx",
                 verbose=False,
                 input_names=['input'],
                 output_names=['output'],
                 export_params=True,
                 opset_version=13
                 )
</code></pre>
<p>When I run the onnx export, I get the error:</p>
<pre><code class="lang-auto">RuntimeError: Type 'Tuple[Tensor, NoneType]' cannot be traced. Only Tensors and (possibly nested) Lists, Dicts, and Tuples of Tensors can be traced
</code></pre>
<p>Meanwhile, I do not get this error when I am exporting the un-quantized model. Thank you very much for your help.</p>

Andrew_Lavin_Phantom: <p>I got the same error and figured out the cause. Not sure if your error has the same cause as mine, but even so this explanation will likely help someone, as it took me hours to figure it out.</p>
<p>The error <code>RuntimeError: Type 'Tuple[Tensor, NoneType]' cannot be traced ...</code> will happen if you try to export a quantized model, and the model’s <code>forward</code> method has a second argument with a default value of <code>None</code>.</p>
<p>The <code>torch.onnx.export</code> function first puts your Tensor example input in a tuple. Then it inspects the signature of the model <code>forward</code> method, and if it finds the method takes a second argument with a default value of <code>None</code>, it automatically adds a second “input” to the tuple with the value None.</p>
<p>Next it sees that you are exporting a quantized model, and calls the function <code>_pre_trace_quant_model</code> which simply calls <code>torch.jit.trace(model, args)</code>. The <code>args</code> is the tuple containing <code>(example_input, None)</code> at this point. The error message you see is from <code>torch.jit.trace</code> complaining that it does not support <code>None</code> as an argument value.</p>
<p>In my case, I resolved the error by modifying the <code>forward</code> method of my model to remove the second argument that had the default value of <code>None</code>.</p>
<p>Hope this helps someone!</p>
<p>Edit: I used PyTorch 2.0.0</p>
Batching images in each folder together

Thomas_Lavender: <p>Hi!</p>
<p>I have a bunch of training images which I have organized into folders grouped by image size (200x600 images in 200x600 folder).</p>
<p>When initalizing my Dataloader and passing the batch size, I’d like it that images who share the same folder are processed together in one batch. Currently what’s happening is that images of different sizes are being batched together and processed leads to an error.</p>
<p>I do not know how to achieve my goal and would appreciate any feedback. If there any questions to my issue then please let me know. Below is the code for my custom Dataset class which may be relevant.</p>
<pre><code class="lang-python">class ImageDataset:
  def __init__(self, directory: str):
    self.files: list[str] = glob(os.path.join(directory, "*/*.png"), recursive=False)

  def __len__(self):
    return len(self.files)
  
  def __getitem__(self, item_id: int) -&gt; tuple[npt.NDArray, npt.NDArray]:
    with open(self.files[item_id], "rb") as fh:
        with Image.open(fh) as image:
            target_data = np.array(image)
            input_data = np.array(image.resize((int(target_data.shape[1]/4), int(target_data.shape[0]/4)), Image.BILINEAR))
            input_data = input_data.transpose(2, 1, 0)
            target_data = target_data.transpose(2, 1, 0)

    return input_data, target_data
</code></pre>

nivek: <p>One option is to write a custom BatchSampler that ensures all indices within a batch belong to the same folder. Then, you can pass that to <code>DataLoader</code>.</p>
Change dtype of torch::nn::Linear

Matthias_Moller: <p>How can I change the <code>dtype</code> of a <code>torch::nn::Linear</code> layer. It seems to be <code>float32</code> be default because when I pass a torch::Tensor with <code>dtype=float64</code> I get the following error:</p>
<pre><code class="lang-auto">libc++abi: terminating due to uncaught exception of type c10::Error: expected scalar type Double but found Float
</code></pre>

eqy: <p>Could you check if either casting the layer or the model works for you? <a href="https://discuss.pytorch.org/t/how-to-use-kdouble-type-input-for-linear-layer-in-c/90164" class="inline-onebox">How to use kDouble type input for Linear layer in C++</a></p>
<p>If you know the dtype when the layer is created I would also check if the layer accepts a dtype parameter on when it is created similar to the Python API.</p>
Dataloader with attribute error

Maria_de_Jesus_Garci: <p>I have a problem  with my dataloader, it is define like this</p>
<pre><code class="lang-auto"># Define dataloader
class Mental(Dataset):
    def __init__(self, X,y = None, _config = None):
        self.X = X
        self.y = y
        self.tokenizer = AutoTokenizer.from_pretrained(_config['model_name'], do_lower_case=True)
    def __len__(self):
        return len(self.X)
    def __getitem__(self, idx):
        out=torch.tensor(self.tokenizer.encode(self.X[idx], max_length=128, pad_to_max_length=True, add_special_tokens=True))
        if(self.y):
            return {"text": out, "attention":(out!=1).float(), "label":torch.tensor(self.y[idx])}
        else:
            return {"text": out, "attention":(out!=1).float()}
</code></pre>
<p>but when I call the dataloader, I have this error</p>
<pre><code class="lang-auto">AttributeError                            Traceback (most recent call last)
/002/usuarios/maria.garcia/anaconda3/lib/python3.7/site-packages/IPython/core/formatters.py in __call__(self, obj)
    700                 type_pprinters=self.type_printers,
    701                 deferred_pprinters=self.deferred_printers)
--&gt; 702             printer.pretty(obj)
    703             printer.flush()
    704             return stream.getvalue()

/002/usuarios/maria.garcia/anaconda3/lib/python3.7/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    403                         if cls is not object \
    404                                 and callable(cls.__dict__.get('__repr__')):
--&gt; 405                             return _repr_pprint(obj, self, cycle)
    406 
    407             return _default_pprint(obj, self, cycle)

/002/usuarios/maria.garcia/anaconda3/lib/python3.7/site-packages/IPython/lib/pretty.py in _repr_pprint(obj, p, cycle)
    693     """A pprint that just redirects to the normal repr function."""
    694     # Find newlines and replace them with p.break_()
--&gt; 695     output = repr(obj)
    696     lines = output.splitlines()
    697     with p.group():

/002/usuarios/maria.garcia/.local/lib/python3.7/site-packages/datasets/arrow_dataset.py in __repr__(self)
   1678 
   1679     def __repr__(self):
-&gt; 1680         return f"Dataset({{\n    features: {list(self.features.keys())},\n    num_rows: {self.num_rows}\n}})"
   1681 
   1682     @property

/002/usuarios/maria.garcia/.local/lib/python3.7/site-packages/datasets/arrow_dataset.py in features(self)
    180     @property
    181     def features(self) -&gt; Features:
--&gt; 182         return self._info.features
    183 
    184     @property

AttributeError: 'Mental' object has no attribute '_info'
</code></pre>
<p>Everything was fine until yesterday, I don know what happened</p>

eqy: <p>Are you importing another dataset class e.g., from <code>datasets.arrow_dataset</code> and accidentally overwriting the <code>Dataset</code> class name in your script?</p>
Language model, intermediate training on a word instead of a whole sentence

geenidee005_geenidee: <p>I have an LSTM language model to predict the next word, which normally trains on sentences, or rather a batch of sentences. For an experiment I would like the model to backpropagate gradient for one word of a sentence and optimize accordingly. For that, I would think the following code would be sufficient:</p>
<blockquote>
<p>for x in range(0, test_data.size(0)-1, args.bptt):</p>
<blockquote>
<pre><code>   print()
   sent_data, targets = data.get_batch(test_data, x, args) 
   first_eos = int((sent_data == eos).nonzero()[1][0])  # Cut off data at end of sentence
   sent_data = sent_data[:first_eos]
   targets   = targets[:first_eos]
   #print(sent_data)
   #print(sent_data.size())
   print(model)
   hidden = model.init_hidden(args.eval_batch_size)
   for word, target in zip(sent_data, targets):
</code></pre>
<blockquote>
<pre><code>       model.zero_grad()
       output, hidden = model(word.view(1,1), hidden)
       loss = crit(log_softmax(output.view(-1, ntypes), dim=1), target.view(1))
       loss.backward()
       optimizer.step()
</code></pre>
</blockquote>
</blockquote>
</blockquote>
<p>But this gives a runtimeerror, specifically:<br>
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
<p>I am not familiar with retain_graph=True, but as I read it, it would solve this issue. However, when retain_graph=True, this brings a new error, namely:<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2400, 300]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).<br>
How I solve the second error, I am not certain, looking for help.</p>

ptrblck: <p>Using <code>retain_graph</code> usually doesn’t solve the issue as it should only be used if you explicitly want to call <code>backward</code> multiple times.<br>
Based on the code I would guess <code>hidden</code> creates the issue as it’s not detached.<br>
Try to use:</p>
<pre><code class="lang-python">output, hidden = model(word.view(1,1), hidden.detach())
</code></pre>
<p>to detach the <code>hidden</code> state from the last iteration and check if this would solve the issue.</p>

geenidee005_geenidee: <p>Thank you, indeed detach was necessary(for h and c in my tuple hidden=(h,c)).</p>
Deploying onnx model with TorchServe

thisisjim2: <p>Hi, I am currently looking at ways to deploy ONNX model simply because inference speed is a lot faster. I took a look at TorchServe, which has many features that I would like in production (logging, batch inference, version control etc.). Does anyone know if it’s possible to deploy ONNX model with TorchServe?</p>

marksaroufim: <p>There’s an easy way to do this by just loading the model from a torchserve handler which are quite general in what you can use. There’s probably a better way to do this but at a high level solution would look something like</p>
<pre><code class="lang-auto">    def load_model(self, model_path):
        options = ort.SessionOptions()
        return ort.InferenceSession(model_path, options)
</code></pre>

Mohamed_Nabil1: <p>Hi Mark,<br>
That seems pretty plausible to me<br>
However I’m trying to use torch-model-archiver, would that simply work?</p>

marksaroufim: <p>Oh yeah you can just pass in an onnx file to the archiver like this</p>
<p><code>torch-model-archiver -f --model-name onnx --version 1.0 --serialized-file linear.onnx --export-path model_store --handler onnx_handler.py</code></p>
<p>This test might help <a href="https://github.com/pytorch/serve/blob/master/test/pytest/test_onnx.py#L62" class="inline-onebox" rel="noopener nofollow ugc">serve/test_onnx.py at master · pytorch/serve · GitHub</a> - you’ll still need to worry about preprocessing data to the ort runtime</p>
VHP with model parameters

umarbek_nasimov: <p>i am trying to use <code>torch.autograd.functional</code> vhp function with respect to model parameters. i defined a loss function that takes the model parameters as input but I am having trouble loading the parameters into the model (without inadvertently copying them).</p>
<p>given some parameter p, when i use <code>setattr(model, param_name, p)</code>, i get an error that says p needs to be wrapped in a Parameter. when i wrap it in a Parameter with <code>setattr(model, param_name, torch.nn.Parameter(p))</code>, p is copied so the reference to the original p is lost and vhp returns 0.</p>
<p>this is related to this old thread: <a href="https://discuss.pytorch.org/t/hvp-w-r-t-model-parameters/83520/5" class="inline-onebox">HVP w.r.t model parameters - #5 by dedeswim</a></p>
<p>thanks!</p>
How can I save best model state_dict during training?

pvtien96: <p>Hi,<br>
I want to get the best model state_dict as in the following code snippet:</p>
<pre><code class="lang-auto">def finetune(model, train_loader, val_loader, num_epochs, max_num_epochs, ori_acc, acc_drop_threshold):

    _, best_top1_acc, _ = validate(val_loader, model, criterion)
    best_model_state = model.state_dict()
    epoch = 0
    while epoch &lt; max_num_epochs:
        train(epoch, train_loader, model, criterion, optimizer, scheduler)
        _, valid_top1_acc, _ = validate(val_loader, model, criterion)

        if valid_top1_acc &gt; best_top1_acc:
            best_top1_acc = valid_top1_acc
            best_model_state = model.state_dict()

        acc_drop = ori_acc - best_top1_acc
        if (acc_drop &lt; acc_drop_threshold) and (epoch &gt; num_epochs-1):
            break

        epoch += 1

    model.load_state_dict(best_model_state)

    return model, epoch
</code></pre>
<p>I think that with the above code, the model always loads the best state dict (where it reached the best accuracy) before coming out of the function. But the problem is that when I test the code, the function always returns the model with the last state it got during the while loop.<br>
I don’t want to save the model to disk and then reload it because this’s just a small part of the program.<br>
Thank you. Any recommendation is appreciated!</p>

albanD: <p>Hi,</p>
<p>The <code>.state_dict()</code> method does not copy the parameters but returns a view into the ones in the model.<br>
So if you want to get an independent version (that will not be updated inplace by training), you need to deepcopy it: <code>best_model_state_dict = copy.deepcopy(model.state_dict())</code></p>
Different memory consumption on GPU when training models in parallel

neilkimn: <p>Hi,</p>
<p>I am experiencing strange GPU memory consumption by my training processes when training multiple of the same model in parallel. For instance, if I am training three ResNet18 models in parallel with a batch size of 50, the memory consumption for each training process is reported as follows, by nvidia-smi:</p>
<pre><code class="lang-auto">GPU-2022369e-2f16-0362-7dc3-ea36ded90774, 51077, 1840 MiB
GPU-2022369e-2f16-0362-7dc3-ea36ded90774, 51075, 1840 MiB
GPU-2022369e-2f16-0362-7dc3-ea36ded90774, 51076, 1840 MiB
</code></pre>
<p>If I instead increase the batch size past 70 to, for instance, 80, this is what nvidia-smi shows:</p>
<pre><code class="lang-auto">GPU-2022369e-2f16-0362-7dc3-ea36ded90774, 23441, 2428 MiB
GPU-2022369e-2f16-0362-7dc3-ea36ded90774, 23443, 2172 MiB
GPU-2022369e-2f16-0362-7dc3-ea36ded90774, 23442, 2428 MiB
</code></pre>
<p>I am wondering what causes this uneven utilization of GPU memory. I could imagine it would have something to do with maybe the GPU memory page size or memory swapping on GPU perhaps? It only happens once I surpass a certain threshold for the batch size - for my data that turns out to be a batch size of 70. I am running torch 1.13.1+cu117, CUDA 11.7 and NVIDIA driver 530.30.2.</p>
<p>Perhaps this issue could be resolved by running my desktop machine in headless mode, SSH to it and try again, since I do see a bunch of GPU processes consuming ~750MB of memory of the 8GB memory on my RTX 2080. Maybe my machine is trying to accomodate all of those processes + the training processes at once instead of throwing an OOM exception?</p>

neilkimn: <p>Running my machine headless, I can verify that the differences is due to the graphics processes on the GPU. It makes sense, although I am surprised that rather than strictly requiring the same memory for each process and OOM’ing if not possible, some memory swapping appears to be happening. I suppose this behavior is beyond PyTorch but an interesting observation to make, nonetheless.</p>
ValueError: invalid literal for int() with base 10: 'rdered/img_52'

Muhammad_Anwar: <p>I made some changes in the names of moving and fixed images, like I replaced fi with si, mi with ki, fl with si, ml with kl and then got this error, the further detail is<br>
File “/home/ubuntu/Anwar/XMorpher_reproduce/AU_train.py”, line 142, in <br>
RSTNet = ASeg(OASIS_img_dir, OASIS_seg_dir, LPBA40_img_dir, LPBA40_seg_dir, resized_shape=(80, 96, 112), n_channels=1, lr=1e-4, batch_size=1, data_random_seed=None, model_name=‘XMorpher’)<br>
File “/home/ubuntu/Anwar/XMorpher_reproduce/AU_train.py”, line 33, in <strong>init</strong><br>
self.train_OASIS = torch_Dataloader_OASIS(OASIS_img_dir, OASIS_seg_dir, ‘train’, resized_shape, batch_size, data_random_seed)<br>
File “/home/ubuntu/Anwar/XMorpher_reproduce/utilss/dataloader.py”, line 56, in torch_Dataloader_OASIS<br>
Dataset_OASIS = torch_Dataset_OASIS(img_dir, seg_dir, mode, resized_shape)<br>
File “/home/ubuntu/Anwar/XMorpher_reproduce/utilss/dataloader.py”, line 12, in <strong>init</strong><br>
self.img.sort(key=lambda x: int(x[43:-7]))<br>
File “/home/ubuntu/Anwar/XMorpher_reproduce/utilss/dataloader.py”, line 12, in <br>
self.img.sort(key=lambda x: int(x[43:-7]))</p>

ptrblck: <aside class="quote no-group" data-username="Muhammad_Anwar" data-post="1" data-topic="177848">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/muhammad_anwar/48/59375_2.png" class="avatar"> Muhammad_Anwar:</div>
<blockquote>
<p>I made some changes in the names of moving and fixed images, like I replaced fi with si, mi with ki, fl with si, ml with kl and then got this error</p>
</blockquote>
</aside>
<p>I don’t know how to understand this change, but the error is raised in <code>int(x[43:-7])</code> as it seems you are passing a string to it:</p>
<pre data-code-wrap="python"><code class="lang-python">x = 'rdered/img_52'
int(x)
# ValueError: invalid literal for int() with base 10: 'rdered/img_52'
</code></pre>
<p>so you might need to index the string in another way assuming you want to transform the <code>52</code> to an integer value e.g. via <code>int(x[-2:])</code>.</p>

Muhammad_Anwar: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I try and then reply</p>
Mismatched results for partitioned operators

ralzq01: <p>I found matmul result is inconsistent with partitioned-matmal result using torch APIs/</p>
<p>Consider the matmul calculation: Z = XY, which can be partitioned by Z = [X1 X2][Y1 // Y2] = X1Y1 + X2Y2, where X is partitioned by the second dimension and Y is partitioned by the first dimension. The results need to be reduced to get Z.</p>
<p>Following is the implementation and I run it with PyTorch 1.13 on one nvidia V100-32GB GPU, but it failed at assertation on with <code>atol=1e-2</code>.</p>
<p>Can anyone help explain this?</p>
<pre><code class="lang-py">import torch


def matmul(x, y):
    return torch.matmul(x, y)


def partitioned_matmual(x, y, num):
    xs = x.chunk(num, dim=1)
    ys = y.chunk(num, dim=0)
    out = torch.zeros(x.size(0), y.size(1), dtype=x.dtype, device=x.device)
    for x, y in zip(xs, ys):
        out += matmul(x, y)
    return out

if __name__ == '__main__':
    x = torch.randn(128, 128, dtype=torch.float16).cuda()
    y = torch.randn(128, 128, dtype=torch.float16).cuda()
    
    z = matmul(x, y)
    pz = partitioned_matmual(x, y, 2)

    print(z)
    print(pz)
    
    assert torch.allclose(z, pz, atol=1e-1)
    assert torch.allclose(z, pz, atol=1e-2)
    assert torch.allclose(z, pz, atol=1e-3)

</code></pre>

ptrblck: <p>These errors might be expected in <code>float16</code> and you could use a wider dtype to increase the precision.<br>
E.g. <code>float32</code> yields an <code>abs().max()</code> error of ~1e-5 and <code>float64</code> of ~1e-14.</p>

ralzq01: <p>Thank you! I’m curious that whether this mismatching is due to different ordering of multiplication and reduction in the underlying hardware?</p>

ptrblck: <p>Yes, numerical mismatches due to the limited floating point precision can be caused by a difference in the operation order as seen also in this small example:</p>
<pre><code class="lang-python">x = torch.randn(100, 100)
s1 = x.sum()
s2 = x.sum(0).sum(0)
s3 = x[torch.randperm(x.size(0))].sum()
print(s1 - s2)
# tensor(-5.7220e-06)
print(s1 - s3)
# tensor(1.1444e-05)
</code></pre>
How can i make this network using pytorch

alok_tripathi: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/69f7ee9276c77ff41ddabd70c5f54ec2e7ee840d.png" data-download-href="https://discuss.pytorch.org/uploads/default/69f7ee9276c77ff41ddabd70c5f54ec2e7ee840d" title="Screenshot 2023-04-18 at 12-50-43 1-s2.0-S2215098621001427-main.pdf"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69f7ee9276c77ff41ddabd70c5f54ec2e7ee840d_2_377x500.png" alt="Screenshot 2023-04-18 at 12-50-43 1-s2.0-S2215098621001427-main.pdf" data-base62-sha1="f7rkduBWbpwg5W2glXakLPqJpp3" width="377" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69f7ee9276c77ff41ddabd70c5f54ec2e7ee840d_2_377x500.png, https://discuss.pytorch.org/uploads/default/original/3X/6/9/69f7ee9276c77ff41ddabd70c5f54ec2e7ee840d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/9/69f7ee9276c77ff41ddabd70c5f54ec2e7ee840d.png 2x" data-dominant-color="D7E1E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-04-18 at 12-50-43 1-s2.0-S2215098621001427-main.pdf</span><span class="informations">479×634 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

sinatv52: <p>Hi</p>
<p>This is somehow a simple UNet implementation, you can use available great UNet tutorials like <a href="https://www.youtube.com/watch?v=IHq1t7NxS8k&amp;list=PLhhyoLH6IjfxeoooqP9rhU3HJIAVAJ3Vz&amp;index=42" rel="noopener nofollow ugc">This</a> to code your model in Pytorch.<br>
The main difference might be your input and output image where you should add noise to have those noisy inputs.</p>

alok_tripathi: <p>do you have any template code implementation regarding the same? I am taking ultrasound nerve segmentation dataset for speckle noise removal from ultrasound images using batch renormalization.</p>
How to provide ctx for torch native functions manually

zisisnotzis: <p>I’m implementing a reversible convolution layer, so that forward pass does not need to save <code>x</code>.</p>
<p>I have the inverse function already implemented. I’m wondering how do I reuse the original implementation of conv backward, without “re-implementing” it in python, but provide <code>ctx</code> manually this time.</p>
<p>Or is there any way to add “hooks” to these pytorch layers, that will “remove” certain <code>ctx</code> immediately after forward, and “insert” it back before backward</p>

ptrblck: <p>You could call the backward methods via:</p>
<pre><code class="lang-python">torch.nn.grad.conv2d_input
&lt;function torch.nn.grad.conv2d_input(input_size, weight, grad_output, stride=1, padding=0, dilation=1, groups=1)&gt;

torch.nn.grad.conv2d_weight
&lt;function torch.nn.grad.conv2d_weight(input, weight_size, grad_output, stride=1, padding=0, dilation=1, groups=1)&gt;
</code></pre>