akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
ValueError: expected 4D input (got 3D input)

Aleksandr_Samofalov: <p>Hello, guys! I am trying to check my trained model, but when I run my code I got a mistake which I can`t solve for few hours.</p>
<p>Maybe, someone have ideas what is wrong?</p>
<blockquote>
<p><a class="hashtag-cooked" href="/c/data/37" data-type="category" data-slug="data" data-id="37"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>data</span></a> preparation<br>
preprocess = transforms.Compose([<br>
transforms.Resize(size=(64, 64)),<br>
transforms.ToTensor(),<br>
transforms.RandomRotation(degrees = 5),<br>
transforms.Normalize(mean=[0.485, 0.456, 0.406],<br>
std=[0.229, 0.224, 0.225])<br>
])</p>
<p>def show_images(img):<br>
plt.imshow(transforms.functional.to_pil_image(img))<br>
plt.show()</p>
<h1>Check my own x-ray chest image</h1>
<p>image_valid = Image.open(‘Samofalov_xray_chest.jpeg’).convert(‘RGB’)<br>
image_valid = preprocess(image_valid)</p>
<p>device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)</p>
<p>image_valid = image_valid.to(device)<br>
model.to(device)<br>
model.eval()<br>
valid_result = model(image_valid)</p>
<p>if valid_result == 1:<br>
print(f’Model says it is PNEUHMONIA’)<br>
elif valid_result == 0:<br>
print(f’Model says it is NORMAL’)<br>
else:<br>
print(‘Something wrong with binary classification - check twice’)</p>
<p>show_images(image_valid)</p>
</blockquote>

Matias_Vasquez: <p>Your model expects an input with 4 dimensions which correspond to <code>BxCxHxW</code> = (Batch x Channel x Height x Width).</p>
<p>Since you are testing it with only one image, you are missing the Batch (<code>B</code>) dimension.</p>
<p>To solve this, you can add this dimension by using <code>unsqueeze</code></p>
<pre><code class="lang-python">valid_result = model(image_valid.unsqueeze(0))
</code></pre>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

Xinxin_Li: <p>It helps! Thank you very much!</p>

xinghao: <p>thank youIt helps! Thank you very much!</p>
Multi GPU training using a fixed model

Shanka_Subhra_Mondal: <p>I am trying to train on multiple GPUs over multiple nodes using distributed data parallel (DDP). Specifically, I have two models A and B. Model B uses the output of A to perform the task. I have model A trained already using a single GPU and I don’t want to train it further. However, I want to train model B using the outputs of model A on a larger batch size so I need to set up both models using DDP. However, DDP raises an error (<a href="https://gessisc.com/click?redirect=https%3A%2F%2Fgithub.com%2Fpytorch%2Fpytorch%2Fissues%2F25550&amp;dID=1659725312121&amp;linkName=https://github.com/pytorch/pytorch/issues/25550" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/25550</a>) when a module has no parameter that requires gradient. Please let me know on how to implement this properly. Thanks in advance.</p>
Best way to implement validation / train split using torchdata?

Vedant_Roy: <p>I’m using torchdata, and I was wondering what the best way was to implement a validation/train split.</p>
<p>Currently, I’m using <code>fork</code> + <code>header</code> to split the datapipe into 2 and take the first N samples for the validation dataset, then I use <code>enumerator</code> + <code>filter</code> to skip those same N samples from the training dataset but I wonder if there’s a better way.</p>
<p><strong>Edit</strong>: I was also wondering if there was a good way to set the length of a data pipe, if we know its length ahead of time. (For example: maybe we know the number of files in a directory).</p>

tom: <p>This isn’t exactly what you asked, but to me, validation/train (/test?) split is more a (crucial!) processual/organization topic than a pure data loading one.</p>
<p>In particular, I would highly recommend to define the split ahead of time (early on) and record it (e.g. as a set of .csv files) rather than doing it on the fly. This gives you the opportunity to easily re-use the same split, audit the split for inadvertent information leaks etc.</p>
<p>Best regards</p>
<p>Thomas</p>

nivek: <p>One possible solution right now is to use:</p>
<pre><code class="lang-python">train_dp, eval_dp = dp.demux(num_instances=2, classifier_fn=rand_fn)
</code></pre>
<p>where your classifier function will use a RNG and you will have to reset the seed of the RNG after each epoch (or at the beginning).</p>
<p>We are currently working on a DataPipe that can do that more easily. If that doesn’t meet your use or you have more feature request, feel free to upvote or comment on this <a href="https://github.com/pytorch/data/issues/712" rel="noopener nofollow ugc">GitHub issue</a>.</p>

nivek: <p>Also, the easiest way to set a length of DataPipe is using <a href="https://pytorch.org/data/main/generated/torchdata.datapipes.iter.Header.html?highlight=header" rel="noopener nofollow ugc"><code>.header(length)</code></a>.</p>
Multi-gpu training hangs due to an `if`

TigerYan86: <p>Hi,</p>
<p>I discovered recently my 8-GPU training will hang if I have this <code>if</code> (using DDP, all GPUs saturate at 100%, happens randomly at some epoch in the middle of a job):</p>
<p>(modification of <code>backproject()</code> from <a href="https://github.com/magicleap/Atlas/blob/master/atlas/model.py#L77-L78" class="inline-onebox" rel="noopener nofollow ugc">Atlas/model.py at master · magicleap/Atlas · GitHub</a>)</p>
<pre><code class="lang-auto">    ...
    volume = torch.zeros(
        batch, channels, nx * ny * nz, dtype=features.dtype, device=device
    )
    # `valid` shape: [b, nx*ny*nz]
    if valid.any():
      for b in range(batch):
          volume[b, :, valid[b]] = features[b, :, py[b, valid[b]], px[b, valid[b]]]

    volume = volume.view(batch, channels, nx, ny, nz)
    valid = valid.view(batch, 1, nx, ny, nz)

    return volume, valid
</code></pre>
<p>after removing the <code>if</code> my model trains well. The purpose of the if was to avoid unnecessary index to save time. I do know this might cause GPUs execute different graph and diverge between samples, but is that the main reason? Is it generally not encouraged to have such data-dependent branching code in training? (I found post like <a href="https://discuss.pytorch.org/t/conditional-computation-that-saves-computation/14878">this</a> that obviously uses <code>if</code> which could lead to even more divergence I assume.)</p>
<p>Any insights are appreciated!</p>

Yanli_Zhao: <p>do you want to try passing ‘find_unused_parameters=True’ in the DDP wrapper? it is trying to support different graph training across ranks. But it is possible your case is a edge case and can not be supported by DDP</p>

TigerYan86: <p>thanks Yanli. Let me give it a try.</p>

TigerYan86: <p>in fact that flag is by default <code>True</code> in pytorch-lightning (which I’m using), and it only outputs warning:</p>
<pre><code class="lang-auto">find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters, consider turning this flag off. Note that this warning may be a false positive your model has flow control causing later iterations to have unused parameters.
</code></pre>
<p>so looks like there is no unused parameters.</p>
Virtual batch sizing with multi GPU training

timudk: <p>Hi there,</p>
<p>I was wondering if virtual batch sizing (as explained, for example, in <a href="https://opacus.ai/tutorials/building_text_classifier" class="inline-onebox" rel="noopener nofollow ugc">Opacus · Train PyTorch models with Differential Privacy</a>) is supposed to work with multi-GPU training? For me, virtual batch sizing work well (even if model is wrapped in DifferentiallyPrivateDistributedDataParallel) when using a single GPU but fails when using multiple GPUs. The first couple iterations work fine but at some point the <code>self.pre_step():</code> in <code>DistributedDPOptimizer</code> returns True for one rank while returning False for the other rank. The code then gets stuck because the first optimizer takes a “real step” while the second doesn’t.</p>
<p>EDIT: After further investigation it seems that the problem is the following. Due to Poisson sampling, the <code>BatchSplittingSampler</code> may split a batch into <code>n</code> physical batches, where <code>n</code> is not divisible by the number of GPUs used.</p>

karthikprasad: <p>Hmmm. That does sound like a bug. If not solve, Opacus should, at the very least, warn the users about this. Thanks for flagging <a class="mention" href="/u/timudk">@timudk</a> , could you please open an issue on github? <a href="https://github.com/pytorch/opacus/issues/new?assignees=&amp;labels=&amp;template=bug-report.md" class="inline-onebox" rel="noopener nofollow ugc">Sign in to GitHub · GitHub</a></p>
Worker_init_function deprecated in DataLoader

srishti-git1110: <p>Hi, I read the <a href="https://pytorch.org/docs/stable/data.html#multi-process-data-loading" rel="noopener nofollow ugc">docs</a> and understood that we do not need to worry about data duplication with map-style datasets. But, with iterable-style need to be careful of data duplication, and so use of <code>worker_init_function</code> is recommended in the <code>DataLoader</code>.</p>
<p>First, did I understand it right?</p>
<p>Secondly, when I was trying the <code>IterDataPipes</code> from <code>TorchData</code>, I used the <code>worker_init_function</code> argument in the DataLoader to which PyTorch gave a deprecation warning.</p>
<blockquote>
<p>/usr/local/lib/python3.7/dist-packages/torch/utils/data/backward_compatibility.py:4: UserWarning: Usage of backward_compatibility.worker_init_fn is deprecated as DataLoader automatically applies sharding in every worker<br>
warnings.warn(“Usage of backward_compatibility.worker_init_fn is deprecated”<br>
/usr/local/lib/python3.7/dist-packages/torch/utils/data/backward_compatibility.py:4: UserWarning: Usage of backward_compatibility.worker_init_fn is deprecated as DataLoader automatically applies sharding in every worker<br>
warnings.warn(“Usage of backward_compatibility.worker_init_fn is deprecated”</p>
</blockquote>
<p>So, does this mean we do not need to worry about duplication with <code>IterDataPipes</code> as sharding is automatically applied in every worker according to this warning?</p>
<p>If yes, why do we then need to chain the <code>sharding_filter()</code> datapipe? Isn’t the purpose of <code>sharding_filter()</code> to make sure sharding is done in every worker process?</p>
<p>Thanks in advance!</p>

nivek: <p>Prior to the latest TorchData release (0.3.0 or before), you needed both <code>backward_compatibility.worker_init_fn</code> for DataLoader and <code>.sharding_filer()</code> in your pipeline for the sharding to work properly in every worker process.</p>
<p>You no longer need to import and use <code>backward_compatibility.worker_init_fn</code> in the latest TorchData version (0.4.0) because of a change to <code>DataLoader</code>. However, you still need to add <code>.sharding_filer()</code> in your data pipeline in the location when/where you would like to shard your data. See this <a href="https://pytorch.org/data/main/tutorial.html#working-with-dataloader" rel="noopener nofollow ugc">tutorial</a> for an example.</p>
Dist.all_reduce_multigpu NCCL error messages

taipeiallen15: <p>Hey all,<br>
I am training my data using Distributed Data Parallel.<br>
I have two separate machines and have 2 GPUs on each machine.<br>
I’ve tried the codes in the documents like this</p>
<pre><code class="lang-auto">        tensor_list = []
        for dev_idx in range(torch.cuda.device_count()):
            tensor_list.append(torch.FloatTensor([1].cuda(dev_idx))

        dist.all_reduce_multigpu(tensor_list)
</code></pre>
<p>But got some error messages :</p>
<p>RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1656352465323/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, invalid usage, NCCL version 2.10.3<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).</p>
<p>Can anyone help me out with this?</p>
Would dataloader take the advantage of multiple workers if setting Shuffle = False?

chengzegang: <p>I am recently working on contrastive learning and use one dataloder to load data randomly for negative samples and one unshuffled so I can keep loading data from the same categories. In my understanding, multi workers run in parallels so the dataloader can pick up whatever batch that is ready at the moment. So if I have to keep the order of batches, it will generally slow down the loading process. I knew there is better and simple implementation for positive sampling that can goarround this issue, it is just my curiosity taking me here.</p>
<p>Btw, the data are un-preprocessed images, variies in size. And I cant preprocess and save them into my own place cuz its imagenet and its read only on that host.</p>
OOM when using multiprocessing (even though memory is there)

kl_divergence: <p>I’m getting the following error:</p>
<p>RuntimeError: [enforce fail at CPUAllocator.cpp:65] . DefaultCPUAllocator: can’t allocate memory: you tried to allocate 2454416848 bytes. Error code 12 (Cannot allocate memory)</p>
<p>Here is the relevant part of the code that triggered it:</p>
<p><code>from multiprocessing import Pool</code></p>
<p><code>pool = Pool(70)</code></p>
<p><code>start = time.time()</code></p>
<p><code>print("length of lines: ", len(lines))</code></p>
<p><code>line_tensors = pool.map(process, lines)</code></p>
<p><code>print("lines processed")</code></p>
<p><code>print(time.time() - start)</code></p>
<p><code>line_tensors = [x for x in line_tensors if x is not None]</code></p>
<p><code>size = 0</code></p>
<p><code>for l in line_tensors:</code></p>
<p><code>size += l.shape[0]</code></p>
<p><code>print("total size: ", size)</code></p>
<p><code>megatensor = torch.cat(line_tensors, dim=0).flatten()</code></p>
<p>The total size that is printed is 306802106. The error occurs on the last line:</p>
<p><code>Traceback (most recent call last):</code></p>
<p><code>File "bert.py", line 73, in &lt;module&gt;</code></p>
<p><code>megatensor = torch.cat(line_tensors, dim=0).flatten()</code></p>
<p><code>RuntimeError: [enforce fail at CPUAllocator.cpp:65] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 2454416848 bytes. Error code 12 (Cannot allocate memory)</code></p>
<p>I thought it’s OOM, but I’m pretty sure I have 2.5G on my machine. This similar (?) piece of code also seems to run fine:</p>
<p><code>import torch</code></p>
<p><code>x = []</code></p>
<p><code>for i in range(100):</code></p>
<p><code>x.append(torch.randint(1, 10000, (100000000,)))</code></p>
<p><code>print(i)</code></p>
<p><code>mega = torch.cat(x, dim=0).flatten()</code></p>
<p>Any thoughts on what might be wrong here?</p>

ptrblck: <p>Maybe you are running out of shared memory, so check if your system limits it and increase it if needed.</p>
Floating point values representing images

AjeelAhmed1998: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65b564ac5890a381c17eca77e8f02daa448454b4.png" data-download-href="https://discuss.pytorch.org/uploads/default/65b564ac5890a381c17eca77e8f02daa448454b4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65b564ac5890a381c17eca77e8f02daa448454b4_2_570x500.png" alt="image" data-base62-sha1="evKRbgWcWoBhOmtIuMXBRqXI6Es" width="570" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65b564ac5890a381c17eca77e8f02daa448454b4_2_570x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65b564ac5890a381c17eca77e8f02daa448454b4_2_855x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/5/65b564ac5890a381c17eca77e8f02daa448454b4.png 2x" data-dominant-color="E7E8E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">891×781 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I’m normalizing images I use the ToTensor() function which squeezes all the values between 0 and 1.<br>
In the image you can see I want to see some images in the batch so I unnoramlized it but didn’t undo my ToTensor() function, yet it’s showing images just fine.<br>
My point is that the pixels are not integers anymore and are actually decimal values between 0 and 1, how are they still be able to display images?</p>

JuanFMontesinos: <p>Matplotlib expects uint8 images or float values between 0 or 1.</p>

AjeelAhmed1998: <p>okay but we know that pixels are represented in terms of integer values of [0,256), what happens inside matplotlib? Does it convert them back from decimal values to to a range of 0 to 256?</p>

JuanFMontesinos: <p>I’m afraid I don’t know the source code of matplotlib as it’s a super complex library.<br>
I neither think it’s relevant at all. Matplotlib’s main purpose is creating graphics, and your eye cannot really distinguish the difference between values of 255 and 253 (for example).</p>
<p>Anyway, if it’s really important to you, I think you can ask in a more specialized forum such us Stack Overflow  or Superuser.</p>

AjeelAhmed1998: <p>fair enough, thank you.</p>

ptrblck: <p>I think <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> is correct and <code>matplotlib</code> would rescale the image to the standard <code>uint8</code> format before clipping invalid values. You could check the source code (e.g. with a debugger) and should see some calls into e.g. <a href="https://github.com/matplotlib/matplotlib/blob/149a0398b391cfc4eddb5e659f50b3c66f32ea65/lib/matplotlib/image.py#L1240">this transformation</a> and <a href="https://github.com/matplotlib/matplotlib/blob/149a0398b391cfc4eddb5e659f50b3c66f32ea65/lib/matplotlib/image.py#L713-L724">this clipping</a>.</p>
How to return torch model weights from endpoint Flask

amlarraz: <p>Hi all!</p>
<p>I’m developing a project in PyTorch which requires returning the weights (state_dict) of a pytorch model as a Flask endpoint response. To better explain it, the simplest code could be:</p>
<pre><code class="lang-auto">@app.endpoint('/send_weights', methods=['GET', 'POST'])
def send_weights():
    model_weights = model.state_dict() # It is a dict[str, torch.tensor]
    return model_weights
</code></pre>
<p>However it is not as simple because the <em>torch.tensor</em> is not JSON serializable, so, I’ve tried to convert them to a list (JSON serializable object) and it works:</p>
<pre><code class="lang-auto">@app.endpoint('/send_weights', methods=['GET', 'POST'])
def send_weights():
    model_weights = model.state_dict() # It is a dict[str, torch.tensor]
    model_weights = {k:v.tolist() for k,v in model_weights.items()}
    return model_weights
</code></pre>
<p>However this process is very slow and it doesn’t meet my requirements. I was trying to convert the tensors to bytes but the code gives the same problem, bytes is not JSON serializable. So, I’m thinking that the json response won’t be the solution. I’m not an expert in Flask but I’ve read about the flask <a href="https://tedboy.github.io/flask/generated/flask.send_file.html" rel="noopener nofollow ugc"><em>send_file</em></a> method, however I not sure how to use it in this case (not even sure this would be a possible solution), I haven’t got a mimetype for the dictionary.</p>
<p>Does anybody know a better way to do this?</p>

amlarraz: <p>I’ve just found a solution. It is based on return the weights as a file using the <em>send_file</em> method from flask with the weights saved on a binary using <em>torch.save</em> and the mimetype '<em>application/octet-stream’</em> as appears in <a href="https://stackoverflow.com/questions/6783921/which-mime-type-to-use-for-a-binary-file-thats-specific-to-my-program" rel="noopener nofollow ugc">this question</a>. The final endopint code will be:</p>
<pre><code class="lang-auto">@app.endpoint('/send_weights', methods=['GET', 'POST']) 
def send_weights():
    model_weights = model.state_dict()
    to_send = io.BytesIO()
    torch.save(model_weights, to_send, _use_new_zipfile_serialization=False)
    to_send.seek(0)
    return send_file(to_send, mimetype='application/octet-stream')
</code></pre>
<p>And, to load it in the other side:</p>
<pre><code class="lang-auto">weights = torch.load(io.BytesIO(response.content))
</code></pre>
<p>Hope it will help somebody.</p>
Modified bessel function arbitrary order

Wesley_Neill: <p>I am looking for a PyTorch analog to <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.iv.html#r3a80f286b9e7-1" rel="noopener nofollow ugc"><code>scipy.special.iv</code></a> which excepts an arbitrary order as one of its arguments. I only see fixed 0th and 1st order implementations in <code>torch.special</code>’s docs. Does anyone know if this analog exists elsewhere?</p>

KFrank: <p>Hi Wesley!</p>
<aside class="quote no-group" data-username="Wesley_Neill" data-post="1" data-topic="158733" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wesley_neill/48/25709_2.png" class="avatar"> Wesley_Neill:</div>
<blockquote>
<p>I only see fixed 0th and 1st order implementations in <code>torch.special</code>’s docs.</p>
</blockquote>
</aside>
<p>I am not aware of any pytorch implementation of higher Bessel functions.</p>
<p>If you need gradients (backpropagation), you could wrap scipy’s <code>iv()</code> in<br>
a <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="noopener nofollow ugc">custom autograd Function.</a>  You would (presumably using scipy again)<br>
implement the derivative of <code>iv()</code> in your <code>Function</code>’s <code>backward()</code> method.</p>
<p>This would come at the cost of not using the gpu for your Bessel-function<br>
computations.  (For convenience, you could have your custom <code>Function</code><br>
move relevant tensors from the gpu to cpu as necessary so that it would<br>
work with gpu tensors, but the computations would still be being done on<br>
the cpu.)</p>
<p>Best.</p>
<p>K. Frank</p>

Wesley_Neill: <p>Thank you. I think I need to stick with an implementation that will allow GPU usage. I’m trying to implement a power series expansion in PyTorch instead. I’m not sure how well batching is going to work out though. If that doesn’t work I’ll have to get help from our resident C++ expert to help me dive under the hood.</p>
How to create a dataset containing one datapoint and its augmentations?

KimiaKN: <p>Hi,<br>
I have an imagenet dataset. I want to extract only one image of the dataset and apply random augmentations on this <strong>single</strong> data point. So basically the transform I define is as following:</p>
<pre><code class="lang-auto">sample_transform = transforms.Compose([
        transforms.RandomResizedCrop(img_size, scale=(0.2, 1.0)),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(mean=_IMAGENET_RGB_MEANS, std=_IMAGENET_RGB_STDS),
    ])
</code></pre>
<p>And assume that my one sample is as follows:</p>
<pre><code class="lang-auto">from torch.utils.data import Subset
def extract_sample(data, idx):
    return Subset(data, [idx])
single_sample = extract_sample(data, idx = 0)
</code></pre>
<p>Not I want to create a dataset by applying <code>sample_transform</code>, 64 times on <code>single_sample</code> (in order to get 64 different augmentations of <code>single_sample</code>). How can I do that?</p>
<p>Thanks in advance for your help!</p>

ptrblck: <p>You can just iterate 64 times and apply <code>sample_transform(single_sample)</code> inside the loop.</p>

KimiaKN: <p>Thanks!<br>
Could you also tell me how I can concatenate them in a single final dataset?</p>

ptrblck: <p>You could wrap <code>list</code> of the transformed sample in e.g. a <code>TensorDataset</code> afterwards:</p>
<pre><code class="lang-python">samples = []
for _ in range(64):
    for sample in single_sample:
        samples.append(sample)
        
samples = torch.stack(samples)
print(samples.shape)
# torch.Size([64, 3, 32, 32])

dataset = torch.utils.data.TensorDataset(samples)
</code></pre>
Video annotation software

franfram: <p>Hello everyone,<br>
For my PhD, I’m building an accelerometer data decoder using covnets. But first I need to annotate videos of animal behaviour, and I’d like to have some software or code (R, python or julia) that allows me te predefine the classes of behaviours to annotate, assign them to numbers on my keyboard, and then just press those buttons as I watch the video to annotate it. And ideally that would yield a csv with the times of when each behaviour happened. There’s nothing on google that I could find that fits this description.</p>
<p>Any suggestiones are greatly appreciated!</p>
<p>Thanks</p>
Prediction thresholding - avoid scattered results

toobeeh: <p>Hey!<br>
I have a model for binary semantic segmentation for cables.<br>
The results good enough for now:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afb002c05ebfb511ae3b36f96482a9c1404beb19.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/afb002c05ebfb511ae3b36f96482a9c1404beb19" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/afb002c05ebfb511ae3b36f96482a9c1404beb19_2_534x499.jpeg" alt="image" data-base62-sha1="p4cEPIMT7p3X3b89yJ2x5gE0x9D" width="534" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/afb002c05ebfb511ae3b36f96482a9c1404beb19_2_534x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/afb002c05ebfb511ae3b36f96482a9c1404beb19_2_801x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/f/afb002c05ebfb511ae3b36f96482a9c1404beb19.jpeg 2x" data-dominant-color="535152"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">928×868 72.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Wehen i create the masks from the predictions, i apply a threshold to get a binary image.<br>
As it can be seen here, the contours by the contours found by opencv are rather scattered; i can’t find a definite threshold that is “perfect” for all images.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6bdb4095a00c64640aeaa09fe42289a84a6164c.png" alt="image" data-base62-sha1="q4BkTq7ZUvZhat1gaNRAoXeKlVq" width="562" height="235"></p>
<p>My question is, maybe is there a library available that processes the predictions in a manner that exceeds the pure thresholding?<br>
Like, finding contours depending on the maybe a bit too low values next to definitive truths?</p>
<p>I hope i could explain my problem well, thank you in advance!</p>
Num classes in dataset

maxrivera: <p>How do I find the num classes in a PyTorch dataset? I expect it to be a certain number but I want to be able to verify after I loaded the data with ImageFolder. I currently only know how to find the length of the dataset.</p>

mxahan: <p>Can you please check the unique value in the labels?</p>

manudwd: <p>You can get the classes by using the <code>.classes</code> attribute</p>
<p>For example if you’ve loaded to the train data using ImageFolder:</p>
<pre><code class="lang-auto">train_set = datasets.ImageFolder(os.path.join(data_dir, "train"),
                                         data_transforms["train"])

train_set.classes 
</code></pre>
<p>Will output the unique values in your training/test/val set</p>

maxrivera: <p>Thank you that works!</p>
Dead Kernal when loading a quantized model

manudwd: <p>Hi,</p>
<p>I’ve a pretrained quantized model which I trained on Colab, I moved the files on my system to run ONNX runtime inference. When loading the model however with</p>
<pre><code class="lang-auto">quantized_model = torch.load('quantizedmodel.pt')
</code></pre>
<p>My kernel proceeds to die, non-quantized models seem to load just fine.</p>
<p>My torch version is ‘1.11.0’, one thing I’ve done different is that I was mapping the model to CUDA on Colab, however I was mapping to the device instead with <code>map_location = "cpu:0"</code>, I tried changing it back to cuda device with <code>cuda:0</code> instead to no development.</p>
<p>Running <code>nvidia-smi </code> gives me:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b30258b0ecbe6e940288184a09c504db8a69af3.png" alt="image" data-base62-sha1="fief4lzvikKbgfjlO9IikwTt7X5" width="656" height="181"></p>
<p>Which seems to check out. Looking for some help with debugging this.</p>

ptrblck: <p>Try to run the script with <code>gdb</code> and see what the <code>backtrace</code> points to after the crash.</p>

Zafar: <p><a class="mention" href="/u/manudwd">@manudwd</a> Could you provide a small script that reproduces that?</p>

manudwd: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I’m not really familiar with debugging using gdb so I pretty much ended up getting stuck there. But the solution seemed to be pretty simple. I’ll add <a class="mention" href="/u/zafar">@Zafar</a> here as well so you both know, the problem seemed to be with my conda environment itself rather than anything with torch, there seemed to be issues with meta data generation due to inconsistent numpy packages (this is just a guess but after hours of me debugging my packages that was the only outlier I found). After starting a clean environment, the quantized model seemed to load just fine. Thanks!</p>
Profilig Memory Usage

dnlwbr: <p>Hey everybody,</p>
<p>I am currently trying to figure out how much memory different models need for the forwardpass on the CPU (I know GPU is much faster ;)). I came across the <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" rel="noopener nofollow ugc">PyTorch Profiler</a>, but I have problems to interpret the results.</p>
<p>With</p>
<pre><code class="lang-auto">    with profile(activities=activities, record_shapes=True, profile_memory=True) as prof:
        with record_function("model_inference"):
            model.eval()
            prediction = model([inp])
    print(prof.key_averages().table(top_level_events_only=True, row_limit=10))
</code></pre>
<p>I get the following output:</p>
<p>===============================================================================<br>
This report only display top-level ops statistics</p>
<hr>
<pre><code>                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls  

                            aten::zeros         0.00%      58.000us         0.31%      18.145ms       1.008ms   98039.65 Kb     -19.59 Kb            18  
                        model_inference         3.15%     182.338ms       100.00%        5.788s        5.788s  592648.17 Kb  -1799765.56 Kb             1  
</code></pre>
<hr>
<p>Self CPU time total: 5.788s</p>
<p>I know that Self CPU refers only to the particular function without the child processes and CPU total includes them. With memory I think it will be analogous. However, CPU Mem states 592648.17 Kb, while including the child processes 1799765.56 Kb have been released. What does this mean exactly? Can I say that a total of 1799765.56 Kb was needed or how do I get the total memory needed from the whole model? Is this a reasonable way at all, or should I rather use psutil (e.g. psutil.Process().memory_info().vms)? However, here the copmlete Python process is specified including imported libraries as far as I know.</p>
<p>I would be happy if someone can help me. Thank you!</p>
How to observe the affects of quantisation

puranjay_mishra: <p>Hello everyone<br>
I went through the documentation and blogs on pytorch but did not find a way to compare the model before and after quantisation (specially the weight compression aspect of it). It would also be really helpful if the method to compare works on custom quantisation implementations as this is something I desire.<br>
Thank you!</p>

Zafar: <p>Could you, please, provide an example of what you are doing?</p>
<p>There are some ways to see the effects: You can use the <a href="https://pytorch.org/tutorials/prototype/numeric_suite_tutorial.html" rel="noopener nofollow ugc">NS</a> to see what happens to the weights during the quantization or you could just save the model and compare the sizes.</p>

puranjay_mishra: <p>Hey. I am trying to produce a quantized VIT model for image classification. I wanted tools for comparing the difference in space complexities and time required for the forward propagation of the model. I have figured it out though , so thank you! I will also look into NS, seems like something that would be helpful for similar tasks. Good day! <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
Early stopping - Which model to use

Caipi: <p>Hey fellows,</p>
<p>just a quick question. I trained a network with an early stopping patience of five with the validation loss as the dependent metrics.</p>
<p>Now the network stopped at 38 epochs but what should I use as the final model?</p>
<p>The best from validation loss point of view of course is epoch 33 (38-5) but from the training loss point of view the last one…</p>
<p>Which one to use for the presentation of the numerical and visual results?</p>

mxahan: <p>I would suggest, using the last one. Did you save the best model? Sometimes, it may be due to the particular data samples that provided the best result.</p>

Caipi: <p>I saved all models because I was unsure about this certian point…</p>
<p>The thing is the validation is closer to the test set than the training set because the model does not learn on it… So wouldn’t I choose the best validation performance to achieve better test accuracy?</p>

jayz: <p>If you care about test set performance, you should go with the best validation loss model, because at this point the model generalizes to unseen samples best. If you simply care to show that your model can (over-)fit any dataset, you should go with the lowest training loss model. But presenting it’s performance will simply show that your model (with enough parameters) can find some local optimum that fits the training set well. However, there is no guarantee then that the model “understands” the true underlying structure of the data.</p>
backward() takes 4 positional arguments but 5 were given

gpwiggy: <p>I need help with my .backward() method as I don’t know how to fix it. I believe it is due to the custom loss function taking as input a shape of (1001, 1, 3) (where 1001 is the batch_size) and outputting a shape of (1) (as the loss function acts over the batches given), but I can’t be sure. Is there a way to fix the error of <code>TypeError: backward() takes 4 positional arguments but 5 were given</code>? My code is below:</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
# model is designed to have input (batch_size, 1, 3) and output (batch_size, 1, 3)
# I would like the loss function to consider elements of the batch given, too
# loss_fn take (x, y), each of shape (batch_size, 1, 3) 
# and outputs (d1, d2, idx1, idx2) which are (float, float, int, int)

def train_loop(model, loss_fn, optimizer, epochs=1000, source=source_shape, target=target_shape):
    # source_shape = target_shape = (1001, 1, 3)
    for epoch in range(epochs):
        d1, d2, idx1, idx2 = loss_fn(model(source), target)
        loss = (torch.mean(d1)) + (torch.mean(d2))
        
        # backprop
        optimizer.zero_grad()
        loss.backward() # &lt;- here is where the error is
        optimizer.step()
</code></pre>
<p>The loss function is Chamfer’s distance, taken from a Github <code>chamfer_distance</code> library. How it’s computed is that for two point clouds Sx Sy, (say, each of shape n, 1, 3), for each point in Sx find the minimal L2 distance to any point in Sy, summing this over points in Sx (this is how <code>d1</code> is calculated in my code above). The same is then done for points in Sy, and the two sums are added to get the final loss.</p>

ptrblck: <p>Could you explain the implementation of the custom loss function in more detail or post its definition, please?</p>

gpwiggy: <p>My post has just been edited to correct for this.</p>
Can we add shape check when rewrite the computation graph in torchScript?

jiahuil: <p>I want to rewrite the computation graph, transform the origin op to custom op, but my custom op has tensor shape restriction, can we add shape check when rewrite?</p>

AlphaBetaGamma96: <p>Can you share a minimal reproducible example so people can debug your problem?</p>

jiahuil: <pre><code class="lang-auto">pattern = """ 
    graph(%x,%1,%2,%3): 
        %0:Tensor=aten::index_select(%x,%3,%1) 
        %out:Tensor=aten::mul(%0,%2)
        return(%out) 
""" 
replacement = """ 
    graph(%x,%1:Tensor,%2,%3): 
        %out:Tensor=index_mul::index_mul(%x,%2,%1)
        return(%out) 
""" 
torch._C._jit_pass_custom_pattern_based_rewrite_graph(pattern, replacement,graph)
</code></pre>
<p>For operation  <code>index_mul::index_mul(%x,%2,%1)</code>, it wants %x to be 2 dim tensor, %2: 2 dim tensor, %1:  1 dim tensor, can we add shape check in subgraph rewrite?</p>
Loading GPU trained checkpoint to cuda device doesn't utilise GPU for testing

Steven_Tuften: <p>Have trained a U-Net model for image segmentation on my win64 environment with:<br>
NVIDIA RTX 2060<br>
Python 3.8.13<br>
CUDA Version: 11.3<br>
PyTorch Version: 1.12.1<br>
PyTorch Lightning Version: 1.5.10</p>
<p>I saved the checkpoint files and the training fully utilised my GPU.<br>
It took about 2hrs per epoch to train the model.</p>
<p>I then loaded my model from the final checkpoint and used map_location=torch.device(‘cuda’) .<br>
I then ran the test , using pytorch lighting Trainer.test().</p>
<p><strong>Estimated time for 1 epoch 58hrs.</strong><br>
The GPU Cuda is 0% but the copy area on the card is at 80% constantly.</p>
<p>Not sure what is happening.<br>
Probably a newbie issue <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_diagonal_mouth.png?v=12" title=":face_with_diagonal_mouth:" class="emoji" alt=":face_with_diagonal_mouth:" loading="lazy" width="20" height="20"> but any insight most appreciated!</p>

JuanFMontesinos: <p>Hi, the map location only specify where the weights are loaded. When you isntantiate the model, it uses those or a copy of those depending on where the model is allocated.</p>
<p>If you are using plain pytorch you should allocate the model in the gpu as:<br>
<code>model=model.cuda()</code><br>
if you use pytorch lightning you have to pass the same kwarg as for training when you build the trainer.<br>
This is <code>gpus=1</code> (or more)<br>
Also note that if you use pytorch lightning, they handle the model loading by passing<br>
<code>ckpt_path=...</code></p>

Steven_Tuften: <p>Thankyou Juan. Appreciate the quick response. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
<p>I’m passing the checkpoint file location to the trainer and the same kwargs …however I’ll double check all the args just to be sure!</p>
PyTorch Autograd for Regression

ArkadyBogdanov: <p>another PyTorch newbie here trying to understand their computational graph and autograd.</p>
<p>I’m learning the following model on potential energy and corresponding force.</p>
<pre><code class="lang-auto">model = nn.Sequential(
    nn.Linear(1, 32),
    nn.Linear(32, 32), nn.Tanh(),
    nn.Linear(32, 32), nn.Tanh(),
    nn.Linear(32, 1)
)

optimizer = torch.optim.Adam(model.parameters())
loss = nn.MSELoss()
</code></pre>
<pre><code class="lang-auto"># generate data
r = torch.linspace(0.95, 3, 50, requires_grad=True).view(-1, 1)
E = 1 / r
F = -grad(E.sum(), r)[0]

inputs = r

for epoch in range(10**3):
    E_pred = model.forward(inputs)
    F_pred = -grad(E_pred.sum(), r, create_graph=True, retain_graph=True)[0]

    optimizer.zero_grad()
    error = loss(E_pred, E.data) + loss(F_pred, F.data)
    error.backward()
    optimizer.step()
</code></pre>
<p>However, if I change the <code>inputs = r</code> to <code>inputs = 1*r</code>, the training loop breaks and gives the following error</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
</code></pre>
<p>Could you please explain why this happens?</p>

tom: <p>I’m surprised that the view doesn’t cause trouble to you, too.</p>
<p>When you do an assignment <code>inputs = r</code>, the variable on the left hand side is assigned exactly what is the right hand side, in this case the very object (i.e. the tensor Tensor) that is <code>r</code>.<br>
This contrasts with <code>1 * r</code>, which is a computed result, even if the computation is not very interesting, and then might get assigned to <code>inputs</code>.<br>
Similarly, the <code>.view</code> should be “computation” to autograd.</p>
<p>Now, if <code>inputs</code> is an “interior” node, i.e. the result of computation, autograd wants to go back through the computational graph that produced <code>inputs</code>. In the second iteration, it will cause the problem because you already did the first time.</p>
<p>The fix is to only keep leaves (i.e. non-computed values, like the ones you pass to the optimizer) or non-grad-requiring tensors outside the loop and to all computation from them within the for loop.<br>
For some cases (e.g. the <code>view</code>), it may be beneficial to not use <code>requires_grad</code> in the factory method (<code>linspace</code>) but instead add <code>.requires_grad_()</code> after the view. This way, <code>r</code> will be a leaf again.</p>
<p>Best regards</p>
<p>Thomas</p>
Generating text/csv file for image path and mask path for semantic segmentation

siddharth_MV: <p>I have a huge set of images(60k) and masks(60k) that need to be loaded into a PyTorch dataloader for semantic segmentation.</p>
<pre><code class="lang-auto">Directory Structure:

 - Segmentation
       -images
           -color_left_trajectory_3000_00001.jpg
           -color_left_trajectory_3000_00002.jpg
           ...
       -masks
           -color_segmentation_3000_00001.jpg
           -color_segmentation_3000_00002.jpg
           ...
</code></pre>
<p>I want to know the most efficient way to load these into a dataloader in Pytorch. I was thinking of generating a csv file with the paths to images and masks. How will I go about generating the same? Any other suggestions are appreciated!</p>

tom: <p>In my use cases I lazily use glob.glob to get a list of images and then use string manipulation (.replace in particular) to compute the corresponding mask paths. This has some hazard for capitalization and .jpg vs. .jpeg extensions, but works well if you have relatively tidy path names.</p>
<p>A more modern way is to use <code>pathlib.Path</code>, maybe with <code>.iterdir()</code> instead of the (also available) <code>.glob(...)</code>. If you care about cleanliness of your code, that might be worth looking at.</p>
<p>Best regards</p>
<p>Thomas</p>

siddharth_MV: <p>Okay makes sense, will check it out. Thank you!</p>
Visualizing pytorch's transformer feats

JuanFMontesinos: <p>Hi there! Does anyone know a library to visualize pytorch’s tranformers?<br>
thanks!</p>

osama-usuf: <p>Here’s a couple packages I’ve used to visualize transformers. Both of them end up plotting details that I personally didn’t require for my use case, so you might have to tweak your plots as well.</p>
<ul>
<li><a href="https://github.com/szagoruyko/pytorchviz" rel="noopener nofollow ugc">PyTorchViz</a></li>
<li>
<a href="https://github.com/lutzroeder/netron" rel="noopener nofollow ugc">Netron</a> (this requires converting your model to ONNX - <a href="https://deci.ai/blog/how-to-convert-a-pytorch-model-to-onnx/" rel="noopener nofollow ugc">this</a> guide was really helpful)</li>
</ul>
Ineffective Backward Propogation of losses

SHIVANSH_ANAND: <pre><code class="lang-auto">class ModifiedNet(nn.Module):
    def __init__(self):
        super().__init__() # just run the init of parent class (nn.Module)
        self.conv1 = nn.Conv2d(1, 16, 11, padding = 5)
        self.conv2 = nn.Conv2d(16, 16, 7, padding = 3)
        self.conv3 = nn.Conv2d(16, 16, 5, padding = 2)
        self.conv4 = nn.Conv2d(16, 16, 3, padding = 1)
        self.conv5 = nn.Conv2d(16, 16, 3, padding = 1)
        self.conv6 = nn.Conv2d(16, 1, 1)

    def convs(self, x, mask):
        # max pooling over 2x2
        x = F.relu(self.conv1(x * mask))
        x = x / torch.sum(mask)
        mask = F.max_pool2d(mask, kernel_size=11, stride = 1, padding = 5)
        x = F.relu(self.conv2(x * mask))
        x = x / torch.sum(mask)
        mask = F.max_pool2d(mask, kernel_size=7, stride = 1, padding = 3)
        x = F.relu(self.conv3(x * mask))
        x = x / torch.sum(mask)
        mask = F.max_pool2d(mask, kernel_size=5, stride = 1, padding = 2)
        x = F.relu(self.conv4(x * mask))
        x = x / torch.sum(mask)
        mask = F.max_pool2d(mask, kernel_size=3, stride = 1, padding = 1)
        x = F.relu(self.conv5(x * mask))
        x = x / torch.sum(mask)
        mask = F.max_pool2d(mask, kernel_size=3, stride = 1, padding = 1)
        x = self.conv6(x * mask)
        x = x / torch.sum(mask)
        return x

    def forward(self, x, mask):
        x = self.convs(x, mask)
        return x
</code></pre>
<p>I wanted to implement a sparse convolutional neural network for interpolation of depth maps. I implemented the algorithm but the problem which is arising now is since I am normalising the output at end of every layer the the autograd while backpropogating dilutes the error at subsequent layers which is causing very minimum or no change in weights and my network is not able to fit the data the losses being same in each epoch. Please help me with this I am really stuck I checked every forum and I started using pyTorch a week before.</p>

osama-usuf: <p>Seems like you’re running into vanishing gradients.</p>
<p>You seem to be normalizing your network’s activations (and not layer outputs). Could you elaborate why this is needed?</p>
<p>If the normalization is a necessary step, try playing around with network-level parameters (larger learning rates, smaller batch size, optimizers, etc).</p>
<p>Another thing I’d recommend is changing the <code>reduction</code> parameter of your loss function to <code>"sum"</code> instead of the default <code>"mean"</code> - as given <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">here</a> for CrossEntropyLoss.</p>