akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Issue with backproping log probabilities

username: Zeeyuu
cooked: <p>For the task i want to solve, I need to backdrop the average log probability. It looks like the following:</p>
<pre><code class="lang-auto">tensor(-24.9345, device='cuda:2', grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<p>so it seems to have gradient computation enabled. However, when I do backwards, it gives me the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "/home/zhan1130/anaconda3/envs/nano/lib/python3.8/site-packages/torch/_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "/home/zhan1130/anaconda3/envs/nano/lib/python3.8/site-packages/torch/autograd/__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Unexpected floating ScalarType in at::autocast::prioritize
</code></pre>
<p>not sure what it means or how to solve it.</p>
<p>Edit: i am using code modified from nanoGPT. The default dtype is ‘bfloat16’, but somehow the output logits is float32 and hence the log probability is also float32. simply using log_ probability = log_probability.bfloat16() still gives the same error</p>

username: eqy
cooked: <p>This looks like an autocast error, could you post a runnable script that reproduces the issue or try disabling autocast e.g., by forcing the null context here <a href="https://github.com/karpathy/nanoGPT/blob/a82b33b525ca9855d705656387698e13eb8e8d4b/sample.py#L32" class="inline-onebox" rel="noopener nofollow ugc">nanoGPT/sample.py at a82b33b525ca9855d705656387698e13eb8e8d4b · karpathy/nanoGPT · GitHub</a> ?</p>

username: Zeeyuu
cooked: <p>I guess it would be very hard, because I don’t have enough memory on my machine. After posting, I tried to change all dtype to float16 and the issue is gone, however I faced out of memory issue. I presume bfloat16 is to save memory, so if I disable autocast there shouldn’t be this issue, but i simply don’t have enough memory to run it.</p>
RTX 4090 2way cannot train model; it stuck

username: Hyunjin_Kim
cooked: <p>One RTX4090 can train a model normally with Pytorch-cuda. But with 2x RTX4090, it stuck at dataloader. Does NVIDIA block P2P communication between GPUs? I cannot understand why this serious problem is not announced in public? If I know it, I will not buy this kind of machine. My machine uses ThreadRipper Pro 5955wx and 2 GPUs.</p>

username: ptrblck
cooked: <p>Your 4090s do not support P2P and some NVIDIA drivers created hangs. Drivers 525.107 and 520.86 should fix it and avoid the hang.</p>

username: Hyunjin_Kim
cooked: <p>Thanks for quick answer. But I have another question. Do all 4090 GPUs not support P2P communication? Or no plan for allow P2P communication?</p>

username: Hyunjin_Kim
cooked: <p>And where can I download the stable drivers? Thanks again.</p>

username: ptrblck
cooked: <p>Sorry, the driver should be released by end of this month and might not be published yet. As a workaround you could use <code>export NCCL_P2P_DISABLE=1</code>.</p>
<aside class="quote no-group" data-username="Hyunjin_Kim" data-post="5" data-topic="176174">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hyunjin_kim/48/58935_2.png" class="avatar"> Hyunjin_Kim:</div>
<blockquote>
<p>Do all 4090 GPUs not support P2P communication? Or no plan for allow P2P communication?</p>
</blockquote>
</aside>
<p>P2P is not enabled on 4090s as it’s an unsupported configuration. The driver fix doesn’t enable it, but avoids hangs and data corruption.</p>

username: Hyunjin_Kim
cooked: <p>I think that the release of these drivers require more time. Is it right? On 30th March, 525.105 driver was released.</p>

username: Hyunjin_Kim
cooked: <p>I want to know whether 525.105 driver was safe for the hangs and data corroptions.</p>

username: ptrblck
cooked: <p>Yes, this seems to be the case based on <a href="https://github.com/NVIDIA/nccl-tests/issues/117#issuecomment-1494863061">this comment</a>.</p>
Using vmap instead of explicit batch dimension: good idea?

username: harpone
cooked: <p>I’m in a situation where it’s a lot easier to design a layer without an explicit batch dimension. Is it in general now a good idea to just design an entire NN without batch dimension and then just vmap over the batch dim?</p>
<p>I could see problems with BatchNorm, but everybody hates BatchNorm anyway right?</p>

username: ptrblck
cooked: <p>Based on the <a href="https://pytorch.org/functorch/stable/generated/functorch.vmap.html#functorch.vmap">docs</a> it seems this might indeed be a targeted use case:</p>
<blockquote>
<p><a href="https://pytorch.org/functorch/stable/generated/functorch.vmap.html#functorch.vmap"><code>vmap()</code></a> can be helpful in hiding batch dimensions, leading to a simpler model authoring experience.</p>
</blockquote>
<p>However, I would also be interested in learning more about your actual use case and how you are writing your code. I know some ops might not accept a batched input in which case <code>vmap</code> sounds like the right tool, but it seems your code can be simplified by dropping the batch dimension?</p>

username: harpone
cooked: <p>Yes, simplification is the primary motivation for me. I’m not using any NN layers except nn.Linear (in this case), which works with any shapes, so I figured it would be a nice opportunity to just drop the batch dimension completely.</p>
<p>I was explicitly interested in some CUDA benchmarks since it’s not very clear to me how the various parallelization optimizations happen under the hood… but I guess I could do some simple benchmarks with e.g. nn.Linear myself.</p>

username: ptrblck
cooked: <p>Yes, I would also recommend benchmarking the actual use case to see if any differences would be visible or if <code>vmap</code> can properly use the optimal ops.<br>
Also, CC <a class="mention" href="/u/richard">@richard</a> for visibility in case you have already benchmarked different workloads.</p>
Is it important for an (image-based) dataloader to return a torch.Tensor instead of a numpy array?

username: fabian_schutze
cooked: <p>The <a href="https://detectron2.readthedocs.io/en/latest/_modules/detectron2/data/dataset_mapper.html#DatasetMapper.__call__" rel="noopener nofollow ugc">datasetmapper</a> of detectron has the following interesting comment in the codebase: (for the function returning the inference / train dataset)</p>
<pre><code>    # Pytorch's dataloader is efficient on torch.Tensor due to shared-memory,
    # but not efficient on large generic data structures due to the use of pickle &amp; mp.Queue.
    # Therefore it's important to use torch.Tensor.
</code></pre>
<p>The comments is just above the following line of code:</p>
<pre><code class="lang-auto">dataset_dict["image"] = torch.as_tensor(np.ascontiguousarray(image.transpose(2, 0, 1)))
</code></pre>
<p>Is there some source for this statement? Can somebody explain possible reasons for this? How could I test this?</p>

username: ptrblck
cooked: <p>I would guess the comment might refer to <a href="https://github.com/pytorch/pytorch/issues/13246">this issue and description</a>.</p>

username: fabian_schutze
cooked: <p>Thanks for your answer, <a class="mention" href="/u/ptrblck">@ptrblck</a> .</p>
<p>I am not sure if this refers to the same problem. I think the issue you mentioned refer to using Python object for the member variable of the data loader. I think the same issue was discussed here: <a href="https://ppwwyyxx.com/blog/2022/Demystify-RAM-Usage-in-Multiprocess-DataLoader/" class="inline-onebox" rel="noopener nofollow ugc">Demystify RAM Usage in Multi-Process Data Loaders - Yuxin's Blog</a></p>
<p>The comment in the detectron codebase to me rather seems about returning python/numpy objects vs pytorch objects from the dataloader, or? Maybe I should ask directly on the detectron github page…</p>

username: ptrblck
cooked: <p>I might be wrong and the linked issue was the only one which sounded familiar. Yes, this sounds like a good idea and please post the answer here in case you get one as I would also be interested to learn more about their concerns.</p>

username: fabian_schutze
cooked: <p>An very interesting answer to the question was posted, which I repeat here:</p>
<blockquote>
<p>Large tensors are better returned as torch.Tensor than numpy arrays. Because torch.Tensor are pickled through shared memory <a href="https://github.com/pytorch/pytorch/blob/fd3a7264ae80332ba5ec8f60446e6d7a2c2276c1/torch/multiprocessing/reductions.py#L379-L395" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/fd3a7264ae80332ba5ec8f60446e6d7a2c2276c1/torch/multiprocessing/reductions.py#L379-L395</a> while numpy arrays are not.</p>
</blockquote>
<p>The answer can be found <a href="https://github.com/facebookresearch/detectron2/issues/4874#issuecomment-1498137745" rel="noopener nofollow ugc">here</a>.</p>
Backwards() through a graph iteratively and efficiently

username: imgeorgiev
cooked: <p>I want to iteratively roll out a computation graph while checking the gradients at each step efficiently, ideally within the same time needed to compute the gradients from the full graph.</p>
<p>This is probably best understood via a minimal example. Here is a standard graph</p>
<pre><code class="lang-auto">x = tensor([1.0], requires_grad=True)
y = [zeros(1) for _ in range(100)]
y[0] += x**2
for i in range(1, len(y)):
    y[i] += y[i - 1] ** 2
y[-1].backward()
</code></pre>
<p>which produces gradients <code>tensor([1.2677e+30])</code> in 0.0022s on my machine (on cpu)</p>
<p>Now, what I want to do is compute the gradients at each step without any hit to performance. Here is a minimal example that achieves that using <code>retain_graph</code></p>
<pre><code class="lang-auto">x = tensor([1.0], requires_grad=True)
y = [zeros(1) for _ in range(100)]
y[0] += x**2
y[0].backward(retain_graph=True)
for i in range(1, len(y)):
    y[i] += y[i - 1] ** 2
    x.grad.zero_()  # introduces extra compute but makes gradients correct
    y[i].backward(retain_graph=True)
</code></pre>
<p>This produces the correct gradients <code>tensor([1.2677e+30])</code>, however, it does so in 0.0618s. My goal is to bring this computing time down as much as possible.</p>
<p>I tried doing <code>retain_grad()</code> on all intermediate variables which indeed retained the gradients at each intermediate variable. However, it looks like <code>backward()</code> isn’t reusing the gradients that it already computed and does the whole backward pass from scratch. Assuming that is correct, my question boils down to how can I make autograd reuse the available gradients in the graph without necessarily computing them again?</p>
<p>Complete script I used for ease of reproducing:</p>
<pre><code class="lang-auto">from torch import tensor, zeros
from time import time

start = time()
x = tensor([1.0], requires_grad=True)
y = [zeros(1) for _ in range(100)]
y[0] += x**2
for i in range(1, len(y)):
    y[i] += y[i - 1] ** 2
y[-1].backward()
end = time() - start
print("real y grads in {:.4f}s".format(end), x.grad)

# now get iterative grads
start = time()
x = tensor([1.0], requires_grad=True)
y = [zeros(1) for _ in range(100)]
y[0] += x**2
y[0].backward(retain_graph=True)
for i in range(1, len(y)):
    # y[i - 1].retain_grad()  # makes this retain the gradients
    y[i] += y[i - 1] ** 2
    x.grad.zero_()  # introduces extra compute but makes gradients correct
    y[i].backward(retain_graph=True)
end = time() - start
print("y grads in {:.4f}s".format(end), x.grad)
</code></pre>

username: soulitzer
cooked: <p>A dumb way do this is to manually chain the gradients together, if y_2 = fn(y_1) you could do dy_1 = torch.autograd.grad(fn, y_1, grad_output=(dy_2,))</p>
<p>Another way is to cache the outputs of operations that happen during the backward pass,  I don’t think that is better though because (due to a bug) it probably breaks when you have inplace ops in backward formulas. It would also save a bunch of tensors unnecessarily.</p>
Where is <ATen/ops/to_ops.h>

username: jinyuan.feng
cooked: <p>Hi， I can’t find the header file “ATen/ops/to_ops.h” in <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/core/Tensor.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/Tensor.cpp at master · pytorch/pytorch · GitHub</a></p>

username: soulitzer
cooked: <aside class="quote no-group" data-username="jinyuan.feng" data-post="1" data-topic="176642">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/8baadc/48.png" class="avatar"> jinyuan.feng:</div>
<blockquote>
<p>to_ops.h</p>
</blockquote>
</aside>
<p>You have to build pytorch to see it, after building you can find it in <code>build/aten/src/ATen/ops/to_ops.h</code></p>

username: jinyuan.feng
cooked: <p>Thanks, And Do you know how to compile ATen alone?</p>

username: jinyuan.feng
cooked: <p><a class="mention" href="/u/soulitzer">@soulitzer</a><br>
Thanks, And Do you know how to compile ATen alone?</p>

username: soulitzer
cooked: <p>You can set certain environment variables to reduce the things you need to build., e.g. <code>USE_DISTRIBUTED=0 USE_MKLDNN=0 USE_CUDA=0 BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 python setup.py develop</code>, but I’m not sure how to build only aten.</p>
Backward hook with specified gradient

username: szhang1103
cooked: <p>I want to set up a backward hook to modify the gradient while the gradient is specified.</p>
<p>First, I set up a forward hook on a ReLU module of ResNet34 to get intermediate output.</p>
<pre><code class="lang-auto">def mid_out_hook(self):
    def hook(module,inp,out):
         self.mid_out = out
    return hook
def get_mid_out(self, inp, module):
    hook = module.register_forward_hook(mid_out_hook())
    self.model(inp)
    hook.remove()
    return self.mid_out
mid = get_mid_out()
</code></pre>
<p>I omit the wrapper class to save space. Then for the same module, I register a backward hook to modify the gradient and backward the middle output with specified gradient,</p>
<pre><code class="lang-auto">mask = torch.zero_like(mid)
def grad_hook(mask):
    def hook(module,grad_inp,grad_out):
        new_grad = torch.mul(grad_out[0], mask)
        return (new_grad,)
    return hook
module.register_full_backward_hook(grad_hook(mask))
mid.backward(gradient = torch.ones_like(mid))
</code></pre>
<p>However, I got errors</p>
<blockquote>
<p>RuntimeError: Module backward hook for grad_input is called before the grad_output one. This happens because the gradient in your nn.Module flows to the Module’s input without passing through the Module’s output. Make sure that the output depends on the input and that the loss is computed based on the output.</p>
</blockquote>
<p>A possible solution is to modify the gradient first and backward through the module input instead of output to avoid the backward hook. However, I am wondering is there other way to resolve the conflict between backward hook gradient modification and specified gradient.</p>

username: soulitzer
cooked: <p>Since your module only takes a single input you could just register a hook to the input tensor of the module instead. <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.register_hook — PyTorch 2.0 documentation</a></p>
<p>Note that Tensor hooks would behave differently in a subtle way. You can read <a href="https://pytorch.org/docs/stable/notes/autograd.html#backward-hooks-execution" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 2.0 documentation</a> for more details on that.</p>

username: szhang1103
cooked: <p>It could work but I guess I need to register the hook for every different input, which is a little annoying.</p>

username: szhang1103
cooked: <p>I realize that the problem is that the middle output is obtained by the forward hook instead of by the module output. In that way, the backward hook function would not be successfully applied to the module output as it does not record the grad_fn. However, I still don’t know how to fix it at this point.</p>

username: soulitzer
cooked: <p>Another way is to pass the module inputs through a dummy module that returns all the inputs as-is. Then register a module full backward hook to <em>that</em> module.</p>
AttributeError: 'list' object has no attribute 'seek'

username: nwe_lay
cooked: <p><strong>when I run this code on google colab. The following error occur.</strong><br>
<strong>Please help me.</strong></p>
<blockquote>
<p>if cuda:<br>
state_dic=torch.load(model_path[0])<br>
encoder.load_state_dict(state_dic)<br>
state_dic2=torch.load(model_path[1])<br>
decoder.load_state_dict(state_dic2)</p>
</blockquote>
<p>AttributeError: ‘list’ object has no attribute ‘seek’. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.</p>

username: Andrei_Cristea
cooked: <p>Could you provide the part of the code where you declare model_path?</p>

username: nwe_lay
cooked: <p>yes. I declare the model path as follow…</p>
<blockquote>
<p>import glob<br>
encoder_pkl=glob.glob(“/content/drive/MyDrive/remote_sensing_image_captioning/test/encoder.pkl”)<br>
decoder_pkl=glob.glob(“/content/drive/MyDrive/remote_sensing_image_captioning/test/decoder.pkl”)<br>
encdoder_pkl=encoder_pkl[0]<br>
decoder_pkl=decoder_pkl[0]</p>
</blockquote>
<blockquote>
<p>model_path = [encoder_pkl, decoder_pkl]</p>
</blockquote>
<blockquote>
<p>def predict(img_name, model_path):<br>
encoder = Encoder()<br>
decoder = Decoder()<br>
if cuda:<br>
encoder = encoder.cuda()<br>
decoder = decoder.cuda()<br>
if model_path:<br>
print(‘Loading the parameters of model.’)<br>
if cuda:<br>
state_dic=torch.load(model_path[0])<br>
encoder.load_state_dict(state_dic)<br>
state_dic2=torch.load(model_path[1])<br>
decoder.load_state_dict(state_dic2)</p>
</blockquote>
<p>The error occur when the model_path is passed to the parameter of predict function.</p>

username: ptrblck
cooked: <p>It seems <code>model_path[0]</code> and/or <code>model_path[1]</code> return a <code>list</code> instead of a file path or a buffer.</p>

username: Rohith_Bachu
cooked: <p>Iam also having same issue while loading my network. I understand that model path is returning a list instead of file path or a buffer. can you suggest possible reasons or solutions for this?</p>

username: ptrblck
cooked: <p>You could index the <code>list</code> to return the desired string.</p>
How to split an image into N sub-images?

username: nima_pw
cooked: <p>Hi,</p>
<p>I have an image dataset with different sizes.<br>
Instead of resizing, I want to split every image into <code>N</code> sub-images with the same size <code>(320*320)</code>.<br>
Certainly, <code>N</code> will be different for each image, and maybe there will be a need for zero-padding.<br>
For example, if the size of the image is <code>1920x1080</code>, <code>Height</code> will be split into <code>6</code> parts, and the width will be split into 3 with zero padding to keep the size of <code>320</code>. So there will be <code>18 sub-images</code> in this case. For another image with a different size, it will be different.<br>
But I don’t know how to do this in practice.</p>

username: sinatv52
cooked: <p>Hi Nima,<br>
This topic has already discussed in multiple posts. To get your solution faster you can check them . in <a href="https://discuss.pytorch.org/t/how-to-extract-patches-from-an-image/79923">this post</a> and <a href="https://discuss.pytorch.org/t/creating-nonoverlapping-patches-from-3d-data-and-reshape-them-back-to-the-image/51210">this post</a></p>
Hdf5 file into pytorch dataloader

username: Daniel_Hen
cooked: <p>Hi,<br>
I have some hdf5 files which are splitted by X,Y and train/va/test (e.g. 1 file is train_X.h5, another file is train_y.h5, etc.)</p>
<p>I’m trying to load each of them into pytorch dataloader, but I feel that I need to somehow first unite the files (meaning - train should be 1 file) and then load them?</p>
<p>The problem is that I’m a bit newbiew <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">  and don’t have experience with working with hdf5, so I would love to have some help with that.</p>
<p>I don’t have control over the hdf5 files and these are static, so I have to load them as it is.</p>
<p>What should I create for that? Which classes?</p>
<p>Thanks in advance,<br>
Daniel</p>

username: Daniel_Hen
cooked: <p>Hi,</p>
<p>I have solved it using a great post - <a href="https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/16?fbclid=IwAR2jFrRkKXv4PL9urrZeiHT_a3eEn7eZDWjUaQ-zcLP6BRtMO7e0nMgwlKU" class="inline-onebox">DataLoader, when num_worker &gt;0, there is bug - #16 by piojanu</a></p>
<p>Basically, since my HDF5 were different (each file, different X / Y files), I had to separately load them.</p>

username: msmjn04
cooked: <p>Hi <a class="mention" href="/u/daniel_hen">@Daniel_Hen</a>!</p>
<p>What is the size of your .h5 data? By using the solution you shared, was the overhead reduced when you load the entire train dataset?</p>
<p>Thank you.</p>
How to batch dataset with variable target length?

username: greedsin
cooked: <p>Hello all,</p>
<p>i am wondering what my options are:</p>
<ol>
<li>
<p>My goal is object detection. So I have an image and then a list of entries with the following form [[class1, x0, y0, width, height], [class2, x0, y0, width, height], …]. The entries denote what and where objects are located in the image.</p>
</li>
<li>
<p>The Problem : For each image there is a variable amount of objects to detect which means the list of entries have different lengths.</p>
</li>
<li>
<p>My Questions: Do I need to pad here? Most of the time I read about padding the input but in my case the images  are all the same size. I tried to use a different collate_fn where I just zip everything but this leads to an dimension error for a linear layer I use.</p>
</li>
</ol>
<p>This is how I construct my dataset:</p>
<pre><code class="lang-auto">        img = Image.open(self.root_dir+"/"+self.prefix+"0"+str(idx)+".png").convert('RGB')
        txt_file = open(self.root_dir+"/"+self.prefix+"0"+str(idx)+".txt",'r')

        bounding_boxes=[]
        class_labels=[]
      
        lines =txt_file.read().splitlines()
        for idx,line in enumerate(lines):
            tmp = line.split(" ")
            bounding_box = [int(tmp[1]),int(tmp[2]),int(tmp[3]),int(tmp[4])]
            class_label = [int(tmp[0])]
            class_labels.append(class_label)
            bounding_boxes.append(bounding_box)
           
        sample = {'image':img,'bounding_boxes':bounding_boxes,"class_labels":class_labels}

        if self.transform:
           sample= self.transform(sample)

        return sample["image"],sample["bounding_boxes"],sample["class_labels"]

</code></pre>
<p>Many thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: flowkcaj
cooked: <p>I’m facing a similar issue now, did you solve this?</p>
ValueError: n_splits=5 cannot be greater than the number of members in each class

username: codecoaster
cooked: <p>Using Skorch for the first time for hyperparameter turning a neural network made with PyTorch and having issues in getting it to run. I have used this as a guide: <a href="https://debuggercafe.com/hyperparameter-search-with-pytorch-and-skorch/" class="inline-onebox" rel="noopener nofollow ugc">Hyperparameter Search with PyTorch and Skorch</a></p>
<p>However I have used my own dataset, consisting of 7 classes, with a total dataset size of 10292 images, with the smallest class having 480 images. All help I have found online mentions</p>
<pre><code class="lang-auto">StratifiedKFold
</code></pre>
<p>However I have not used this within the code. The Skorch part is as below:</p>
<pre><code class="lang-auto">    net = NeuralNetClassifier(
        module = CNN,
        criterion = nn.CrossEntropyLoss,
        optimizer = torch.optim.Adam,
        max_epochs = EPOCHS,
        lr = LR,
        verbose = False
        )

    param_grid = {
        'module__dropout_rate': [0.0, 0.16, 0.21, 0.28, 0.12, 0.04, 0.09],
        }
    grid = GridSearchCV(net, param_grid, refit=False, n_jobs = -1, cv = 2)

    search_batches = 1
    counter = 0
    for i, data in enumerate(trainDataLoader):
        counter += 1
        image, labels = data
        image = image.to(device)
        labels = labels.to(device)
        outputs = grid.fit(image, labels)
        if counter == search_batches:
            break
</code></pre>
<p>Any help on how to help solve this issue and get some solid hyperparameter tuning on the go would be much appreciated!</p>
<p>First time poster.</p>

username: AbdulsalamBande
cooked: <p>t seems you are trying to run the grid search for hyperparameters using Skorch inside the training loop, which is incorrect. You should perform the grid search before training the model, and the input should be the entire dataset rather than individual batches.<br>
After running the grid search, you can use the best hyperparameters found to train your model.</p>
L2 Normalization Layer

username: eaah
cooked: <p>Hi, I am trying to implement an L2 normalization layer. But I see that the F.normalize is not accepted by the sequential module as it requires an input. Is there an L2 normalization layer in pytorch?</p>
<pre><code class="lang-auto">self.backbone = torch.nn.Sequential(self.conv_b1,
                                            self.conv_b2,
                                            self.conv_b3,
                                            self.conv_b4_1,
                                            self.conv_b4,
                                            self.conv_b5,
                                             F.normalize())
</code></pre>
<p>TypeError: normalize() missing 1 required positional argument: ‘input’</p>

username: AbdulsalamBande
cooked: <p><code>F.normalize()</code> is a function from the <code>torch.nn.functional</code> module and cannot be used directly inside a <code>nn.Sequential</code> container. However, you can easily create a custom normalization layer by subclassing <code>nn.Module</code> and wrapping the <code>F.normalize()</code> function. Something like below</p>
<pre><code class="lang-auto">class L2NormalizationLayer(nn.Module):
    def __init__(self, dim=1, eps=1e-12):
        super(L2NormalizationLayer, self).__init__()
        self.dim = dim
        self.eps = eps

    def forward(self, x):
        return F.normalize(x, p=2, dim=self.dim, eps=self.eps)
</code></pre>

username: eaah
cooked: <p>Thanks so much for your response. My input tensor to the L2 normalization layer is a 128 x 512 x 14 x 14. N x C x H x W. Will this be performing the normalization over the Batch or over the channel dimension?</p>

username: AbdulsalamBande
cooked: <p>If you would like to perform normalization over the batch dimension, you can change the <code>dim</code> parameter to 0 .However, normalizing over the batch dimension is not a common practice, as it would mix information between different samples in the batch.</p>
N-dimensional convolutions coming?

username: malbergo
cooked: <p>Hi there, just curious if the n-dimensional convolution is in the pipeline for pytorch : ) Does anyone know if this feature is available / will be available sometime in the future?</p>
<p>Thanks very much for the information.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think there are any plans of supported anything else than 1D, 2D or 3D convs.<br>
Do you have a specific use case where you would need something else?</p>

username: SphericalCow
cooked: <p>I would like n-dimensional convolutions in order to multiply n-dimensional polynomials (in a batched manner).</p>
Conversion of torch GraphModule back to nn.Module

username: Chame_call
cooked: <p>Hello.<br>
I’ve done some changes with my nn.Module using torch.fx and after I did recompile() call I eventually got GraphModule object.<br>
So the question is how to convert it back to usual nn.Module or it’s better to leave it in this form cause otherwise my changes won’t be applied or smth?</p>

username: AbdulsalamBande
cooked: <p>You can use the GraphModule object obtained after applying torch.fx and recompile() like a regular nn.Module, as it is a subclass of nn.Module. There is no need to convert it back to the original nn.Module. The changes you made using torch.fx are preserved within the GraphModule, and you can use it for training, evaluation, and other operations just like you would with a regular nn.Module.</p>

username: Chame_call
cooked: <p>thanks a lot, have a good day or something)</p>
When i run the code to train the model, the first epoch time is gives this error :

username: Sardor_Mamarasulov
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e184ce8810c99394eaab2a7daf055a439e8715c0.png" data-download-href="https://discuss.pytorch.org/uploads/default/e184ce8810c99394eaab2a7daf055a439e8715c0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e184ce8810c99394eaab2a7daf055a439e8715c0_2_690x167.png" alt="image" data-base62-sha1="wb1Y16EiIto1SoT8fDztxarBru8" width="690" height="167" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e184ce8810c99394eaab2a7daf055a439e8715c0_2_690x167.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e184ce8810c99394eaab2a7daf055a439e8715c0_2_1035x250.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e184ce8810c99394eaab2a7daf055a439e8715c0.png 2x" data-dominant-color="343434"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1321×321 47 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Could you post a minimal and executable code snippet to reproduce the error as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>
<p>You can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>
Turning arm from i386 in an MacBook M1 Chip

username: Killpit
cooked: <p>I have M1 and checked MPS support, but when I look at processor, I get i386 instead of arm.</p>
<pre><code class="lang-auto">import platform

# should print "arm"
print(platform.processor())
</code></pre>
<p>Output: i386</p>

username: Killpit
cooked: <p>Also I get this when I write this</p>
<pre><code class="lang-auto">import torch
print(f"PyTorch version: {torch.__version__}")

# Check PyTorch has access to MPS (Metal Performance Shader, Apple's GPU architecture)
print(f"Is MPS (Metal Performance Shader) built? {torch.backends.mps.is_built()}")
print(f"Is MPS available? {torch.backends.mps.is_available()}")

# Set the device
device = "mps" if torch.backends.mps.is_available() else "cpu"
print(f"Using device: {device}")
</code></pre>
<p>/Users/atatekeli/PycharmProjects/PyTorchZTMWork/venv/bin/python /Users/atatekeli/PycharmProjects/PyTorchZTMWork/processor.py<br>
PyTorch version: 1.13.1<br>
Is MPS (Metal Performance Shader) built? True<br>
Is MPS available? True<br>
Using device: mps</p>

username: Yiii
cooked: <p>Hi,<br>
you can take a look at this instead of platform, since platform may have an issue with python module version :<br>
import os<br>
system_info = os.uname()<br>
print(system_info.machine)</p>
Using DataLoader with WebDataSet

username: JAllsopp
cooked: <p>Hi,<br>
I’ve a structure of 132 tar files, each containing 500 images (png, greyscale, 641x481) and json labels. I’m trying to load them like this;</p>
<pre><code class="lang-auto">preproc = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.Grayscale(num_output_channels=1),
    transforms.ToTensor(),
    transforms.Normalize((0.0227),(0.0071))
])

def identity(x):
    return x

dataset = (wds.WebDataset("training/training-out-{000000..000131}.tar").shuffle(100).decode("pil").to_tuple("png","json").map_tuple(preproc,identity))
batch_size = 50
dataset1=dataset.with_length(batch_size) # https://stackoverflow.com/questions/73918904/size-of-webdataset-in-pytorch
train_loader = torch.utils.data.DataLoader(dataset1.batched(batch_size), num_workers=1, batch_size=None) 
</code></pre>
<p>I’ve been looking in the documentation and trying to experiment with different options, but I was wondering if someone could confirm that this is pretty much the right way to go about things, as I’m just starting out. Batch size of 50 is a bit odd, but I grouped my files into 500 file tar files, whereas a power of 2 might have been more consistent. I was getting errors without the with_length (dataset has no length method) and</p>
<p>I’m trying to use this with;</p>
<pre><code class="lang-auto">        loss_train = 0.0
        for imgs, labels in train_loader:
            outputs = model(imgs)
            loss = loss_fn(outputs, function_that_handles_label(labels))
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            loss_train += loss.item()
            
        if epoch == 1 or epoch% 10 ==0:
            print(f'{datetime.datetime.now()} Epoch {epoch}, Training loss {loss_train/len(train_loader)}')
</code></pre>
<p>If anyone has any suggestions on how to improve this training loop, best practice, or “Well you can do it like that but you’ll run into problems here, here and here…” I would be very grateful,</p>
<p>Thanks</p>

username: JAllsopp
cooked: <p>I’m also getting lots of errors like this in the second epoch - if anyone has any ideas, please let me know.</p>
<blockquote>
<p>/home/allsoppj/.conda/envs/pytorch/lib/python3.10/site-packages/torch/utils/data/dataloader.py:640: UserWarning: Length of IterableDataset &lt;webdataset.pipeline.WebDataset_Length object at 0x7f73449b6bf0&gt; was reported to be 50 (when accessing len(dataloader)), but 1289 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 2.1 documentation</a> for examples.</p>
</blockquote>
Can I use the pre-training of weights as I add input channels?

username: SimonBet
cooked: <p>Hello !</p>
<p>I’m trying to use a ResNet18 model to process Sentinel-2 satellite imagery for a regression problem. I have weights that were pre-trained on 13 channels, but I’d like to add channels for additional information (temperature, sea depth etc.). As I have a very limited amount of data, I can’t afford to re-train the whole model, and I’d like to use those weights.</p>
<p>Looking around on forums, I found that the common solution is to modify the first layer to accept more channels, then clone the pre-trained weights, and initialize the new weights (either randomly or using the first weights). However, i’m not sure about about the integrity of the weights in the next layers. If I understand correctly, the weights for each layer after the first one will initially stay the same. Now if I feed the model new information in the form of my new channels, won’t this mess up the pre-trained weights ? Should I freeze some of them, change the learning rate ?..</p>
<p>Thanks in advance for any input !</p>

username: ptrblck
cooked: <p>I think your concern is valid and I would also expect that at least finetuning of the first layer is needed. I don’t know if retraining the already pretrained layers is needed or helps.<br>
As an alternative you could also consider using a new “mapping” layer, which would transform your inputs channels to the desired one used in your pretrained model. If you are using a kernel size of <code>1x1</code> for this layer you could see it as a “trainable color transformation”.</p>

username: SimonBet
cooked: <p>Thank you for your reply. I’m concerned that this mapping layer would introduce error in the final prediction, as some of my new channels are not at all correlated with the imagery. Would it be possible, with torch, to run two CNNs in parallel with a common loss function, and merge their output at the last layer ? This way i would keep one CNN on the 13 band Sentinel imagery with the frozen pre-trained weights, and train the other from scratch in parallel on the new channels only</p>

username: ptrblck
cooked: <p>Yes, you can create such an architecture, but would then of course need to check how these features should be handled. E.g. you could concatenate them or use any reduction operation.</p>

username: SimonBet
cooked: <p>Thank you, I will try a few things on parallel networks and compare the fusion methods.</p>
Using single optimizer to train both generators in CycleGan

username: Asjad_Murtaza
cooked: <p>I was looking at cycleGAN’s implementation in PyTorch by the author of the paper. In the code author chained the parameters of both generators and passed them to Adam optimizer. I don’t understand the intuition behind training both networks with single optimizer. Shouldn’t we use different optimizers for different networks ? Where am I wrong ?</p>

username: vmirly1
cooked: <p>The code I looked upas two optimizers: <code>optimizer_G</code> and <code>optimizer_D</code>. Can you point to the code that you are referring to?</p>

username: Asjad_Murtaza
cooked: <p>Yes the code uses optimizer_G to train Generator AB and Generator BA. They are different models and need different gradients so why did the author use one optimizer ? Perhaps I am understanding it wrong.</p>

username: Asjad_Murtaza
cooked: <pre><code class="lang-auto">loss_G = loss_identity_A + loss_identity_B + loss_GAN_A2B + loss_GAN_B2A + loss_cycle_ABA + loss_cycle_BAB
        loss_G.backward()
        
        optimizer_G.step()
</code></pre>
<p>why are all the losses being added when both models have different parameters ?</p>

username: dragonHyeon
cooked: <p>Check out <a href="https://stackoverflow.com/a/75938118/8423105" rel="noopener nofollow ugc">this answer</a>.</p>
<p>What optimizer does is just updating its tracking, corresponding weights. For example, if network <code>GA</code> should be updated but network <code>GB</code> doesn’t need to, then this single optimizer which is tracking both <code>GA</code> and <code>GB</code> will just gonna update <code>GA</code>’s parameter but not <code>GB</code>’s parameter.</p>
<p>Official repo’s <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/1381" rel="noopener nofollow ugc">issue</a> says this simplifies the code but the effect should be the same.</p>