akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.load not able to find a class while importing a python script

anshs: <p>I am trying to load a model for further processing, from a .pt save file after finishing the training. But while loading I getting the following error message:</p>
<p>Does this mean there has been some issue while saving the model? I saved the model with <code>torch.save(unet, CONFIG['model_path2'])</code></p>
<p>While loading, I tried: <code>unet_model = torch.load(f"{CONFIG['model_path2']}", map_location=torch.device(device=device))</code> and <code>unet_model.load_state_dict(torch.load(f"{CONFIG['model_path2']}", map_location=torch.device(device=device)))</code> but both are giving the same error message.</p>
<p>Is there any way I can check whether the saved .pt file has attributes for <code>SinusoidalPosEmb</code> or not?</p>
<p><strong>Error:</strong></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/Users/anshumansinha/Diff_structrepgen/gen_diff_1.py", line 67, in &lt;module&gt;
    unet_model.load_state_dict(torch.load(f"{CONFIG['model_path2']}", map_location=torch.device(device=device)))
  File "/Users/anshumansinha/miniconda3/lib/python3.10/site-packages/torch/serialization.py", line 809, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/Users/anshumansinha/miniconda3/lib/python3.10/site-packages/torch/serialization.py", line 1172, in _load
    result = unpickler.load()
  File "/Users/anshumansinha/miniconda3/lib/python3.10/site-packages/torch/serialization.py", line 1165, in find_class
    return super().find_class(mod_name, name)
AttributeError: Can't get attribute 'SinusoidalPosEmb' on &lt;module '__main__' from '/Users/anshumansinha/Diff_structrepgen/gen_diff_1.py'&gt;
</code></pre>

ptrblck: <aside class="quote no-group" data-username="anshs" data-post="1" data-topic="184760">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anshs/48/50729_2.png" class="avatar"> anshs:</div>
<blockquote>
<p>Does this mean there has been some issue while saving the model? I saved the model with <code>torch.save(unet, CONFIG['model_path2'])</code></p>
</blockquote>
</aside>
<p>It seems you have saved the entire model instead of the <code>state_dict</code>. Saving the model directly requires you to restore the same script structure when trying to load the model. Based on the error message I guess some scripts are missing or their definition changed.</p>
[Pytorch Lightning] Gradient accumulation not working for n>1

Thibault712: <p>In my trainer if I set accumulate_grad_batches=n, it doesn’t work (n&gt;1, for n=1 it trains perfectly so the problem shouldn’t come from my model/implementation). I really can’t find where is the trouble maybe one of you got the same error and struggled with it before me and found a solution?</p>
<pre><code class="lang-auto">venv/lib/python3.8/site-packages/torch/autograd/__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
SystemError: &lt;built-in method run_backward of torch._C._EngineBase object at 0x7f2624d4a9f0&gt; returned NULL without setting an error
Epoch 0:   0%|          | 2/2864 [00:07&lt;2:49:55,  3.56s/it, v_num=5]  
</code></pre>
<p>if it might be related to the hardware, I’m running on 2 GPUs:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A6000    On   | 00000000:1B:00.0 Off |                  Off |
| 30%   38C    P8    31W / 300W |      5MiB / 49140MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A6000    On   | 00000000:1C:00.0 Off |                  Off |
| 30%   33C    P8    24W / 300W |      5MiB / 49140MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
</code></pre>
Iterate over 5d tensor

Rohan_Bareja: <p>Hello,</p>
<p>I want to iterate over a 5d tensor ‘images’, extract 4d tensors and do some operation on the 4d tensors which is an individual bag of (batch size, channels, height, width).So in a 5d tensor, 1st dimension is batch size, 2nd dimension is bag size which is number of images in a bag (lets say 20),3rd dimension is RGB, 4th and 5th dimension are height and width.here is my for loop which I am using but this very slow.is there any other way to make this code faster? the for loop is very slow.</p>
<pre><code class="lang-auto">                new_output = []
                for i in range(args.bag_size):
                    individual_bag = images[:,i,:,:,:]
                    print('individual_bag shape',individual_bag.shape)
                    intermediate_output = model.get_intermediate_layers(individual_bag, n)
                    output = torch.cat([x[:, 0] for x in intermediate_output], dim=-1)
                    new_output.append(output)
                output = torch.mean(torch.stack(new_output),dim = 0)
</code></pre>
<p>Thanks,<br>
Rohan</p>

ptrblck: <p>I don’t know what each “individual bag” represents, but since you are indexing each separately I assume they don’t have any dependency between each other. If so, would it be possible to flatten <code>dim1</code> into <code>dim0</code> and just increase the batch size for a single forward pass?</p>

Rohan_Bareja: <p>Each individual bag represents lets say 20 images, so instead of passing each image, I am passing a bag of images.I am extracting features from the network for each bag in a batch, and doing this for each batch.The purpose of taking a bag of images is to get mean representation but I think I would need to do it for each batch separately.</p>

ptrblck: <p>I’m unsure why you would need to perform this operation for each bag separately. You could still reshape the output back before applying the <code>mean</code> operation.</p>
Adding hand selected features to one hot encoding?

anteater: <p>I am a beginner in training machine learning models and am faced with a situation where I’d like to combine a one hot encoding of my input with some features that I hand selected using domain knowledge using PyTorch. For example, the one hot encoding could be of size 100 and I want to add say, 3 or 4 hand selected numerical features.</p>
<p>I attempted to do this by simply concatenating the two tensors together but the results weren’t very good. The loss did go down, but the accuracy was very low. I am looking for any feedback or guidance to be pointed in the right direction on how to accomplish this (that is, assuming my current approach won’t work for some reason). The exact encoding of the input can be changed - I have also considered things like Embedding layers, etc.</p>
<p>Thanks everyone for all the help.</p>

dreidizzle: <p>I am guessing one-hot means one-hot per token in your vocabulary. So your feature per token is the concatenation of one-hot and some sort of custom feature for that token. I would probly have a model where you don’t use one-hot, but you use embeddings for each token (you can use glove or fast text or even bert). This gives you an embedding per token.</p>
<p>You can concatenate your to eac of these embeddings hand features to get a larger vector representation, but I don’t know if there is some rule which gaurantees you’ll do much better. BERT for example would take into account other tokens arund your token, so it might implictily contain your hand features. Do you have a spelled out example?</p>

vdw: <p>In principle, you can simply concatenate your one-hot vector with additional numerical features. You could also first push your one-hot vectors through 1 or more linear layers and then concatenate this output with your hand-selected features. The sky’s the limit :).</p>
<p>Regarding the effects on the results, did you check the accuracy using only the one-hot vectors vs. only the hand-selected features.</p>
<p>On a more fundamental level, the question is also where the one-hot vectors come from. Since you posted in the NLP category, are your inputs words? Then 100 seems kind of small. Or do you work with characters? What is your exact task?</p>
<p>As <a class="mention" href="/u/dreidizzle">@dreidizzle</a> , you could always consider an Embedding layer, which kind of mimics the idea above to first push the one-hot vectors through 1 or more linear layers (see above). The advantage of embeddings, in general, are that words/tokens with similar semantic meaning (e.g., quick and fast) tend to have similar embeddings. However, in contrast, characters don’t really feature something like semantic similarity (at least not like words). In this case, this advantage of embeddings would not as pronounced.</p>
<p>Can you give some insights into your training setup and goals?</p>
Can I extract a 'built-in-function' node by using the 'named_modules' function?

Sanghyeon_Lee: <p>Now, I’m trying to synthesize a node definition consisting of the ResNet-50 model.</p>
<p>The ResNet model has a ‘add’ node as a ‘built-in function’.</p>
<p>However, this ‘add’ node is not readable when I use the ‘named_modules’ function in the ResNet model.</p>
<p>Other functions like ‘modules()’, and ‘named_children’ are the same as well.</p>
<p>How can I extract the ‘built-in function’ node information by using the ‘named_modules’?</p>
<p>If my solution is wrong, please kindly let me know about that.</p>
<p>Thanks</p>
Replicating training with single gpu and adjusting batch size

serafej: <p>I am attempting to replicate (not reproduce so I don’t need identical seeding) the training of vision models provided in <a href="https://github.com/pytorch/vision/tree/main/references/classification" rel="noopener nofollow ugc"><code>vision.references.classification</code></a>. According to the ReadMe, these models have been trained in a distributed way across 8 GPUs (i.e. <code>--nproc_per_node=8</code>) while I would like to train using identical hyperparameters on a single (or at least fewer) GPUs.</p>
<p>Therefore, I would like to change the command from e.g.<br>
<code>torchrun --nproc_per_node=8 train.py --batch_size 32 --model resnet18</code><br>
to<br>
<code>torchrun --nproc_per_node=1 train.py --batch_size 32 --model resnet18</code></p>
<p>However it is not clear to me if this is equivilent as it depends on whether <code>model.optimize()</code> is carried out within each batch or not. If not (which I believe to be the case?) the effective batch size would change from 32 * 8 to 32 * 1 thereby effecting the implicit bias of stochastic gradient descent.</p>
<p>Therefore I suspect the solution to match these experiments consists of multiplying the batch size and instead running:</p>
<p><code>torchrun --nproc_per_node=1 train.py --batch_size 256 --model resnet18</code></p>
<p>Is this correct?</p>
<p>And as a follow up question, suppose that this increased batch size requires too much memory from my machine. The solution I have come up with in this case is to only apply <code>torch.optimize()</code> every 8th batch. Specifically I would run:</p>
<p><code>torchrun --nproc_per_node=1 train.py --batch_size 32 --model resnet18</code></p>
<p>but I would additionally change the source code in <code>vision.references.classification.train.py</code> in the training logic (L46) from:</p>
<pre><code class="lang-auto">optimizer.step()
</code></pre>
<p>to:</p>
<pre><code class="lang-auto">if (i + 1) % 8. == 0:
   optimizer.step()
</code></pre>
<p>Again, is this equivilent to the original hyperparameters implemented? Is this the best solution?</p>
I am current having the problem which shows that TypeError: 'Tensor' object is not callable

Yeasin_Al_Yeasa: <p>from ultralytics import YOLO<br>
import cv2</p>
<p>cap = cv2.VideoCapture(0)<br>
cap.set(3, 1280)<br>
cap.set(4, 720)</p>
<p>model = YOLO(“…/yolo weights/yolov8n.pt”)</p>
<p>while True:<br>
success, img = cap.read()<br>
results = model(img, stream=True)<br>
for r in results:<br>
boxes = r.boxes<br>
for box in boxes:<br>
x1, y1, x2, y2 = box.xyxy(0)<br>
print( x1, y1, x2, y2)</p>
<pre><code>cv2.imshow("Image", img)
cv2.waitKey(1)
</code></pre>
<p>and the error is<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c093374d507c051fb42b6b0c8becdf966268434.png" data-download-href="https://discuss.pytorch.org/uploads/default/4c093374d507c051fb42b6b0c8becdf966268434" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c093374d507c051fb42b6b0c8becdf966268434_2_690x187.png" alt="image" data-base62-sha1="aQE036LVglFYjiNzG9c1e0AJLUg" width="690" height="187" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c093374d507c051fb42b6b0c8becdf966268434_2_690x187.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c093374d507c051fb42b6b0c8becdf966268434_2_1035x280.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c093374d507c051fb42b6b0c8becdf966268434.png 2x" data-dominant-color="312E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1258×341 36.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>box.xyxy(0)</code> since you are calling it as a function. In case you want to index the tensor, use <code>box.xyxy[0]</code>.</p>

Yeasin_Al_Yeasa: <p>Thanks mate, it worked</p>
Modifying parts of tensors while preserving gradient computation

CTZ: <p>Hi,<br>
I want to train a network by taking the gradient of a simulation rollout. Here is a small snippet of what I intend to differentiate:</p>
<pre><code class="lang-auto">for n steps do:
obs = get_observations(state)
  actions = get_actions(obs)
  next_state = simulation_step(state,actions)
  reward = get_reward(next_state)
</code></pre>
<p>Since I need all observations and rewards for loss computation after the rollout, I want to have something more like this:</p>
<pre><code class="lang-auto">for n steps do:
  obs_buf[i,...] = get_observations(state)
  actions = get_actions(obs_buf[i,...])
  next_state = simulation_step(state,actions)
  reward_buf[i,...] = get_reward(next_state)
loss = get_loss(obs_buf, rew_buf)
</code></pre>
<p>However, doing this with Torch tensors gives different error messages like “A view of a leaf variable that requires grad is being used in an in-place operation”, “Trying to backward through the graph a second time”, or that the version of a tensor does not match during gradient calculation. My current solution avoids these issues by using lists:</p>
<pre><code class="lang-auto">for n steps do:
  obs_list.append(get_observations(state))
  actions = get_actions(obs_list[i])
  next_state = simulation_step(state,actions)
  reward_list.append(get_reward(next_state))
loss = get_loss(obs_list, rew_list)
</code></pre>
<p>But I am worried about performance. I am still struggling with fully understanding how tensors are handled in autograd. A deeper insight into why Torch tensors don’t work and any suggestions on my Issue would be highly appreciated.</p>
<p>Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

CTZ: <p>Hey,<br>
for anyone having similar issues, what I was missing is the fact that changing parts of a tensor by slicing is handled as an in-place operation. Therefore, adding <code>.clone()</code> before modifying the tensor solves the problem. The loop would then look like:</p>
<pre><code class="lang-auto">for n steps do:
  obs_buf = obs_buf.clone()
  obs_buf[i,...] = get_observations(state)
  actions = get_actions(obs_buf[i,...])
  next_state = simulation_step(state,actions)
  reward_buf = reward_buf.clone()  
  reward_buf[i,...] = get_reward(next_state)
loss = get_loss(obs_buf, rew_buf)
</code></pre>
<p>However, I am not sure if this is a smart way to do it since memory consumption for the buffers is basically multiplied by the number of rollout steps n. If anyone has a better idea than this or the list example above please feel free to share it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Compile error PyTorch 2.0.1 / GCC 13.1.0

Matthias_Moller: <p>I came across a strange error when compiling PyTorch (only C++ API, Python disabled) on A64FX using GCC 13.1.0.</p>
<p>When compiling the file <code>aten/src/ATen/native/mkldnn/Pooling.cpp</code> GCC turns on <code>-Werror</code> without turning on <code>-Wno-dangling-reference</code> so that</p>
<pre><code class="lang-auto">aten/src/ATen/core/IListRef_inl.h:171:17: error: possibly dangling reference to a temporary [-Werror=dangling-reference]
  171 |     const auto&amp; ivalue = (*it).get();
</code></pre>
<p>This seems to be triggered by line 534 in <code>caffe2/CMakeLists.txt</code>:</p>
<pre><code class="lang-auto"># Required workaround for LLVM 9 includes.
if(NOT MSVC)
  set_source_files_properties(${TORCH_SRC_DIR}/csrc/jit/tensorexpr/llvm_jit.cpp PROPERTIES COMPILE_FLAGS -Wno-noexcept-type)
  # Force -Werror on several files
  set_source_files_properties(${CMAKE_CURRENT_LIST_DIR}/../aten/src/ATen/native/mkldnn/Pooling.cpp PROPERTIES COMPILE_FLAGS "-Werror")
endif()
</code></pre>
<p>Is this special treatment of the file really necessary? If so, one should maybe add <code>-Wno-dangling-reference</code>.</p>

ptrblck: <p>Could you create a Github issue for this warning/error so that the actual warning might get fixed, please?</p>

Matthias_Moller: <p>Done <a href="https://github.com/pytorch/pytorch/issues/105728" rel="noopener nofollow ugc">#105728</a>. Thank you!</p>
How to perform a shallow copy of the torch module?

amine_ammor: <p>Hello,<br>
I want to copy perform a “shallow” copy of the “torch” module .The reason for that is that I want to modify the torch.Tensor module, by adding to it some custom behavior, in specific context.</p>
<p>Here is the code that I tried so far (I works by replacing torch by the os library)</p>
<pre><code class="lang-auto">
import sys
import importlib.util

SPEC_torch = importlib.util.find_spec('torch')
torch1 = importlib.util.module_from_spec(SPEC_torch)
SPEC_torch.loader.exec_module(torch1)
sys.modules['torch1'] = torch1

torch2 = importlib.util.module_from_spec(SPEC_torch)
SPEC_torch.loader.exec_module(torch2)
sys.modules['torch2'] = torch2
del SPEC_torch

assert torch1 is not torch2
</code></pre>
<p>Though, when executing the line <code>torch1 = importlib.util.module_from_spec(SPEC_torch)</code>, pytorch fails<br>
with the following output <code>"NameError: name '_C' is not defined"</code></p>
<p>I found an issue that reference this problem, but with a different context “<a href="https://github.com/pytorch/pytorch/issues/1633" class="inline-onebox" rel="noopener nofollow ugc">Error during import torch, NameError: name '_C' is not defined · Issue #1633 · pytorch/pytorch · GitHub</a>”, and whose proposed solutions, does not resolve mine.</p>
<p>I also tried to modify , the attributes of torch.Tensor after applying a deepcopy on it, but it resulted on the modification of the original torch.Tensor class, which is not the targeted behavior  … (maybe it is linked to the fact that torch.Tensor is a builtin class).</p>
<p>I am looking for a solution  which is thread_safe ( the proposed method would be thread safe, because we have a duplicate class of torch.Tensor, which could be concurrentialy used with the original “torch.Tensor”'s class ), and which works with the routines of torch.nn.functionnal (defining a new class of torch.Tensor with the “type” method doesn’t enable this features,unfortunatly).</p>
<p>Thank you again for your time.</p>

ptrblck: <p>I don’t know how to solve your original error but:</p>
<aside class="quote no-group" data-username="amine_ammor" data-post="1" data-topic="184694">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amine_ammor/48/58265_2.png" class="avatar"> amine_ammor:</div>
<blockquote>
<p>The reason for that is that I want to modify the torch.Tensor module, by adding to it some custom behavior, in specific context.</p>
</blockquote>
</aside>
<p>sounds as if <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-with-a-tensor-like-type">extending the tensor class</a> might fit your use case.</p>
PyTorch package to calculate SSIM score between images

shrbrh: <p>Hello, is there a PyTorch package or in-built method in PyTorch that I can use to find the SSIM score between two sets of images?</p>

ptrblck: <p><a href="https://pytorch.org/ignite/generated/ignite.metrics.SSIM.html">Ignite</a> and <a href="https://torchmetrics.readthedocs.io/en/stable/image/structural_similarity.html">torchmetrics</a> seem to provide these method.</p>
How to extract KITTI 3D BOUNDING BOX DATASET?

chi_hiwhy: <p>please anyone can know how to extract KITTI 3d-bounding box object detection dataset for training deep neural networks.</p>
<p>Last of fall what data look like:<br>
Image and 3d-bounding-box similar to 2d-bounding box dataset.</p>
<p>Than I train model input as image and predict 3d-bounding box</p>
Can't reproduce training process even if i have fix random seed on different platforms

jimjimjimjimjimjimji: <p>I want to reproduce on two different platforms. I fixed the random seed with the following code:</p>
<pre><code class="lang-python">def FixedSeed(seed: int = 1122) -&gt; None:
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True
</code></pre>
<p>The training process is reproducible on each platform, but the training process is not exactly the same on different platforms. Does anyone know what is causing the problem?</p>

KFrank: <p>Hi Jim!</p>
<aside class="quote no-group" data-username="jimjimjimjimjimjimji" data-post="1" data-topic="184645" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jimjimjimjimjimjimji/48/59923_2.png" class="avatar"> jimjimjimjimjimjimji:</div>
<blockquote>
<p>the training process is not exactly the same on different platforms.</p>
</blockquote>
</aside>
<p>This is to be expected.  Pytorch does not offer any assurance that<br>
computations will be exactly reproducible across platforms.</p>
<p>As long as your results agree within reasonable round-off error (say,<br>
after a single forward and backward pass – deviations may accumulate<br>
if you train for many iterations), you are doing as well as can be expected.</p>
<p>Best.</p>
<p>K. Frank</p>

jimjimjimjimjimjimji: <p>Thanks for your answer!</p>
Quantization error about _conv_transpose2d

sharkghost4: <p>Hey,Guys!I had tried to use dequant and quant to ignore the convtranspose2d like this:</p>
<pre><code class="lang-auto">class Conv2dTranspose(nn.Module):
    def __init__(self, cin, cout, kernel_size, stride, padding, output_padding=0, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.conv_block = nn.Sequential(
                            nn.ConvTranspose2d(cin, cout, kernel_size, stride, padding, output_padding),
                            nn.BatchNorm2d(cout),
                            nn.ReLU()
                            )

        #self.act = nn.ReLU()
        self.quant = torch.ao.quantization.QuantStub()
        self.dequant = torch.ao.quantization.DeQuantStub()

        def get_middle_todequant(module,args):
            output = self.dequant(args[0])
            module.qconfig = None
            return (output)

        def get_middle_toquant(module,args,output):
            output = self.quant(output)
            return output
        self.handle = self.conv_block[0].register_forward_pre_hook(get_middle_todequant)
        self.handle2 = self.conv_block[0].register_forward_hook(get_middle_toquant)
    def forward(self, x):
        out = self.conv_block(x)
        # out = self.dequant(out)
        return out

</code></pre>
<p>but still has an error like this:</p>
<pre><code class="lang-auto">NotImplementedError: Could not run 'aten::slow_conv_transpose2d.out' with arguments from the 'QuantizedCPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build).
</code></pre>
<p>My torch version is 2.0.1+cu17,Is there any problems?</p>

HDCharles: <p>either your stubs aren’t getting converted to actual quant/dequant or your prehook isn’t actually updating the output (i think you might need to either modify args directly or not use hooks for this)</p>

sharkghost4: <p>Thank you for providing the idea. In fact, the hook did not work properly. I have modified the code to ensure that the input and output of the ConvTranspose2d layer use quant and dequant</p>
How can I disable quantization for specific layers

_moonlight: <p>Hi all, how can i disable quantization for specific layers or op in torch.fx?</p>

HDCharles: <p>you set it to match one of the QConfigMappings with qconfig=None</p>

_moonlight: <p>I got it, i tried to use module_name option in qconfig to do mix quantization, but i didn’t realized it can also be set as None. Thank you a lot!</p>
When to stop training on model that isn't improving

Spencerfonbuena: <p>I’m training a time series transformer on 60GB of data. Previously, when my model wasn’t improving after a few epochs, I would stop it, make some tweaks to the model, data, or hyperparameters, and re-run it. However, that was when I was training the model with much less data. Now that I am training it on a much larger dataset, I am unsure if the model isn’t learning due to limited capacity of the model itself, or simply because it needs more time to go through the data. For context, it will take my model about a week to go through 1 epoch, which is a lot of time to leave a broken model running. I unfortunately don’t have access to more computing power, and have parallelized and increased batch size to my systems limits.</p>
<p>When I went through previous training runs, there were instances that loss would plateau at the beginning, and then learn after a few hours of training. My concern with stopping it before it has gone through one epoch, is that i may be stopping a good model too soon. However, if my model has been running for a few days with no real improvement, is that always indicative of a poor model? Any ideas would be much appreciated.</p>
Custom function with nn.module inside nn.module class

ykovachev: <p>Hi, using <code>nn_module</code> (<a href="https://torch.mlverse.org/docs/reference/nn_module.html" class="inline-onebox" rel="noopener nofollow ugc">Base class for all neural network modules. — nn_module • torch</a>)  in R I can do the following:</p>
<pre><code class="lang-auto">SomeClass &lt;- nn_module(
     initialize = function(some_params){
     ...
     ...
     ...
     },
    forward = function(some_params){
    ...
    ... 
    ...
    },
    custom_function = function(some_params){
         nn_module(
               initialize = function(some_params){
               ...
               ...
               ...
               },
               forward = function(some_params){
               ...
               ...
               ...
              }
         )
     }
)
</code></pre>
<p>and so I can call <code>custom_function()</code> on <code>SomeClass</code> instances and they will return an <code>nn_module</code>. In PyTorch can I write inside of a custom <code>nn.Module</code> class a custom function which returns a <code>nn.Module</code> similar to what the R pseudocode above does?</p>
<p>I tried using a nested class and passing <code>some_params</code> to the <code>__init__()</code> function of the inner class but that did not work.</p>
How to get the vocal quickly from huge list of texts?

RKnowledge: <p>Hi!</p>
<p>I have been following the <a href="https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html" rel="noopener nofollow ugc">tutorial: Language Translation With Torchtext</a> and in the tutorial they used this function to get a <code>vocab</code> object:</p>
<pre><code class="lang-auto">def build_vocab(filepath, tokenizer):
  counter = Counter()
  with io.open(filepath, encoding="utf8") as f:
    for string_ in f:
      counter.update(tokenizer(string_))
  return vocab(counter, specials=['&lt;unk&gt;', '&lt;pad&gt;', '&lt;bos&gt;', '&lt;eos&gt;'])
</code></pre>
<p>(By the way, I have changed the <code>Vocab</code> in the tutorial to <code>vocab</code> which is up to date with the current version of torch text)</p>
<p>So I have tried it on my own data which is a text file containing about 10M sentences and it is estimate to end after 3 hours (I’m using the CPU from Colab). I was wondering if there is a more efficient way to do this.</p>
<p>Thank you!</p>

RKnowledge: <p>I have changed my function to:</p>
<pre><code class="lang-auto">def build_vocab(data_iter, tokenizer):
    def yield_tokens(data_iter, tokenizer):
      for data_sample in data_iter:
        yield tokenizer(data_sample)

    special_symbols = special_symbols_dic.keys()
    vocab = build_vocab_from_iterator(
        yield_tokens(data_iter, tokenizer),
        min_freq = 1,
        specials = ["&lt;unk&gt;", "&lt;pad&gt;", "&lt;bos&gt;", "&lt;eos&gt;"],
        special_first = True
    )
    vocab.set_default_index(0)
    return vocab
</code></pre>
<p>I have also changed the tokenizer to:</p>
<pre><code class="lang-auto">from torchtext.data.utils import get_tokenizer
tokenizer = get_tokenizer("spacy", "en_core_web_sm")
</code></pre>
<p>And this reduce the necessary time to process my data to about 15minutes.</p>
<p>I’m sure we can optimize even longer with tokenizers from Huggingface and using batching and maybe also use a different vocab object that can be built without iterators but does batching too, but I haven’t checked that out yet.</p>
How do I pass a cuda event from python to a c++ operation?

92hyungjun: <p>Hello,<br>
I’m trying to pass a “cuda event” from python to cpp function, but I’m getting an error.</p>
<p>torch version is 1.13.0a0+d0d6b1f.</p>
<p>python code</p>
<pre><code class="lang-auto">event = torch.cuda.Event()
my_fn(event)
</code></pre>
<p>c++ code</p>
<pre><code class="lang-auto">void my_fn(at::cuda::CUDAEvent event){
   ...
}
</code></pre>
<p>err msg.</p>
<pre><code class="lang-auto">/opt/conda/lib/python3.8/site-packages/torch/include/pybind11/cast.h:1441:37: error: use of deleted function ‘at::cuda::CUDAEvent::CUDAEvent(const at::cuda::CUDAEvent&amp;)’
</code></pre>
<p>Is there any other way to pass “cuda event” from python to cpp?<br>
Thanks in advance!</p>
How to install PyTorch 1.7.1 on Mac M1?

AZH: <p>Hi there, anyone has successfully installed PyTorch 1.7.1 on Mac M1?<br>
I failed to do so using either Pip or Conda. Basically the error messages say there is no suitable package found from the usual channels.<br>
I also checked <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a> , and found there is indeed no *.whl file for v1.7.1 working for arm64.<br>
Any suggestions? I’d much appreciate it!</p>

ptrblck: <p>Based on the <a href="https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/">announcement blog post</a> <code>torch==1.12</code> was the first release supporting this OS with binaries.</p>

AZH: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> ! Yes that makes sense!</p>
Tensor.backward does not work unless create_graph = true

Yeb02: <p>Windows 10, using libtorch 2.0.1 GPU version but running on a single CPU.</p>
<p>I am running an experiment with a model that has an atypical architecture. I can not call .backward more than once unless I specify create_graph = true (i.e. <code>out.backward({}, true)</code>). In this case I get the warning :</p>
<blockquote>
<p>[W C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\autograd\engine.cpp:1156] Warning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (function operator ())</p>
</blockquote>
<p>I fear the code’s behaviour is not what I intended. The architecture’s training on one sample goes like this:</p>
<p>Net A takes a 1D vector as input, outputs a 1D vector.<br>
Nets B0 takes as input subparts of Net A’s output, that are stitched and reshaped into a batch of 1D vectors. It outputs a batch of 1D vectors, which is compared (MSE loss) to the expected output. Loss.backward is then called.<br>
The process is repeted sequentially for nets B1, B2, … BN. All B nets have exactly the same architecture, but different weights. Their respective inputs are made of different (but overlapping) subparts of Net A’s output.</p>
<p>In pseudo code:</p>
<p><code>A_Input = from_blob(rawInput)    </code><br>
<code>A_Output = NetA(A_Input);</code></p>
<p><code>for i from 0 to N {</code></p>
<p><code>   B_Input = assemble(A_Output, i); // operations in this function let gradients flow through</code><br>
<code>   B_Output = NetB(B_Input);</code><br>
<code>   out = mse_loss(B_output, targets[i]);</code><br>
<code>  out.backward();  // does not work, I have to use out.backward({}, true)</code></p>
<p><code>}</code></p>
<p>When using <code>out.backward()</code>, I get a runtime error, <strong>on the second loop iteration</strong>, in <em>libtorch_2.0.1_gpu_DEBUG_cu1_18\libtorch\include\ATen\core\ivalue.inl.h</em>,   line 975,  in <code>IValue value()</code>.</p>
<p>Do you have any idea of what could cause this issue ?</p>
Any way to get model name

DivyanshJha: <p>I am asking if I have a resnet34 model. I can do something like <code>model.__name__</code> and it returns a string as <code>resnet34</code>.</p>

ptrblck: <p>Try <code>model.__class__.__name__</code> to get the name.</p>

DivyanshJha: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="12877" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Try <code>model.__class__.__name__</code> to get the name.</p>
</blockquote>
</aside>
<p>It returns only the base class name which is <code>resnet</code> not <code>resnet34</code></p>

ptrblck: <p>Could you post the model definition?<br>
This seems to work or did I misunderstand your question?</p>
<pre><code>class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
    
    def forward(self, x):
        return x + 1

model = MyModel()            
print(model.__class__.__name__)
&gt;&gt; MyModel</code></pre>

DivyanshJha: <p>I am using <code>torchvision.models.resnet34</code>.<br>
What I want is</p>
<pre><code class="lang-auto">model = torchvision.models.resnet34(pretrained=True)
print(model.__class__.__name__)
</code></pre>
<p>to print</p>
<p><code>resnet34</code></p>
<p>but currently it prints<br>
<code>ResNet</code></p>

ptrblck: <p>Ah ok, I see.<br>
<code>resnet34()</code> is just a function for constructing the appropriate model. The model itself is just the <code>ResNet</code>.</p>
<p>However, you could simply add a new parameter to your model:</p>
<pre><code>model = MyModel()
model.name = 'ResNet34'
print(model.name)
</code></pre>
<p>Will this meet your needs?</p>

DivyanshJha: <p>Yeah! I also came up with a workaround for myself similar to this.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="12877">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Will this meet your needs?</p>
</blockquote>
</aside>
<p>Thank you for the help!</p>

AreTor: <p>Has this been somehow integrated in PyTorch? I often need the model name when storing results generated by a specific model. This requires me to either bring together a <code>model_name</code> string along with the model itself during the function calls or to monkey patch the class (or an instance) by adding a <code>model.name</code> attribute.</p>
<p>As of today, are there better ways to access the name of a model?</p>
Check the batch size of dataloader

plant3032: <p>Hi, i’m using the dynamic batch size (changing the batch size during training, and i implemented that by using the custom batch sampler, and partly fixing the pytorch’s dataloader code. plus, the method of my custom batch sampler is quite similar to pytorch’s Batchsampler), and i want to check whether actual batch size is changed or not.</p>
<p>in this case, what should i have to do? at first, i tried to check the batch size by printing like this code,</p>
<pre><code class="lang-auto">print(train_dataset.batch_size)
</code></pre>
<p>but in here, i checked that batch size isn’t change as i expected,  but at the custom batch sampler, when i checked the batch size, it changed right as i expected</p>
<p>and sorry that i can’t attach the whole code of CustomBatchSamler because of the security, and if anyone want to see whole code for solving the problem, i can send it to your e-mail or any way you prefer</p>
<p>thanks a lot for your all response <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Need help for tesla k40m gpu

themohal: <p>Hi, i am working as intern at company and they have provided me gpu for learning and training. So far what i have done is I installed tf==2.10 with cuda toolkit 11.2.0 and cudnn 8.1.0 i was able to detect gpu with this. Now i want to install torch so that it detects the gpu which version of torch and cuda should i install? As i am unable to detect gpu with torch 2.0.1+cpu. I have also tried to find cuda version with torch.cuda.version but output is none. i am gonna use ultralytics and easyocr library for my work please help. My environment is just CLI and nothing else like in cmd: python example.py and i am using pip to download packages and then installing them as my server is offline and has OS windows server 2016 running on it.</p>

ptrblck: <p>Your Kepler GPU is too old and is not supported in the PyTorch binaries anymore, as the binaries with CUDA 11.8 support sm_37-sm_90 while the binaries with CUDA 12.1 support sm_50-sm_90.<br>
You could try to rebuild PyTorch from source for compute capability 3.5.</p>

themohal: <p>Thank you for the RepPly. From where i can rebuild and which version?</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch#from-source">Here</a> are the build instructions.<br>
You should be able to build PyTorch from <code>main</code> as long as your used CUDA toolkit supports your architecture. Since your TF installation seems to work I would recommend using 11.2 to match it.</p>

themohal: <p>There is not pytorch 11.2 with cuda. Please suggest if i install cudatookit 11.3 with <code>torch==1.12.1+cu113</code> will this work ? As i can see the drivers only till 11.2</p>

ptrblck: <p>It’s not a question about the used CUDA toolkit, but rather which PyTorch version supports your GPU. As mentioned before, the PyTorch binaries dropped support for sm_35 a while ago and I don’t remember which version was supporting it, but think it was removed in <code>torch==1.9.0</code>.</p>

themohal: <p><a href="https://github.com/moi90/pytorch_compute_capabilities/blob/main/table.md" rel="noopener nofollow ugc">https://github.com/moi90/pytorch_compute_capabilities/blob/main/table.md</a> can you check this table and confirm if this is authentic and would work?</p>

ptrblck: <p>No sorry, I didn’t create or maintain this table so don’t know how it was created.<br>
To get the supported GPU architectures for current builds you can use <code>print(torch.cuda.get_arch_list())</code> and compare it to the table (or ask the author).</p>
Use `torch.fx.Graph` from `torch.dynamo`

kilianyp: <p>We are using <code>torch.fx</code> to find functional calls like <code>torch.add(x, x)</code> and replace them with a call to a module.</p>
<p>This sometimes fails because <code>torch.fx.tracing</code> cannot handle some cases.</p>
<p><code>torch.dynamo</code> however seems to be much more robust in generating the <code>torch.fx.Graph</code> then torch.fx.</p>
<p>Is there a path forward to integrate <code>torch.fx.</code> and <code>torch.dynamo</code> closer together?</p>
<p>I managed to get the <code>torch.fx.graph</code> out of by saving the graph through a custom backend. However, it is not as usable as the <code>torch.fx.trace</code> generated graph, because:</p>
<ul>
<li>in some cases one gets multiple graphs. It’s unclear how one needs to connect the multiple graphs, e.g. how does the output from one graph passed to another graph.</li>
<li>the module names are rewritten. They all get names like <code>self_module_conv</code>. I would like to maintain them.</li>
</ul>

marksaroufim: <p>fx is a representation and dynamo is a tracer so they already compose together, the fx tracer is unlikely to continue receiving support though</p>
<p>Also is this the tutorial you’re referring to? <a href="https://colab.research.google.com/drive/1Zh-Uo3TcTH8yYJF-LLo5rjlHVMtqvMdf#scrollTo=MWUvovoo6KYK" class="inline-onebox">Google Colab</a> - I think <a class="mention" href="/u/sherlocknomad">@SherlockNoMad</a> explained to me how you can figure out which graph pipes to which other graph but I’m drawing a balnk</p>

kilianyp: <p>Hi <a class="mention" href="/u/marksaroufim">@marksaroufim</a>, thanks for the quick response <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
<p>Yes, I saw that one and it’s also explained in:<br>
<a href="https://pytorch.org/docs/stable/dynamo/custom-backends.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/dynamo/custom-backends.html</a></p>
<p>There’s a few subtile difference that don’t allow me using it for my usecase when following the tutorial.</p>
<ul>
<li>torch.dynamo kind of flattens the module. It doesn’t seem no longer possible back to link it back to a specific module as far as I can tell. I need that to be able to further configure the model after it has been traced.</li>
<li>the aformentioned multiple graphs. I think I understand the necessity, but an easy way (some way will already exists somewhere deep) to connect the outputs correctly</li>
</ul>
<p>Do you happen to know if there’s a timeline/roadmap on that one can look at? Or what’s the best way to get active myself?</p>

marksaroufim: <p>Probably the best way is to author a longer issue here <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a> describing your use case</p>
Memory Usage of a Modified Convolution

Marute1037: <p>I want to do a modified version of 1D convolution. Instead of computing the weighted sum, I want to compute the channel-wise distance between the kernel and the data. However, after computing the distance, the gradient takes a lot of memory and even more after backpropagation. Here is a sample code I wrote:</p>
<pre><code class="lang-auto">device = torch.device("cuda")
data = torch.rand(128, 5, 1000, device=device)
x = data.unfold(-1, 10, 1).permute(0, 2, 1, 3).unsqueeze(2)
w = torch.rand(200, 5, 10, requires_grad=True, device=device)
print(f"GPU memory usage: {torch.cuda.memory_reserved(0)/1e9:.2f}")
d = (x - w).norm(dim=-1)
print(f"GPU memory usage: {torch.cuda.memory_reserved(0)/1e9:.2f}")
loss = d.mean()
print(f"GPU memory usage: {torch.cuda.memory_reserved(0)/1e9:.2f}")
loss.backward()
print(f"GPU memory usage: {torch.cuda.memory_reserved(0)/1e9:.2f}")
</code></pre>
<p>I got the following output:</p>
<pre><code class="lang-auto">GPU memory usage: 0.53
GPU memory usage: 6.11
GPU memory usage: 6.11
GPU memory usage: 16.39
</code></pre>
<p>This is not an issue when using <code>nn.Conv1D()</code> probably because some low-level optimisation. Is there any way to make the distance computation more memory efficient?</p>
<p>Thanks for your help.</p>

soulitzer: <p>By unfolding data, you are creating an intermediate tensor x that is 10 times the size - you materialize that when you do sub, and then promptly reduce it away. One way to reduce memory when you do this is to do it piece of piece and accumulate the summation in the norm as you go. You’ll need to figure out how to manipulate the shapes in the right way though.</p>
<p>Btw you’ll want to use <a href="https://pytorch.org/docs/stable/generated/torch.cuda.memory_allocated.html#torch.cuda.memory_allocated" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda.memory_allocated — PyTorch 2.0 documentation</a> (or peak_memory_allocated) instead of memory_reserved. Even after tensors get cleared, PyTorch will still hold on to that memory rather than return it to CUDA for efficiency reasons, the amount of data that PyTorch holds onto is the value returned by memory reserved whether or not that memory corresponds to actual tensors that are alive.</p>

Marute1037: <p>Thanks a lot for your answer! I checked the memory usage using <code>torch.cuda.max_memory_allocated()</code> and the peak memory usage is indeed very high because of intermediate variables.</p>
<p>My thought is that, mathematically, the distance computation should have similar complexity as typical convolution. Currently, I am trying to learn <code>autograd.Function</code> which seems like a solution to this issue by first computing the gradient by hand and prevent storing intermediate results <a href="https://pytorch.org/docs/stable/notes/extending.html" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 2.0 documentation</a></p>
<p>Thanks again for your answer.</p>
Custom dataset with unknown length

David_Sriker1: <p>I have a custom mux dataset that hold k different datasets.</p>
<p>I train my model with iteration rather than epochs so I want to make sure the mux dataset is infinite dataset object since the dataloader sometimes get stuck when I recreate the iterator using the following:</p>
<pre><code class="lang-auto">def endless_iterator(iterable: Iterator[any]):
    while True:
        for elem in iterable:
            yield elem
</code></pre>
<p>Now a valid solution is to set the <em>len</em> to return a very large number but I was wondering if there is a way to just make it work endlessly with next(data_loader) without the need of my endless_iterator.</p>

ptrblck: <p>In case you are using <code>num_workers&gt;0</code> you could use <code>persistent_workers=True</code>, which would keep the workers alive without tearing them down at the end of an epoch.</p>