akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Difference between Pruning Techniques

Keshvi_Tomer: <p>So according to the pytorch documentation, it offers a lot of different network pruning techniques. 2 of them being <code>prune.L1Unstructured</code> which Prune (currently unpruned) units in a tensor by zeroing out the ones with the lowest L1-norm and there’s <code>prune.l1_unstructured</code> which Prunes tensor by removing units with the lowest L1-norm. I want to know how these two are different?  And if they cause a change on the neural net’s architecture?</p>
<p>Can someone help?</p>
Unable to run code on Multiple GPUs in PyTorch - Usage shows only 1 GPU is being utilized

abidmeeraj: <p>I am training a Transformer Encoder-Decoder based model for Text summarization. The code works without any errors but uses only 1 GPU when checked with nvidia-smi. However, I want to run it on all the available GPUs (I can access as many as I want). I wrapped my model in Dataparallel.<br>
<strong>Here’s how I have wrapped the model:</strong></p>
<pre><code class="lang-auto">    if torch.cuda.device_count() &gt; 1:
        print(f"Using {torch.cuda.device_count()} GPUs")
        model = nn.DataParallel(model)

    model.to(device)
</code></pre>
<p><strong>This is how I am calling the functions from my model:</strong></p>
<pre><code class="lang-auto">            if torch.cuda.device_count() &gt; 1:
                encoder_output = model.module.encode(
                    encoder_input, encoder_mask
                )  # (B, input_len, d_model)
                decoder_output = model.module.decode(
                    encoder_output, encoder_mask, decoder_input, decoder_mask
                )  # (B, seq_len, d_model)
                proj_output = model.module.project(
                    decoder_output
                ) 
</code></pre>
<p><strong>I am using Python 3.12 and Torch 2.3.0.</strong></p>
<p>MWE is available on <a href="https://github.com/abidmeeraj/MWE-problem-with-MultipleGPUs.git" rel="noopener nofollow ugc">GitHub</a>.</p>
<p>I have also checked if GPUs are configured correctly. Tried this example from <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="noopener nofollow ugc">PyTorch Documentation</a> to test my GPUs configuration and working.</p>

ptrblck: <p>In your provided code snippets you are using the deprecated <code>nn.DataParallel</code> module first, but are then skipping it by accessing the internal <code>.module</code>.<br>
Call the model directly and let <code>DataParallel</code> handle the data splits as well as model copies before it calls the <code>forward</code>.</p>

abidmeeraj: <p>OMG. That was horrible part to miss. Using the forward() resolved the issue.</p>
Frombuffer() → "The given buffer is not writable"

Geremia: <p>¿Why when I run:</p>
<pre><code class="lang-auto">file = open('/tmp/big.bin', 'rb')
file_bytes = file.read()
mytensor = torch.frombuffer(file_bytes, dtype=torch.float32).reshape(a, b)
</code></pre>
<p>do I get this:</p>
<blockquote>
<p>UserWarning: The given buffer is not writable, and PyTorch does not support non-writable tensors. This means you can write to the underlying (supposedly non-writable) buffer using the tensor. You may want to copy the buffer to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /tmp/SBo/pytorch-v2.3.1/torch/csrc/utils/tensor_new.cpp:1524.)</p>
</blockquote>
<p>Why is the file buffer not writable? (It gives this warning regardless if I run <code>file.close()</code> before <code>frombuffer()</code> or not.) And what does the file buffer’s writability have to do with the tensor’s writability?</p>
<p>Also, <code>mytensor</code> has some <code>nan</code>s in it. I don’t get a tensor with any <code>nan</code>s when I use <code>torch.uint8</code> as the <code>dtype</code>.</p>

smth: <p>you’ve opened the file in <code>rb</code> mode which stands for read-only binary mode.</p>
<blockquote>
<p>And what does the file buffer’s writability have to do with the tensor’s writability?</p>
</blockquote>
<p><code>from_buffer</code> uses Python’s buffer interface and attempts to directly map to the underlying buffer without copying the data. So, the tensor is directly mapping to the file, it is not separately materializing or copying the data from the buffer into new memory. So <code>torch</code> is warning you that attempts at changing the <code>tensor</code> values might result in a new tensor and those changes aren’t reflected in the file itself.</p>
<blockquote>
<p>Also, <code>mytensor</code> has some <code>nan</code> s in it. I don’t get a tensor with any <code>nan</code> s when I use <code>torch.uint8</code> as the <code>dtype</code> .</p>
</blockquote>
<p>Sure, that depends on the contents of the file.</p>
One GPU per Thread vs multiple GPUs per CPU thread

dimpo: <p>PyTorch docs make the following statement:</p>
<blockquote>
<p>The multi-GPU functions (which stand for multiple GPUs per CPU thread) are deprecated. As of today, PyTorch Distributed’s preferred programming model is one device per thread, as exemplified by the APIs in this document.</p>
</blockquote>
<p>(see here: <a href="https://pytorch.org/docs/stable/distributed.html#multi-gpu-collective-functions" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 2.3 documentation</a>)</p>
<p>My question is why is that? What are the benefirst of the one device per thread programming model and what are the tradeoffs?</p>

ptrblck: <p>Python’s GIL would block the threads slowing down your code, which is why we recommend using a single process per device.</p>

dimpo: <p>Thanks fot the reply <a class="mention" href="/u/ptrblck">@ptrblck</a>. However, the docs talk about threads not processes. So GIL would be a problem in the case we have one GPU per thread, not one thread accessing multiple GPUs. What am I missing? Except it it is one single-threaded process per GPU, hence one thread per GPU.</p>

ptrblck: <aside class="quote no-group" data-username="dimpo" data-post="3" data-topic="204870">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dimpo/48/53921_2.png" class="avatar"> dimpo:</div>
<blockquote>
<p>one GPU per thread, not one thread accessing multiple GPUs. What am I missing?</p>
</blockquote>
</aside>
<p>If a single thread is responsible to drive the work of multiple devices, you would be back at <code>nn.DataParallel</code> with its shortcoming of serialized launches etc. A lot of models in <code>DDP</code> already suffer from CPU overheads (the CPU is not fast enough to launch the work on the GPU) and thus benefit from using CUDA Graphs. If you put more pressure on the CPU by forcing it to launch the work of <em>all</em> GPUs, the overhead would become even worse and the launches would also be sequential (unless I’m missing something now).</p>

dimpo: <blockquote>
<p>If you put more pressure on the CPU by forcing it to launch the work of <em>all</em> GPUs, the overhead would become even worse and the launches would also be sequential (unless I’m missing something now).</p>
</blockquote>
<p>This is a different problem from the one posed by the Python GIL though, right?</p>
Implementing LeNet architecture

pratap_Vamshidhar: <p>Hello, I am new to Pytorch. I tried to implement LeNet architecture in PyTorch. But my result is bad when compared to the tutorial i followed.</p>
<p>from torch import nn</p>
<p>class LeNet5(nn.Module):</p>
<p>def <strong>init</strong>(self):</p>
<pre><code>super().__init__()
#initialize the layers
# Layer 1 which is convolution layer
self.conv1 = nn.Conv2d(in_channels = 1,
                       out_channels = 6,
                       kernel_size = 5,
                       stride = 1,
                       padding = 2
                       ) #28 X 28 -----&gt; 32 X 32 ----&gt; 28 X 28

# The reason we are adding padding beacause, the image size we have is 28 X 28.
# But in the original paper we have input size as 32 X 32.


#Layer 2 which is Average Pooling
self.pool1 = nn.AvgPool2d(kernel_size = 2,
                          stride = 2) # 14 X 14


# Layer 3 which is Convolution
self.conv2 = nn.Conv2d(in_channels = 6,
                       out_channels = 16,
                       kernel_size = 5,
                       stride = 1,
                       ) # 10 X 10

#Layer 4 AVG Pooling
self.pool2 = nn.AvgPool2d(kernel_size = 2,
                          stride = 2
                          ) # 5 X 5

#Layer 5 Fully connected

self.fc1 = nn.Linear(in_features = 16 * 5 * 5,
                     out_features = 120
                     )
#As the image size for this layer is 16 X 5 X 5 , the input features will
#400 (multiply). Paper used 120 as output features

# Layer 6 Fully connected
self.fc2 = nn.Linear(in_features = 120,
                     out_features = 84)

# Layer 7 Fully connected

self.fc3 = nn.Linear(in_features = 84,
                     out_features = 10)
</code></pre>
<p>def forward(self,input):</p>
<pre><code># conv2D -&gt; TanH -&gt; AVGpool

tanh = nn.Tanh()

c1 = tanh(self.conv1(input))
#In PyTorch, nn.Tanh is a module that applies the hyperbolic tangent function
#element-wise to the input tensor. It does not take any arguments during instantiation.
#Instead, you should call it directly on the input tensor.
p1 = self.pool1(c1)

#  conv2D -&gt; TanH -&gt; AVGpool
c2 = tanh(self.conv2(p1))
p2 = self.pool2(c2)

# AVGpool -&gt; FC1 -&gt; Fc2 -&gt;Fc3
f1 = tanh(self.fc1(torch.flatten(p2,1)))
f2 = tanh(self.fc2(f1))

f3 = self.fc3(f2)
# return the output
return f3
</code></pre>
<p>This is my code. I am not sure what i did wrong</p>
How to count the number of parameters in Yolov4?

Mahdi_Amrollahi: <p>How can I count the number of parameters in <a href="https://github.com/AlexeyAB/darknet" rel="noopener nofollow ugc">Yolov4</a> with custom configs when that is not implemented in Pytorch?</p>

ptrblck: <aside class="quote no-group" data-username="Mahdi_Amrollahi" data-post="1" data-topic="204929">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdi_amrollahi/48/62481_2.png" class="avatar"> Mahdi_Amrollahi:</div>
<blockquote>
<p>when that is not implemented in Pytorch?</p>
</blockquote>
</aside>
<p>You could ask the model / library author in the corresponding GitHub repository how this might work if it’s not written in PyTorch.</p>

Mahdi_Amrollahi: <p>Yes, Thanks, but I have fount no answer yet.</p>
When should we use torch.multiprocessing.spawn?

Smith_Jack: <p>Hello. I looked through some tutorials about <strong>DistributedDataParallel</strong>. Some of them use the <strong>spawn</strong> module however others said <strong>spawn</strong> should not be used (for example, <a href="https://github.com/pytorch/examples/tree/main/distributed/ddp" rel="noopener nofollow ugc">this page</a>, " 1. It must provide an entry-point function for a <em>single worker</em>. For example, it should not launch subprocesses using <code>torch.multiprocessing.spawn</code>").<br>
This make me very confused.</p>
<p>Besides, I have some other questions.</p>
<ol>
<li>If I have a single GPU, do I have to use ONE process when I call <strong>torchrun</strong> ? Can I use more than one process ?</li>
<li>In whcih case, will I need <code>--rdzv_id</code>,  <code>--rdzv_backend</code> and  <code>--rdzv_endpoint</code> ?  Do I have to use them on multiple nodes and multiple GPUs or single node and multiple GPUs or single node and CPU with multiple cores ?</li>
<li>If I use <strong>DistributedDataParallel</strong>. on a slurm system, each node has a CPU with many cores, for example, 32. How should I use torchrun ?<br>
and in this case, my script should be</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">dist.init_process_group("gloo") #do not use rank=rank, world_size=world_size
model = ToyModel() #do not use .to(rank)
ddp_model = DDP(model) #do not use device_ids=[rank]
</code></pre>
<p>is this right ?</p>

ptrblck: <ol>
<li>Since <code>DDP</code> uses a single process per device, you would also use one process for your single GPU or you could also just ignore <code>torchrun</code> and execute your single-GPU script directly.</li>
<li>These <code>rdzv</code> settings are used for <a href="https://pytorch.org/docs/stable/elastic/run.html#note-on-rendezvous-backend">multi-node runs</a> and can be used to launch multiple runs on a single node as seen <a href="https://pytorch.org/docs/stable/elastic/run.html#stacked-single-node-multi-worker">here</a>.</li>
<li>I don’t know how pure CPU DDP works.</li>
</ol>
Running Pytorch on Azure GPU VM NV12ads_V12

kaggle_user: <p>Hello All,<br>
I am facing issue on trying to run Pytorch on Windows GPU VM (Azure NV12ads_V12).<br>
Which has GPU Partition 1/3 (8GB out of  24 GB is allocated to that VM).</p>
<p>Attaching a screenshot where I do very simple calculation.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57ad0eaa0e9b9eafff39cf5a8a8fd88b7acf6420.png" data-download-href="https://discuss.pytorch.org/uploads/default/57ad0eaa0e9b9eafff39cf5a8a8fd88b7acf6420" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57ad0eaa0e9b9eafff39cf5a8a8fd88b7acf6420.png" alt="image" data-base62-sha1="cvCjqT5t5LcQM4uB0TAvQv6oW88" width="690" height="236" data-dominant-color="E4DDE6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">796×273 8.29 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But facing this error.</p>
<p>I have tried all the things that internet suggested,<br>
With various pytorch versions and cuda versions as well but no luck.</p>
<p>Does anybody face the similar issue ?<br>
What can be done to solve the issue ?</p>
<p>Thanks in Advance.</p>

ptrblck: <p>Your Windows driver seems to kill long running kernels as explained <a href="https://learn.microsoft.com/en-us/windows-hardware/drivers/display/timeout-detection-and-recovery">here</a>. I don’t know if and how these thresholds can be changed as I’m not using Windows.</p>
Why no macosx x86_64 build after torch-2.2.2-cp39-none-macosx_10_9_x86_64.whl

Dianjun_Zhao: <p>After torch-2.2.2-cp39-none-macosx_10_9_x86_64.whl, I don’t see any build for 2.3.0 and on in <a href="https://download.pytorch.org/whl/torch/" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch/</a>?<br>
Why?</p>

ptrblck: <p>PyTorch macOS x86 builds were deprecated as explained <a href="https://dev-discuss.pytorch.org/t/pytorch-macos-x86-builds-deprecation-starting-january-2024/1690">here</a> and <a href="https://github.com/pytorch/pytorch/issues/114602">here</a>. <code>torch==2.2.x</code> was the last release for it.</p>

Dianjun_Zhao: <p>Thanks.</p>
<p>Is it possible I build from source code on my MAC?</p>

ptrblck: <p>I would assume building from source might work, but haven’t tried it.</p>
Difference in pip install and conda install pytorch with GPU

sirius777coder: <p>I want to download pytorch with cuda=11.7. I find not only pip install but also conda install will download the cudatoolkit.<br>
The pip comand is</p>
<pre><code class="lang-auto">pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
</code></pre>
<p>The conda comand is :</p>
<pre><code class="lang-auto">conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
</code></pre>
<p>Why does these two methods will all download the cudatoolkit?</p>

ptrblck: <p>I’m not sure I understand the question completely, but both binaries ship with their CUDA dependencies and allow you to execute PyTorch code directly without installing a full CUDA toolkit locally. You would only need to install the NVIDIA drivers.</p>

sirius777coder: <p>That is what I need! Thanks so much. I find there is some slight difference between Pytorch versions. For example in v1.12.1 the download command is :</p>
<pre><code class="lang-auto">conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge
</code></pre>
<p>And in Pytorch v1.13.1 it is :</p>
<pre><code class="lang-auto">conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia
</code></pre>
<p>Does the v1.12.1 download the cudatoolkit while v1.13.1 only download CUDA runtime and other binaries such as cudnn and cublas rather than cudatoolkit?</p>

ptrblck: <p>The install commands can differ depending on our build process but both should install a PyTorch conda binary able to execute workloads on the GPU.</p>

sirius777coder: <p>Thanks! An awesome reply.</p>

liphoss: <p>Hi, I want to install pytorch with gpu on a machine with CUDA 12.3. I tried with a virtualenv with pip and a conda environment. In the conda environment torch.cuda.is_available yields true but in the virtualenv it yields false.<br>
Is there a difference between the binaries sent or how the library is built between the two ?<br>
(I tried installing the 2.3.1 version with cuda 12.1 in both cases for reference)</p>

sirius777coder: <p>I’m sorry I don’t use python virtual environment. In my opinion pytorch installation in conda environment only needs nvidia driver rather than nvcc. Maybe in your virtualenv you need to check your nvcc is installed.</p>

liphoss: <p>I checked with an install of nvcc on the virtualenv for cuda 12.3 but it didn’t work sadly.</p>

ptrblck: <p>The CUDA toolkit (including the compiler) is neither needed when installing the conda binaries nor the pip wheels.<br>
The binaries themselves are build using different processes.</p>
Number of communication kernels don't match between workers in run

Osama_Abu_Hamdan: <p>Distributed View is not available, and I think due to this error</p>
<pre><code class="lang-auto">E0618 17:14:15.276058 131298609845824 loader.py:150] Number of communication kernels don't match between workers in run: gpu_resnet50_cifar10_ddp_batch512_precision32_nodes3
</code></pre>
<p><strong>Data Collection Env:</strong><br>
Python version: 3.11.7<br>
GCC (GCC) 12.2.0<br>
Torch: ‘2.3.1+cu121’<br>
PyTorch lightning: ‘2.3.0’<br>
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch<br>
Distributor ID: CentOS<br>
Description: CentOS Linux release 7.8.2003 (Core)<br>
Release: 7.8.2003<br>
Codename: Core<br>
SLURM environment<br>
Cuda 12.4.1<br>
DeepSpeed 0.14.3</p>
<p><strong>Data Visualization Env:</strong><br>
MacBook Air M2<br>
OS: Version 14.5 (23F79)<br>
tensorboard==2.17.0<br>
tensorboard-data-server==0.7.2<br>
tensorboard_plugin_profile==2.15.1<br>
tensorboardX==2.6.2.2<br>
torch-tb-profiler==0.4.3</p>
Issue with installing torch, torchvision and torchaudio

Ninad: <p>Tried:- pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a></p>
<p>ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.<br>
torch from <a href="https://download.pytorch.org/whl/cu118/torch-2.3.1%2Bcu118-cp312-cp312-win_amd64.whl#sha256=f44c7b64d990a6b1a382d1cd63c359806153974e7db8d16f6780645a8a9c9fe0:" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118/torch-2.3.1%2Bcu118-cp312-cp312-win_amd64.whl#sha256=f44c7b64d990a6b1a382d1cd63c359806153974e7db8d16f6780645a8a9c9fe0:</a><br>
Expected sha256 f44c7b64d990a6b1a382d1cd63c359806153974e7db8d16f6780645a8a9c9fe0<br>
Got        407dc9e3ed038c589cfdb6e9005b0768916b4db075f35e94a953eac3b14f6d33</p>

ptrblck: <p>CC <a class="mention" href="/u/malfet">@malfet</a> in case you’ve seen these Windows wheel issues before.</p>

malfet: <p>Most likely local cache is corrupted, clean it and try downloading again. Correct sha256 for this file is indeed <code>f44c7b64d990a6b1a382d1cd63c359806153974e7db8d16f6780645a8a9c9fe0</code>:</p>
<pre><code class="lang-auto">$ curl -OL https://download.pytorch.org/whl/cu118/torch-2.3.1%2Bcu118-cp312-cp312-win_amd64.whl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2549M  100 2549M    0     0   376M      0  0:00:06  0:00:06 --:--:--  376M
$ sha256sum torch-2.3.1%2Bcu118-cp312-cp312-win_amd64.whl 
f44c7b64d990a6b1a382d1cd63c359806153974e7db8d16f6780645a8a9c9fe0  torch-2.3.1%2Bcu118-cp312-cp312-win_amd64.whl

</code></pre>

Ninad: <p>Hey I was able to solve the issue, I reinstalled pip and it worked it for me. Thank you <a class="mention" href="/u/malfet">@malfet</a> and <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
Use pytorch cuda for ridge regression

nikiguo93: <p>Hi, I have big datasets and can anybody help with using pytorch cuda for ridge regression?</p>

ptrblck: <p>Could you describe what you’ve tried so far and where you are stuck at the moment?</p>

nikiguo93: <p>ok. I used sklearn for ridge regression. For the datasets, the activation features are from various DNN layers, thus it takes a lot of time. I would like to map activation features to fMRI data, from example, activation feature (of a specific layer) of size 10000 * 7896, fMRI feature of size 10000 * 2345. I have 10000 samples here. When I use sklearn it takes a lot of time. Thus I wonder whether I can use torch.nn.linear to realize that, using cuda.</p>

nikiguo93: <p>for example, can we convert this into pytorch code:</p>
<h1><a name="keras-approach-1" class="anchor" href="#keras-approach-1"></a>Keras approach</h1>
<p>import keras<br>
from keras.layers import Input, Dense<br>
from keras.models import Model</p>
<p><span class="hashtag-raw">#Define</span> the model<br>
def regressor_model(m_features, regularizer=None, learning_rate=0.01):</p>
<pre><code>input_x = Input(shape = (m_features,))
lin_fn = Dense(1, activation = None, kernel_regularizer = regularizer)(input_x)
yx_model = Model(inputs = input_x, outputs = lin_fn)
sgd = keras.optimizers.SGD(lr=learning_rate)
yx_model.compile(loss = 'mean_squared_error', optimizer = sgd)

return yx_model
</code></pre>
<p>reg_par = [0, 0.001, 0.01, 0.1, 1]</p>
<p>for i_reg in reg_par:<br>
start = time.time()</p>
<pre><code># Training
yx_model = regressor_model(m_features,learning_rate=0.1,regularizer=keras.regularizers.l2(i_reg))
log_train = yx_model.fit(X, y, epochs = 50, batch_size = N, verbose = 0)

weights = yx_model.get_weights()

b = weights[0]

print("%.2f sec."%(time.time() - start), end=' - ')
print("Coefficients for x_1, x_2, x_3 are %.3f, %.3f, %.3f, respectively" %
      (b[0], b[1], b[2]))

# Testing
y_hat = yx_model.predict(X)
print("R square: %.3f"%r2_score(y,y_hat))           # R^2 (coefficient of determination) regression score function.
</code></pre>
<h1><a name="in-this-case-no-regularization-is-needed-2" class="anchor" href="#in-this-case-no-regularization-is-needed-2"></a>In this case, no regularization is needed!</h1>

ptrblck: <p>Yes, you can port the Keras model to PyTorch. E.g. Keras’ <code>Dense</code> layers would be replaced with PyTorch’s <code>nn.Linear</code> layers etc. If this is your first time creating a PyTorch model, <a href="https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html">this tutorial</a> might be helpful.</p>
Libtorch meson build

Daeyang_Cho: <p>Hi devs.<br>
I’m trying to make a cpp project that uses libtorch (C++ distributions of pytorch) using meson build.<br>
It has one simple cpp file of about 50 lines that runs deep learning on images.</p>
<p>First, I confirmed that my project runs well in two environments.</p>
<ol>
<li>Since meson uses the pkg-config file, I made a simple pkg-config file for libtorch, and it works for a project running on the CPU.</li>
<li>libtorch provides a TorchConfig.cmake file, so I used cmake to confirm that my project runs on the GPU version.</li>
</ol>
<p>However, I don’t know how to build the GPU version of the project using meson.<br>
The TorchConfig.cmake file is more complicated than I thought, so it was very difficult to make it manually with the pkg-config file.<br>
(TorchConfig.cmake file references many cmake files in libtorch directories.)</p>
<p>I also tried to use libtorch_dep = dependency(‘Torch’, method : ‘cmake’), but it only found libtorch.so among many libtorch libraries that are for GPU APIs.</p>
<p>So, how can I build the project using libtorch with only cmake config file like this with meson?<br>
Or is there a way to utilize the cmake config file to write a pkg-config file?<br>
Any comments or suggestions would be appreciated.</p>
<p>P.S.<br>
This is my CMakeLists.txt file for testing GPU version of my project.</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(custom_ops)

set(LIBTORCH "/usr/local/lib/libtorch")
set(CMAKE_PREFIX_PATH "${CMAKE_PREFIX_PATH};${LIBTORCH}")

find_package(Torch REQUIRED)
find_package(OpenCV REQUIRED)

add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
target_link_libraries(example-app "${OpenCV_LIBS}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)
</code></pre>
<p>This is pytorch.pc file for CPU version of the project using meson build.</p>
<pre><code class="lang-auto">prefix=/usr/local/lib/libtorch
exec_prefix=${prefix}
includedir=${prefix}/include
libdir=${exec_prefix}/lib
sharedlibdir=${prefix}/share

Name: libtorch
Description: libtorch library
Version: 1.8.0
Libs: -L${libdir} -L${sharedlibdir} -lc10 -ltorch -ltorch_cpu
Cflags: -I${includedir}
</code></pre>
<p>If these information alone are not enough, I can provide the cpp code for reproduction.</p>
<p>operating system: Ubuntu 18.04<br>
meson version: 0.54.0<br>
cmake version: 3.22.1<br>
libtorch version: 1.8.0</p>

Daeyang_Cho: <p>I tried to leverage “method: cmake” in meson, but it fails.<br>
So, I wrote pkg-config file for libtorch manually and it works now.</p>

JWei: <p>Could you share your pkg-config file please</p>
How to get FP16 working with torch_tensorrt?

Vedant_Roy: <p>When trying to use <code>torch.compile</code> with the <code>tensorrt</code> backend, I get the following error:</p>
<pre><code class="lang-auto">[2024-06-17 17:25:08,351][torch_tensorrt [TensorRT Conversion Context]][ERROR] - 4: [network.cpp::validate::3399] Error Code 4: Internal Error (fp16 precision has been set
 for a layer or layer output, but fp16 is not configured in the builder)      
</code></pre>
<p>for this code:</p>
<pre><code class="lang-auto">    with torch.no_grad(), torch.cuda.amp.autocast(dtype=torch.float16, enabled=True):
        model = torch.compile(model, fullgraph=True, backend="tensorrt")
        outputs = model(inputs)
</code></pre>
<p>How can I configure the backend to allow FP16?</p>

choosehappy: <p>i came up with this:</p>
<pre><code class="lang-auto">modelh=model.half()
modelhctrt=torch.compile(modelh, backend="torch_tensorrt", dynamic=False,
                                 options={"truncate_long_and_double": True,
                                         "enabled_precisions": {torch.float,torch.half},
                                         "debug": False,
                                         "optimization_level": 5})
</code></pre>
How to implement multiprocessing with several GPUs on only one layer of neural network within the forward function

justinS: <p>Hi All, I am running a machine learning experiment in which I need to implement parallel processing across multiple GPUs in order to speed up the forward function of the neural network. I understand the the <code>nn.DistributedDataParallel</code> class can only be used as a wrapper outside of a neural network module. I am wondering if there is an approach that provides the same functionality on the inside of a forward function, as what I am trying to do is to get duplicates of only one of the neural network layers to run simultaneously on multiple GPUs with simultaneous output.</p>
Parameters out of sync over different ranks due to unused parameters

yyk-wew: <p>Hi. When training the model using DDP, I found that the <code>RuntimeError</code> about unused parameters was not thrown as expected even <code>find_unused_parameters</code> was set to <code>False</code>.</p>
<p>Here’s a toy code snippet for reproduction.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.nn as nn
import torch
import torch.distributed as dist
import torch.optim as optim

class Toy(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.a = nn.Linear(1, 1)
        self.b = nn.Linear(1, 1)

    def forward(self, x):
        return self.a(x)

dist.init_process_group(backend='nccl')
model = Toy().to(device=dist.get_rank())
ddp_model = torch.nn.parallel.DistributedDataParallel(model, find_unused_parameters=False)
optimizer = optim.SGD(model.parameters(), lr=1)
optimizer.zero_grad()

x = torch.randn(2, 1).to(device=dist.get_rank())
out = ddp_model(x)
out.mean().backward()

print(dist.get_rank(), ddp_model.module.a.weight)
optimizer.step()
print(dist.get_rank(), ddp_model.module.a.weight)
dist.destroy_process_group()
</code></pre>
<p>Run the code above, I got</p>
<pre data-code-wrap="python"><code class="lang-python">0 Parameter containing:
tensor([[0.6218]], device='cuda:0', requires_grad=True)
0 Parameter containing:
tensor([[0.7547]], device='cuda:0', requires_grad=True)
1 Parameter containing:
tensor([[0.6218]], device='cuda:1', requires_grad=True)
1 Parameter containing:
tensor([[0.9173]], device='cuda:1', requires_grad=True)
</code></pre>
<p>After one step optimization, the weight of <code>model.a</code> is out of sync. It is expected that a <code>RuntimeError</code> should be thrown here to remind me set <code>find_unused_parameters=True</code> but it didn’t.</p>
<p>As expected, comment out the definition of <code>self.b</code> or set <code>find_unused_parameters=True</code> could get the right result.</p>
<p>So why this happens? Please let me know if i’m wrong somewhere.</p>
<p>Best regards.</p>

yyk-wew: <p>Found the solution. The second forward would throw the error as expected.</p>
DataParallel changes `batch_size`

retsuh: <p>I am using <code>DataParallel</code> to train my model.</p>
<p>For each forward pass, my data consists of 6 images and some other values, so right now I am using the batch dimension as 6 to feed the data into the model.</p>
<p>However, since I use <code>DataParallel</code>, my batch of 6 images is divided into 2 batches of 3 images (my GPU only has 10GB RAM), which is not desired to my model.</p>
<p>Are there any way I can keep 6 as my <code>batch</code> dimension?</p>

ptrblck: <p><code>nn.DataParallel</code> will split the input batch by the number of GPUs. Assuming you are using 2 GPUs, you can increase the batch size to 12 so that each GPU will use 12/2=6 samples.</p>

retsuh: <p>In this way, if I do any calculation on the batch dim, will it be based on 6 samples or 12 samples?</p>

ptrblck: <p>It depends where these calculations are done. If it’s outside the model the “global” batch size of 12 will be used. Inside the model in the <code>forward</code> method the “local” batch size of 6 will be used.<br>
We generally recommend using <code>DistributedDataParallel</code> instead as it won’t show the memory imbalance associated with the default device in <code>DataParallel</code> and is not creating copies of the model in each iteration.</p>
Help Needed with Fine-Tuning BERT for Text Classification Using PyTorch

sally2: <p>Hi everyone, <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_three_hearts.png?v=12" title=":smiling_face_with_three_hearts:" class="emoji" alt=":smiling_face_with_three_hearts:" loading="lazy" width="20" height="20"></p>
<p>I am to PyTorch and transformers, and I’m trying to fine-tune BERT for a text classification project. I’ve loaded BERT and the tokenizer from Hugging Face, prepared my dataset, and written a training loop, but I’m having some issues:</p>
<ul>
<li>
<p>I’m getting CUDA out of memory errors even though my dataset isn’t very large. How can I manage memory better during training?</p>
</li>
<li>
<p>My model’s accuracy isn’t improving much. Could someone check my training loop and data preprocessing to see if I’m doing something wrong?</p>
</li>
<li>
<p>I’m not sure about the hyperparameters I’m using (learning rate, batch size, etc.). What settings are recommended for fine-tuning BERT on text classification?<br>
I also check this : <a href="https://discuss.pytorch.org/t/bert-finetuning-for-binary-classification-with-special-tokens-evaluates-badly/203048">https://discuss.pytorch.org/t/bert-finetuning-for-binary-classification-with-special-tokens-evaluates-bad</a><a href="https://www.igmguru.com/data-science-bi/looker-training/" rel="noopener nofollow ugc">looker</a> But I have not found any solution.</p>
</li>
</ul>
<p>Any advice or tips would be greatly appreciated. Thanks!</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=12" title=":innocent:" class="emoji" alt=":innocent:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>You didn’t post any code so unsure how we could help?</p>
Question about batch not coherent

bengao: <p>Hi!<br>
I’m using a standard autonomous car program.<br>
Something is strange and I can’t figure out why:<br>
I’m creating a memory that consists of this information:</p>
<p>self.memory.push((self.last_state, new_state, torch.LongTensor([(self.last_action)]), torch.Tensor([self.last_reward])))</p>
<p>But when I retrieve my memory with the following instruction <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>batch_state, batch_next_state, batch_action, batch_reward = self.memory.sample(10)</p>
<p>I don’t get the same data for batch_action and  batch_reward</p>
<p>For example : I have in my memory the last reward all equal to “-1” and after in my batch i have for examples [-1,-1,0,-1,0,-1,-1,-1,-1,-1]</p>
<p>I don’t understand, if you have any ideas.</p>
<p>Thanks a lot</p>
<p>Ben</p>

bengao: <p>I found, it the memory replay , of course !!</p>
<p>Thanks</p>
How to get center coordinates CSRNet model?

pteran: <p>I am playing whit this interesting repo from github: <a href="https://github.com/AndranikSargsyan/crowd-counting" class="inline-onebox" rel="noopener nofollow ugc">GitHub - AndranikSargsyan/crowd-counting</a><br>
Basically, it is a crowd counting algorithm build using CNN model called CSRNet. it is built using Pytorch.<br>
I am just starting with pytorch and I cannot managed to get the coordinates or bounding boxes of the detections. I tried getting the coordinates using methods used to get bboxes from pretrained without success.<br>
It gives as output a numpy array which is the heatmap.<br>
How could I get the coordinates of each head detected? I would appreciate any help,<br>
Thanks!</p>

farheen_syed: <p>CsrNet model won’t give the point coordinates once go through the Csrnet research paper you will understand it better.</p>
Request an example to use DistributedDataParallel on multiple CPUs and GPU

Smith_Jack: <p>Hello, I am trying to use <strong>DistributedDataParallel</strong> module to parallel the model on multiple CPUs or a single GPU. I have read some tutorials on <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a> and also some codes written by others (e.g. REANN), but now I am confused on how to use the <strong>DistributedDataParallel</strong> module.</p>
<p>For example, in the tutorial, I see the following code</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.multiprocessing as mp
def run_demo(demo_fn, world_size):
    mp.spawn(demo_fn,
             args=(world_size,),
             nprocs=world_size,
             join=True)
</code></pre>
<p>But I didn’t find this part in the REANN code (it also uses the DistributedDataParallel module, but it doesn’t use the  multiprocessing module).<br>
Another confusing example is</p>
<pre data-code-wrap="python"><code class="lang-python">os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12355'
dist.init_process_group("gloo", rank=rank, world_size=world_size)
</code></pre>
<p>In which case do the parameters <strong>rank</strong> and <strong>word_size</strong> should be used ? In the REANN code, I only see this</p>
<pre data-code-wrap="python"><code class="lang-python">dist.init_process_group(backend=DDP_backend)
</code></pre>
<p>That is, I only need to set the backend whether CPU or GPU is used.<br>
There are also other aspects which make me confused, for example, 1) whether I have to divide up the dataset manually and send them to every process, 2) When shoud I use the to() method on a tensor or model to assign a device…</p>
<p>So, are there some standard example about using the DistributedDataParallel  module on CPUs and GPU ? Thanks.</p>
Define layers as dict

waliston: <p>Hi,</p>
<p>I have the following code:</p>
<pre><code class="lang-auto">class LeNet5(nn.Module):

	def __init__(self, num_classes, grayscale=False):
		super(LeNet5, self).__init__()
		
		self.grayscale = grayscale
		self.num_classes = num_classes

		if self.grayscale:
			in_channels = 1
		else:
			in_channels = 3

		# features layers
		self.layer = {}
		self.layer[1] = nn.Sequential(
			nn.Conv2d(in_channels, 6, kernel_size=5),
			nn.Tanh()
		)
		self.layer[2] = nn.Sequential(
			nn.MaxPool2d(kernel_size=2)
		)
		self.layer[3] = nn.Sequential(
			nn.Conv2d(6, 16, kernel_size=5),
			nn.Tanh()
		)
		self.layer[4] = nn.Sequential(
			nn.MaxPool2d(kernel_size=2)
		)

		# classifier, linears layers
		self.layer[5] = nn.Sequential(
			nn.Linear(16*5*5, 120),
			nn.Tanh()
		)
		self.layer[6] = nn.Sequential(
			nn.Linear(120, 84),
			nn.Tanh()
		)
		self.layer[7] = nn.Sequential(
			nn.Linear(84, num_classes)
		)

	def forward(self, x):
		forward_layer = {}

		forward_layer[0] = x
		forward_layer[1] = self.layer[1](forward_layer[0])
		forward_layer[2] = self.layer[2](forward_layer[1])
		forward_layer[3] = self.layer[3](forward_layer[2])
		forward_layer[4] = self.layer[4](forward_layer[3])

		forward_layer[4] = torch.flatten(forward_layer[4], 1)
		forward_layer[5] = self.layer[5](forward_layer[4])
		forward_layer[6] = self.layer[6](forward_layer[5])
		forward_layer[7] = self.layer[7](forward_layer[6])

		logits = forward_layer[7]
		probas = F.softmax(logits, dim=1)
		return logits, probas, x, forward_layer
</code></pre>
<p>It returns the error:</p>
<p>ValueError: optimizer got an empty parameter list</p>
<p>And I verified that the model is empty.</p>
<p>If I change the nams of the layers inside the class it works.</p>
<p>Can I define, inside the class, the layers as dict?</p>

ptrblck: <p>Use <code>nn.ModuleDict</code> instead of a plain Python <code>dict</code> to properly register the modules.</p>

waliston: <p>It doesn’t accept integer number as keys, right?</p>

waliston: <p>I wanted to access the wieghts of the layer dynamically. After using nn.ModuleDict I will share the solution. If someone have some better solution I ask to share. Thanks</p>
<pre><code class="lang-auto">for s in model.layers.items():
    for l in s[1].parameters():
        print(l)
</code></pre>
CUDA Error While Porting TensorFlow Model to PyTorch: "iter.device(arg).is_cuda() INTERNAL ASSERT FAILED"

snehil.chatterjee: <p>Hi everyone,</p>
<p>I’ve been working on porting a TensorFlow model for Real-Time Super-Resolution (<a href="https://github.com/braindotai/Real-Time-Super-Resolution/tree/master" class="inline-onebox" rel="noopener nofollow ugc">GitHub - braindotai/Real-Time-Super-Resolution: 🔥 Real-time Super Resolution enhancement (4x) with content loss and relativistic adversarial optimization 🔥</a>) to PyTorch over the past week. However, I’ve hit a roadblock and could use some help since there aren’t many resources available online that address my specific issue.</p>
<p>The model trains fine on the CPU, but when I try to run <code>train.py</code> on the GPU (‘cuda’), I encounter the following error:</p>
<p><code>RuntimeError: iter.device(arg).is_cuda() INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1695392035891/work/aten/src/ATen/native/cuda/Loops.cuh":89, please report a bug to PyTorch. argument 2: expected a CUDA device but found cpu</code></p>
<p>You can check out my repository here: <a href="https://github.com/snehilchatterjee/Real-Time-Super-Resolution-PyTorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - snehilchatterjee/Real-Time-Super-Resolution-PyTorch</a></p>
<p>To reproduce the error, you just need to run <code>train.py</code> (no additional arguments required).</p>
<p>I’ve been stuck on this for a while and would appreciate any insights or suggestions on how to resolve this issue. Thank you in advance for your help!</p>

ptrblck: <p>Your code fails with a data dependency:</p>
<pre data-code-wrap="python"><code class="lang-python">FileNotFoundError: [Errno 2] No such file or directory: '../../DIV2K_Complete/DIV2K_train'
</code></pre>
<p>so try to narrow down the failing code section and post a minimal and executable code snippet reproducing the issue.</p>

snehil.chatterjee: <p>Hello,</p>
<p>I apologize for the inconvenience caused by the data dependency issue. To help reproduce the error easily, I have uploaded a notebook on Colab which you can run without any data dependency. I have included the necessary code to download a subset of the data required to run the model once.</p>
<p>You can access the Colab notebook here: <a href="https://colab.research.google.com/drive/1zA_wR6C8isdUICCHXcmg9biSh40HMb4O?usp=sharing" rel="noopener nofollow ugc">Colab Notebook</a></p>
<p>If you prefer to run the notebook on a CPU, simply change <code>force_cpu=True</code> when creating the <code>SRGAN</code> class object.</p>
<p>I apologize for not being able to isolate the failing code section. The error message does not provide specific information, and running the code line by line using the debugger in VS Code did not help in identifying the problem.</p>
<p>Thank you for your understanding.</p>
Computational grad without detach.numpy in a function that only accept np.ndarray

eacuellarq: <p>I am attempting to input a tensor created with PyTorch into a simulation library called SimPEG. This tensor has the same matrix format required by SimPEG but is of type (torch.Tensor, requiere_grad=True) and also possesses gradient information. The integration process is proving to be very complex. Therefore, I am seeking a way to input the tensor or a np.ndarray  into SimPEG using a proxy that can record all operations performed on it in memory, just as gradients do in PyTorch.</p>
<p>Thanks for your time!</p>

soulitzer: <p>It sounds like you are looking for <a href="https://pytorch.org/docs/stable/notes/extending.html#when-to-use" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 2.3 documentation</a>?</p>
Static quantizing open clip failed with function' object has no attribute 'dtype'

kechan: <p>I tracked it down to:</p>
<p>torch.ao.nn.quantized.modules.conv.Conv2d has an attribute weight thats a method rather than the actual weight tensor, and this is causing problem since the forward of CLIPVisionEmbeddings is calling</p>
<p>target_dtype = self.patch_embedding.weight.dtype</p>
<p>and this is giving the error: function object has no attribute ‘dtype’</p>
<p>is this a bug on Conv2d impl.? it is differently different from the non quantized counterpart, where weight is not a function, but the actual tensor.</p>
Data Loader NoneType has no len()

Lucky_Magna: <p>DataLoader is throwing, NoneType object has no len() error. Can anyone help</p>
<pre><code class="lang-auto">TypeError: Caught TypeError in DataLoader worker process 1.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 80, in default_collate
    if not all(len(elem) == elem_size for elem in it):
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 80, in &lt;genexpr&gt;
    if not all(len(elem) == elem_size for elem in it):
TypeError: object of type 'NoneType' has no len()
</code></pre>

ptrblck: <p>Based on the error message it seems as if the <code>Dataset.__getitem__</code> method is returning the <code>None</code> object (or no <code>return</code> statement is used which would then return <code>None</code> by default).<br>
Could you check that the <code>Dataset.__getitem__</code> is indeed returning valid data?</p>

Lucky_Magna: <p>Yeah you are right, thank you for the help <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Benison_Sam: <p>I have the exact same error but in my case I checked <code>Dataset.__getitem__</code> as well as <code>Dataloader.dataset</code> if there were any NoneTypes in them and there weren’t any. I am not sure what’s really going on. I am getting this error when iterating through the batches. But if I take the length of the dataloader before iteration, I am getting the following error:<br>
<code>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class 'NoneType'&gt;</code></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cdd72936082488f3140088d99f110077ba15317.png" alt="image" data-base62-sha1="fx40ZA7k8t7FXdAVpvA7dmTyodN" width="277" height="131"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/4/84ebb8ba770882ad9a445dba06e30333a0c74d46.png" data-download-href="https://discuss.pytorch.org/uploads/default/84ebb8ba770882ad9a445dba06e30333a0c74d46" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/4/84ebb8ba770882ad9a445dba06e30333a0c74d46.png" alt="image" data-base62-sha1="iXS3ydiTRuC3lc7cAXhP5h1W1me" width="690" height="117" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">920×157 4.82 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>