akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to find the index of a list of tensors?

sodlqnf123: <pre><code class="lang-auto">A = torch.tensor([[1,2,3],[4,5,6],[7,8,9],[10,11,12]])
B = torch.tensor([[7,8,9],[4,5,6]])
some_function(A, B) -&gt; torch.tensor([2, 1])
</code></pre>
<p>What I want to do is that get indices of A based on each tensor of B.<br>
The reason why output is [2,1] is that each index of [7,8,9] is 2, and of [4,5,6] is 1.<br>
It looks quite simple, but is very hard to find a way. Can anyone help me please?</p>

KFrank: <p>Hi Minhyuk!</p>
<aside class="quote no-group" data-username="sodlqnf123" data-post="1" data-topic="171737" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sodlqnf123/48/56991_2.png" class="avatar"> sodlqnf123:</div>
<blockquote>
<pre><code class="lang-auto">A = torch.tensor([[1,2,3],[4,5,6],[7,8,9],[10,11,12]])
B = torch.tensor([[7,8,9],[4,5,6]])
some_function(A, B) -&gt; torch.tensor([2, 1])
</code></pre>
</blockquote>
</aside>
<p>Try <code>torch.where ((A == B.unsqueeze (1)).prod (dim = 2))[1]</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

sodlqnf123: <p>A == B.unsqueeze(1) looks magical. thanks</p>

sodlqnf123: <p>Could you give me some insight how two tensors of shape (4,3) and (2,1,3) can be compared by ==?</p>

KFrank: <p>Hi Minhyuk!</p>
<aside class="quote no-group" data-username="sodlqnf123" data-post="4" data-topic="171737" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sodlqnf123/48/56991_2.png" class="avatar"> sodlqnf123:</div>
<blockquote>
<p>Could you give me some insight how two tensors of shape (4,3) and (2,1,3) can be compared by ==?</p>
</blockquote>
</aside>
<p>The shapes of the tensors in your original post were <code>[4, 3]</code> and <code>[2, 3]</code>.<br>
So you can just <code>squeeze()</code> away the singleton dimension in your second<br>
tensor and do the same thing that you did before.</p>
<p>Best.</p>
<p>K. Frank</p>

sodlqnf123: <p>It gives error when I compare two tensors of shape (4,3) and (2,3)</p>
<blockquote>
<p>RuntimeError: The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 0</p>
</blockquote>
<p>It definitely seems like unsqueezing the first dimension does something.</p>
Batch-wise processing is quite slow

Nafees: <p>Suppose I have three vectors A, B, C</p>
<blockquote>
<pre><code>A vector size of 256
B vector size of 256
C vector size of 256
</code></pre>
</blockquote>
<p>Now I want to do concatenation in the following way:</p>
<blockquote>
<pre><code>AB= vector size will be 512
AC = vector size will be 512
BC = vector size will be 512
</code></pre>
</blockquote>
<p><strong>However,</strong> I need to restrict all the concatenated vectors to <em>256</em>, like:</p>
<blockquote>
<pre><code>AB= vector size will be 256
AC = vector size will be 256
BC = vector size will be 256
</code></pre>
</blockquote>
<p>One way is to take the mean of each two values of the two vectors like <code>A first index value</code> and <code>B first index value</code>, <code>A second index value</code> and <code>B second index value</code> … etc. Similarly, in the concatenation of other vectors.</p>
<p><strong>How I implement this:</strong></p>
<pre><code>x # torch.Size([32, 3, 256]) # 32 is Batch size, 3 is vector A, vector B, vector C and 256 is each vector dimension

def my_fun(self, x):
        iter = x.shape[0]
        counter = 0
        new_x = torch.zeros((10, x.shape[1]), dtype=torch.float32, device=torch.device('cuda'))
        for i in range(0, x.shape[0] - 1):
            iter -= 1
            for j in range(0, iter):
                mean = (x[i, :] + x[i+j, :])/2
                new_x[counter, :] = torch.unsqueeze(mean, 0)
                counter += 1
        final_T = torch.cat((x, new_x), dim=0)
        return final_T

ref = torch.zeros((x.shape[0], 15, x.shape[2]), dtype=torch.float32, device=torch.device('cuda'))
for i in range (x.shape[0]):
    ref[i, :, :] = self.my_fun(x[i, :, :])
</code></pre>
<p>But this implementation is computationally expensive. One reason is I am iterating <strong>batch-wise</strong> which makes it computationally expensive. Is there any efficient way to implement this task?</p>
Train a Machine Learning or Deep Learning Model

Noorulain_Islam: <p>As in this link one deep learning model using CNN has trained. <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Training a Classifier — PyTorch Tutorials 1.13.1+cu117 documentation</a></p>
<p>My question is how can we explain this thing in very brief in words. How do you train a Machine Learning or Deep Learning (i.e CNN) model?</p>

vdw: <p>This is a very vague question :). Without knowing what you’re really after, this would be my best shot:</p>
<p>The neural network classifier is a complicated function <code>y = f(x, p)</code> that maps an input <code>x</code> (here: an image) to an output <code>y</code> (here: a class label). The function is parameterized by a set of parameters <code>p</code>. For different values of <code>p</code>, function <code>f</code> will do a better or worse job of mapping an image to its correct label. Training classifier now means to systematically trying to find the best values for <code>p</code>, where the “best” means that these values for <code>p</code> map most images in your training data to their correct class labels.</p>
<p>But I assume that’s not what your asking; so you might need to clarify your question.</p>

Noorulain_Islam: <p>I was asking this question in general context. Like, if we train any deep learning model then  in order to explain the training steps in brief, should we need to tell about our features, labels, number of epochs and the loss function? The way of training the model?</p>
Official Object Detection Finetuning tutorial code returns NaN loss after a few batches

kareemamr: <p>I’m following this <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">official tutorial</a> on how to fine tune Fast R-CNN object detection, following the code as it is with the same hyperparameter choices, after training for only a few batches the returned loss is NaN. The only difference between my code and the tutorial’s code is that I’m using a subset of the COCO dataset objects to fine tune the model but I wouldn’t imagine that to be the problem since the model is already pre-trained on COCO. Has anyone run into this?</p>

MD_DIDARUL_ISLAM: <p>Hi,<br>
Were you able to solve the issue ?</p>
<p>Thanks</p>
How does prefetch factor really work?

Divyansh_Rastogi: <p>Hi!<br>
The docs (<a href="https://pytorch.org/docs/stable/data.html" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 1.13 documentation</a>) define prefetch_factor as the “Number of batches loaded in advance by each worker,” implying that if I have <code>W</code> workers and <code>P</code> prefetch factor, the total batches loaded would be <code>WxP</code>, although are all of these batches that have to be loaded in advance? Or are these the max batches that could be loaded in advance?</p>
<p>For example, consider the following code snippet:</p>
<pre><code class="lang-python">for batch in range(data_loader):
  # forward &amp; backward passes on the model
</code></pre>
<p>For the above code snippet, does the <code>data_loader</code> function in the following way:</p>
<ol>
<li>Before the first iteration, <code>WxP</code> batches are loaded</li>
<li>After <code>WxP</code> iterations, another set of <code>WxP</code> batches is loaded</li>
</ol>
<p>Or does the <code>data_loader</code> function in this way:</p>
<ol>
<li>Before the first iteration, <code>WxP</code> batches are loaded</li>
<li>After the first iteration, now in the host memory <code>WxP - 1</code> batches reside, so any worker from the worker pool fetches a new batch to make it to <code>WxP</code>.</li>
</ol>
<p>If in case the latter is correct, does the 2nd step happen synchronously or asynchronously? i.e. do I need to wait to fetch a new batch to make total batches to <code>WxP</code> and then proceed to the iteration, or given I already have <code>WxP - 1</code> batches to feed, I can process the remaining batch asynchronously?</p>
<p>Thanks in advance for answering!</p>

nivek: <p>I don’t think either description is exactly correct.</p>
<p>In general, <code>DataLoader</code> tries to return the first batch as soon as possible upon request (rather than load <code>WxP</code> batches in advance before returning). With <code>prefetch_factor &gt; 0</code>, while the forward and backward passes happen, <code>DataLoader</code> tries to prepare as many subsequent batches as possible up to the limit set by the <code>prefetch_factor</code> and saves those batches in a buffer. When the model requests the next batch, <code>DataLoader</code> immediately pops off the first batch from the buffer, regardless of whether the buffer is full or not. Then, it goes back and tries to fill up the buffer while the model executes.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/when-does-a-pytorch-dataset-worker-pull-new-data/153286/2">This code snippet</a> also illustrates your description in case it’s helpful.</p>
The cpu() call is slow for a model's output tensor only

torchai: <p>I have a model where the model’s output tensor has a shape of [32, 2, 128, 128]. When I copy this tensor object to CPU memory it takes about 63 milliseconds.</p>
<p>When I create a tensor of the same shape [32, 2, 128, 128] using torch.randn() on the same GPU, then the copy of this tensor object to CPU memory takes about 1 milliseconds.</p>
<p>What options are there to increase the speed of cpu() for a model’s output tensor? I already tried various things like nograd. I also tried to copy the model’s output tensor to a different unused GPU. I get the same slow speed when I call cpu() from this GPU. It looks like there is some expensive parsing going on for a model’s output tensor.</p>
<p>Does anyone have more insights, and how I could possible improve it?</p>
<p>Thanks.</p>

eqy: <p>How are you measuring the time here? If you are not adding <code>torch.cuda.synchronize()</code> calls before the output tensor copy, you could be unwittingly including the cost of previous CUDA operations (e.g., that of running of a model if that is what occurs just before the copy) that have been dispatched but not completed yet in your measurements.</p>

torchai: <p>Yes, I’m also using torch.cuda.synchronize() before I measure the time. And there are no CUDA operations or models on the other GPU where I copied the model’s output tenser to. This test showed that the cpu() call is also slow from this GPU.</p>

eqy: <p>That’s interesting, could you post a minimal code snippet that reproduces this issue?</p>

torchai: <p>It turned out that the first cpu() call after each batch inference is always slow, in my case often 150 ms. This is even the case if I move the model’s output tensor to another unused GPU and then to CPU memory.</p>
<p>My current solution is to save the model’s output tensor for each batch in a list. Once I’m done with the inferences I loop through the list and call cpu() for each batch. The first cpu() call after the last inference is still slow but the remaining cpu() calls take now between 1 to 5 ms. This improved the total processing time of my images significantly (total inference time did not change).</p>
DistributedDataParallel with different 2 GPUs

Logo: <p>Hi,</p>
<p>I have a huge image dataset and would like to train on my rtx 2060 and rtx 3060 simultaneously.<br>
Since they do have quite a different memory an compute capability i wonder if this is viable at all or just train on the 3060.</p>
<p>Since they are so different would it be better to run each process with another batch size fitted to their vram but would this make it necessary to customize the DistributedSampler so that they receive an uneven split of the dataset?</p>
<p>Thanks for your help.</p>

ptrblck: <p>Yes, your overall training might run into a bottleneck caused by the slower GPU. Also, you are right that a custom sampler might be needed to create the imbalanced split for both devices.<br>
<a href="https://discuss.pytorch.org/t/multiple-but-different-gpus/63006/10">This post</a> describes potential issues a bit more and you could also profile the actual workload to see how severe the slowdown is.</p>
Instance segmentation on big images

Sylvain_Ard: <p>Hello,<br>
I would an instance segmentation program which take 3000x2500 images at the input of the neural network. I tested with detectron2 but the GPU memory exploded and I can’t do it. I have a 24GB RAM GPU.<br>
Thank you for your help.<br>
Best regards</p>

ptrblck: <p>Somewhat related to <a href="https://discuss.pytorch.org/t/semantic-segmentation-on-big-images/167504/3">this answer</a>.<br>
I don’t know if you are planning to train the model or just to use it for inference.<br>
For training you could try to trade compute for memory via <code>torch.utils.checkpoint</code> or CPU-offloading as described in <a href="https://discuss.pytorch.org/t/modifying-forward-backward-pass/169687/2">this post</a>.<br>
For inference you should wrap the model in a <code>no_grad()</code> context to save memory. If this is still running out of memory, you might need to use multiple GPUs and apply model sharding.</p>
Torch::save error in torch::serialize::OutputArchive. Can't save the model

mas: <p>Hello fellow developers,</p>
<p>I am hoping to achieve the following function:</p>
<blockquote>
<p>target_nn.load_state_dict(nn.state_dict())</p>
</blockquote>
<p>in <code>C++</code> .  Apparently this is not as easily done in <code>C++</code> as it is in <code>Python</code>.</p>
<p>I found this post in the <code>Libtorch </code> Github <a href="https://github.com/pytorch/pytorch/issues/36577#issuecomment-614813294" rel="noopener nofollow ugc">load_state_dict in C++</a>. Basically, suggests to save and load the model like the following:</p>
<blockquote>
<p>std::stringstream stream;<br>
torch::save(policy_net, stream);<br>
torch::load(target_net, stream);</p>
</blockquote>
<p>I have also attempted this with the same error:</p>
<blockquote>
<p>torch::save(model, “stream.pt”);<br>
torch::load(target_model, “stream.pt”);</p>
</blockquote>
<p>The Error on <code>GCC</code>:</p>
<blockquote>
<p>error: no match for ‘operator&lt;&lt;’ (operand types are ‘torch::serialize::OutputArchive’ and ‘const Net’)<br>
44 |   archive &lt;&lt; value;</p>
</blockquote>
<p>and</p>
<blockquote>
<p>error: no matching function for call to ‘torch::serialize::InputArchive::load_from(std::__cxx11::basic_ostringstream&amp;)’</p>
</blockquote>
<p>The Error on <code>CLang</code>:</p>
<blockquote>
<p>error: invalid operands to binary expression (‘serialize::OutputArchive’ and ‘const RLNet’)<br>
archive &lt;&lt; value;</p>
</blockquote>
<p>I am not sure where to start to fix this error.</p>
<p>If you have a better way to implement <code>target_nn.load_state_dict(nn.state_dict())</code> in <code>C++</code> please let me know. Also, a solution to this error would really help.</p>
<p>Thank you.</p>
License Requirements for Merging a PR

gauenk: <p>I am planning to submit a PR to Pytorch soon for a new function. I realize the entire process may take some time, I will need to learn as I go, and it will require extensive preparation.</p>
<p>My question is only about how Open Source Licensing works for Pytorch.</p>
<p>If I plan to integrate my existing code into Pytorch, what does Pytorch expect from my software’s licensing? My university allows me to choose the Open Source License, but they must approve it before I release my code publically. My project has no restrictions to being released as publically as possible, once the license is appoved.</p>
<p>I believe I only need to choose a generic BSD License, but I wanted to gently/unofficially/roughly check with this forum to seek informal/non-official/non-binding advice.</p>
<p>Thank you for the guidance.</p>

ptrblck: <p>You would need to sign the CLA as described <a href="https://pytorch.org/blog/a-contributor-license-agreement-for-pytorch/">here</a> before your PR can be merged.</p>

gauenk: <p>Thank you. To clarify, do I need to use a specific type of Open Source license?</p>

ptrblck: <p>I’m not deeply familiar with licensed, but would assume you are agreeing to the <a href="https://github.com/pytorch/pytorch#license">BSD licence</a> when submitting code.</p>

gauenk: <p>So do you suspect will Pytorch be bothered if I continue to maintain my own version of my code once I submit a PR? Basically, I just want to make sure I am not doing anything obviously wrong. Are there any examples you can think if where the PR is integrated and the repo is still open? Many of the PRs I see are usually forks that get removed right away.</p>
<p>Thank you for the help so far by the way.</p>

ptrblck: <p>Maintaining your branch with e.g. new (experimental) features should not be a problem.<br>
An example could be <a href="https://github.com/pyg-team/pytorch_geometric">PyTorch-Geometric</a>, which is of course not fully integrated into PyTorch Core, but a few utilities were merged such as <code>scatter</code> operations with reductions. This might not be the best example, but I wouldn’t be concerned about keeping your repository alive and trying to upstream new features in case you think it would be beneficial to have them in PyTorch directly.</p>

gauenk: <p>Thank you. I appreciate all this help.</p>
Hidden layers in MLP - Question

Wael_AbuRezeq: <p>Hi all,</p>
<p>I am trying to validate my understanding of hidden layers :). A NN consists of (3) types of layers as of input, hidden and output. Input and output layers shouldn’t be considered as hidden layers. Is that true?</p>
<p>Secondly, if we have the following network strucutre (coming from <a href="https://discuss.pytorch.org/t/hidden-layers-in-mlp/115518">this</a> post)</p>
<p>"</p>
<blockquote>
<p>class Net(torch.nn.Module):<br>
def <strong>init</strong>(self, num_inputs, num_hidden, num_outputs):<br>
super(Net, self).<strong>init</strong>()<br>
self.num_inputs = num_inputs<br>
self.num_hidden = num_hidden<br>
self.num_outputs = num_outputs</p>
<pre><code>    self.lin1 = nn.Linear(num_inputs, num_hidden)
    self.lin2 = nn.Linear(num_hidden, num_hidden)
    self.lin3 = nn.Linear(num_hidden, num_hidden)
    self.lin4 = nn.Linear(num_hidden, num_outputs)
    self.ReLU = nn.ReLU()

def forward(self, x):
    x = x.view(x.size(0),-1)
    out = self.lin1(x)
    out = self.ReLU(out)
    out = self.lin2(out)
    out = self.ReLU(out)
    out = self.lin3(out)
    out = self.ReLU(out)

    return self.lin4(out)
</code></pre>
</blockquote>
<p>"</p>
<p>The self.lin1 is an explicit input, but also a hidden layer as it takes hidden nodes<br>
The self.lin2 and self.lin3 are hidden layers<br>
The self.lin4 is output layer → it isn’t a hidden layer</p>
<p>in total, we have three hidden layers. Is that true?</p>
<p>Lastly, the x.view is to flatten the input; should I use it this way or swap its input?</p>
<p>Thank you,</p>

vdw: <p><code>self.lin1</code> is also a hidden layer. The input layer is represented by your input vector <code>x</code>.</p>
<p>For example, if you represent basic Logistic Regression with <code>N</code> features as a Neural Network (well, having 1 neuron), then your network has an input layer of size <code>N</code> and an output layer of size <code>1</code>; there are no hidden layers.</p>

Wael_AbuRezeq: <p>Thanks Chris,</p>
<p>so in total my network has 3 layers. right?</p>
<p>Second, commenting on your logstic regresion example, you said that the input layer will hae Size N and output will have size 1; so far there are no hidden layers in here. But, you mentioned that we have 1 nueron - I assume you are referring to the output layer nueron. right?</p>
How to resolve "C++ standard version definition detected in environment variable."?

akshaypaygude: <p>Since the GPU on my target machine is of compute capability 3.5, I am trying to compile pytorch from source. I’m following the standard installation steps given on the Github.</p>
<p>I have done all the steps right but I keep getting this warning.</p>
<p>(old) [akshaypaygude@node07 pytorch]$ <em>python setup.py install</em><br>
<em>Building wheel torch-2.0.0a0+gitUnknown</em><br>
<em>– Building version 2.0.0a0+gitUnknown</em><br>
<em>cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_ARGS=-DCMAKE_LINKER=/home/akshaypaygude/anaconda3/envs/old/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/home/akshaypaygude/anaconda3/envs/old/bin/x86_64-conda-linux-gnu-strip -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/akshaypaygude/pytorch/torch -DCMAKE_PREFIX_PATH=/home/akshaypaygude/anaconda3/envs/old/lib/python3.10/site-packages;/home/akshaypaygude/anaconda3/envs/old:/home/akshaypaygude/anaconda3/envs/old/x86_64-conda-linux-gnu/sysroot/usr -DPYTHON_EXECUTABLE=/home/akshaypaygude/anaconda3/envs/old/bin/python -DPYTHON_INCLUDE_DIR=/home/akshaypaygude/anaconda3/envs/old/include/python3.10 -DPYTHON_LIBRARY=/home/akshaypaygude/anaconda3/envs/old/lib/libpython3.10.a -DTORCH_BUILD_VERSION=2.0.0a0+gitUnknown -DUSE_NUMPY=True /home/akshaypaygude/pytorch</em><br>
<em>– The CXX compiler identification is GNU 11.2.0</em><br>
<em>– The C compiler identification is GNU 11.2.0</em><br>
<em>– Detecting CXX compiler ABI info</em><br>
<em>– Detecting CXX compiler ABI info - done</em><br>
<em>– Check for working CXX compiler: /home/akshaypaygude/anaconda3/envs/old/bin/x86_64-conda-linux-gnu-c++ - skipped</em><br>
<em>– Detecting CXX compile features</em><br>
<em>– Detecting CXX compile features - done</em><br>
<em>– Detecting C compiler ABI info</em><br>
<em>– Detecting C compiler ABI info - done</em><br>
<em>– Check for working C compiler: /home/akshaypaygude/anaconda3/envs/old/bin/x86_64-conda-linux-gnu-cc - skipped</em><br>
<em>– Detecting C compile features</em><br>
<em>– Detecting C compile features - done</em><br>
<em><strong>CMake Warning at CMakeLists.txt:36 (message):</strong></em><br>
***  C++ standard version definition detected in environment variable.PyTorch***<br>
***  requires -std=c++17.  Please remove -std=c++ settings in your environment.***</p>
<p>Although, the installation will continue with the warning but I suspect later it will throw errors.<br>
Please help.</p>
How to load data from .dat files classified by folders

klark: <p>Hello,</p>
<p>I stored my data in .dat files as matrices and I classified them in folders (i.e. ‘A’, ‘B’, etc.).<br>
How can I load them in order to use them in dataloader. Is there any function like torchvision.datasets.ImageFolder to load this data and automatically assign a target class ?</p>

ptrblck: <p>You could use <code>DatasetFolder</code> with a custom <code>loader</code> class which should be able to load the <code>.dat</code> format. <a href="https://discuss.pytorch.org/t/loading-npy-files-using-torchvision/28481/3">This post</a> shows how it’s done for numpy arrays.</p>

klark: <p>Thank you so much. It worked wonderfully.</p>
How to get validation in a pretrainied model?

8dzed2: <p>Hi, I’m quite new to pytorch and am currently trying to retrain a pretrained object detection model <code>fasterrcnn_resnet50_fpn_v2(weights=FasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1)</code><br>
with a custom dataset and different classes (I have changed out the last layer to only have 3 classes).</p>
<p>The model seems to work great after some training time and I can track the training loss from the output dict of<br>
<code>loss_dict = model(images, targets)</code>.<br>
However I am now wondering how I can track the validation loss with the same loss functions used in the training process (without actually changing the weights and/or when in model.eval()). How could I achieve this?</p>
<p>Thanks for your help =)</p>
Plotting image histogram in pytorch

shrbrh: <p>Is there a way to plot the histograms of a colored image in pytorch?</p>

ptrblck: <p><code>torch.histogram</code> should work as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.from_numpy(np.array(image)).float()

bins = torch.linspace(0, 256, 257)
hist = [torch.histogram(c, bins=bins) for c in x]

plt.plot(hist[0].bin_edges[:-1], hist[0].hist, color="r")
plt.plot(hist[1].bin_edges[:-1], hist[1].hist, color="g")
plt.plot(hist[2].bin_edges[:-1], hist[2].hist, color="b")
</code></pre>
<p>You can change the <code>bins</code> of course if you don’t want to count each pixel value.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/3/63e9fd6b22fd7ae4a1beece850167c80bdc4b11a.png" alt="image" data-base62-sha1="efSACudikxLSTGpr3hXTFxe4zKa" width="375" height="248"></p>

shrbrh: <p>Thank you for your answer.</p>
YOLOv5, YOLOv8 questions

Alexis: <p>I am working with yolov5 and i have some questions:</p>
<ol>
<li>How to process 4000 x 4000 image with yolov5? I guess yolov5 firstly resizes 4000x4000 image to 608x608 and then calculate output. Results with different resolutions are different.</li>
<li>How to train yolov5 on custom dataset? Where to find dataset for yolov5?</li>
<li>What is the output layer format of yolov5? Is it [width, height, class, competition] or like?</li>
</ol>
Torchaudio failed to open input

ammar_siddiqui: <p>Hi, I am a beginner in using audio files for deep learning. I have an annotation file in which i have stored the name of the audio files and i have audio files in another path. But when i try to load those audio files using torchaudio it’s giving me this error. I have checked the all the paths and they are correct.</p>
<pre><code class="lang-auto">
 File "/home/ammar/VectraCom/qra/train.py", line 25, in __getitem__
    signal, sr = torchaudio.load(audio_sample_path)
  File "/home/ammar/anaconda3/envs/qra/lib/python3.9/site-packages/torchaudio/backend/sox_io_backend.py", line 246, in load
    return _fallback_load(filepath, frame_offset, num_frames, normalize, channels_first, format)
  File "/home/ammar/anaconda3/envs/qra/lib/python3.9/site-packages/torchaudio/io/_compat.py", line 103, in load_audio
    s = torch.classes.torchaudio.ffmpeg_StreamReader(src, format, None)
RuntimeError: Failed to open the input "/home/ammar/VectraCom/qra/assets/QDAT/FINAL_SOUND/S0_1" (No such file or directory).
</code></pre>
Prediction GPU parallel issue

80Ultraman: <p>Hey guys,</p>
<p>I have a trained 3D-CNN model. It takes almost all GPU memory to run one example.</p>
<p>So I wanna use multi GPU to run the prediction at the same time.</p>
<p>Here’s my code:</p>
<pre><code class="lang-auto">import torch
import make_my_model

dummy_input_0 = torch.rand(1, c, h, w, l, device=torch.device('cuda:0')
dummy_input_1 = torch.rand(1, c, h,w, l, device=torch.device('cuda:1')

with torch.no_grad():
    model_0 = make_my_model().to(device0)
    model_1 = make_my_model().to(device1)
    
    out_0 = model_0(dummy_input_0)
    out_1 = model_1(dummy_input_1)
</code></pre>
<p>This code gives the right output but it seems won’t work async.</p>
<p>What is the proper way to run the prediction parallelly on two (maybe more) GPUs?</p>
<p>Thanks, guys!</p>

ptrblck: <p>I would guess you might not see a parallel execution since your CPU might not be able to schedule the kernels fast enough and/or you might have (unwanted) synchronizations in your code.<br>
In the default “eager” mode the CPU needs to dispatch to the internal operator implementation and dispatches the CUDA kernel.<br>
If your model has e.g. 100 layers (and for simplicity let’s claim each layer is calling into a single CUDA kernel), the CPU would need to schedule 100 kernels in <code>model_0</code> before it can start scheduling the workload in <code>model_1</code>.<br>
Assuming the CPU is fast enough and you are not blocking it with synchronizations, you should see an overlap and could use a profiler (e.g. the native PyTorch profiler or Nsight Systems) to check it.</p>
<p>However, if your overall workload is already CPU-limited, you would of course not see any overlap.<br>
This should already be visible while running a single model in a profiler and Nsight Systems would show “whitespaces” between the actual kernel execution.</p>
<p>If you are working with static input shapes (and meet other requirements) you could try to use CUDA Graphs as described <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-graphs">here</a> to reduce the kernel launch overheads.</p>
`zero_grad()` release GPU memory

qiminchen: <p>I have two questions about <code>zero_grad() </code> releasing GPU memory</p>
<ol>
<li>
<p>Does <code>net.zero_grad()</code> release the GPU memory occupied by the gradients computed from previoue epoch?</p>
</li>
<li>
<p>Suppose I have two networks <code>netD</code> and <code>netG</code>, in the below code snippet, can I add extra <code>zero_grad()</code> after <code>optimizer.step</code> to release some GPU memory before going to the next epoch? I could zero grad both <code>netD</code> and <code>netG</code> at the very beginning of loop but what if I want to save some GPU memory for <code>netG</code> after training <code>netD</code>.</p>
</li>
</ol>
<pre><code class="lang-auto">for epoch in epochs:

    # train discriminator
    netD.zero_grad()
    ...
    loss_d.backward()
    optimizer_d.step()
    # netD.zero_grad()  &lt;------- Can I add another zero_grad() here
                                 to release some GPU memory?
    ...
    ...

    # train generator
    netG.zero_grad()
    ...
    loss_g0.backward()
    optimizer_g.step()
    # netG.zero_grad()  &lt;------- Can I add another zero_grad() here
                                 to release some GPU memory?
    ...
    ...

    # train generator again using other losses
    netG.zero_grad()
    ...
    loss_g1.backward()
    optimizer_g.step()
    # netG.zero_grad()  &lt;------- Can I add another zero_grad() here
                                 to release some GPU memory?
</code></pre>

ptrblck: <ol>
<li>
<p>Not in the default <code>.zero_grad()</code> call, since the <code>.grad</code> attributes will just be filled with zeros, but not deleted. To delete the <code>.grad</code> attributes and to save memory, you would need to call <code>.zero_grad(set_to_none=True)</code>.</p>
</li>
<li>
<p>Yes, if you don’t need the gradients from a previous model anymore you can directly delete them after they were used in <code>optimizer.step()</code>.</p>
</li>
</ol>

qiminchen: <p>Thank you! Does calling <code>.zero_grad(set_to_none=True)</code> slow down the training?</p>

ptrblck: <p>No, it does not slow down the training, as it’s only freeing the gradients. The next gradient calculation should be able to just reuse the memory via the CUDA caching allocator.</p>
Custom loss, tensor does not have a grad_fn

LJ-31: <p>Hello,</p>
<p>I am trying to run a custom loss. All native losses from pytorch run well, but not this one.</p>
<p>My loss takes all points predicted and then selects just some of these points and does the same with the mask. This is one of the losses I am trying to implement. This is a segmentation mask.</p>
<p>During this process, the tensors shape gets reduced from say 100 points in the prediction to say, 10 points and then uses these 10 points to get the loss.</p>
<p>Is there a way to solve this?</p>
<p>Thank you.</p>
<pre><code class="lang-auto"> Getting Runtime error: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<pre><code class="lang-auto"> File "C:\Users\LJ\anaconda3\envs\toolkit\lib\site-packages\torch\_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "C:\Users\LJ\anaconda3\envs\toolkit\lib\site-packages\torch\autograd\__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

</code></pre>

thecho7: <p>Could you let me see your loss function?</p>
Cannot install torchvision anaconda linux

George_Kasapoglou: <p>Hello,<br>
after fighting to install for GPU pytorch with anaconda on linux ubuntu, I have a new problem how to download torchvision for GPU any recommendations?<br>
I have installed it and run yolo v7 successfully with pip, but I don’t want to work with pip, because in my anaconda path I have the rapids of NVIDIA.<br>
thank you in advance</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b79d6eea5df7e9d7e6b37179ad7457381675252d.png" data-download-href="https://discuss.pytorch.org/uploads/default/b79d6eea5df7e9d7e6b37179ad7457381675252d" title="Screenshot from 2023-02-05 21-13-17"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b79d6eea5df7e9d7e6b37179ad7457381675252d_2_690x479.png" alt="Screenshot from 2023-02-05 21-13-17" data-base62-sha1="qckFVIaujMD4SJo7CgDli7cXMyF" width="690" height="479" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b79d6eea5df7e9d7e6b37179ad7457381675252d_2_690x479.png, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b79d6eea5df7e9d7e6b37179ad7457381675252d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b79d6eea5df7e9d7e6b37179ad7457381675252d.png 2x" data-dominant-color="401935"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-02-05 21-13-17</span><span class="informations">975×678 119 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The install instructions from <a href="https://pytorch.org/get-started/locally/">here</a> work for me:</p>
<pre><code class="lang-python">conda install pytorch torchvision pytorch-cuda=11.7 -c pytorch -c nvidia
...
&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; torch.__version__
'1.13.1'
&gt;&gt;&gt; torchvision.__version__
'0.14.1'
&gt;&gt;&gt; torch.version.cuda
'11.7'
&gt;&gt;&gt; torchvision.version.cuda
11070
</code></pre>
Model Predicts Always the same Class

iVirus1: <p>Hello, I have problem with my Model.<br>
My Model Predicts always the Same class in Validation and Test. The Prediction in Training works fine, or better, than in Validation and test. I tried to use the same data in Training, Validation and Test but every time comes the same result. I tried to optimize the Learn rate without success. My Data is also good, Equalized over all classes. I’m using a MC3 model.</p>
<p>Can someone help me with this problem, or giving me a hint where my problem might be?</p>
İmage Classification with Lstm and cnn

Directorsim: <p>Hello,<br>
I’m really beginner for the neural network. Any help is really appreciated :).<br>
So my case is that;<br>
I have a video dataset. I extract one video image frame and extract on audio spectrum as image of the video. I have two main folders -one includes video image frames and the other contains audio spectrums of each videos-. Each two main folder have 8 subfolders - which are the classes.</p>
<p>My model has two inputs -one image frame and one audio spectrum image-. Each input is transferred a pretrained model vgg16 paralelly for feature extraction. Then, result of these two inputs are concatinated into 8192 linear and then transferred the classification step. My problems begins here. I have to use LSTM for the Classification part. I could not combine Vgg ang Lstm, maybe it is not possible.</p>
<p>Any ideas?<br>
Thank you,<br>
Best regards</p>
<pre><code class="lang-auto">vggmodel = vgg16(weights=torchvision.models.VGG16_Weights.DEFAULT)
for param in vggmodel.features.parameters():
    param.require_grad = False
class MyModel(nn.Module):
    
    def __init__(self):
        super().__init__()
                m = vggmodel
        for param in m.parameters():
          param.requires_grad = False

        m.classifier[6] = nn.Identity() # replaced final FC layer with identity

        self.vgg16_modified = m
                
        self.rnn = nn.LSTM(
            input_size=8192,
            hidden_size=64,
            num_layers=1,
            batch_first=True)
            
        self.linear = nn.Linear(64, 8)
        
    def forward(self, x):        
        y1 = self.vgg16_modified(x["videoFrame"])   #VGG feature extraction for video image
        y2 = self.vgg16_modified(x["audioImage"])    #VGG feature extraction for video's audio spectrum
        #y1 = y1.view(y1.size(0), -1) #not sure, so commented
        #y2 = y2.view(y2.size(0), -1)#not sure, so commented

        y = torch.concat((y1, y2), 1) #Concatinate y1 and y2, each of results with 4096 size of layer and concatinate them

        #r_in = y.view(batch_size, 100, -1) #not sure, so commented

        r_in = y.view(1, batch_size, 8192)
        r_out, (_, _) = self.rnn(r_in)
        r_out2 = self.linear(r_out[:, -1, :])
        return F.log_softmax(r_out2, dim=1)

model = MyModel()
print(model)
</code></pre>
To(device) changes the value of input

wilsonng.colab: <pre><code class="lang-auto">   for images, _in train_dataloader:
     print(images.min(), images.max())
     images= images.to(device)
     print(images.min(), images.max())
     ...
</code></pre>
<p>After to(device), the range of images changed from [-1, 1] to (-inf, int).<br>
To be more specific, the value of images after to(device) could become Nan or any arbitrary values.<br>
What could be the possible reason for it?</p>
<p>Here are the images before to(device) and after:<br>
I don’t understand why there could be some random horizontal lines or vertical lines which are pixels with arbitrary values<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0877842d63528616a154910b178a5377c09fda6.png" data-download-href="https://discuss.pytorch.org/uploads/default/b0877842d63528616a154910b178a5377c09fda6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0877842d63528616a154910b178a5377c09fda6_2_690x368.png" alt="image" data-base62-sha1="pbEhcuhlDs0avBGbV4Ky2gevHLM" width="690" height="368" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0877842d63528616a154910b178a5377c09fda6_2_690x368.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0877842d63528616a154910b178a5377c09fda6_2_1035x552.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0877842d63528616a154910b178a5377c09fda6.png 2x" data-dominant-color="A6A6A6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1070×571 134 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

wilsonng.colab: <p>I found this problem has to do with pin_memory=True in DataLoader.<br>
When I set pin_memory=False, the problem gone.</p>
<p>Can I know why does this problem occur when pin_memory=True?</p>

ptrblck: <p>You need to synchronize the CPU on the used stream if you are using async device2host copies, otherwise you’ll run into data corruption.</p>
Semantic segmentation returning the same prediction for each class?

johnny69: <p>I’m trying to implement a semantic segmentation model. I’m using <code>deeplabv3_resnet50</code> with transfer learning. The default version of <code>deeplabv3_resnet50</code> returns 21 classes.</p>
<p>However, I’ve created a <code>deeplabv3_resnet50</code> model and tried passing an image in. What I’ve noticed is that the result that the model outputs for the first class is <em>exactly the same</em> as result for the second class, and the third class and so on. This doesn’t make sense, and defeats the purpose of semantic segmentation because you can’t actually segment an image.</p>
<p>This is also using the default implementation of <code>deeplabv3_resnet50</code>, so I’m not sure what’s going wrong or how to fix it.</p>
<p>Does anyone know whether I might have done something wrong?</p>
Access A Variable from torch.autograd.function.GradReverseBackward

Johan_Adler: <p>I have defined a function as follows:</p>
<pre><code class="lang-auto">class GradReverse(torch.autograd.Function):
    def __init__(self, lambd):
        self.lambd = lambd

    @staticmethod
    def forward(self, x):
        return x.view_as(x)

    @staticmethod
    def backward(self, grad_output):
        return (grad_output * -self.lambd)
</code></pre>
<p>But during the back propagation I get “AttributeError: ‘GradReverseBackward’ object has no attribute ‘lambd’”, even though I set it in the forward pass as GradReverse(lambd).apply(x). Is there something I am missing?</p>

ptrblck: <p>The <code>forward</code> and <code>backward</code> methods are defined as <code>staticmethods</code> and are thus expecting <code>cls</code> as the first argument, while you are expecting to access the <code>self</code> attribute of an object (not the class).<br>
Pass <code>lambd</code> into the <code>forward</code> and use <code>ctx.save_for_backward()</code> or assign it to <code>cls</code> to use it in the <code>backward</code>. The <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">tutorial</a> uses the same approach to store the <code>input</code> tensor which is needed for the gradient computation.</p>

Johan_Adler: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>! That worked!</p>
Using Tensor functions in CUDA kernels

bhqin: <p>HI, I’m new to using the C++ API for PyTorch.</p>
<p>In my current application, I have a 256 by 14 by 4 by 4 tensor that needs to be dealt with. The application logic is something like this: For each 14 by 4 by 4 tensor in the larger tensor, execute some function <code>f</code> that produces some cumulative matrix product of 4 by 4 tensors, so the output of <code>f</code> will still be a 14 by 4 by 4 tensor. Since this is a time critical application, I wish that I can have 256 threads to run the cumulative matrix product on GPU in parallel. However, I found that I cannot directly pass <code>Tensor</code> into CUDA kernels, and I cannot call <code>mm</code> method since it is not a <code>__device__</code> function.</p>
<p>I’m wondering is there any abstractions in PyTorch allows doing so. Thanks.</p>

ptrblck: <p>You can pass the data pointer to the kernel instead of the tensor object via <code>tensor.data&lt;scalar_t&gt;()</code> as described in <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-mixed-c-cuda-extension">this part of the Custom Extensions tutorial</a>.</p>

bhqin: <p>I understand that I can do it, but this returns raw data pointer. I would like to have some tensors on the GPU with capabilities of <code>mm</code> and etc.</p>

ptrblck: <p><code>torch.mm</code> will itself call a CUDA kernel, which uses data pointers to perform the actual operation and you can call torch functions from the C++ functions. The CUDA kernel is the wrong place to call into PyTorch.</p>

bhqin: <p>Thanks. Is there any chance for having tensor functions as well as parallelism in a higher level?</p>

ptrblck: <p>Could you explain this feature a bit more, please?<br>
As already mentioned, <code>torch:mm</code> will accept tensors and will launch a CUDA kernel performing the matrix multiplication using parallelism on the GPU (assuming your tensors are stored on the GPU).<br>
Custom CUDA kernels can be used in case you want to write CUDA code (working on the data via pointers) and launch it via your custom function call on accepting tensors.<br>
What would “having tensor functions as well as parallelism in a higher level” mean?</p>

bhqin: <p>In fact, I would like to operate on the level of matrices in CUDA kernels. In the application, I would like to compute the cumulative product of a series of matrices, i.e., <code>out[i] = prod(input, 0:i)</code> where each entry would be a matrix, and input, output are vectors of matrices. In addition, I have 256 such problems to solve in parallel.</p>
Text classification tutorial confusion about dataset

dev1: <p>I’m following <em><a href="https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html" rel="noopener nofollow ugc">Text classification with the torchtext library</a></em></p>
<pre><code class="lang-auto">def collate_batch(batch):
    label_list, text_list, offsets = [], [], [0]
    for (_label, _text) in batch:
        label_list.append(label_pipeline(_label))
        processed_text = torch.tensor(text_pipeline(_text), dtype=torch.int64)
        text_list.append(processed_text)
        offsets.append(processed_text.size(0))
    
    label_list = torch.tensor(label_list, dtype=torch.int64)
    offsets = torch.tensor(offsets[:-1]).cumsum(dim=0)
    text_list = torch.cat(text_list)
    return label_list.to(device), text_list.to(device), offsets.to(device)

train_iter = AG_NEWS(split='train')
dataloader = DataLoader(train_iter, batch_size=8, shuffle=False, collate_fn=collate_batch)
</code></pre>
<p>after these operations I call<br>
<code>print(next(iter(dataloader))[1])</code></p>
<p>to get the text values for each batch. However I find out that it’s actually a tensor that merged all 8 sentences(batch size is 8) like they’re one. It gives me output 338 which is the number of tokens for this batch of 8. Is this a normal behavior?</p>