akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why is the discriminator trained more than the generator in GAN training?

shrbrh: <p>In the original GAN <a href="https://meet.google.com/tzn-buon-bai" rel="noopener nofollow ugc">paper</a>, in section 4, the authors have trained the generator once for every k times of the discriminator.<br>
Why does the discriminator require more training than the generator?</p>

J_Johnson: <p>You can think of the discriminator as a teacher. Do you want the teacher more or less experienced than the student?</p>
<p>Hence, the discriminator needs to stay ahead of the curve. Additionally, once the discriminator has a high rate of being fooled, such that the generator produces images indistinguishable from real images, you’ll know training is finished.</p>

shrbrh: <p>Thank you for your answer. That was a very clear and simple analogy.<br>
Also, when you talk about the training being finished, do you mean perceptually verifying the produced images to see how good they are or is there any particular trend of values we need to observe for the adversarial loss?</p>

J_Johnson: <p>You can decide how long you wish to train, but as you continue training, you’ll see diminishing returns. What I mean is the discriminator, being more sufficiently trained than the generator, will give you a way to gauge with some certainty how well the generator is doing and where you may want to stop training.</p>
<p>Suppose you had been training your GAN several weeks. And you establish a save and sample every 1,000 batches. And you have a batch size of 16. So a total of 16,000 samples per epoch. And let’s suppose the discriminator guessed 15,900 of those samples were real. So 99.375% generator accuracy. Keeping in mind, the discriminator is only useful during training and later disgarded, once the generator is sufficiently producing realistic images.</p>
<p>But you might decide to keep training a few more weeks in hopes of getting another 0.1-0.2% improvement.</p>
Detectron 2 and image size

Sylvain_Ard: <p>Hello, I asked it on their github but they’re don’t answer me so I ask my question here.<br>
I want to put images of size 2500x3000 pixels at input of the neural network of detectron2 maskrcnn instance segmentation program.<br>
What I have to change in the code please ?<br>
Thank you<br>
Best regards</p>

Sylvain_Ard: <p>I found it : it is MAX_SIZE_TRAIN in config</p>
Question about attn_mask in torch.nn.MultiheadAttention

mendelsontau: <p>Hi,<br>
I’m trying to pass a set of queries, keys and values through a Multi head attention layer but have some of the queries attend to just part of the keys. I think I need to use an attention mask but not quite sure from the documentation how I should use it.<br>
Let’s say I have 3 queries, keys and values and I want:</p>
<ol>
<li>the first query to attend just to the first and third keys</li>
<li>the second query to attend to all keys</li>
<li>the third query to attend just to the first and third keys<br>
would my attention mask then need to look like this?<br>
[[False,True,False],<br>
[False,False,False],<br>
[False,True,False]]<br>
Thanks</li>
</ol>
Eigendecomposition failed because matrix is ill-conditioned

Arthur_F: <p>Hello,</p>
<p>I am currently trying to train a network involving an eigendecomposition step. I keep running into the same error : torch._C._LinAlgError: torch.linalg.eigh: (Batch element 0): The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated eigenvalues (error code: 50).</p>
<p>I know that matrices can be ill-conditionned for some task (e.g. inversion) but well-conditionned for another (e.g. eigendecomposition). Does anyone know what is meant by ‘ill conditionned’ in the above error ?<br>
Related question : when computing the condition number of a matrix with torch.linalg.cond, what task does the result relates to ?</p>
<p>Thanks in advance !</p>
Compare two models layer by layer

spiegelball: <p>I have two instances of the same model, living on different devices. The inference results differ. Is it possible to compare the models “step-by-step” to figure out which calculation is faulty?</p>

ptrblck: <p>One possible approach would be to use forward hooks and to compare the intermediate outputs of each layer. <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">This post</a> gives you an example on how to register the hook and how to print it.<br>
You could use different <code>dict</code>s or store the desired outputs using different keys etc.</p>

spiegelball: <p>This sounds very good, thank you!</p>
Passing Argument into generate_sp_model()

J_Johnson: <p>I’m running <code>generate_sp_model()</code> on a file and getting the following error message:</p>
<pre><code class="lang-auto">
trainer_interface.cc(346) LOG(WARNING) Found too long line (8849 &gt; 4192).
trainer_interface.cc(348) LOG(WARNING) Too long lines are skipped in the training.
trainer_interface.cc(349) LOG(WARNING) The maximum length can be changed with --max_sentence_length=&lt;size&gt; flag.
</code></pre>
<p>I’d like to modify the <code>max_sentence_length</code> but when I try to pass it in as an argument, it says:</p>
<pre><code class="lang-auto">generate_sp_model(filename, vocab_size, model_type="unigram", max_sentence_length=10000)
TypeError: generate_sp_model() got an unexpected keyword argument 'max_sentence_length'
</code></pre>
<p>How does one go about passing this argument in?</p>

ptrblck: <p>The warning is raised by <code>sentencepiece</code> in <a href="https://github.com/google/sentencepiece/blob/31656da0c9cccfc47d4f0e69fc32d55faac3e1e9/src/trainer_interface.cc#L376-L381">these lines of code</a> and <code>torchtext</code>’s <code>generate_sp_model</code> does indeed not expect this argument and thus fails.<br>
I’m not familiar enough with <code>torchtext</code> and don’t know how it exactly calls into <code>sentencepiece</code>, but <a class="mention" href="/u/zhangguanheng66">@zhangguanheng66</a> might know it.</p>
Model performs terrible on the same dataset after setting it to eval mode

Bogdan_Khamelyuk: <p>I had really good results(at least in test and accuracy), as I’ve trained model for 3 epochs on the thermal images dataset to output coordinates of moving object. For that purpose I used cnn, maxpool, lstm and linear layers.</p>
<p>As deviation with the target coordinates was in 1e-2 field, I decided to try out printing coordinates on the video itself, so i changed model from train() to eval() modes and gave the model the before used training video frames, with torch.zero_grad() decorator above.</p>
<p>Results were way far away from those I had previously. My accuracy(sum of the absoluted difference of predict and target x,y coordinates) in the training case was in 80% of all frames not greater than one pixel. Now im having worst ever results. Can somebody say how it can be possible that evaluation on the same samples, which were used for training can be soo bad??</p>
<p>Here Im gonna also post the general network performance and its structre.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15ba207ad85630fd917f6d3a5696f661b657a176.png" alt="image" data-base62-sha1="36cNInNtRWmnMsvBlhyGuhPz0VM" width="286" height="208"><br>
Model:</p>
<pre><code class="lang-auto">  (conv1): Conv2d(1, 40, kernel_size=(6, 6), stride=(5, 5))
  (conv2): Conv2d(40, 40, kernel_size=(5, 5), stride=(2, 2), padding=(1, 1))
  (conv3): Conv2d(40, 120, kernel_size=(3, 4), stride=(3, 3))
  (relu0): ReLU()
  (maxpool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (flatten1): Flatten(start_dim=1, end_dim=-1)
  (lstm): LSTM(20, 175, num_layers=2)
  (flatten2): Flatten(start_dim=0, end_dim=-1)
  (linear3): Linear(in_features=21000, out_features=2, bias=True)
)
</code></pre>
<p>Optimizer:</p>
<pre><code class="lang-auto">Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    capturable: False
    differentiable: False
    eps: 1e-08
    foreach: None
    fused: False
    lr: 0.0005
    maximize: False
    weight_decay: 0
)
</code></pre>

eqy: <p>If you haven’t already, I would start with some sanity checks that the model is outputting similar predictions on the same data in train and eval mode. If it is, then I would check that normalization/augmentations at training time are compatible with those that are/are not being applied at eval time.</p>

Bogdan_Khamelyuk: <p>The problem is that in eval mode it predicts awfully bad on the same samples I used for training. I dont know why, since i even deleted batchnorm and dropout layers.</p>

eqy: <p>Right, it’s useful to isolate exactly why it’s performing poorly e.g., exactly where the model outputs diverge in training vs. eval and depending on the model even something as simple/straightforward as printing intermediate activations could help.</p>
Data Entry, Accounting Clerk Bookkeeper, Accounting Executive

hiring202: <p><em>Job Title: Data Entry, Accounting Clerk Bookkeeper, Accounting Executive, Executive Administrative Assistant, Customer Service, Clerical Administrator, Payroll.</em></p>
<p><em>Location: 95 Wellington Street West Suite 800 Toronto, Ontario, Canada</em></p>
<p><em>Duration: 12 Months</em></p>
<p><em>Pay Rate : $19.43/hr on w2</em></p>
<p><em>Monday- Friday 12:00pm-8:30pm CST</em></p>
<p><em>Training will be Monday-Friday 8-am 4:30pm CST</em></p>
<p><em>Description:</em></p>
<p><em>Fairfax Financial Holdings Limited is a holding company which, through its subsidiaries, is engaged in property and casualty insurance and reinsurance and investment management. Fairfax’s corporate objective is to achieve a high rate of return on invested capital and build long-term shareholder value. Fairfax seeks to differentiate itself by combining disciplined underwriting with the investment of its assets on a total return basis, which Fairfax believes provides above-average returns over the long-term.</em><br>
<em>Fairfax was founded in 1985 by the present Chairman and Chief Executive Officer, V. Prem Watsa. The company has been under present management since 1985 and is headquartered in Toronto, Canada. Its common shares are listed on the Toronto Stock Exchange under the symbol FFH and in U.S. dollars under the symbol FFH.U.</em></p>
<ul>
<li>
<ul>
<li>
</li>
</ul>
</li>
</ul>
<p><em>Required Qualifications</em></p>
<p><em>High School Diploma or equivalent.</em></p>
<p><em>Require a minimum of 35 wpm.</em></p>
<p><em>Must be able to read, write, interpret, and speak English.</em></p>
<p><em>Strong interpersonal written &amp; verbal communication skills</em></p>
<p><em>Ability to pay close attention to detail, work with speed and accuracy under pressure of heavy workloads and deadlines.</em></p>
<p><em>Proficient with complex data entry concepts.</em></p>
<p><em>Ability to navigate independently and understand complex instructions to make decisions.</em></p>
<p><em>Strong critical-thinking and problem-solving skills.</em></p>
<p><em>PC skills required – Must be proficient with MS Office Suite and Skype for Business</em></p>
<p><em>Must be able to attend full duration of training period.</em></p>
<p><em>Desired Skills</em></p>
<p><em>Ability to quickly learn and adapt to new concepts.</em></p>
<p><em>Ability to self-learn based on client instructions.</em></p>
<p><em>General knowledge of CPT, HCPCS and ICD-10 coding, and modifier usage.</em></p>
<p><em>In-depth understanding of explanation of benefits (EOBs).</em></p>
<p><em>Understanding of all HIPAA guidelines</em></p>
<p><em>Fairfax Financial Holdings Limited</em></p>

ptrblck: <p>Could you describe if and how this job might be related to PyTorch?<br>
I’ll also move it into the <code>Jobs</code> category for now.</p>
Create tensor having all elements set to 2

shrbrh: <p>Hello!<br>
I want to check whether each value of a tensor of shape [1, 1, 16, 16] is greater than 2. I tried doing it simply by</p>
<pre><code class="lang-auto">if(myTensor &gt; 2)
</code></pre>
<p>but it throws the following error:</p>
<pre><code class="lang-auto">RuntimeError: Boolean value of Tensor with more than one value is ambiguous
</code></pre>
<p>Is there any way to create a tensor of shape [1, 1, 16, 16] with all its values set to 2. Will I then be able to compare each individual value of myTensor with 2?</p>

ptrblck: <p>The <code>if</code> condition expects a single <code>True</code> or <code>False</code> statement while your comparison <code>myTensor &gt; 2</code> could yield multiple outputs as it’s an elementwise comparison.<br>
If you want to check if <strong>all</strong> values in <code>myTensor</code> are <code>&gt; 2</code> you could use:</p>
<pre><code class="lang-python">if (myTensor &gt; 2).all():
</code></pre>
<p>To create a tensor with all values set to <code>2</code> you could use:</p>
<pre><code class="lang-python">myTensor = torch.full((1, 1, 16, 16), fill_value=2.)
</code></pre>

shrbrh: <p>Thank you. That worked!</p>
In-place operation in training time

_guyyariv: <p>The situation is as follows:<br>
I have 3 blocks of NNs: A, B and C. B is nn.Embedding<br>
I want A’s output will replace only one embedding in B. for instance: B[idx]=A_output<br>
and I want that only A will train (B&amp;C parameters freezes):</p>
<pre><code class="lang-auto">for step, batch in enumerate(train_dataloader):
    A_output = A(batch['input_1'])
    B.weight.data[index] = torch.zeros(A_output.shape)
    B.weight[idx] = B.weight[idx] + A_output
    B_output = B(batch['input_2'])
    C_output = C(B_output)
    loss = F.mse_loss(C_output, batch['target'], reduction="mean")
    loss.backward(retain_graph=True)
    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<p>I’ve got:<br>
<code>one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [512, 768]], which is output 0 of AsStridedBackward0, is at version 3; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
<p>Any ideas?</p>
<p>Thanks</p>

ptrblck: <p>These lines of code are most likely causing the issue:</p>
<pre><code class="lang-python">    B.weight.data[index] = torch.zeros(A_output.shape)
    B.weight[idx] = B.weight[idx] + A_output
</code></pre>
<p>since you are directly manipulating the internal <code>.data</code> attribute (which is deprecated as it could yield unwanted side effects) and manipulating the parameter inplace.<br>
I don’t fully understand your use case since <code>A_output</code> would have a shape of <code>[batch_size, *]</code> (where <code>*</code> denotes additional dimensions) while the embedding matrixin <code>B</code> will have the shape <code>[num_embeddings, embedding_dim]</code>, which thus does not have a dependency on the batch size.<br>
Are you forcing the training to work with a batch size of <code>1</code> or how do you guarantee that the assignment would work?</p>
Error: inner dimensions differ

MikeTN: <p>Hi guys, I am very new to pytorch. I am trying to build an NN for digit and alphabet recignition with ReLU as the recognition function. The error message confuses me a lot. I am wondering if it has anything to do with my images in the dataset (torch.Size([64, 3, 32, 32])). I would appreciate any type of hint and help.<br>
Thanks in advance:)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/5462d245a8e169ebfab3b42a35af559a2799eca4.png" data-download-href="https://discuss.pytorch.org/uploads/default/5462d245a8e169ebfab3b42a35af559a2799eca4" title="Screenshot 2023-01-22 at 3.08.46 pm"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/5462d245a8e169ebfab3b42a35af559a2799eca4_2_690x424.png" alt="Screenshot 2023-01-22 at 3.08.46 pm" data-base62-sha1="c2vPxF990dYuyFudfUgakIM89OA" width="690" height="424" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/5462d245a8e169ebfab3b42a35af559a2799eca4_2_690x424.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/5462d245a8e169ebfab3b42a35af559a2799eca4_2_1035x636.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/5462d245a8e169ebfab3b42a35af559a2799eca4_2_1380x848.png 2x" data-dominant-color="232323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-01-22 at 3.08.46 pm</span><span class="informations">2546×1566 457 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

thecho7: <p>Flatten (64, 3, 32, 32) would be (64, 3072).<br>
If you want to use <code>nn.Linear</code> as you wrote, <code>in_feature</code> should be 3072 which is the same as your input dimension.</p>
<p>In short, the first ‘nn.Linear’ should be<br>
<code>nn.Linear(3072, 512)</code> instead of <code>nn.Linear(1024, 512)</code></p>

MikeTN: <p>Thanks for the explanation. Sounds like linear is not the best approach here, is there a better way to set up the network for 3 dimensional image?</p>

thecho7: <p>Convolution is the most popular way to do that.<br>
See <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Training a Classifier — PyTorch Tutorials 1.13.1+cu117 documentation</a><br>
to know how to classify the image with cifar10 tutorial</p>
Train model with multiple independent parts and manualy modified tensor

MartinPerry1: <p>Hi, I am not sure how to train my architecture.</p>
<p>I have input x and each batch consist of several parts. Each part goes through a different model. At the end, parts are merged together and passed through simple CNN.</p>
<p>My model  forward method is like:</p>
<pre><code class="lang-auto">forward(self, x):
   #switch layers and batch
   x = x.permute(1, 0, 2, 3, 4, 5)

   x0 = self.model0(x[0])
   x1 = self.model1(x[1])
   x2 = self.model2(x[2])

   #merge layers together
   x = torch.stack([x0, x1, x2], dim=1)

   #threshold to 0 / 1
   x = (x &gt; 0.5).float()
   xInd = torch.argmin(x, dim=4)  
   #calculate merged data based on argmin, self.threshold is custom value in &lt;0, 1&gt;
   xMerge= ((xInd - 1) * 2 + 1) * self.threshold 

   xMerge = self.cnn(xMerge)

   return [x0, x1, x2, xMerge]
</code></pre>
<p>I want to train the network to not only predict xMerge, but also have the loss for layers. In my custom loss function, I use multiple <code>BCEWithLogitsLoss</code>. To calculate, I do this:</p>
<pre><code class="lang-auto">class MultiBceLoss(nn.Module):
    __init__(self, layersCount):
       nn.Module.__init__(self) 
       self.layersCount = layersCount
       self.bce_loss = nn.BCEWithLogitsLoss()

    forward(self, output, gt):
       loss = self.bce_loss(output[-1], gt[-1])
       for i in range(0, self.layersCount):
           loss += self.bce_loss(output[i], gt[I])
       return loss
</code></pre>
<p>However, the network learns incorrectly. <code>xMerge</code> gradient from layers is loss, since it is a newly generated output from <code>argmin</code>. How to train this type of network? Can it be done with a single loss?</p>

ptrblck: <p>I’m not sure which part exactly you want to train, but note that you are detaching <code>x</code> and <code>xMerge</code> from <code>x0, x1, x2</code> since you are using non-differentiable operations such as <code>(x &gt; 0.5).float()</code>.</p>

MartinPerry1: <p>I want to train <code>model0</code>, <code>model1</code>, <code>model2</code> and <code>cnn</code> together.</p>

ptrblck: <p>This won’t be possible since the <code>modelX</code> are already detached by the threshold operation as well as the <code>argmin</code>.<br>
You would thus have to use a differentiable / “soft” threshold operation as described e.g. <a href="https://discuss.pytorch.org/t/training-with-threshold-in-pytorch/145962/3">here</a> by <a class="mention" href="/u/kfrank">@KFrank</a>.</p>
Difference of implementation between torch.svd and torch.linalg.svd

cyberguli: <p>Hello, can I ask the difference between the implementations of torch.svd and torch.linalg.svd?<br>
I have a laptop with 8GB of ram.</p>
<p>If I do:</p>
<pre><code class="lang-python">X=torch.rand([600, 154875, 3])
X=X.reshape(-1,1)
U,S,V=torch.svd(X)
</code></pre>
<p>all the commands compute fine.</p>
<p>Instead if I try</p>
<pre><code class="lang-python">X=torch.rand([600, 154875, 3])
X=X.reshape(-1,1)
U,S,V=torch.linalg.svd(X)
</code></pre>
<p>pytorch tries to allocate too much ram and the command fails.</p>

KFrank: <p>Hi Guglielmo!</p>
<aside class="quote no-group" data-username="cyberguli" data-post="1" data-topic="170829" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cyberguli/48/56632_2.png" class="avatar"> cyberguli:</div>
<blockquote>
<p>Hello, can I ask the difference between the implementations of torch.svd and torch.linalg.svd?</p>
</blockquote>
</aside>
<p>From the documentation for the two versions of <code>svd()</code>, <a href="https://pytorch.org/docs/1.13/generated/torch.svd.html" rel="noopener nofollow ugc">torch.svd()</a> and<br>
<a href="https://pytorch.org/docs/1.13/generated/torch.linalg.svd.html" rel="noopener nofollow ugc">torch.linalg.svd()</a>, <code>torch.svd()</code> computes, by default, the “reduced” SVD:</p>
<blockquote>
<p>If <code>some</code> is True (default), the method returns the reduced singular value decomposition.</p>
</blockquote>
<p>while <code>torch.linalg.svd()</code> computes, by default, the full SVD:</p>
<blockquote>
<p>The parameter <code>full_matrices</code> chooses between the full (default) and reduced SVD.</p>
</blockquote>
<p>Given that your <code>X</code> is very large an <em>exceedingly</em> non-square, <code>torch.svd()</code>'s<br>
default reduced SVD could well require much less memory.  You might try<br>
<code>U,S,V = torch.linalg.svd (X, full_matrices = False)</code> and see if<br>
that works for you.</p>
<p>As an aside, <code>torch.svd()</code> is deprecated in favor of <code>torch.linalg.svd()</code>,<br>
so if you can get <code>torch.linalg.svd()</code> working, you should probably use it.</p>
<p>Best.</p>
<p>K. Frank</p>

cyberguli: <p>Thank you, I should have read the documentation better. Using</p>
<pre><code class="lang-python">U,S,V = torch.linalg.svd (X, full_matrices = False)
</code></pre>
<p>indeed gets the work done.</p>
ResNet50 training using additional intermediate layers

norbertmarko: <p>Hello!</p>
<p>I’m trying to implement a custom network by <strong>utilizing the pre-trained ResNet50 network from torchvision</strong>.</p>
<p>What I would like to achieve is taking the features after the first 2 stages (so essentially the <code>self.layer1</code> if I understood the architecture right) and use that output along with the swapped out <code>self.fc</code> layer <strong>so I can concatenate them or even use them in two different routes.</strong></p>
<p>It would look something like this in the forward method (ideally):</p>
<pre><code class="lang-python">def forward(self, x):
    fc_out, layer1_out = (unkown)(x)
    return fc_out, layer1_out
</code></pre>
<p>It’s not exact, but you get the idea, both layers could be used for forward pass and backward pass also.</p>
<p>To understand the context a little more, I’m using the convolutional network with image sequences (4D inputs) and then I utilize the output features in a many-to-many RNN network. I would like to add the aforementioned intermediate layer to the training either as a skip connection by concatenating with the last fc layer, or feeding it to a second RNN layer.</p>
<p>So you see, it seems a little bit complicated but I hope there is a simple and elegant solution for this problem.</p>
<p>Thanks in advance!</p>

norbertmarko: <p>Anyone have any guesses? Might this be achieveable by forward or backward hooks?</p>

tiramisuNcustard: <p>I know that additional layers can be added at the end of the built-in models. Not sure if you can add intermediate layers using built-in functions. You can create your own ResNet50 model. Please see the following link on ResNet50.</p>
<p><a href="https://dhruvs.space/posts/understanding-resnets/" rel="noopener nofollow ugc">https://dhruvs.space/posts/understanding-resnets/</a></p>
Working example of CUDA accessor

Matthias_Moller: <p>I would like to know how to integrate the CUDA accessor example given here</p>
<p><a href="https://pytorch.org/cppdocs/notes/tensor_basics.html#cuda-accessors" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/notes/tensor_basics.html#cuda-accessors</a></p>
<p>in a CMake project. Just copy-pasting the code into a <code>cpp</code> file does not compile.</p>

ptrblck: <p>I don’t know what exactly is failing as you didn’t give any information about the issues, but maybe the right includes are missing?</p>

Matthias_Moller: <p>Just pasting the code in a cpp file and compiling it with Clang or Gcc does not work because of the <strong>global</strong> keyword. I assume that I need to place it in a separate cu file and compile it with nvcc into a library. How is it done with the CUDA kernels in the C++ API’s libraries?</p>

ptrblck: <p>You might need to add <code>CUDA</code> as a language to your <code>CMakeLists.txt</code> e.g. via:</p>
<pre><code class="lang-auto">project(custom_ops LANGUAGES CXX CUDA)
</code></pre>
<p>Also make sure the file extension is <code>.cu</code> for your custom CUDA code.</p>

Matthias_Moller: <p>I still try to reverse engineer how Torch is doing it internally. Assume that I have a C++ application written in the file <code>app.cxx</code>. Depending on whether the tensors are on the CPU or GPU I need to dispatch between a CPU accessor and a GPU accessor variant, e.g.</p>
<pre><code class="lang-auto">auto a = torch::zeros({10});
if (a.is_cuda()) 
{
  auto a_accessor = a.packed_accessor64&lt;float,1&gt;();
  packed_accessor_kernel&lt;&lt;&lt;1, 10&gt;&gt;&gt;(a_accessor); // does not compile by g++!!!
} 
else 
{
  auto a_accessor = a.accessor&lt;float,1&gt;();
  for (int64_t i=0; i&lt;10; ++i)
    a_accessor[i] = i;
}
</code></pre>
<p>The corresponding CUDA kernel would read as follows</p>
<pre><code class="lang-auto">__global__ void packed_accessor_kernel(
    torch::PackedTensorAccessor64&lt;float, 1&gt; a) {
  int i = threadIdx.x;
  a[i] = i;
}
</code></pre>
<p>I could put the CUDA kernel into a <code>.cu</code> file and put the entire dispatch routine into a library that I link to my application. But this would introduce some overhead as I would have to call the dispatch routine instead of implementing it inline (so far all my code is header-only). Any help on how this is done efficiently in Torch is appreciated.</p>
Is it possible to used shared GPU memory while loading Stable diffusion model?

Dip_Deb: <p>I have 8GB memory graphics card. When I’m running Stable diffusion (using <strong>torch.float16</strong>) if the model big then my dedicated memory become full and showed error like<br>
<strong>CUDA out of memory. Tried to allocate 2.32 GiB (GPU 0; 8.00 GiB total capacity; 5.66 GiB already allocated; 0 bytes free; 6.41 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</strong></p>
<p>By searching through internet I found to set data_type to <strong>torch.float16</strong> which does not help all the time.</p>
<p>But GPU shared memory is not using. Is there any way to use shared GPU memory to bypass the out of memory error?</p>

tiramisuNcustard: <p>If you have set pin_memory = True then set it to False in your Dataloader and see if that helps.</p>

Dip_Deb: <p>I’m actually using diffusers library where i didn’t find any pin_memory variable.</p>

tiramisuNcustard: <p>pin_memory is used to allocate space in GPU RAM for faster data transfer. If there is a way to do the same thing in the diffusers library then you might want to check that. Here is something that I found on the project’s github page, I hope this is the library you are using. <a href="https://github.com/huggingface/diffusers" rel="noopener nofollow ugc">https://github.com/huggingface/diffusers</a></p>
<p>“If you are limited by GPU memory, you might want to consider chunking the attention computation in addition to using <code>fp16</code>. The following snippet should result in less than 4GB VRAM.”</p>
<pre><code class="lang-auto">pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
pipe = [pipe.to](http://pipe.to)("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
pipe.enable_attention_slicing()
image = pipe(prompt).images[0]

</code></pre>
Running time and BP under condition

Yanik_Kovlev: <p>I want to train a network but apply the gradient only for images (out of a minibatch) that their output is under some certain condition - for example, the maximal value of the output for given image is above some threshold.</p>
<p>Before trying to solve this I tried to feedforward a minibatch of 100 images and calculated the time it took to compute ff and bp and I got something like 25 seconds while for 10 images it took 7 seconds</p>
<p>but when I tried to do the following (taking only 10 outputs out of the 100) it still took 25 seconds</p>
<p>output = model(images)</p>
<p>output = output[0:10,:]<br>
labels = labels[0:10,:]</p>
<p>optimizer.zero_grad()<br>
loss= criterion(output, labels)<br>
loss.backward()<br>
optimizer.step()</p>
<p>thanks</p>

eqy: <p>How are you doing the timing here (e.g., with appropriate <code>torch.cuda.synchronize()</code> calls before timing starts and and stops if the execution is happening on GPU)</p>
<p>Note that it could be also expected that the time is not changing substantially if the bottlenecks lie elsewhere or the hardware is not being fully utilized.</p>

Yanik_Kovlev: <p>I use time.time() before and after the backprop part.<br>
I don’t think it lie somewhere else …</p>
<p>I think there is something that does a matrix multiplication of sizes [ 100,10] instead of [10,10]<br>
in the backprop.</p>
<p>Maybe some placeholder is created after the forward ?</p>

eqy: <p>These problem sizes are small enough that they are essentially “free” compared to the cost of launching the kernels and dispatching from Python.<br>
For example, on an A6000, we need around 1e5 iterations to even get a stable measurement of the time per iteratIon:</p>
<pre><code class="lang-auto"># cat matmul.py
import time
import torch

iters = 100000

a = torch.randn(10, 10, device='cuda')
b = torch.randn(100, 10, device='cuda')

torch.cuda.synchronize()
t1 = time.perf_counter()
for _ in range(iters):
  torch.matmul(a, a)
torch.cuda.synchronize()
t2 = time.perf_counter()

print(f"10,10 x 10,10 took {t2-t1}, {(t2-t1)/iters} per iter")

torch.cuda.synchronize()
t1 = time.perf_counter()
for _ in range(iters):
  torch.matmul(b, a)
torch.cuda.synchronize()
t2 = time.perf_counter()

print(f"100,10 x 10,10 took {t2-t1}, {(t2-t1)/iters} per iter")
</code></pre>
<pre><code class="lang-auto"># python matmul.py
10,10 x 10,10 took 0.6885399222373962, 6.885399222373963e-06 per iter
100,10 x 10,10 took 0.6861815741285682, 6.861815741285682e-06 per iter
</code></pre>
<p>I would check if you see the same behavior on a larger model or with a greater difference in input sizes.</p>

Yanik_Kovlev: <p>I don’t think my problem relates to cuda runtimes at all.</p>
<p>Let me try to define my problem differently -</p>
<p>Assume I want to do BP only on 10 images out of the 100 images minibatch - how can I do it efficiently ?</p>

Yanik_Kovlev: <p>Does anyone have any suggestions?</p>
Inference with mps backend on M1 returns +/- Inf

ivanv03x: <p>I load a superresolution model with</p>
<pre><code class="lang-auto">device = torch.device('cpu')
model = torch.load("model_srresnet.pth",map_location=device)["model"]
image = Variable(torch.from_numpy(im_input / 255.0).float())
image = image.to(device)
with torch.inference_mode():
    output = model(image)
</code></pre>
<p>With device ‘cpu’ all works as expected the the inference output looks good, i.e. if I do output.min(), output.max() I see tensor(0.0403) tensor(1.0430). However, if I change ‘cpu’ to ‘mps’ the output min-max is tensor(inf, device=‘mps:0’) tensor(-inf, device=‘mps:0’).</p>

ivanv03x: <p>Just more info: I am using torch version 1.13.1 on MacOS Ventura (13.0).</p>

ivanv03x: <p>Problem solved, installed the latest nightly pytorch (torch 2.0.0.dev20230121) and it’s all good.</p>
Random global shift in data transformation/augmentation

Crispolo: <p>I’m trying to reproduce a model described in a paper that I found online, you can found it <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9014480/" rel="noopener nofollow ugc">here</a>.</p>
<p>In section <strong>“D. Finetuning the Models”</strong> I read:<br>
<em>To encourage the model to be more robust to the exact location of the  pleural line we applied a random global shift to the signed distance channel (max ±8%).</em></p>
<p>I’m not sure what <strong>“random global shift to the signed distance channel”</strong> means and what kind of transformation I should apply.</p>
<p>Can you help me to understand it please?</p>
<p>I’m using PyTorch in Colab environment.</p>
What is linear regime of the non-linearity for a non-linear activation function?

shrbrh: <p>We use batch normalization layers in between the input layers of CNNs to reduce the internal covariate shift, as per my understanding from this <a href="https://arxiv.org/pdf/1502.03167v3.pdf" rel="noopener nofollow ugc">paper</a>. However, in section 3 there’s a part that says-</p>
<pre><code class="lang-auto">normalizing the inputs of a sigmoid would constrain them to the linear regime 
of the nonlinearity. To address this, we make sure that the transformation 
inserted in the network can represent the identity transforms.To accomplish
this, we introduce, for each activation, a pair of parameters γ and β, which
scale and shift the normalized value.
</code></pre>
<p>What does “linear regime of the nonlinearity” mean and how do scaling and shifting help?</p>

eqy: <p>I’m not an expert on batchnorm, but I would interpret this as a roundabout way of saying the region (near zero) where the first derivative of the sigmoid function is not changing much, and scaling/shifting moves the input of the sigmoid toward zero.</p>
Is there any solution other than increasing memory?

NagaYu: <pre><code class="lang-auto">model.eval()
with torch.no_grad():
        inputs = torch.tensor(test_df.iloc[:,:].values).to(device)
        test_Y = model(inputs.float())
        survived = torch.max(test_Y, dim=1)[1]
        test_paID = pd.read_csv('sample_submission.csv')['customer_ID']
        sub_df = pd.DataFrame({'customer_ID':test_paID.values, 'prediction':survived})
        print(sub_df)
        sub_df.to_csv('sample_submission.csv', index=False)
</code></pre>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
Cell In[9], line 4
      2 with torch.no_grad():
      3         inputs = torch.tensor(test_df.iloc[:,:].values).to(device)
----&gt; 4         test_Y = model(inputs.float())
      5         survived = torch.max(test_Y, dim=1)[1]
      6         test_paID = pd.read_csv('sample_submission.csv')['customer_ID']

File ~/opt/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py:1488, in Module._call_impl(self, *args, **kwargs)
   1483 # If we don't have any hooks, we want to skip the rest of the logic in
   1484 # this function, and just call forward.
   1485 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1486         or _global_backward_pre_hooks or _global_backward_hooks
   1487         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1488     return forward_call(*args, **kwargs)
   1489 # Do not call functions when jit is used
   1490 full_backward_hooks, non_full_backward_hooks = [], []

Cell In[6], line 30, in Net.forward(self, x)
     28 x = self.f7(x)
     29 x = self.relu(x)
---&gt; 30 x = self.f8(x)
     31 x = self.relu(x)
     32 x = self.f9(x)

File ~/opt/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py:1488, in Module._call_impl(self, *args, **kwargs)
   1483 # If we don't have any hooks, we want to skip the rest of the logic in
   1484 # this function, and just call forward.
   1485 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1486         or _global_backward_pre_hooks or _global_backward_hooks
   1487         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1488     return forward_call(*args, **kwargs)
   1489 # Do not call functions when jit is used
   1490 full_backward_hooks, non_full_backward_hooks = [], []

File ~/opt/anaconda3/lib/python3.9/site-packages/torch/nn/modules/linear.py:114, in Linear.forward(self, input)
    113 def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 114     return F.linear(input, self.weight, self.bias)
</code></pre>
<p>RuntimeError: MPS backend out of memory (MPS allocated: 4.80 GB, other allocations: 7.36 GB, max allowed: 18.13 GB). Tried to allocate 7.03 GB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).</p>

eqy: <p>It looks like you are loading the entire dataframe for inference on the model at once. Would it be possible to split this with some kind of batching if the dataframe is large?</p>
Performance drops when setting persistent_workers=True

simone: <p>I encounter a performance drop when setting <code>persistent_workers</code> equals <code>True</code> in the datamodule. Should it just handle the workers? Why does it affect in some way also the optimization of the model?</p>
<p>So, what is the proper way to use persistent workers?</p>
<p>A working <a href="https://colab.research.google.com/drive/1vKIkxm2oVudHM6t-ivI6SMfTBUbLWnI8?usp=sharing" rel="noopener nofollow ugc">Colab</a> to reproduce the behavior.</p>

ptrblck: <p>Are you seeing the same issue without wrapping your model into Lightning and if so could you share a minimal and executable code snippet, please?</p>
OpenAI's whisper to mobile

Sriram: <p>Has anyone been successful in porting OpenAI’s whisper to torchscript?</p>
<p>I have been unable to get jit.script/trace working. I get errors like this:</p>
<p><em>RuntimeError: Can’t redefine method: forward on class: <strong>torch</strong>.whisper.model.Conv1d</em></p>
<p>Strangely, I was able to convert the .pt model translated to tflite, but I don’t know (yet) if that works.</p>

Joao_Simoes: <p>Looking for the same solutions <img src="https://discuss.pytorch.org/images/emoji/apple/raised_hand.png?v=12" title=":raised_hand:" class="emoji" alt=":raised_hand:" loading="lazy" width="20" height="20"></p>

kimishpatel: <p>Is there more detail to this message?</p>

Eliran_Abdoo: <p>I’ve managed to do so (it required quite a few changes), but I’m unable to publish it currently due to IP.<br>
You guys think there’s a demand for that?</p>

Sriram: <p>Yes, please. What’s the IP issue?</p>

Eliran_Abdoo: <p>IP as in intellectual property, I’m just unable to publish it. I’m trying to understand if it’s an interesting enough problem so it that it worths publishing</p>
Reproducibility in numerical operations

AleTL: <p>Hello!</p>
<p>I have noticed that, even though I am actively fixing the seed, there are small differences between runs when the I multiply two tensors. More specifically, I have located a run in which I perform an element wise multiplication of two tensors and, in one run, the multiplication of <code>torch.sigmoid(torch.tensor(0.0000))*torch.sigmoid(torch.tensor(0.1744))</code> returns <code>0.2717</code> and other in which returns <code>0.2718</code>. I am highly confused and I don’t really know how can I fix it. It’s important since this small differences accumulated and provoke higher differences in the long term. I set the same seed in both experiments and both run in the same GPU model.</p>
<p>I have investigated more the issue and apparently the difference is not in the multiplication but in the output of a loss function, the difference is quite small and cannot be captured with the precision of the print I wrote, but in some cases I get <code>0.0053490624</code> vs <code>0.0053490633</code>. Could this be fixed simply changing the data type?</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>You are most likely seeing the expected small errors caused by the limited floating point precision and a potentially different order of operations in the used operators.<br>
Take a look at the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> to see how deterministic algorithms could be enabled (which could be slower, but should yield the same results).</p>
<p>Increasing the <code>dtype</code> width (e.g. by casting the tensors to <code>float64</code>) will increase the precision for a performance penalty (on GPUs this could yield a significant slowdown).</p>
Torch.fft.rff2 and a combination of fft mismatch

PaTrickWwW: <p>According to the torch.fft doc, the rfft2 is equivalent to a combination of 'fft()'and ‘rfft()’, but when I apply it to an image, the results of the combination of separable fft mismatch with the rfft2, why?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/e/cec76c4f1de11d80a27df65f2620b1bb13810577.png" data-download-href="https://discuss.pytorch.org/uploads/default/cec76c4f1de11d80a27df65f2620b1bb13810577" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/cec76c4f1de11d80a27df65f2620b1bb13810577.png" alt="image" data-base62-sha1="tvfCuIOUGYdCmA5438WKoecZGUD" width="690" height="151" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1033×227 11.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fec5b6999c7405debc914dfa074ff02cfd411191.png" alt="image" data-base62-sha1="AlOSkhVCynaHMPfB3Lm5taYxggV" width="542" height="95"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c3bb3b4947bc19f65f144c9262ebaa60533ec00.png" alt="image" data-base62-sha1="6jiV0TVO4z1q5i1oyDMmm87p2wg" width="661" height="309"></p>

ptrblck: <p>The docs use <code>assert_close</code> which compares floating point numbers with a small <code>eps</code> depending on their <code>dtype</code> while you are comparing both results for bitwise-identical results, which will easily fail for floating point numbers due to the limited precision and a potentially different order of operations.</p>
ResNet18+GRU results

Sara176: <p>Hi i am trying to use pretrained resnet 18 and gru for automatic lipreaing<br>
here is my model<br>
class CNN3D(nn.Module):                                                                         <span class="hashtag-raw">#classes_num</span><br>
def <strong>init</strong>(self, t_dim=29, img_x=88, img_y=88, drop_p=0.2, fc_hidden1=256, fc_hidden2=128, num_classes=5):<br>
super(CNN3D, self).<strong>init</strong>()</p>
<pre><code>    resnet = models.resnet18(pretrained=True)
    resnet.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3,bias=False)
    modules = list(resnet.children())[:-1]      # delete the last fc layer.
    
    self.resnet = nn.Sequential(*modules)

    self.gru = nn.GRU(input_size=512, hidden_size=1024, num_layers=3, batch_first=True, bidirectional=True, dropout=0.2)
    self.fc = nn.Linear(1024*2, num_classes)
    self.dropout = nn.Dropout(0.4)

def forward(self, x):
    #print(x.shape) #[70, 1, 29, 88, 88]
    x = x.transpose(1, 2)
    x = x.contiguous()
    x = x.view(-1, 1, x.size(3), x.size(4))

    #print(x.shape)  #[2030, 1, 88, 88]
    x = self.resnet(x)
    #print(x.shape) #[2030, 512, 1, 1]

    x= x.view(-1, 29, 512)
    #print(x.shape) #[70, 29, 512]
    x,_ = self.gru(x)
    #print(x.shape) #[70, 29, 2048]

    x = self.dropout(x).mean(1)
    #print(x.shape) #[70, 2048]

    x= self.fc(x)
    #print(x.shape) #[70, 2]


    return x
</code></pre>
<p>and the results i got were<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/161cb34e792a6cdb6f7b828c843e11b43f756f7e.png" data-download-href="https://discuss.pytorch.org/uploads/default/161cb34e792a6cdb6f7b828c843e11b43f756f7e" title="Picture1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/161cb34e792a6cdb6f7b828c843e11b43f756f7e_2_690x274.png" alt="Picture1" data-base62-sha1="39BZFEKfdnCpCSAdFtqdcXQBrMW" width="690" height="274" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/161cb34e792a6cdb6f7b828c843e11b43f756f7e_2_690x274.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/161cb34e792a6cdb6f7b828c843e11b43f756f7e_2_1035x411.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/6/161cb34e792a6cdb6f7b828c843e11b43f756f7e.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Picture1</span><span class="informations">1035×412 56.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>can someone help me to determine what is causing this?</p>

ptrblck: <p>It looks like your model is overfitting the training data and cannot generalize to unseen data.<br>
A common approach is to add regularization to the training.</p>
Guided backprop interpretation for multilayer perceptron

neurospacer: <p>Hi,</p>
<p>I want to use interpretation algorithm for simple feed-forward neural network (multilayer perceptron with 13 input neurons, 2 layers deep - 10 neurons each, output is 2 class) to classify voxels.</p>
<p>Each input neuron comes as voxel from 13 feature maps channels of the CT image (extracted statistical maps).  The question is can I use <a href="https://captum.ai/api/guided_backprop.html" rel="noopener nofollow ugc">Guided Backpropagation</a> for interpretation? Or it better to use model-agnostic methods like LIME?</p>
<p>I didn’t find any use cases of guided backpropagation for MLP, most of the examples are used with deep convolutional NNs.</p>