akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How does the multinomial sampling work?

username: JiaMingLin
cooked: <p>Hi,<br>
I am dealing an imbalanced dataset, say #(postive) = 1K and #(negative) = 50K.<br>
Then I find a library <a href="https://github.com/ufoym/imbalanced-dataset-sampler" rel="noopener nofollow ugc"> imbalanced-dataset-sampler</a>(which is based on <code>torch.multinomial</code>) to resample to reduce the skewness.</p>
<p>For the basic usage, it pass and array of data weight to <code>torch.multinomial</code> then return the sampled indices(with replacement).</p>
<p>Example</p>
<pre><code class="lang-auto"># weight for each data point, 2e-5 = 1/#(negative), 1e-3 = 1/#(positive)
weights = [2e-5, 1e-3, 2e-5, ...] 
sampled_indices = torch.multinormail(
                           weights, 
                           num_samples = (num_pos+num_neg),
                           replacement = True
)
</code></pre>
<p>However, when I query original dataframe with this sampled indies, the #(positive) and #(negative) are almost equal. I am wondering the reason behind, can someone give me some tips?<br>
Moreover, I would like to adjust the pos to neg ratio, how do I implement with <code>torch.multinomial</code>?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="JiaMingLin" data-post="1" data-topic="169711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jiaminglin/48/56188_2.png" class="avatar"> JiaMingLin:</div>
<blockquote>
<p>However, when I query original dataframe with this sampled indies, the #(positive) and #(negative) are almost equal. I am wondering the reason behind, can someone give me some tips?</p>
</blockquote>
</aside>
<p>This would be the desired outcome as the sampler would try to balance the class indices in the batch.<br>
The reason is that samples with a higher weight will be sampled more likely compared to samples with a lower weight. Since you are defining the weight as the class frequency the minority class samples will be oversampled.</p>
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 4 and 3 in dimension 2. My inputs are numpy files that I converted to tensor

username: peony
cooked: <p>I am getting this error when I try batch size more than 1</p>
<pre><code class="lang-auto">RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 5 and 4 in dimension 2 at C:\w\1\s\tmp_conda_3.6_095855\conda\conda-bld\pytorch_1579082406639\work\aten\src\TH/generic/THTensor.cpp:612
</code></pre>
<p>All the solution I found here and in StackOverflow talk about images as inputs, but my inputs are numpy arrays (.npy) that I converted to tensors separately. Here is my code for the conversion:</p>
<pre><code class="lang-auto"> tensor1 = torch.from_numpy(np.load(NPY))
 tensor1 = tensor1.type(torch.FloatTensor)
 tensor1 = torch.einsum('h w c -&gt; c h w', tensor1)
</code></pre>
<p>I understand that by default numpy data is given as</p>
<pre><code class="lang-auto">[batch_size, depth, height, width, channels] 
</code></pre>
<p>however torch tensors have this dimension</p>
<pre><code class="lang-auto">[batch_size, channels, depth, height, width]. 
</code></pre>
<p>From this, I am guessing dimension 2 refers to depth.<br>
Is there any way I can make the sizes of the tensors in dimension 2 similar? If so, how can I do this with numpy arrays?</p>
<p>Here is the code of the model I am experimenting with for reference:</p>
<pre><code class="lang-auto">        elif arch.startswith('resnet50'):
            self.features = nn.Sequential(*list(original_model.children())[:-1])
            
            for i, param in enumerate(self.features.parameters()):
               
                param.requires_grad = False
            self.fc_pre = nn.Sequential(nn.Linear(2048, fc_size), nn.Dropout())
            self.rnn = nn.LSTM(input_size = fc_size,
                        hidden_size = hidden_size,
                        num_layers = lstm_layers,
                        batch_first = True)
            self.fc = nn.Linear(hidden_size, num_classes)
            self.modelName = 'resnet50_lstm'

        else:
            raise Exception("This architecture has not been supported yet")

    def init_hidden(self, num_layers, batch_size):
        return (torch.zeros(num_layers, batch_size, self.hidden_size).cuda(),
                torch.zeros(num_layers, batch_size, self.hidden_size).cuda())

    def forward(self, inputs, hidden=None, steps=0):
        length = len(inputs)
        fs = torch.zeros(inputs[0].size(0), length, self.rnn.input_size).cuda()
        
        for i in range(length):
            
            f = self.features(inputs[i])
            f = f.view(f.size(0), -1)
            f = self.fc_pre(f)
            fs[:, i, :] = f

        outputs, hidden = self.rnn(fs, hidden)
        outputs = self.fc(outputs)
        return outputs

</code></pre>
<p>Thank you.</p>
Keep inter-state variable through loss.backward()

username: hao_chen
cooked: <p>I am trying on time-series control simulation by means of a LSTM controller, which uses response sensor signal to gernerate driving signal. The target is to minimize the response.</p>
<h2>
<a name="when-apply-lossbackward-the-inter-state-variable-intermediate-value-will-be-auto-deleted-1" class="anchor" href="#when-apply-lossbackward-the-inter-state-variable-intermediate-value-will-be-auto-deleted-1"></a>When apply loss.backward(), the inter-state variable (intermediate value) will be auto-deleted.</h2>
<p>It tells<br>
“Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.”</p>
<h2>
<a name="however-this-inter-state-variable-record-the-final-state-of-last-step-which-is-critical-for-the-simulation-2" class="anchor" href="#however-this-inter-state-variable-record-the-final-state-of-last-step-which-is-critical-for-the-simulation-2"></a>However, this inter-state variable record the final state of last step, which is critical for the simulation.</h2>
<h2>
<a name="i-have-tried-lossbackwardretain_graphtrue-3" class="anchor" href="#i-have-tried-lossbackwardretain_graphtrue-3"></a>I have tried loss.backward(retain_graph=True),</h2>
<p>it tells<br>
“RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [32, 6]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!”</p>
<p>How could I keep the inter-state variable through loss.backward(),<br>
Any suggestion?</p>
<p>I learn a lot from the post below. However, I still cannot solve it.</p>
<p><a href="https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time/6795/127">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time - PyTorch Forums</a></p>
<p>Best regards.<br>
Chen</p>
PyTorch2.0 dev20230105 crashes kernel in jupyternotebook when starting training

username: Robert_Avram
cooked: <p>I removed the last version of PyTorch2.0 from December 26th that had due to the security fix.</p>
<pre><code class="lang-auto">pip3 uninstall -y torch torchvision torchaudio torchtriton
</code></pre>
<p>I then reinstalled PyTorch 2.0<br>
<code>pip3 install numpy --pre torch --force-reinstall --index-url https://download.pytorch.org/whl/nightly/cu117</code></p>
<p>I have CUDA 11.7, 4 A6000 GPUs, Driver 525.60.13.<br>
The training doesn’t start and the kernel dies immediately after filling the GPU memory.</p>
<p>If I revert back to the PyTorch version from December 26th, everything works fine.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Robert_Avram" data-post="1" data-topic="169694">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/robert_avram/48/56184_2.png" class="avatar"> Robert_Avram:</div>
<blockquote>
<p>If I revert back to the PyTorch version from December 26th, everything works fine.</p>
</blockquote>
</aside>
<p>The vulnerable binaries were deleted so I assume you are using your cache?<br>
In any case, <strong>do not reinstall</strong> these binaries as you might also reinstall the <code>triton</code> malware binary.</p>
<p>I’ve just installed the latest binaries on an A6000 and it works properly:</p>
<pre data-code-wrap="python"><code class="lang-python">python -c "import torch; print(torch.__version__); print(torch.randn(1).cuda())"
2.0.0.dev20230105+cu117
tensor([0.6857], device='cuda:0')
</code></pre>
Can I keep running torch==1.13 with CUDA=11.1?

username: Kendos93
cooked: <p>Hi there!</p>
<p>I wanted to upgrade the torch version I am currently using. I am using a K80 GPU, which, if I understand correctly supports CUDA Driver Version 11.4 and CUDA Runtime Version 11.1. So I am using nvidia/cuda:11.1.1-cudnn8-runtime base image for my docker.</p>
<p>I have been using torch==1.11 before, but recently, I have updated it to torch==1.13. Even though it is not supposed to work with my CUDA 11.1, it seems to be running correctly.</p>
<p>I am wondering why is it the case and if it is possible that something will break if I keep using the newest version of torch.</p>
<p>Best,<br>
Jakub</p>

username: ptrblck
cooked: <p>K80s are still supported in all of our binaries, since we are building for compute capability 3.7-8.6. You should not expect to see any issues.</p>
Dropout2d (channel dropout) but for 1D input?

username: carbocation
cooked: <p>I like the idea of trying channel dropout, what PyTorch refers to as <a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout2d.html" rel="noopener nofollow ugc">Dropout2d</a>, to drop entire filters from my convnet. But I am working with 1D signal, not 2D images. The shape required by Dropout2d is <code>(N,C,H,W) or (C, H, W)</code>, which is just a wrapper that calls into <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.dropout2d.html#torch.nn.functional.dropout2d" rel="noopener nofollow ugc">F.Dropout2d</a>.</p>
<p>This isn’t compatible with my input, which is <code>(N, C, W) or (C, W)</code>. Am I missing anything in PyTorch that would enable channel dropout for 1D convnets?</p>

username: mMagmer
cooked: <p>i don’t know about existing one.<br>
but you can unsqueeze input and use the same module.</p>
<pre><code class="lang-auto">d = nn.Dropout2d()
out = d(in.unsqueeze(-1)).squeeze()
</code></pre>

username: carbocation
cooked: <p>It looks like this works. Because of the unsqueeze/squeeze steps, this would normally require a bunch of entries in the <code>forward()</code> function, from what I can tell. One workaround is below: I define a module and that can be used in the <code>init()</code> function of other modules. There is probably a more elegant solution, and if so I’d like to learn it.</p>
<pre><code class="lang-auto">class ChannelDropout1d(nn.Module):
    def __init__(self, *args):
        super().__init__()
        self.d = nn.Dropout2d()
    
    def forward(self, x):
        out = x
        out = out.unsqueeze(-1)
        out = self.d(out)
        out = out.squeeze()
        
        return out
</code></pre>

username: carbocation
cooked: <p>A slight change - most important part is specifying the squeeze dimension, otherwise this plays badly with batch sizes of 1. I also added kwargs so that <code>p</code> (dropout probability) can be specified.</p>
<pre><code class="lang-python">class ChannelDropout1d(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()
        self.d = nn.Dropout2d(**kwargs)
    
    def forward(self, x):
        out = x
        out = out.unsqueeze(-1)
        out = self.d(out)
        out = out.squeeze(-1)
        
        return out
</code></pre>

username: crownz
cooked: <p>Please see pytorch’s source code. They have provided the 1d version of dropout. Link: <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/dropout.html#Dropout" rel="noopener nofollow ugc">torch.nn.modules.dropout — PyTorch 1.13 documentation</a></p>

username: carbocation
cooked: <p>Nice, thanks! Based on looking at prior versions of the docs, it looks like it got <a href="https://pytorch.org/docs/1.12/generated/torch.nn.Dropout1d.html" rel="noopener nofollow ugc">introduced in 1.12</a>.</p>
Gpu asynchronous and processparallel

username: ximing
cooked: <p>Since gpu is asynchronous in Pytorch. Can I regard it as a process parallel?</p>

username: ximing
cooked: <p>depends on how you use it.</p>
<p>you can generate different process and each run on a GPU<br>
or you can run all the GPUs on one process. but different GPU has different steam. They are parallel with each other. by default, each GPU has one default stream.</p>
Failing to Replicate sklearn LogisticRegression in PyTorch. Any help?

username: BlakeWest
cooked: <p>Hi, I’ve been using sklearn for a while in a personal project, and it’s generally been very good. I’ve built a model that works pretty well using their built in LogisticRegression. However I know there are some features that are not linear responses. And I also know generally how to use PyTorch so wanted to try out  incorporating that into my project. Ideally, I’m looking to run what I believe are the non-linear features through some shallow NN’s, and then concatenate those with the linear features into a basic LogisticRegression model. One key thing I need is for the probabilities to be well-calibrated. NN’s typically don’t provide this. But sklearn’s LogisticRegression does this extremely well.<br>
So! I want to replicate sklearn’s logistic regression in PyTorch, and then build the model described above in an end-to-end fashion. However, I can’t seem to get it to replicate, and I’m hoping I can get some advice. If you look at the sklearn <a href="https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression" rel="nofollow noopener">docs</a>, they show the loss function, and and from the source code, they appear to use LibLinear, and are essentially optimizing an NLLLoss with an L2 penalty. Seemed easy enough with PyTorch, but no luck!</p>
<p>Here’s what I’ve tried…</p>
<pre><code class="lang-auto">class DeepLogisticRegression(nn.Module):
    def __init__(self, num_in):
        super().__init__()
        output_units = 2
        self.linear = nn.Linear(num_in, output_units)
        self.sigmoid = nn.Sigmoid()
        self.sequential = nn.Sequential(self.linear, self.sigmoid)

    def forward(self, X):
        return self.sequential(X.float())
# Optimizer = Adagrad(weight_decay=2, lr = 0.001, batch_size=4096)
</code></pre>
<p>Note I’m fitting the above using Adagrad, and the weight decay is meant to replicate sklearn’s L2 penalty. I run this for about 30 epochs. I’ve tried various configs of the params, including SGD, different weight decay, different lr, etc. But my PyTorch version always ends up just getting to a local minima, where it essentially always picks the class that is slightly more common (62.26% of total samples). I can’t seem to get it to actually “train”, and find a legit model. But sklearn’s implementation has no problem with this.</p>
<p>Any ideas or thoughts would be so much appreciated! Thanks you thank you!</p>
<p>PS: I guess theoretically if I could create transformations of my non-linear features and just hand those directly into an sklearn LogisticRegression, and train the whole thing end to end, that would be sweet too. I don’t think that’s possible though?</p>

username: travellingbones
cooked: <p>I’m having the same problem using L1 regularization on the weighted Adult Dataset. SciKit implementation gets ~80%AUC using liblinear optimizer, but I’m struggling to get over 70% in most runs w/ PyTorch with various optimization settings.</p>
RuntimeError: Mask shape should match input shape; transformer_mask is not supported in the fallback case

username: Kong
cooked: <p>I receive the error below when doing a forward pass with a Transformer. I read from the post here <a href="https://github.com/pytorch/pytorch/issues/83142" class="inline-onebox" rel="noopener nofollow ugc">Transformer encoder error when encoding long sequence (more than 1024 tokens) · Issue #83142 · pytorch/pytorch · GitHub</a> that it is because my sequence length is greater than 1024. Indeed my sequence is of length. 1300.</p>
<p>The same thread on GitHub mentions with a link that the issue has been fixed. <a href="https://github.com/pytorch/pytorch/pull/83639" class="inline-onebox" rel="noopener nofollow ugc">Fix issue in softmax.cu with transformer error when mask seqlen &gt; 1024 by erichan1 · Pull Request #83639 · pytorch/pytorch · GitHub</a></p>
<p>But I don’t see the solution there. What should I do? What version of PyTorch do I need? How do I fix the same issue? I am using PyTorch and CUDA versions 1.12.0 and 11.6 respectively.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home_nfs/haziq/Keystate-Forecasting/train.py", line 234, in &lt;module&gt;
    va_output = loop(net=net,inp_data=va_data,optimizer=None,counter=va_counter,epoch=i,args=args,mode="val")
  File "/home_nfs/haziq/Keystate-Forecasting/train.py", line 155, in loop
    out_data = net(inp_data, mode=mode)
  File "/home_nfs/haziq/cenvs/ptg/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home_nfs/haziq/Keystate-Forecasting/models/action2pose/stf.py", line 215, in forward
    transformer_encoder_out = self.transformer_encoder(key, src_key_padding_mask=mask) # [batch, 2 + pose_padded_length* obj_wrist_padded_length, transformer_encoder_units]
  File "/home_nfs/haziq/cenvs/ptg/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home_nfs/haziq/cenvs/ptg/lib/python3.8/site-packages/torch/nn/modules/transformer.py", line 238, in forward
    output = mod(output, src_mask=mask, src_key_padding_mask=src_key_padding_mask)
  File "/home_nfs/haziq/cenvs/ptg/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home_nfs/haziq/cenvs/ptg/lib/python3.8/site-packages/torch/nn/modules/transformer.py", line 437, in forward
    return torch._transformer_encoder_layer_fwd(
RuntimeError: Mask shape should match input shape; transformer_mask is not supported in the fallback case.
</code></pre>

username: Kong
cooked: <p>Sorry I just created a new environment with the latest version of PyTorch 1.13.1 and I no longer get the error.</p>
How to perform the MaxSim operator leveraging torch procedures?

username: Aldebaran
cooked: <p>Let <code>T</code> and <code>L</code> be two batches of matrices <code>(MxN)</code> and a function <code>f(ti,lj)</code> that calculates a score for matrices <code>ti</code> and <code>lj</code>.  For instance, if</p>
<pre><code class="lang-python">T, L= torch.rand(4,3,2), torch.rand(4,3,2)
# T = tensor([[[0.0017, 0.5781],
#          [0.8136, 0.5971],
#          [0.7697, 0.0795]],

#         [[0.2794, 0.7285],
#          [0.1528, 0.8503],
#          [0.9714, 0.1060]],

#         [[0.6907, 0.8831],
#          [0.4691, 0.4254],
#          [0.2539, 0.7538]],

#         [[0.3717, 0.2229],
#          [0.6134, 0.4810],
#          [0.7595, 0.9449]]])
</code></pre>
<p>and the score function is defined as shown in the following code snippet:</p>
<pre><code class="lang-python">def score(ti, lj):
    """MaxSim score of matrix ti and lj
    """
    m = torch.matmul(ti, torch.transpose(lj, 0, 1))
    return torch.sum(torch.max(m, 1).values, dim=-1)
</code></pre>
<p>How to return a score matrix <code>S</code>, where <code>S[i,j]</code> represents the score between <code>T[i]</code> and <code>L[j]</code>?</p>
<pre><code class="lang-python">#S = tensor([[2.3405, 2.2594, 2.0989, 1.6450],
#            [2.5939, 2.4186, 2.3946, 2.0648],
#            [2.9447, 2.3652, 2.3829, 2.1536],
#            [2.8195, 2.3105, 2.2563, 1.8388]])
</code></pre>
<p><strong>NOTE</strong>: This operation must be differentiable.</p>

username: dreidizzle
cooked: <aside class="quote no-group" data-username="Aldebaran" data-post="1" data-topic="169363">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aldebaran/48/23963_2.png" class="avatar"> Aldebaran:</div>
<blockquote>
<p><code>m = torch.matmul(ti, torch.transpose(lj, 0, 1))</code></p>
</blockquote>
</aside>
<p>I think you want to do batch matrix multiplication first of all no? This would be more efficient. See the code below. The max is usually approximated in a differentiable way by the softmax, so maybe you can do the softmax below. What is the objective etc?</p>
<pre><code class="lang-auto"># Get something of dimension B X M X M where T ~ M X N and L ~ M X N and B is the batch size.
m = torch.bmm(T, torch.transpose(L, 1, 2))

# For each matrix, the row sums are 1. 
nn.Softmax(dim=2)(m).shape

# Then, you'd want to do a cross entropy loss across the m classes, where the true label is the (actual) max? 
</code></pre>
<p>BTW - ReLU technically has no derivative but we can still use it bc it has a SUB-gradient.</p>
<p>The max of two things can be expressed as</p>
<p>f(x, y) = (x + y  + |x - y|) / 2 and I’m sure you can express the max of K things in the same way. Basically, using RELUs and absolute values. These things can be used, since they have subgradients.</p>
Updating link in PyTorch ecosystem page

username: sarthakpati
cooked: <p>Hi,</p>
<p>I wanted to know whom to get in touch with regarding updating the link for GaNDLF in the PyTorch Ecosystem page. When it was inculcated in the ecosystem, it was part of one of the University of Pennsylvania’s research centers. However, it has since moved under the MLCommons umbrella and has a new home: <a href="https://github.com/mlcommons/GaNDLF" class="inline-onebox" rel="noopener nofollow ugc">GitHub - mlcommons/GaNDLF: A generalizable application framework for segmentation, regression, and classification using PyTorch</a></p>
<p>Would it be possible for you to update the link on the PyTorch ecosystem website to either this or <a href="https://mlcommons.github.io/GaNDLF" class="inline-onebox" rel="noopener nofollow ugc">GaNDLF by mlcommons</a>?</p>
<p>Cheers,<br>
Sarthak</p>

username: ptrblck
cooked: <p>CC <a class="mention" href="/u/suraj.pt">@suraj.pt</a> might be able to forward this request.</p>

username: suraj.pt
cooked: <p><a class="mention" href="/u/sarthakpati">@sarthakpati</a> made the change, it should be updated in a a few hours</p>
Backward operation does not scale as expected

username: adaien
cooked: <p>Hello everyone,</p>
<p>I am trying to understand why the computational time of the backward pass of a given operation do not scale as expected.</p>
<p>Consider the following code</p>
<p>J = 1000<br>
S = 10<br>
L = 10<br>
M = 10<br>
J, S, L, M = 1000, 10, 1, 10<br>
N = M * J<br>
p = torch.nn.Parameter(torch.zeros(L, S, N), requires_grad=True)<br>
p_prod = torch.stack([torch.prod((p[:, :, range(i * M, (i + 1) * M - 1)]), 1) for i in range(J)])<br>
out = p_prod.sum()</p>
<p>start_time = time.time()<br>
out.backward(inputs = [p])<br>
print(“— %s seconds —” % (time.time() - start_time))</p>
<p>which essentially compute the products of p across several indexes and separately across the first and second dimension.</p>
<p>For L = 1, the operations takes 0.19535231 seconds, but for L = 10, the operation takes 5.62876 seconds, which is more than x10 what I was expecting it to take.</p>
<p>Is there a reason why the operation does not scale as expected? Moreover, is there a way to speed up the computation of the gradient when I perform operations across several dimensions (either by computing the product in a different way or by using parallelisation)?</p>
<p>Thanks in advance</p>
Pytorch Guru needed for Visual Effects Company

username: VFX
cooked: <p>We’re looking for a freelance Pytorch Guru to help implement existing models into our Nuke software and also customise and create new models with applications image processing for use in Visual Effects for film and television. More details on application. Please email <a href="mailto:info@midasvfx.com">info@midasvfx.com</a></p>

username: cerebruminfotech
cooked: <p>Hey there,<br>
This <a href="https://www.cerebruminfotech.com" rel="noopener nofollow ugc">software company</a> offers many services please try to contact for your project<br>
Thankyou<br>
Jasica James</p>

username: Lauren1
cooked: <p>Hello,<br>
Hope all is well!</p>
<p>I would like to assist you.</p>
<p>Kindly reach me via skype/email.<br>
Skype-live:lauren_8606<br>
Email-lauren(at)cisinlabs(dot).com</p>
<p>Thanks,<br>
Lauren W.</p>

username: cerebruminfotech
cooked: <p>hi there,<br>
PyTorch needs external programs for visualization. For production, it also needs an API server.<br>
If you want to know more about this Then visit our <a href="https://www.cerebruminfotech.com/" rel="noopener nofollow ugc">software company</a>.</p>
Mypy classifies torchvision imports as Any by default

username: ash95
cooked: <p>I noticed that mypy seems to classify all torchvision imports as <code>Any</code> whereas this isn’t the case for <code>torch</code>.</p>
<p>For example, running the follow script</p>
<pre><code class="lang-auto">from torch import Tensor
from torchvision.models._api import Weights
from torchvision.transforms import Compose

reveal_type(Tensor)
reveal_type(Weights)
reveal_type(Compose)
</code></pre>
<p>yields</p>
<pre><code class="lang-auto">test.py:5: note: Revealed type is "Overload(def (*args: Any, *, device: Union[torch._C.device, builtins.str, builtins.int, None] =) -&gt; torch._tensor.Tensor, def (storage: torch.types.Storage) -&gt; torch._tensor.Tensor, def (other: torch._tensor.Tensor) -&gt; torch._tensor.Tensor, def (size: Union[torch._C.Size, builtins.list[builtins.int], builtins.tuple[builtins.int, ...]], *, device: Union[torch._C.device, builtins.str, builtins.int, None] =) -&gt; torch._tensor.Tensor)"
test.py:6: note: Revealed type is "Any"
test.py:7: note: Revealed type is "Any"
Success: no issues found in 1 source file
</code></pre>
<p>Could someone point out why this occurs?</p>
Gradient problem - variable modified, cannot find where

username: MartinPerry1
cooked: <p>I have received this error:</p>
<blockquote>
<p>one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [4754, 300]] is at version 4; expected version 3 instead.</p>
</blockquote>
<p>when I call <code> loss.backward()</code> on the output from <code>customNegativeLoss</code>.</p>
<p>My code:</p>
<pre><code class="lang-auto">def customNegativeLoss(self, embedInRes, targets):
        batchSize = targets.shape[0]
        wordsCount = targets.shape[1]

        embedInRes = embedInRes.view(batchSize * wordsCount, -1)
        targets = targets.view(batchSize * wordsCount)

        emb_context = self.embedOut(targets)
       
        emb_product = torch.mul(embedInRes, emb_context)
        emb_product = torch.sum(emb_product, dim=1) 
       
        pos_loss = F.logsigmoid(emb_product) 

        if (self.negativeSamples == -1):      
            #this works          
            return -(pos_loss).mean()
                
        noise_dist = torch.ones(self.vocabSize).to(targets.device)  
       
        num_neg_samples_for_this_batch = (batchSize * wordsCount) * self.negativeSamples
        negative_example = torch.multinomial(noise_dist, num_neg_samples_for_this_batch, replacement = True) 
            
        negative_example = negative_example.view(batchSize * wordsCount, self.negativeSamples)
            
        emb_negative = self.embedOut(negative_example)

        emb_product_neg_samples = torch.bmm(emb_negative.neg(), embedInRes.unsqueeze(2)) 
            
        noise_loss = F.logsigmoid(emb_product_neg_samples).squeeze(2)
        noise_loss = noise_loss.sum(1)
          
        total_loss = -(pos_loss + noise_loss).mean()
            
        #this crashes
        return total_loss
</code></pre>
<p>Embedings are inited as:</p>
<pre><code> self.embedIn = nn.Embedding(
        num_embeddings=self.vocabSize,
        embedding_dim=300,
        padding_idx=paddingIdx, #index of padding
        max_norm=1,
        sparse=True
    )

    self.embedOut = nn.Embedding(
        num_embeddings=self.vocabSize,
        embedding_dim=300,
        padding_idx=paddingIdx, #index of padding
        max_norm=1,
        sparse=True
    )
</code></pre>
<p>if <code>self.negativeSamples</code> is <code>-1</code>, the backward pass is working. So the error must be in the “else” branch, but I am unsure where. The code is for Word2Vec training with Skip-Grams (based on <a href="https://github.com/n0obcoder/" class="inline-onebox" rel="noopener nofollow ugc">n0obcoder · GitHub</a>).</p>

username: ptrblck
cooked: <p>Could you post a minima, executable code snippet to reproduce the issue, please?</p>

username: MartinPerry1
cooked: <p>The problem was in Embedding.</p>
<pre><code class="lang-auto">sparse=True
</code></pre>
<p>and in used optimizer <code>torch.optim.SparseAdam</code>.</p>
<p>If I set sparse to <code>False</code> and use classic <code>AdamW</code>, the problem is gone</p>
Initialise all model weights the same way each time

username: avyavkumar
cooked: <p>Hi,</p>
<p>I fine-tune a transformer + linear layer on different few-shot data and I then evaluate the model on a test set. However, it looks like the model weights are being initialised differently each time. While ideally they should converge as training converges, I was wondering if it is possible to initialise all linear weights with the same values each time.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>You could seed to code via <code>torch.manual_seed</code> before creating a new model instance. This would however initialize the global pseudorandom number generator (PRNG) and thus also define the random sequence of every other call into it.<br>
Alternatively, you could initialize the model once, save its <code>state_dict</code>, and reload it afterwards for each new model.</p>
How to PyTorch Install on FreeBsd 12.1?

username: Tristan_Yu
cooked: <p>PyTorch have support FreeBSD?<br>
If support how to install ?</p>

username: ptrblck
cooked: <p>Based on <a href="https://discuss.pytorch.org/t/the-installation-of-stable-diffusion-web-ui-natively-on-freebsd-fails-because-cant-install-torch-and-torchvision/166395/9">this post</a> it doesn’t seem to be supported and I don’t know if <a class="mention" href="/u/mario_marietto">@Mario_Marietto</a> was interested to start working on it.</p>

username: Mario_Marietto
cooked: <p>Hello. I don’t know what’s the project of the OP,but I’m making some progress by installing the AUTOMATIC1111 web-ui for stable diffusion,not installing pytorch natively,but using the linuxulator.</p>
What is the most efficient way to shuffle each row of a tensor with different shuffling order for each of the row?

username: yxchng
cooked: <p>Suppose I have a tensor of size (3,5). I need to shuffle each of the three 5 elements row independently. All the solutions that I found shuffle all the rows with the same shuffling order (eg. using torch.randperm). Is there a way to shuffle each row with independant random shuffling order, without using a for loop to shuffle each of the row one by one?</p>

username: Eta_C
cooked: <p>Try</p>
<pre><code class="lang-python">import torch 

x = torch.rand(3, 5)
indices = torch.argsort(torch.rand(*x.shape), dim=-1)
result = x[torch.arange(x.shape[0]).unsqueeze(-1), indices]
</code></pre>

username: francois-rozet
cooked: <p>Instead of indexing by hand, one can use the <a href="https://pytorch.org/docs/stable/generated/torch.gather.html" rel="noopener nofollow ugc"><code>torch.gather</code></a> function.</p>
<pre><code class="lang-python">import torch 

x = torch.rand(3, 5)
indices = torch.argsort(torch.rand_like(x), dim=-1)
result = torch.gather(x, dim=-1, index=indices)
</code></pre>

username: shwinshaker
cooked: <p>A slightly more complicated request, in case anyone is interested. I have a multi-D tensor and I am looking to shuffle the 1st dimension with different orders for the 0th dimension. Here is a solution based on the above replies.</p>
<pre><code class="lang-auto">import torch 

x = torch.rand(3, 5, 4)
indices = torch.argsort(torch.rand(*x.shape[:2]), dim=1)
result = torch.gather(x, dim=1, index=indices.unsqueeze(-1).repeat(1, 1, x.shape[-1]))
</code></pre>
Module 'torch' has no attribute 'Module'

username: Mukesh1729
cooked: <p>I get this error after installing the latest version of torch .module ‘torch’ has no attribute ‘Module’. i did re-install it. Not sure how to go about it</p>

username: ptrblck
cooked: <p><code>Module</code> is defined in the <code>torch.nn</code> namespace, so <code>torch.nn.Module</code> should work.</p>

username: tifa
cooked: <p>hahaha,may be you use<br>
<code>import torch as nn</code><br>
the correct is<br>
<code>from torch import nn</code></p>

username: play201861
cooked: <p>import torch.nn as nn</p>
Does non_blocking=True use my stream or a new one?

username: rinkujadhav2013
cooked: <p>I’m using stream context manager (<code>with torch.cuda.stream(s)</code>) to write my custom autograd function. If I do some data transfer from host to device using <code>to(device=d, non_blocking=True)</code> or the <code>copy_</code> function (again with <code>non_blocking=True</code>), will the data be transferred using my stream <code>s</code> or will PyTorch create yet another stream to transfer the data? So far, my results have been correct so I’m assuming that it’s the former.</p>

username: ptrblck
cooked: <p>The kernel will use the current stream, so your custom stream assuming you are using the <code>to()</code> or <code>copy_</code> op in the context manager based on <a href="https://github.com/pytorch/pytorch/blob/0d2c2110f178da19aaf89259a2034c9c0653fcee/aten/src/ATen/native/cuda/Copy.cu#L249-L270">these lines of code</a>.</p>
Calculating cosine similarity loss from two tensors stored in register buffer

username: pauleezz
cooked: <p>Hello, I’m trying to train a neural network related to the differentiable neural computer (DNC). The loss is determined by the cosine similarity loss of two slots from memory stored in the register buffer, which means it cannot be gradient descent. In the training process, the model will output the memory, and the loss is updated by</p>
<pre><code class="lang-auto">loss = cos_loss(memory[j].detach(), memory[k].detach())
loss.backward()
</code></pre>
<p>But I got the following error:</p>
<pre><code class="lang-auto">RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.
</code></pre>
<p>If I use the <code>with torch.no_grad():</code> to the cos_loss, will it still update the memory parameter?<br>
I have no idea how to fix this error… Anyone has any idea? Thanks</p>
Cross no longer supports broadcasting

username: Florian_Bruckner
cooked: <p>Why does torch.linalg.cross no longer support broadcasting like e.g.<br>
I am using expressions like:</p>
<pre><code class="lang-auto">x = torch.linalg.cross(m, p)
</code></pre>
<p>where <code>m</code> is a vector field with shape (nx, ny, nz, 3) and <code>p</code> is a material parameter.<br>
<code>p</code> may either be of the same shape as <code>m</code> or of shape (3) if the material is constant.</p>
<p>Both versions worked with a previous version of pytorch, but this behaviour was changed in ‘Strenghten preconditions of linalg.cross (<span class="hashtag">#83798</span>)’. Now the second version raises the following exception:</p>
<pre><code class="lang-auto">RuntimeError: linalg.cross: inputs must have the same number of dimensions.
</code></pre>
<p>I would now need to check the dimension of <code>p</code> first and reshape / expand accordingly.<br>
Is there a more elegant way?</p>
<p>best wishes<br>
Florian</p>

username: ptrblck
cooked: <p>I think expanding is the right approach and something like this should work:</p>
<pre><code class="lang-python">m = torch.randn(6, 5, 4, 3)
p = torch.randn(3)

x = torch.linalg.cross(m, p.expand_as(m))
</code></pre>

username: Florian_Bruckner
cooked: <p>Ah, cool. Thanks for the fast solution.</p>
<p>best wishes<br>
Florian</p>
Torch 2.0 a bit slower without compile?

username: Kendos93
cooked: <p>Hi there!</p>
<p>I just wanted to ask, is it normal, that torch 2.0 is slightly slower than torch 1.11 if I don’t use <code>torch.compile</code>? I am getting an average of 45sec/epoch vs 40sec/epoch. I am using Resnet18 from timm, doing some metric learning training.</p>
<p>I am using K80 and CUDA 11.1, but I also tried CUDA 11.7.</p>
<p><code>torch.compile</code> won’t work on K80 unfortunately ;(</p>
<p>Thanks!</p>
Training 10 models at once with torch distributed

username: Cuong_Nguyen
cooked: <p>Hi everyone,<br>
I have 10 ConvNet models (e.g. ResNet, DenseNet, …). I was successfully trained these models one by one with torch distributed, but I want to write a script that train these model at once with for loop.</p>
<p>def main(args):<br>
…<br>
return a</p>
<p>for model in models:<br>
args.model = model<br>
a = main(args)<br>
But if I end the main function with return the process is hanged so I replace it by sys.exit(0) that cause the loop only execute one time.<br>
Any recommend for this situation.</p>