akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Fastai problem/Input tensor issue

username: Ten_Canmotcai
cooked: <p>I created a model using fastai and save into torchscript.<br>
When I run torchscript model it wrong input<br>
My model: resnet50<br>
then error:<br>
Expected 4-dimensional input for 4-dimensional weight 64 3 7 7, but got 3-dimensional input of size [39, 28, 3] instead</p>
<p>code:<br>
import torch<br>
example = torch.rand(1, 3, 64, 64)<br>
traced_script_module = torch.jit.trace(torch_model, example)<br>
traced_script_module.save(“model.pt”)<br>
img = cv2.imread(‘image.jpg’)<br>
traced_script_module.forward(torch.from_numpy(img))</p>
<p>Have anybody help me?</p>

username: Michael_Suo
cooked: <p>Is the result of <code>torch.from_numpy(img)</code> the same dimensions as your example inputs? When using tracing, you must make sure that the example inputs given are representative.</p>
Convert a list of scipy sparse matrices to PyTorch 3D sparse tensor

username: as1949078
cooked: <p>I have a list of scipy lil_matrix and want to convert it to a 3D PyTorch sparse tensor.</p>
<p>I read the document <a href="https://pytorch.org/docs/stable/sparse.html" rel="noopener nofollow ugc">TORCH.SPARSE</a> but I did not find too much information about how to construct a 3D sparse tensor. The document mentions</p>
<blockquote>
<p>You can also construct hybrid sparse tensors, where only the first n dimensions are sparse, and the rest of the dimensions are dense.</p>
</blockquote>
<p>In my case, I want to construct the sparse tensor with the shape like <code>batch_size * M * N</code>.</p>
Turning loop into matrix operation

username: blade
cooked: <p>I have an operation, that can be reproduced using the following code</p>
<pre><code class="lang-auto">reg_sig = torch.randn([32, 9, 5])
reg_adj = torch.randn([32, 9, 9, 4])

n_node = 9
n_bond_features = 4

SM_f = nn.Softmax(dim=2)
SM_W = nn.Softmax(dim=3)

p_f = SM_f(reg_sig)
p_W = SM_W(reg_adj)

V = torch.zeros([reg_sig.size(0), n_node])

for i in range(n_node):
    for j in range(n_node):
        if i is not j:
            for k in range(n_bond_features):
                V[:, i] += k * p_W[:, i, j, k]
</code></pre>
<p>I would like to get rid of the loop at the end. Here is my implementation</p>
<pre><code class="lang-auto">h_vec = torch.arange(n_bond_features).float()
inner_sum = torch.einsum('i,bjki-&gt;bjk', h_vec, p_W)
V = inner_sum-torch.diag_embed(torch.einsum('...ii-&gt;...i', inner_sum))
V = V.sum(2)
</code></pre>
<p>I was wondering if this is the most efficient way to do it.</p>
nn.CrossEntropyLoss and batching

username: maria
cooked: <p>I’m trying to use cross entropy loss on sequential batched data.<br>
I have a many to many language model.<br>
My nets output size is torch.Size([20, 54, 9999]), and the labels are torch.Size([20, 54]).<br>
Batch size is 20, Sentence Length is 54 and vocab 9999.<br>
When trying to pass this to cross entropy it passes the net output as (20, 9999) and therefore expects the target also to be of this size.<br>
How can I use cross entropy loss in this case?</p>

username: Asad_Ullah
cooked: <aside class="quote no-group" data-username="maria" data-post="1" data-topic="65532">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria/48/14253_2.png" class="avatar"> maria:</div>
<blockquote>
<p>I have a many to many language model.</p>
</blockquote>
</aside>
<p>I think one option will be to add a linear final layer (nn.Linear(9999, 54)) to get prediction ([20,54]) similar to output size.</p>

username: maria
cooked: <p>The 54 is just given as an example, I have varying sequence lengths.<br>
Using transpose fixes that, however I think it’s wrong. and now I’m getting this error:<br>
<code>RuntimeError: CUDA error: device-side assert triggered</code><br>
Any suggestions?</p>
<p>Logically, I tried to max out the vocab dimension to get the actual prediction, but it seems that cross entropy loss should ‘take care’ of that (+ it gave another error about not supporting long).</p>
Examples utilizing tensor cores

username: mahmoodn
cooked: <p>Hi<br>
I ran some tutorials such as <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noopener">seq2seq</a>, <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" rel="nofollow noopener">neural_style</a>, <a href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html" rel="nofollow noopener">spatial_transformer</a>, and others.</p>
<p>Using the default code, no FP16 has been observed and tensor unit is idle according to the profiler’s report.<br>
I would like to know what should I do in order to use them? All FP instructions are 32-bit.</p>

username: ptrblck
cooked: <p>To use TensorCores, you would have to apply some operations in FP16. You could transform (some) operations manually via <code>.half()</code> or I would recommend to use our mixed-precision approach in <a href="https://github.com/NVIDIA/apex">apex</a>.</p>
<p>We are currently working on a direct PyTorch integration.</p>
Weight argument equivalence of TF loss in PT CrossEntropyLoss

username: blade
cooked: <p>I’m trying to extend a binary classification problem to multi-class and implement it using PyTorch. The implementation of the binary problem is available in TensorFlow framework. The cost function is defined as</p>
<p><code>cost = norm * tf.reduce_mean(tf.nn.weighted_cross_entropy_with_logits(logits=preds_sub, targets=labels_sub, pos_weight=pos_weight))</code></p>
<p>To train the network in my PyTorch implementation, I use <code>CrossEntropyLoss</code> to compare the output scores of classes from the network with target labels. My question is if <code>pos_weight</code> argument of TensorFlow’s <code>weighted_cross_entropy_with_logits</code> is the same as <code>weight</code> argument in PyTorch’s <code>CrossEntropyLoss</code>. TF documentation states that</p>
<blockquote>
<p><code>pos_weight</code> allows one to trade off recall and precision by up- or down-weighting the cost of a positive error relative to a negative error.</p>
</blockquote>
<p>My understanding is that in the case of <code>CrossEntropyLoss</code>, I’m not differentiating between false classifications (e.g. a class C being classified as A has the same weight as being classified as B) and for <code>weight</code> argument of <code>CrossEntropyLoss</code> to do the same thing as <code>weighted_cross_entropy_with_logits</code>, I think that I need some sort of modification to the weights, like a weight matrix of size (class number) X (class number) but I’m not sure. I would appreciate any comments that might clarify this.</p>
Weights of `CrossEntropyLoss` and biasing toward the rare event

username: blade
cooked: <p>I have a VAE that takes molecular graphs as input and is trained to generate molecular graphs in the output. The output is a vector of scores of different classes for each node and each edge. Different classes for edges include different bond types [single, double, triple, or no edge] and classes for nodes include different atom types. To train this model, I compare these scores with target labels using <code>torch.nn.CrossEntropyLoss</code>. Since the number of single bonds in the training dataset is way higher than the number of double and triple bonds, I am trying to use weights for the loss function. The way I impose this weights is using a normalized vector of w_class = (1/total number of occurrence for class). My expectation is that when occurrence of a class in dataset is low, the algorithm is penalize more for misprediction and hence, my model is better trained.</p>
<p>In reality, instead of better training, this biasses the output toward the rare class, i.e. when I generate output from latent space samples, the occurrence of rare classes are significantly increased. For instance, if my dataset had molecular graphs with the following occurrence of edge class [single=100, double=10, triple=5, or no edge=200], when generating samples from z space, I get molecular graphs that have mostly edges of class ‘double’.</p>