akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Different tensor indexing results on different machines

username: fermat97
cooked: <p>Hi, I have two different machines both Ubuntu 18.04 with torch <code>1.9.0</code> and <code>1.9.0+cu102</code>. When I try the followings:</p>
<pre><code class="lang-auto">V = torch.tensor([[1,2,3],[4,5,6]])
idx = torch.tensor([0])
</code></pre>
<p>Now if I try:</p>
<pre><code class="lang-auto">print(V[idx]) 
</code></pre>
<p>I ll get different outputs:</p>
<pre><code class="lang-auto">tensor([[1, 2, 3]]) # for torch 1.9.0
tensor([1, 2, 3]) # for torch 1.9.0+cu102
</code></pre>
<p>Do you have any idea?</p>

username: ptrblck
cooked: <p>Assuming the difference between both installations would be a CPU-only vs. CUDA10.2 install, I cannot reproduce the issue and both pip wheels give <code>tensor([[1, 2, 3]])</code> as the output using your code snippet.</p>

username: fermat97
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> this is very strange because the torch <code>1.9.0</code> is definitely using GPU.</p>

username: ptrblck
cooked: <p>Yes, that’s indeed strange, as I’ve installed the CPU and CUDA versions of <code>1.9.0</code> for this particular test and wasn’t able to reproduce it.</p>

username: fermat97
cooked: <p>The problem is my own code which hooks torch and somehow affects the <code>__getitem__</code> functionality of  torch.</p>
(scalar*vector).mean() != (vector.mean())*scalar

username: Gil_Gilad
cooked: <p>I have a loss value that becomes negative. when testing it i notices that  if i am doing  a scalar multiplication of the vector and then mean i am getting a different results then doing the mean and then multiply, i assume this is due to representation error.  since the final mean is small.</p>
<p>But how will it affect the loss ?  any ideas how to avoid the issue?</p>
Backward() takes 2 positional arguments but 4 were given

username: ChiTseHuang
cooked: <p>I modify a custom autograd function like this<br>
I thought this error is caused by multiple return values from function A<br>
but I need min scale to make other codes run properly.<br>
class A(torch.autograd.Function):<br>
<span class="mention">@staticmethod</span><br>
def forward(ctx, input, qparams=None):<br>
…<br>
return output, min, scale<br>
<span class="mention">@staticmethod</span><br>
def backward(ctx, grad_output):<br>
grad_input = grad_output<br>
return grad_input, None<br>
class B(nn.Module):<br>
def <strong>init</strong>(self):<br>
…<br>
def forward(self, input, num_bits, qparams=None):<br>
…<br>
output, min, scale = A().apply(input, qparams)<br>
return output</p>

username: ptrblck
cooked: <p>The backward would expect the same number of input arguments as were returned in the <code>forward</code> method, so you would have to add these arguments as described in the <a href="https://pytorch.org/docs/stable/notes/extending.html"><code>backward</code> section of this doc</a>.</p>
How can we use autograd for sequence optimization (in a for loop)?

username: 335621
cooked: <p>I want to optimize a sequence in a for loop using Pytorch Autograd.</p>
<pre><code class="lang-auto">for i in range(10):
     x = f(x,z[i])
</code></pre>
<p>Say the sequence length is 10. I want to optimize <code>x</code> as well as <code>z</code>(z is a tensor array). Note the <code>x</code> will be updated in the loop.</p>
<p>To run this, I have to open:</p>
<pre><code class="lang-auto">loss.backward(retain_graph=True)
</code></pre>
<p>Is there a better way to do so (To make it run faster)?</p>

username: ptrblck
cooked: <p>I’m not sure why you need to use <code>detect_anomaly</code>, as it’s a debugging tool and is expected to slow down the code. Could you explain the reason a bit more?</p>

username: 335621
cooked: <p>Sorry, my mistake.<br>
I am using</p>
<pre><code class="lang-auto">loss.backward(retain_graph=True)

</code></pre>

username: ptrblck
cooked: <p>Using <code>retain_graph</code> would only be necessary if you need to keep the intermediate tensors alive after a backward operation. I’m still unsure why it’s needed in your case, so would need more information about the actual use case.</p>
Does 'autoencoder' work?

username: Research_ZZ
cooked: <p>Hi Guys, recently i saw a post on the forum related LSTM auto encoder, and want to play with it to have some fun.  I tried a few different architectures ( lstm, covn1d, conv2d ) to model timeseries and image restoration. I found two common problems discussed on this forum, also other place such as stack overflow, but never seem to be sorted out.</p>
<ol>
<li>
<p>restored data is simply an average of the original input - a problem with my lstm and conv1d based AE;</p>
</li>
<li>
<p>loss not decreasing - however,  for a covn2d image autoencoder, it does reproduce an image similar to original.</p>
</li>
</ol>
<p>just wondering if your guys noticed that same, and what is your solution  ?</p>
<p>thanks<br>
Feng</p>

username: Research_ZZ
cooked: <p>two sample problems :</p>
<ol>
<li>
<p><a href="https://stackoverflow.com/questions/54411662/lstm-autoencoder-always-returns-the-average-of-the-input-sequence/54480128" class="inline-onebox" rel="noopener nofollow ugc">python - LSTM autoencoder always returns the average of the input sequence - Stack Overflow</a></p>
</li>
<li>
<p>this auto encoder loss is not decreasing, but can restore images.<br>
<a href="https://www.kaggle.com/ljlbarbosa/convolution-autoencoder-pytorch#" class="inline-onebox" rel="noopener nofollow ugc">Convolution Autoencoder - Pytorch | Kaggle</a></p>
</li>
</ol>

username: Research_ZZ
cooked: <p>finally tried auto encoder with FNN and CNN,  use MSELoss and adjusted weight decay, and learning rate, seems to work. Loss decreasing, and can restore image reasonable well.</p>
How to Properly Evaluate a Mask-RCNN Model

username: LucasGCardoso
cooked: <p>Hello. I am trying to train and evaluate a Mask-RCNN model based on the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">Pytorch Torchvision Object Detection Finetuning Tutorial</a>. I’ve modified the code to fit in my own dataset. So, everything runs great, but I wanted to evaluate the model during the training process, so for each epoch, I could save the model if it has better scores. The problem is that the scores in the output don’t change at all, and sometimes the model even returns empty scores.<br>
Here is the core of the training code and the part that i added to evaluate the model.</p>
<pre><code class="lang-auto">    bestScore = 0
    for epoch in range(int(num_epochs)):
        model.train()
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, writer, print_freq=10)
        # update the learning rate
        lr_scheduler.step()

        #####################################
        #evaluate and save the model 
        model.eval()
        soma = 0
        qtd = 0      
        for image in data_loader_test:
            #Here we create a list, because the model expects a list of Tensors
            lista = []
            #It is important to send the image to CUDA, otherwise it will try to execute in the CPU
            x = image[0][0].cuda()
            lista.append(x)
            output = model(lista)

            for item in output[0]['scores']:
                soma += item.data.item()
                qtd += 1
        averageScores = soma/qtd
        print("Epoch ", epoch, " score: ", averageScores)
       
        if((averageScores) &gt; bestScore):
            print(averageScores, "is bigger than ", bestScore)
            bestScore = averageScores
            print("#### Saving the model ####")
            path = "model.pth"
            torch.save(model.state_dict(), path)
        # #####################################
</code></pre>
<p>And this is a sample output of the model in “eval mode” that I often get:</p>
<pre><code class="lang-auto">[{'boxes': tensor([], device='cuda:0', size=(0, 4), grad_fn=&lt;StackBackward&gt;), 'labels': tensor([], device='cuda:0', dtype=torch.int64), 'scores': tensor([], device='cuda:0', grad_fn=&lt;IndexBackward&gt;), 'masks': tensor([], device='cuda:0', size=(0, 1, 480, 720))}]
</code></pre>
<p>I don’t know if this average of the scores is correct, and if this is a valid and proper way to evaluate this mask-RCNN model during training. Could you guys help me out with this one? Thanks in advance!</p>

username: MikeDoho
cooked: <p>I am dealing with this same problem. Did you ever find out what was wrong?</p>
Parameters for LSTM with CNN in a sequential data

username: Jyoti_Rawat
cooked: <p>I am doing a classification problem with ECG data. I built a LSTM model but the accuracy of the model is not quiet good. Hence, I am thinking to implement it with CNN. I am planning to pass the data from CNN, then passing the output from CNN to LSTM. Howver, I have noticed that CNN is mostly used in Image classifications. I have sequential data with 4000 time steps. Could you please help me to define the parameters of the CNN model.</p>
<pre><code class="lang-auto">Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', device=None, dtype=None)
</code></pre>
<p>Can someone explain me what would be the in_channels, out_channels, kernel_size and stride for a sequence data having 4000 time steps?</p>
How to mask a tensor in c++?

username: baris
cooked: <p>Let’s say I have the tensor:</p>
<p>[22.2, 3.3, 3.2, 10.2, 3.2]</p>
<p>let’s also say that i have another tensor:</p>
<p>[0, 3, 1, 1, 3]</p>
<p>I’d like to turn the first tensor to:</p>
<p>[22.2, 0.0, 3.2, 10.2, 0.0]</p>
<p>That is where the second tensor’s value is 3, i want the first tensor’s value to be reset to 0.</p>
<p>If someone can write the 1-2 lines of c++ that corresponds to this it’d help me a lot.</p>
<p>Thank for the help.</p>

username: baris
cooked: <p>I found the answer. It’s something like this:</p>
<pre><code class="lang-auto">torch::where(t2 != 3, t1, torch::zeros_like(t1));
</code></pre>
Correct data augmentation with k-fold cross validation

username: sguerin
cooked: <p>Hi!<br>
I’m performing 10 k-fold cross validation on my neural network model. I’ve done the splits (k=10: 90% train, 10% validation), but It’s not clear for me if I should apply transforms (random horizontal flip, random rotation, etc) only on training set or both training and validation. What is the correct way to do it?</p>
<p>Thanks!</p>
Is this available in pytorch?

username: sss9616948
cooked: <p>I was working on the SIR pandemic model with neural network.The input of the model is the daily increasing cases and outputs the parameters alpha and beta(size:N-&gt;2N)<br>
so I take the outputs into the SIR model and compute the prediction of the daily increasing cases.<br>
And finally,the loss fuction is MSELoss(input,prediction of the daily increasing cases)<br>
But it turns out to not train at all,here’s my code,how can I fix it?Thanks a lot.<br>
By the way,I was doing it on jupyter notebook.</p>
<p>those are my code and reading file:<br>
<a href="https://drive.google.com/file/d/1SLMlW2xLiZvNv8p3YUue6B4JGW4XiiGh/view?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/file/d/1SLMlW2xLiZvNv8p3YUue6B4JGW4XiiGh/view?usp=sharing</a><br>
<a href="https://drive.google.com/file/d/17-1zr8Im1i4DwYvNaqhqtuGOrD7RZFkD/view?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/file/d/17-1zr8Im1i4DwYvNaqhqtuGOrD7RZFkD/view?usp=sharing</a></p>
Compute class weight

username: nimning
cooked: <p>To handle unbalanced data, I would like to weight each class according to their data distribution.</p>
<p>It is very straightforward in Tensofrflow as the foloowing</p>
<pre><code class="lang-auto">from sklearn.utils.class_weight import compute_class_weight
generator_train = datagenerator_train.flow_from_directory(directory=train_dir,
                                                          target_size=input_shape,
                                                          batch_size=batch_size,
                                                          shuffle=True)
cls_train = generator_train.classes
class_weight = compute_class_weight(class_weight='balanced',
                                    classes=np.unique(cls_train),
                                    y=cls_train)
</code></pre>
<p>Does pytorch have similar native support? It seems I have to write some function to compute the weights</p>

username: ptrblck
cooked: <p>If you don’t know the targets beforehand, you would need to iterate all samples once and count all class occurrences. Once this is calculated, you could use the <code>sklearn.utils.class_weight.compute_class_weight</code> or just these two lines of code:</p>
<pre><code class="lang-python">class_sample_count = np.unique(target, return_counts=True)[1]
weight = 1. / class_sample_count
samples_weight = weight[target]
samples_weight = torch.from_numpy(samples_weight)
</code></pre>

username: Nishanth_Sasankan
cooked: <p>I was trying to understand how to account for class imbalance while performing semantic segmentation.</p>
<p>So when you say</p>
<p>"If you don’t know the targets beforehand, you would need to iterate all samples once and count all class occurrences. Once this is calculated, you could use the "</p>
<p>do you mean how many pixels in a label map database correspond to a certain class or how many times does a given unique label occur in a dataset?</p>
<p>Thanks<br>
Nishanth</p>

username: MA_CASANDRA_QUILANG
cooked: <p>Hi! I have the same question. Did you find any answer to this?</p>

username: doem97
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="28379">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>don’t know</p>
</blockquote>
</aside>
<p>ptrblck’s answer is clear and concise I think.</p>
Dlib + Pytorch, if you import dlib first, it does not work

username: Eren_Cerman
cooked: <p><span class="hashtag">#Did</span> ayn of you had this weird problem??<br>
############################################<br>
import dlib<br>
print(dlib.DLIB_USE_CUDA)<br>
cnn_face_detector = dlib.cnn_face_detection_model_v1(‘mmod_human_face_detector.dat’)<br>
import torch<br>
print(torch.rand(4))<br>
<span class="hashtag">#OSError:</span> [WinError 127] The specified procedure could not be found. Error loading #“C:\Python38\lib\site-packages\torch\lib\cudnn_adv_train64_8.dll” or one of its dependencies.</p>
<p>######################  while there is no problem if torch is imported first:)</p>
<p>import torch<br>
print(torch.rand(4))<br>
import dlib<br>
print(dlib.DLIB_USE_CUDA)<br>
cnn_face_detector = dlib.cnn_face_detection_model_v1(‘mmod_human_face_detector.dat’)</p>
<p><span class="hashtag">#this</span> works without any problems as:<br>
#&gt;tensor([0.1777, 0.0763, 0.1909, 0.5742])<br>
#&gt;True</p>
Insert a new Conv Block in a pretrained model

username: Mriganka_Nath
cooked: <p>I am working with 3D medical data but wanted to use the pre-trained models. So I thought of adding a Conv3D block between the layers of the pre-trained module. And while training the whole model will be trained, the pre-trained weights and the new weights in the 3D Conv blocks.<br>
If someone can guide me on how to do that, I would be really gladful.</p>
<p>I really don’t know if this will work, so if you have some other idea, I will be pleased to hear it, Thanks.</p>
DataListLoader out of memory in pytorch geometric

username: snehith_routhu
cooked: <p>I’m trying to train a graph cnn on a sample data of 8 meshes. I’m facing out of memory error in dataloader after few epochs. My dataset and dataloader are</p>
<p>class CustomDataset(Dataset):<br>
def <strong>init</strong>(self, root, transform=None, pre_transform=None):<br>
super(CustomDataset, self).<strong>init</strong>(root, transform, pre_transform)<br>
self.root=root<br>
self.df = pd.read_csv(root,header=None)</p>
<pre><code>def len(self):
    return len(self.df)

def get(self, idx):


    # Load Meshes and RBG Image
    x = np.load(self.df.iloc[idx,0])['v']

    #gt_pts = np.load(self.df.iloc[idx,1])['g']
    gt_pts = np.load('buff_gt/'+str(idx)+'/gt.npz')['g']

    edge_index = np.load(self.df.iloc[idx,2])['e']

    faces = np.load(self.df.iloc[idx,3])['f']

    # Generate data to be returned
    edge_index = torch.from_numpy(edge_index).type(torch.LongTensor)
    faces = torch.from_numpy(faces).type(torch.LongTensor)
    x = torch.from_numpy(x).type(torch.float)
    gt_pts = torch.from_numpy(gt_pts).type(torch.float)

    data = Data(x = x, edge_index = edge_index,gt = gt_pts, faces = faces)
    return data
</code></pre>
<p>dataset = CustomDataset(’/home/snehith/Pixel2Mesh/buff_train_data.csv’)<br>
train_data_loader = DataListLoader(dataset, batch_size = self.options.train.batch_size, num_workers=40, shuffle = True,pin_memory=False)</p>
<p>TIA</p>

username: benedekrozemberczki
cooked: <p>Did you manage to solve this?</p>
About saving state_dict/checkpoint in a function

username: banikr
cooked: <p>I am trying to implement the following function to save the model_state checkpoints:</p>
<pre><code class="lang-auto">def train_epoch(self):
    for epoch in tqdm.trange(self.epoch, self.max_epoch, desc='Train Epoch', ncols=100):
        self.epoch = epoch      # increments the epoch of Trainer
        checkpoint = {} # fixme: here checkpoint!!!
        # model_save_criteria = self.model_save_criteria
        self.train()
        if epoch % 1 == 0:
            self.validate(checkpoint) 
        checkpoint_latest = {
            'epoch': self.epoch,
            'arch': self.model.__class__.__name__,
            'model_state_dict': self.model.state_dict(),
            'optimizer_state_dict': self.optim.state_dict(),
            'model_save_criteria': self.model_save_criteria
        }
        checkpoint['checkpoint_latest'] = checkpoint_latest
        torch.save(checkpoint, self.model_pth)
</code></pre>
<p>Previously I did the same by just running a for loop:</p>
<pre><code class="lang-auto">train_states = {}
for epoch in range(max_epochs):
    running_loss = 0
    time_batch_start = time.time()
    model.train()
    for bIdx, sample in enumerate(train_loader):
    ...
    train...
    validation...
    train_states_latest = {
        'epoch': epoch + 1,
        'model_state_dict': model.state_dict(),
        'optimizer_state_dict': optimizer.state_dict(),
        'model_save_criteria': chosen_criteria}
    train_states['train_states_latest'] = train_states_latest
    torch.save(train_states, FILEPATH_MODEL_SAVE)
</code></pre>
<p>Are there ways to initiate the <code>checkpoint={}</code> and update it every loop? Or <code>checkpoint={}</code> in every epoch is fine since model itself is holding the <code>state_dict()</code>. Just I am overwriting the <code>checkpoint</code> each time.</p>
LSTM train max length vs length at prediction time

username: Mika_S
cooked: <p>Does LSTM train max length restriction carry over to restrictions for inference as well? That is can I expect an lstm model trained over sequences of max length 100 to give good results well for sequences of length 200?</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="Mika_S" data-post="1" data-topic="127430">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mika_s/48/14253_2.png" class="avatar"> Mika_S:</div>
<blockquote>
<p>That is can I expect an lstm model trained over sequences of max length 100 to give good results well for sequences of length 200?</p>
</blockquote>
</aside>
<p>It is possible, but not guaranteed. Simply speaking, if distributions of your training vectors (both x and y) are stationary over time, extrapolation into future should be good, otherwise - it is dubious, but may work, at least for some number of extra time steps. GRU may fare better in this regard, because if has no separate ‘forget’ and ‘update’ gates, that can cause state drift over time.</p>

username: Mika_S
cooked: <p>Do you think it’s ok to have the timestamp as part of the features for every timestep where difference between timestamps of two timesteps is not exactly the same.</p>

username: googlebot
cooked: <p>Irregular time series are not handled that well by RNNs in general, if disparity is big. Timestamps are not good, as they’re absolute; even if you shift to zero base - you’ll get position dependent forecasts, inferior beyond your training length. Time differences (like numpy.diff) should work better, another approach is adding “periodic” features {sin,cos}(2<em>pi</em>t/period).</p>

username: Mika_S
cooked: <p>Do you think it makes sense to just move my set up to transformers?</p>

username: googlebot
cooked: <p>It depends. Look at what approaches are used in your problem domain. Irregularity issue can be tackled to some extent with resampling, and transformers are more “lookup” than “forecast” models (i.e. with bigger internal memory), so their applicability scenarios are somewhat different.</p>
FasterRCNN models with single channel input (surprisingly?) work

username: jtjohnston
cooked: <p>I’ve been using the pre-trained FasterRCNNResnet50FPN model for object detection and noticed a behavior that was unexpected (at least to me).  If one inferences with a single channel image (e.g. grayscale) the normalization process subtracts a 3-channel mean (RGB) and the result is a 3-channel tensor–so inference <em>works</em> (because the ResNet50 backbone expects a 3-channel input).  I understand that the way Python broadcasts arrays (tensors) is what causes the shape to be altered in this way.  For reference, it is this <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/transform.py#L138" rel="noopener nofollow ugc">line</a> that causes the reshaping.</p>
<p>My question is: is this reshaping expected behavior?  I mean that in the sense of, if I input a grayscale image, would I expect the model to implicitly make it RGB (by duplicating channels and subtracting the RGB mean)?  or should I expect the model to just “crash” (and complain that the input shape is not correct)?  Or at least provide a warning–that an input shape was implicitly resized?</p>

username: ptrblck
cooked: <p>I personally am a big fan of loud and proper error messages, which would also be raised in case you are using the <code>torchvision.transforms.Normalize</code> transformation, since it’s using an inplace op as seen <a href="https://github.com/pytorch/vision/blob/bdc88f5245b276143f23da6a5df8f3f1fdbf9c7e/torchvision/transforms/functional.py#L335">here</a>.<br>
I’m unsure why the <code>detection.transform</code> implementation doesn’t use <code>torchvision.transforms.functional.normalize</code> to get the same behavior and avoid code duplication, but it might have been used exactly to add this type of “flexibility” so that users are able to pass grayscale images.</p>
<p>CC <a class="mention" href="/u/fmassa">@fmassa</a> is my speculation correct or should these transformations be rather “unified”?</p>
Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)

username: Mai_Magdy
cooked: <p>how can I hide this warring when I’m using number of workers &gt; 0</p>

username: ptrblck
cooked: <p>The issue is tracked <a href="https://github.com/pytorch/pytorch/issues/57273">here</a> and fixed <a href="https://github.com/pytorch/pytorch/commit/567e6d3a8766133f384eb1e00635b21ed638d187">here</a> so you could either ignore the warning or update PyTorch to a nightly release.</p>
Weird GPU behavior with LSTM in pytorch 1.9.0

username: cwitkowitz
cooked: <p>Hello,</p>
<p>I am attempting to train an LSTM on a specific GPU, but keep encountering weird behavior where the forward call adds data to both the device I intend (“cuda:1”), as well as the first available device (“cuda:0”). The following code snippet reproduces this behavior.</p>
<blockquote>
<p>import torch<br>
lstm = torch.nn.LSTM(input_size=100, hidden_size=200, batch_first=True, bidirectional=False).to(“cuda:1”)<br>
data = torch.randn(10, 6, 100).to(“cuda:1”)<br>
h0 = torch.randn(1, 10, 200).to(“cuda:1”)<br>
c0 = torch.randn(1, 10, 200).to(“cuda:1”)<br>
_ = lstm(data, (h0, c0))<br>
print(torch.__version__)<br>
<em>1.9.0+cu102</em></p>
</blockquote>
<p>As far as I know, I did not encounter this issue with previous versions of PyTorch, and I haven’t change my code which initializes and uses the LSTM.</p>
<p><strong>EDIT</strong> - I tried version 1.8.0 and can confirm the exact same code only places data on the intended GPU.</p>
Cuda error when using GPU

username: el_youssfi_azeddine
cooked: <p>Hello, I use torch 1.9.0 and uda 10.2 , GeForce RTX3090 but I got this error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52d0c428cee600963bb970bd628596f61e7d38ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/52d0c428cee600963bb970bd628596f61e7d38ce" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52d0c428cee600963bb970bd628596f61e7d38ce.png" alt="image" data-base62-sha1="bOCqNnhicm6XZ8nWDVUFevd5TWu" width="690" height="58" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">983×84 3.84 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You would need to install the CUDA11.1 binary to use your Ampere GPU.</p>
Writing to a Log - Error: Bound Method

username: Tecia_Marshall
cooked: <p>Any help is appreciated. I was able to get it to print to the log, but I received the bound method error. I think it has something to do with my + str in my exception handling</p>
<p>def main():<br>
“”“This function controls the main thread of the program.”""<br>
print(‘Please enter a floating point value’)<br>
response = input()<br>
try:<br>
max_counter = int(response)<br>
except ValueError:<br>
error_message()<br>
else:<br>
if max_counter &lt;= 0:<br>
error_message()<br>
else:<br>
Leibniz_object = MarshallHomework6.Pi_LeibnizClass(10)<br>
<span class="hashtag">#Collect</span> the number from the user to use for the radius<br>
print (‘Please enter your radius number: ‘)<br>
response = input()<br>
try:<br>
max_counter = float(response)<br>
except ValueError:<br>
print(‘You have entered an invalid value. You must enter a positive floating point value or an integer.’)<br>
print (‘Please restart the program and try again.’)<br>
else:<br>
Leibniz_object = MarshallHomework6.Pi_LeibnizClass(10)<br>
logging.basicConfig(filename=‘example.log’,<br>
format=’%(asctime)s %(levelname)s: %(message)s’,<br>
level=logging.INFO)<br>
msg1 = 'The circumference of circle is: ’ + str(Leibniz_object.get_circumference)<br>
logging.info(msg1)<br>
msg2 = 'The area of a circle is: ’ + str(Leibniz_object.get_area)<br>
logging.info(msg2)<br>
msg3 = 'The volume of a circle is: ’ + str(Leibniz_object.get_volume)<br>
logging.info(msg3)<br>
msg4 = 'The surface area of a circle is: ’ + str(Leibniz_object.get_surfacearea)<br>
logging.info(msg4)<br>
ending()</p>

username: ptrblck
cooked: <p>The question doesn’t seem to be PyTorch-related so e.g. StackOverflow might be a better place to ask these Python questions in order to get a faster response. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Also, your code formatting is unfortunately still not working (click on the “Preformatted text” button to add code snippets).</p>

username: Tecia_Marshall
cooked: <p>This <strong>is</strong> Python. I’m coding in Python.</p>

username: Tecia_Marshall
cooked: <p>thanks for the help.</p>

username: ptrblck
cooked: <p>Yes, I see that you are using Python, but this discussion forum is specialized on PyTorch (the ML framework). While we can certainly try to help solving plain Python issues, the majority of the discussion is focused on PyTorch, which is why I suggested other boards for a faster response <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Differnece in pytorch computational graphs

username: aswamy
cooked: <p>Hello,</p>
<p>I am trying the compare the two computational graphs(graph_1, graph_2). Basically, both perform exactly the same operations in terms of their architecture construction. However, in graph_1 all parameters are initialized and in graph_2 none of the parameters are initialized. When I compare the two graphs there are some differences, but I am not able to understand the difference and justify the reason for the difference.</p>
<p>Any help in understanding these two graphs and their differences is highly appreciated!<br>
<strong>graph_1:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/684d49c0622f78afc76e21fb38b372c75434507e.png" data-download-href="https://discuss.pytorch.org/uploads/default/684d49c0622f78afc76e21fb38b372c75434507e" title="graph_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/684d49c0622f78afc76e21fb38b372c75434507e_2_417x500.png" alt="graph_1" data-base62-sha1="eSHffeMbPbXaXkorfD5Mpm1AVtA" width="417" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/684d49c0622f78afc76e21fb38b372c75434507e_2_417x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/684d49c0622f78afc76e21fb38b372c75434507e_2_625x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/684d49c0622f78afc76e21fb38b372c75434507e_2_834x1000.png 2x" data-dominant-color="F3F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">graph_1</span><span class="informations">2398×2870 200 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>graph_2:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbec5583ff2ce52d85298d7c1caeccd737cb9dea.png" data-download-href="https://discuss.pytorch.org/uploads/default/dbec5583ff2ce52d85298d7c1caeccd737cb9dea" title="graph_2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbec5583ff2ce52d85298d7c1caeccd737cb9dea_2_318x500.png" alt="graph_2" data-base62-sha1="vnwUnFYIsAJuDs6OqGGOwgAnC7o" width="318" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbec5583ff2ce52d85298d7c1caeccd737cb9dea_2_318x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbec5583ff2ce52d85298d7c1caeccd737cb9dea_2_477x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbec5583ff2ce52d85298d7c1caeccd737cb9dea_2_636x1000.png 2x" data-dominant-color="F4F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">graph_2</span><span class="informations">2284×3589 225 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you!</p>
Torch.repeat for sparse matrix

username: samsepiol
cooked: <p>I want to copy a 2D sparse matrix along a new third axis. With a dense matrix, I’d use <code>unsqueeze</code> followed by <code>repeat</code>, but with a sparse matrix, I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: sparse tensors do not have strides
</code></pre>
<p>How do I fix this?</p>
<p>Any help would be greatly appreciated. Thank you!</p>

username: AlphaBetaGamma96
cooked: <p>Would you mind sharing the code?</p>
<p>As you said <code>unsqueeze</code> and then <code>repeat</code> don’t work,</p>
<pre><code class="lang-auto">matrix=torch.sparse_coo_tensor(size=(3,3))
matrix.unsqueeze(2).repeat(1,1,4)
#RuntimeError: sparse tensors do not have strides
</code></pre>
<p>However, you can get around this via using list comprehension and <code>torch.stack</code>, by,</p>
<pre><code class="lang-auto">matrix=torch.sparse_coo_tensor(size=(3,3))
new_matrix = torch.stack([matrix for _ in range(4)], dim=2)
#returns 
#tensor(indices=tensor([], size=(3, 0)),
#       values=tensor([], size=(0,)),
#       size=(3, 3, 4), nnz=0, layout=torch.sparse_coo)
</code></pre>
<p>I hope this helps! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Is the gradient of tensor slice right?

username: forever
cooked: <p>As shown in the code, why the gradient of <code>tt</code> is not <code>[2., 4., 6., 8.]</code>, but <code>[0., 4., 6., 8.]</code>.<br>
Is it affected by the design principle of <code>a[0] = value</code>? Why is it designed like this?Is it right?</p>
<pre><code class="lang-python">
import torch
import numpy as np


def func(t, value):
    a = t * t
    a[0] = value
    return a.sum()


array = np.array([1, 2, 3, 4], dtype='float32')
value = np.array([5.], dtype='float32')

tt = torch.tensor(array, requires_grad=True)
tvalue = torch.tensor(value, requires_grad=True)

l1 = func(tt, tvalue)
l1.backward()

print('torch:array', tt.grad)  # torch:array tensor([0., 4., 6., 8.])
print('torch:value', tvalue.grad)  # torch:value tensor([1.])


</code></pre>

username: Varal7
cooked: <p>The gradient calculation is correct.<br>
When you set <code>a[0] = value</code>, the first value of the <code>tt</code> array is not modified.<br>
Then, the computation of <code>l1</code> never uses this first value of <code>tt</code> so its gradient will be 0.</p>
Make subclass Tensor has grad attributes

username: ralzq01
cooked: <p>Hi,</p>
<p>I want to subclass the Tensor and make it has <code>.grad</code> attribute during backward.</p>
<p>As <code>.grad</code> is only available for leaf tensor, I made the following simple demo but found the behavior is not consistent with original torch.Tensor. There will be <code>AliasBackward</code> introduced in the subclass tensor, but I want to remove it to make it looks like original torch.Tensor.</p>
<p>Current it has the behavior like this:</p>
<pre><code class="lang-auto">import torch

class TensorFoo(torch.Tensor): pass
tensor = TensorFoo([1,2,3,4]).cuda().requires_grad_()
print(tensor.is_leaf, tensor.requires_grad)
tensor = torch.Tensor([1,2,3,4]).cuda().requires_grad_()
print(tensor.is_leaf, tensor.requires_grad)
</code></pre>
<p>The output will be</p>
<pre><code class="lang-auto">False True
True True
</code></pre>
<p>Have any ideas about this? Thanks in advance.</p>

username: Varal7
cooked: <p>Hi,</p>
<p>I can reproduce this on 1.9.0 but it seems to be fixed on master.</p>
<pre><code class="lang-python"># !pip install --pre torch -f https://download.pytorch.org/whl/nightly/cu111/torch_nightly.html
print(torch.version.__version__) # '1.10.0.dev20210723+cu111'
class TensorFoo(torch.Tensor): pass
tensor = TensorFoo([1,2,3,4]).requires_grad_()
print(tensor.is_leaf, tensor.requires_grad) # True True
</code></pre>
<p>As a workaround, you can use <code>tensor._base</code></p>
Accuracy results

username: Mai_Magdy
cooked: <p>Epoch: [1][0/25] Loss 0.5619 (0.5619)Prec@1     73.438 (73.438)   Prec@5 100.000 (100.000)</p>
<p>Epoch: [1][20/25] Loss 0.5804 (0.7834)Prec@1   73.438 (67.113)   Prec@5 100.000 (100.000)</p>
<p>I have two classes and prec 5 always 100 is it okay or I have a problem ?</p>

username: ptrblck
cooked: <p>I don’t think a <code>Prec@5</code> metric would make sense, since it would return:</p>
<pre><code class="lang-auto">(number of relevant classes in top 5 predictions) / (number of relevant classes)
</code></pre>
<p>Since you are working with two classes, the “relevant” class would always be in the “top5” predictions.</p>

username: Mai_Magdy
cooked: <p>should I use another accuracy metric or it’s okay if I used top1 &amp; top 5 in my case</p>

username: ptrblck
cooked: <p>I think the plain accuracy calculation would be sufficient for a binary classification use case and you could remove the Prec@5 one.</p>
Question about RNN Implementation

username: drbeethoven
cooked: <p>An example RNN implementation contains this code for the training section:</p>
<h1>
<a name="train-the-model-1" class="anchor" href="#train-the-model-1"></a>Train the model</h1>
<p>n_total_steps = len(train_loader)</p>
<p>for epoch in range(num_epochs):</p>
<pre><code>for i, (images, labels) in enumerate(train_loader):  

    # origin shape: [N, 1, 28, 28]

    # resized: [N, 28, 28]

    images = images.reshape(-1, sequence_length, input_size).to(device)

    labels = labels.to(device)

    

    # Forward pass

    outputs = model(images)

    loss = criterion(outputs, labels)

    

    # Backward and optimize

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()

    

    if (i+1) % 100 == 0:

        print (f'Epoch [{epoch+1}/{num_epochs}], Step [{i+1}/{n_total_steps}], Loss: {loss.item():.4f}')
</code></pre>
<p>My question is this: here, all entries within a sequences have the same label. This was an example where MNIST was used, and a “sequence” was a row of an image. So, it makes sense that every “column” has the same image classification. However, I’m trying to make an implementation such that each entry in a sequence can have a different output and that subsequent outputs depend on the predictions of previous outputs.</p>
<p>Does anyone know of any implementations of RNNs that have this?</p>
Torch::zeros creation

username: afshin67
cooked: <p>I want to create a <code>torch::zeros</code> based on a dimension which comes from a <code>std::vector&lt;int&gt; vec</code>. Apparently it is not possible to create a tensor by <code>torch::zeros(vec)</code>, since it only accepts the rvalue of arrays like <code>{a,b,c}</code> or <code>c10::ArrayRef</code>. Since the length of my size <code>vec</code> changes, I cannot use <code>{a,b,c}</code> and I have to follow the second approach. Any idea how can I convert a <code>std:vector</code> to <code>c10::ArrayRef</code>?</p>
<p>I also tried passing an array by <code>int* vec = &amp;vec[0];</code>, still got type conversion error for the api.<br>
Also, I tried to create an <code>arrayRef</code> via <code>auto dims = c10::ArrayRef&lt;int64_t&gt;(vec.size());</code>, but torch did not like its syntax.</p>

username: Matej_Kompanek
cooked: <p>Following code compiles and runs as expected (at least with MSVC and win 10), or maybe I am missing something ? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>
<pre><code class="lang-auto">std::vector&lt;int64_t&gt; vec;

vec.push_back(1);
vec.push_back(2);
vec.push_back(3);
auto t = torch::zeros(vec);
std::cout &lt;&lt; t &lt;&lt; std::endl;
</code></pre>

username: afshin67
cooked: <p>Thanks!, it worked now. The problem was that my input <code>std::vector</code> initially was not <code>int64_t</code> and it was <code>int</code>.</p>
CodeReview: Combining index_put_ operations

username: F10w10
cooked: <p>Hi there,</p>
<p>I have a small function that performs cropping/padding in fourier space. Input is a tensor of which the last two (image) or alst three (volume) dimensions are results from an <code>fft.rfftn</code> call. Therefore I needed multiple <code>index_put_</code> operations to correctly fill the cropped/padded FFT:</p>
<pre><code class="lang-auto">using namespace torch::indexing;

torch::Tensor fft_crop(torch::Tensor&amp; fft_volume, int dim, int new_x, int new_y, int new_z) {

    auto n = fft_volume.ndimension();

    auto newDims = fft_volume.sizes().vec();
    newDims[n - 1] = (int64_t)new_x / 2 + 1;
    if (dim &gt;= 2)
        newDims[n - 2] = new_y;
    if (dim == 3)
        newDims[n - 3] = new_z;

    auto newVol = torch::empty(newDims, fft_volume.options());

    int old_x = (fft_volume.size(-1)-1)*2;
    int x = std::min(old_x, new_x);
    int old_y = fft_volume.size(-2);
    int y = std::min(old_y, new_y);
    if (dim == 2) {
        newVol.index_put_({ "...", Slice(0, y / 2 + 1), Slice(0, x / 2 + 1) },
            fft_volume.index({ "...",Slice(0, y / 2 + 1), Slice(0, x / 2 + 1) }));

        newVol.index_put_({ "...", Slice(new_y - y + (y / 2 + 1), None), Slice(0, x / 2 + 1) },
            fft_volume.index({ "...", Slice(old_y - y + (y / 2 + 1), None), Slice(0, x / 2 + 1) }));
    }

    if (dim == 3) {
        int old_z = fft_volume.size(-3);
        int z = std::min(old_z, new_z);
        newVol.index_put_({ "...", Slice(0, z / 2 + 1), Slice(0, y / 2 + 1), Slice(0, x / 2 + 1) },
            fft_volume.index({ "...", Slice(0, z / 2 + 1), Slice(0, y / 2 + 1), Slice(0, x / 2 + 1) }));

        newVol.index_put_({ "...", Slice(0, z / 2 + 1), Slice(new_y - y + (y / 2 + 1), None), Slice(0, x / 2 + 1) },
            fft_volume.index({ "...", Slice(0, z / 2 + 1), Slice(old_y - y + (y / 2 + 1), None), Slice(0, x / 2 + 1) }));

        newVol.index_put_({ "...", Slice(new_z - z + z / 2 + 1, None), Slice(0, y / 2 + 1), Slice(0, x / 2 + 1) },
            fft_volume.index({ "...", Slice(old_z - z + z / 2 + 1, None), Slice(0, y / 2 + 1), Slice(0, x / 2 + 1) }));

        newVol.index_put_({ "...", Slice(new_z - z + z / 2 + 1, None), Slice(new_y - y + (y / 2 + 1), None), Slice(0, x / 2 + 1) },
            fft_volume.index({ "...", Slice(old_z - z + z / 2 + 1, None), Slice(old_y - y + (y / 2 + 1), None), Slice(0, x / 2 + 1) }));
    }
    return newVol;
}
</code></pre>
<p>I am now wondering about the efficiency of this code. Basically, I am just copying different parts of the <code>fft_volume</code> into my <code>new_volume</code>. Is there any way to increase performance here, i.e. combining this into a single operation?</p>
How to save tensors on mobile with lite interpreter

username: lytch
cooked: <p>Based on the discussion in <a href="https://github.com/pytorch/pytorch/pull/30108" rel="noopener nofollow ugc">#30108</a> it’s clear that <code>pickle_save</code> is not supported on mobile, because <code>/csrc/jit/serialization/export.cpp</code> is not included when building for lite interpreter; producing the following runtime error:</p>
<pre><code class="lang-auto">#else 
AT_ERROR( "pickle_save not supported on mobile " "(see https://github.com/pytorch/pytorch/pull/30108)"); 
#endif
</code></pre>
<p>For loading there’s an option of using <code>torch::jit::_load_for_mobile</code>.</p>
<p><strong>However, are there any methods, or alternative approaches, of serialising and saving <code>c10::IValue</code> objects on the mobile device?</strong></p>

username: cccclai
cooked: <p>lite interpreter doesn’t support saving models. The model is generated from desktop and deployed on mobile device. See the deployment workflow in <a href="https://pytorch.org/mobile/home/" class="inline-onebox" rel="noopener nofollow ugc">Home | PyTorch</a></p>

username: lytch
cooked: <p>While true, the answer is misleading. I modified the PyTorch source code a little bit to enable save support for the lite interpreter. I’m surprised it’s not allowed by default. It appears to work without any issues when enabled.</p>
How can I get the gradients of the weights of each layer?

username: d099f6849ae8a1462cd0
cooked: <p>Since my network (rnn used) does not converge, I want to see the gradient of the weights of each layer. I tried using tensor.grad to get the gradient, however, the output is always None. How can I print the gradient in each layer? Thanks</p>

username: ptrblck
cooked: <p>Once you’ve called <code>backward</code> to calculate the gradients, you can directly print them using something like this:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.Linear(10, 2)
)
...
loss.backward()
print(model[0].weight.grad)
</code></pre>
<p>In your case the model definition will look a bit different. So depending how you’ve implemented the model, you might need to index the layers like in my example or call the layers directly with their names:</p>
<pre><code class="lang-python">print(model.my_layer.weight.grad)
</code></pre>

username: d099f6849ae8a1462cd0
cooked: <p><span class="mention">@thank</span> you so much. Problem solved</p>

username: cqzzg
cooked: <p>Can you tell me how you finally get the gradients of each layer？</p>
Logging multiple losses efficiently

username: yiftachbeer
cooked: <p>Hi everyone,</p>
<p>I’m using a loss which is a sum of multiple losses.<br>
While for the actual training I can work with the sum only, I want to log the values of each loss in every iteration.<br>
Naively, I would call log(loss1.item()) and then log(loss2.item()) but to my understanding each call to item() results in a cuda sync which I want to avoid as much as possible.</p>
<p>Am I right to be worried? What is usually done is such situations? Should I concatenate the losses (on the GPU), call item() once on a vector containing all of them and then split them back on the CPU? Should I avoid logging them so often in the first place?</p>
<p>Thanks,<br>
Yiftach</p>
Inferring CrossEntropy loss value

username: arya47
cooked: <p>Hey,<br>
I’m performing a binary classification on a 3d MRI scan dataset while using nn.CrossEntropy as my loss function.</p>
<p>In my training loop i calculate the loss as</p>
<pre><code class="lang-auto">loss = loss_fn(pred, y)
</code></pre>
<p>which is pretty standard.<br>
What i’m trying to learn is what can be inferred from the loss value. i.e</p>
<ul>
<li>What do these loss values represent?</li>
<li>What classifies as a good value?</li>
<li>What do these values tell me about my model, data…?</li>
</ul>
<p>The results i’m currently getting are</p>
<pre><code class="lang-auto">loss: 530.178589  [    0/  474]
loss: 231.394638  [   70/  474]
loss: 61.357143  [  140/  474]
loss: 482.464294  [  210/  474]
loss: 452.214294  [  280/  474]
loss: 173.785721  [  350/  474]
loss: 117.030411  [  420/  474]
</code></pre>
<p>Which intuitively does not feel satisfactory.<br>
Would really appreciate if someone can help me understand what is a good score (if any).</p>
<p>Thanks</p>
Tensorboard: Plot scalar to custom horizontal/x-axis

username: MoRoBe
cooked: <p>Hi everyone,</p>
<p>I’m working on a temporal prediction model predicting the state of a graph for several timesteps. I calculate all metrics per prediction distance. That is, for predicting the first, second, third… timestep after the input graph. I’d like to plot the errors with this prediction distance as x-axis. However from what I found I can only use step, batch or epoch as horizontal axis. As I calculate all the metrics in the test step, step, batch and epoch are the same so the scalars are all on one vertical line. Is there a way to provide a custom x-value to the tensorboard logger?</p>
<p>Best regards<br>
MoRoBe</p>