akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Replace diagonal of matrix with new diagonal values

username: mayurat22
cooked: <p>Numpy Version of the Code:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2f2497e40fdb45c2397068e508eb3abeed5eef3.png" alt="image" data-base62-sha1="u67mIJfJtQW2TIr6mOKp5WCgiu7" width="301" height="263"></p>
<p>For Pytorch version, It seems like we cannot replace diagonal elements with a vector of elements and only a scalar value can be filled on diagonal.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/383072579e09313642bfa95296bdae2642296d87.png" data-download-href="https://discuss.pytorch.org/uploads/default/383072579e09313642bfa95296bdae2642296d87" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/383072579e09313642bfa95296bdae2642296d87.png" alt="image" data-base62-sha1="814x9OSscbl0P63B9fzZhPO10JF" width="690" height="330" data-dominant-color="141620"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">745×357 27.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/replace-diagonal-elements-with-vector/15311">This older post</a> provides some approaches using direct indexing to fill the diagonal with a tensor.</p>

username: mayurat22
cooked: <p>Thank you. It worked. I should have checked it better.</p>
Porting PixelCNN tensorflow code to Pytorch

username: KarthikR
cooked: <p>Hi,<br>
I am trying to recreate “PixelCNN” model written in Tensorflow at this <a href="https://github.com/PacktPublishing/Hands-On-Image-Generation-with-TensorFlow-2.0/blob/master/Chapter01/ch1_pixelcnn.ipynb" rel="noopener nofollow ugc">link</a>, in Pytroch.</p>
<p>My code is as below:</p>
<pre><code class="lang-auto">class MaskedConv2D(nn.Module):
    
    def __init__(self, mask_type, kernel=5, filters=1):
        super(MaskedConv2D, self).__init__()
        self.kernel = kernel
        self.filters = filters
        self.mask_type = mask_type
        
        self.weight = torch.nn.Parameter(torch.Tensor(self.kernel,self.kernel,1,self.filters), requires_grad=True)
        self.bias   = torch.nn.Parameter(torch.Tensor(self.filters,), requires_grad=True)
        nn.init.xavier_uniform_(self.weight)
        nn.init.constant_(self.bias, 0)
        
        mask = np.ones(self.kernel**2, dtype=np.float32)
        center = len(mask)//2
        mask[center+1:] = 0
        if self.mask_type == 'A':
            mask[center] = 0
            
        mask = mask.reshape((self.kernel, self.kernel, 1, 1))
        self.mask = torch.tensor(mask, dtype=torch.float32, requires_grad=False)

# Keras code seem to pass "SAME to the padding parameter. How do I do it with Conv2d here?        
    def forward (self, inputs):
        # If i try to do padding like below, I get "RuntimeError: Boolean value of Tensor with more than one value is ambiguous", during the Conv2d step.
        inputs = F.pad(inputs, (1,1,0,0))        
        masked_w = self.weight.mul(self.mask)
        output = torch.nn.Conv2d(in_channels=inputs,out_channels=64,kernel_size=masked_w, stride=1) # + self.bias
        return torch.nn.ReLU(output)

    
class ResidualBlock(nn.Module):
    def __init__(self, h=32):
        super(ResidualBlock, self).__init__()

        self.forward = nn.Sequential(MaskedConv2D('B', kernel=1, filters=h),
                                     MaskedConv2D('B', kernel=3, filters=h),
                                     MaskedConv2D('B', kernel=1, filters=2*h))         
    def call(self, inputs):
        x = self.forward(inputs)
        return x + inputs  


class PixelCNN(nn.Module):
    def __init__(self, hidden_features=64, output_features=64, resblocks_num=7):
        super(PixelCNN, self).__init__()
        self.model = nn.Sequential(
             MaskedConv2D('A',kernel=7, filters=2*hidden_features),
             ResidualBlock(hidden_features),
             ResidualBlock(hidden_features),
             ResidualBlock(hidden_features),
             ResidualBlock(hidden_features),
             ResidualBlock(hidden_features),
             ResidualBlock(hidden_features),
             ResidualBlock(hidden_features),
             nn.Conv2d(in_channels = output_features, kernel_size = (1,1),out_channels=1),
             torch.nn.ReLU(),
             nn.Conv2d(in_channels = 1,out_channels = 1, kernel_size = (1,1)),
             torch.nn.Sigmoid()
        )
        
    def forward(self, x):
        return self.model(x)


net = PixelCNN()
</code></pre>
Convert torch.save model to matrices that can be handled by numpy

username: mperez2
cooked: <p>Hello everyone,</p>
<p>Is there a way to transform the torch model into algebraic matrices that can be operated with numpy or basic Python, without the need to install torch and other libraries in a device (system on module)? The libraries need a lot of space and they cannot be handled. In an afirmative case, could you please give me some hints or link with explanation? Thank you very much.</p>

username: ptrblck
cooked: <p>You could certainly rewrite the model in numpy, but note that you wouldn’t be able to train the model directly, since Autograd isn’t implemented (and you couldn’t use other devices such as the GPU).<br>
Are you running into space limitations on your hard disk?</p>

username: mperez2
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> The thing is that I did not develop the model and was expecting for a tool to obtain the Python code of matrices and more simple operations I would need to perform and handle without torch. Is there something like this?</p>
<p>I only would like to get new forecasts with this model working on a hardware that incorporates Python, and that due to space limitations I cannot install torch. No need to train again.</p>
<p>Thank you very much.</p>

username: ptrblck
cooked: <p>I’m not aware of any library exporting a PyTorch model to plain Python code, as it would most likely be too slow for the majority of use cases.<br>
I’m not sure how space-limited you are, but maybe you could use the CPU-only PyTorch package or build PyTorch from source and remove all accelerators (CUDA, cudnn, NCCL, MKL, etc.).</p>
Slightly Different Results When Evaluating Same Model on Different Machines

username: John_Watkins
cooked: <p>I am currently working on a NLP project, and have trained a Seq2Seq Transformer model. Due to certain project requirements, I need to check that my model (and code) can work on different machines (with different operating systems).</p>
<p>I have noticed that when conducting evaluation on the different machines with the same model, code, and dataset, the results are all slightly different. Specifically, once every while (once per 200 cases), my output sequences on the different machines will be different by a few tokens. Just to double check, I am wondering if this behaviour, that evaluation on different machines will result in slightly different results, is expected. Additionally, what might be the cause of this phenomenon?</p>
<p>Thank you in advance.</p>

username: ptrblck
cooked: <p>Yes, this is generally expected due to different hardware, algorithms, library versions, optimizations etc.<br>
E.g. a different CPU might use other optimizations for e.g. SIMD instructions and could thus yield slightly different results; the same applies for GPUs, especially since different kernels can be used.</p>
Key error 'iscrowd' while finetuning

username: Miafr
cooked: <p>I’m trying to fine tune an existing model, I’ve created a custom Dataset class and included iscrowd key in my <strong>get_item</strong>() (based on what is done <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">here</a>),<br>
but I keep getting:</p>
<blockquote>
<p>site-packages/pycocotools/cocoeval.py", line 188, in <br>
iscrowd = [int(o[‘iscrowd’]) for o in gt]<br>
KeyError: ‘iscrowd’</p>
</blockquote>
<p>it seems to run fine without GPU.<br>
Can anyone help?</p>

username: ptrblck
cooked: <p>Could you post a minimal code snippet using random input shapes, which would reproduce this issue, please? The error points to a missing key, but since the code seems to apparently work on the CPU, we would like to debug it on the GPU.</p>
How to extract the tensor element from tuple?

username: ADONAI_TZEVAOT
cooked: <p>I have a prior which is multivariate gaussian and a tensor zs.<br>
<code>self.prior.log_prob(zs[-1])</code></p>
<p>the above line of code is throwing the folowing error:</p>
<pre><code class="lang-auto">AttributeError: 'tuple' object has no attribute 'log_prob'
</code></pre>
<p>Please help</p>

username: ptrblck
cooked: <p>Based on the error message <code>self.prior</code> seems to be a <code>tuple</code>, so you would have to unwarp or index it to access the distribution, which defines the <code>log_prob</code> method.</p>
Change size of a tensor

username: nafise
cooked: <p>Hi<br>
how can convert a tensor with size[3,1000] to a tensor with size[128, 512, 3, 3]?</p>

username: ptrblck
cooked: <p>You cannot directly convert a tensor in the first shape to the second one, since the number of elements is different (3000 vs. 589824), so you would have to increase the size somehow (e.g. via upsampling/interpolation, transposed convolutions etc.).</p>
BCELossWithLogits negative outputs

username: Jam
cooked: <p>Hi am having a binary classification problem and using BCELossWithLogits. However, my predictions are negative values (e.g. <code>[-0.263, -0.308, -0.449, -0.279]</code>). Before, when I was using BCELoss with Sigmoid, I received probabilities as output as I was expecting. I am using auc_roc_score to compare the predictions to my targets (which are either 0 or 1). I read the docs and it says that the output of BCELossWithLogits is the negative logarithm of the sigmoid (but the example stills shows a positive value). Does someone know how I have to modify my predictions in order to get the probabilities or am I doing something wrong?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Jam" data-post="1" data-topic="120028">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/f6c823/48.png" class="avatar"> Jam:</div>
<blockquote>
<p>However, my predictions are negative values (e.g. <code>[-0.263, -0.308, -0.449, -0.279]</code>). Before, when I was using BCELoss with Sigmoid, I received probabilities as output as I was expecting.</p>
</blockquote>
</aside>
<p><code>nn.BCEWithLogitsLoss</code> expects logits (as given in the name), which are unbounded and can contain values in <code>[-Inf, Inf]</code> (of course they should contain valid values, since an <code>Inf</code> output would also yield an invalid loss value). If you want to convert these logits to probabilities for further processing, you could use <code>torch.sigmoid</code> on them. Note that you should <em>not</em> pass the probabilities (sigmoid outputs) to the criterion, but the logits.</p>
<aside class="quote no-group" data-username="Jam" data-post="1" data-topic="120028">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/f6c823/48.png" class="avatar"> Jam:</div>
<blockquote>
<p>I read the docs and it says that the output of BCELossWithLogits is the negative logarithm of the sigmoid (but the example stills shows a positive value).</p>
</blockquote>
</aside>
<p>The output of <code>nn.BCEWithLogitsLoss</code> is the binary cross-entropy loss and would take values in <code>[0,+Inf]</code> for the expected inputs.</p>
Fine-tuning beginner Question about changing Parameters

username: newbieee
cooked: <p>Hi guys</p>
<p>I have a trained model which is intended exactly for my problem. I want to improve this model by training it further on new data. Now I have a question how to proceed in choosing the hyperparameters, especially the learning rate. Currently I am trying to train 1 epoch on all training data for a certain learning rate and then compare the validation of the models on the test data. Here I try to build up certain ranges e.g. from 1e-3 to 1e-4 and then I take the best final result and continue from there, if e.g. 1e-4 was better, then I would test 0.5e-4 to 1.5e-4 as the next range until I notice no or minimal changes at some point. Does this procedure make sense? An epoch lasts about 8 hours and the procedure therefore takes a very long time.</p>
<p>With real numbers i first tested<br>
5E-3 and 1e-4, and 1e-4 was better<br>
so i tested 1e-4 1e-5, 1e-5 won etc</p>
<p>Thanks</p>
Can packed sequence be put in a linear layer?

username: ironmanaudi_leon
cooked: <p>I am right now using the <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.PackedSequence.html#torch.nn.utils.rnn.PackedSequence" rel="noopener nofollow ugc">PackedSequence</a> as an input to the <code>LSTM</code> network. However, the hidden layer and output have different lengths of features, which I have to transform the output of the LSTM to by a linear layer with softmax activation. The question is can I get this done by using the <code>PackedSequence</code>? The reason is that if I simply pad the output (<code>PackedSequence</code>) of the LSTM; then, the new output will have a lot of <code>0</code> in it, which will later be transformed to none-zero values by the linear transformation and softmax funciton. In this case, the output is uncorrect becasue the padded parts of the sequence should always stay zero.</p>
Changing batch_size and setting num_workers > 0 or 1 throws the following: 'TypeError: __init__() missing 1 required positional argument: 'dtype'

username: E_Quitee
cooked: <p>Hi all, still learning this module (and about 6-8 months of python coding experience). I was wondering if some one can point me to the correct direction in what this may be telling me? (error in the image below)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee13ef2ffae8abd5e6146f2b56ae07b951f6af22.png" alt="Trains but stops and complains" data-base62-sha1="xY8lw6QCb6j9tObWInjcL06nQc2" width="675" height="464"></p>
<p>I am following a book along (Deep Learning with Pytorch by Eli Stevens and Thomas Viehmann) about implementing some deep learning models but I am not entirely sure what is going on in this error (been stuck for 1 day or 2 now). If I set the num workers to 0 and some batch_size, the model trains but if I start setting the num_workers to 2 (in the figure above, the num_workers was set to 2 by default) or higher along with increasing the batch_sizes this error usually will be thrown…if i set num_workers higher like 12 I get an <em>“OSError: [WinError 1455] The paging file is too small for this operation to complete”</em></p>
<p>Other things I notice:</p>
<ul>
<li>from the windows task manager GPU utilization for cuda: it barely goes over 10% (i see spikes)</li>
<li>the memory usage does go up</li>
</ul>
<p>I have the following:<br>
OS: Windows 10<br>
CPU: AMD Ryzen 9 3900X 12-Core Processor<br>
GPU: GTX 1660ti<br>
32gb ram</p>
<p>So from these, do you think it could be a hardware issue or what? I’ve been trying to figure this out, no idea what else to do so this is my first post…so hopefully someone sees this and can point me to some direction. Thank you very much in advanced</p>