akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Converting a python list to tensor

username: Samah_Abu_saleem
cooked: <p>I am trying to convert a list to tensor but I got an error.</p>
<pre><code class="lang-auto">ValueError: expected sequence of length 631 at dim 1 (got 284)
</code></pre>
<p>This code computes the pixel values for each detected object in the image.</p>
<pre><code class="lang-auto">imgs = PIL.Image.open(img_path).convert('RGB')
image_width, image_height = imgs.size
imgArrays = np.array(imgs)

     
X = (xCenter*image_width)          
Y = (yCenter*image_height)        
W = (Width*image_width)          
H = (Height*image_height)       

cropped_image = np.zeros((image_height, image_width, 3))

for i in range(len(X)):

           x1, y1, w, h = X[i], Y[i], W[i], H[i]

          x_start = int(x1 - (w/2))

          y_start = int(y1 - (h/2))

          x_end = int(x_start + w)

          y_end = int(y_start + h)

            
          temp = imgArrays[y_start: y_end, x_start: x_end]

            
          cropped_imagesList.append(temp)

          cropped_images = torch.as_tensor(cropped_imagesList)
</code></pre>

username: tom
cooked: <p>The error sounds to me as if rows in the prospective tensor are not of the same length.<br>
Tensors are strictly regular in this respect, if you want to process data of different lengths, you would need to resort to padding or something like that.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Samah_Abu_saleem
cooked: <p>Sir, I do resize for all the images but I still got the error:</p>
<blockquote>
<p>RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 5 but got size 31104000 for tensor number 1 in the list.</p>
</blockquote>

username: tom
cooked: <p>This means that the shape of the second tensor (at index 1) is incompatible with the shape of the first.</p>
Basic installation

username: kenclr
cooked: <p>I went to Anaconda Prompt (32-bit) and got a prompt “(base)”. There I entered the command “conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch”. After 5-10 minutes, I got the message “The following packages are not available from current channels: cudatoolkit=10.2, pytorch, torchaudio”. I was suggested to look at other “channels”. Am I supposed to do something else? This is not explained.</p>

username: ptrblck
cooked: <p>In case you are using a 32bit Python environment, update to 64bit Python and try to install PyTorch again.</p>

username: kenclr
cooked: <p>I did install the 64bit Python environment and was successful in importing <strong>torch</strong>. In addition, I had installed Anaconda3 and went to its prompt, where I was also able to import <strong>torch</strong>. Thus, there are two methods.</p>
Fuse Modules - How to fuse modules in the Sequential block

username: lhkhiem28
cooked: <p>I am trying to fuse some modules in the AlexNet</p>
<pre><code class="lang-auto">model = torchvision.models.alexnet()
</code></pre>
<p>with the architecture:</p>
<pre><code class="lang-auto">AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace=True)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace=True)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(6, 6))
  (classifier): Sequential(
    (0): Dropout(p=0.5, inplace=False)
    (1): Linear(in_features=9216, out_features=4096, bias=True)
    (2): ReLU(inplace=True)
    (3): Dropout(p=0.5, inplace=False)
    (4): Linear(in_features=4096, out_features=4096, bias=True)
    (5): ReLU(inplace=True)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
)
</code></pre>
<p>This is the code I use to fuse Conv2d and ReLU</p>
<pre><code class="lang-auto">torch.quantization.fuse_modules(model, [["features[0]", "features[1]"]])
</code></pre>
<p>or</p>
<pre><code class="lang-auto">torch.quantization.fuse_modules(model, [["features0", "features.1"]])
</code></pre>
<p>but got the same error<br>
<code>AttributeError: 'AlexNet' object has no attribute 'features[0]'</code><br>
How can I fuse modules within a Sequential block?</p>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <a class="mention" href="/u/lhkhiem28">@lhkhiem28</a> , <code>[["features.0", "features.1"]]</code> should work.  More generally, you could print out the fully qualified name of every module in the model with something like</p>
<pre><code class="lang-auto">for name, mod in model.named_modules():
  print(name, mod)
</code></pre>
Gradient and Cuda out of memory

username: As_Ran
cooked: <p>Hello !<br>
I’m actually implementing a complex custom model which look like :</p>
<pre><code class="lang-auto">for step in range(18):
      "data calculation with big matrixes"
      x = lstmcell(data)
      "data calculation with x to get a new_x"
      data = new_x
</code></pre>
<p>My problem is I got my gpu (and cpu btw) out of memory because the gradient is calculated over the data calculation 18 times so at the end I have a very big graph stocked on my gpu (memory usage over 20 GB). Each iteration needs the data of the previous one.</p>
<p>So do you guys know some tricks to help me to reduce the memory usage with this kind of model structure, I tried to torch.no_grad the parts where I do calculation with the data and that works but my network loss function doesn’t reduce and I think its because of the no_grad block. Do I really need to compute the gradient of the calculation part ?</p>
<p>Thanks</p>
sparsely represent 2 numbers in tensor

username: cg_trial
cooked: <p>Torch.sparse results in a sparse representation of only zeros.</p>
<p>But if my vector/tensor is something like [1,1,1,1,1,1,0,0,0,0,0], is there any way I can make the 1’s sparsely represented as well and further reduce memory usage?</p>
<p>I have large tensors consisting of 1s and 0s (masks), and I need to further reduce the memory used so that I may be able to increase the batch size in training. Please share any pointers you may have to optimize the memory usage during training.</p>
How can I train my model on nn.CosineEmbeddingLoss() but with weights?

username: jungminc88
cooked: <p>Like in the code below, I’m trying to train my model on nn.CosineEmbeddingLoss(), and since the dataset has severe label imbalance (the examples where the pair of sentences are not similar far outnumber the ones where they are similar), I need to train it with weights. What I gathered from the document is that nn.CosineEmbeddingLoss() does not directly support weights? If so, is there a workaround? How can I attain weighted loss? I would greatly appreciate your advice. Thank you!</p>
<pre><code class="lang-auto">    criterion = nn.CosineEmbeddingLoss()
    for i, batch in enumerate(dataloader):
        sent_a_input_ids = batch[0].to(device)
        sent_a_attention_mask = batch[1].to(device)
        sent_b_input_ids = batch[2].to(device)
        sent_b_attention_mask = batch[3].to(device)
        labels = torch.where(batch[4]==0, -1, 1).to(device) # -1 if dissimilar 1 if similar
        a_out = model(input_ids=sent_a_input_ids, attention_mask=sent_a_attention_mask).last_hidden_state
        b_out = model(input_ids=sent_b_input_id, attention_mask=sent_b_attention_mask).last_hidden_state
        a_emb = _mean_pooling(a_out, sent_a_attention_mask)
        b_emb = _mean_pooling(b_out, sent_b_attention_mask)

        loss = criterion(a_emb, b_emb, labels)
        loss.backward()
</code></pre>
How to do Surrogate Loss Learning in pytorch

username: Tan_j
cooked: <p>Hello,<br>
I have two models defined as:</p>
<p>model_1 = Net1()<br>
model_2 = Net2()<br>
opt1 = Adam(model1.parameters())<br>
opt2 = Adam(model2.parameters())</p>
<p>When training, the model_1 the output of model_1 is input to model_2 and model_2 acts as a loss function for model_1.<br>
So in this scenario, I wish to update the gradients of model_1 and not model_2 and backpropagate the output of model_2.<br>
def train_model_1():<br>
model_1.train()<br>
model_2.eval()<br>
output1 = model_1(data)<br>
output2 = model_2(output1)<br>
loss = Variable(output2, requires_grad=True)<br>
and then,<br>
loss.backward()</p>
<p>Similarly, for training model_2 I wish to update the gradients of model_2 and not model_1,<br>
def train_model_2():<br>
model_2.train()<br>
model_1.eval()<br>
output1 = model_1(data)<br>
output2 = model_2(output1)<br>
output2_real = get_target_value()<br>
loss = l1_Loss(output2, output2_real)<br>
loss.backward()</p>
<p>Main outer loop looks like this:<br>
pretrain model_1 with some builtin loss function.<br>
pretrain model_2 till its stable<br>
for n epochs:<br>
train_model_1()<br>
train_model_2()</p>
How to transpose a tensor and apply it to cross entropy loss?

username: KawaiiNotHawaii
cooked: <p>Hi guys. If the dimension of the tensor is [x, y], then the dimension of the labels for the loss would be [x]. I want to create a loss - crossEntropyLoss([x, y], [x]) + crossEntropyLoss([y, x], [y]). For the second part, I transpose the original [x, y] tensor, but it keeps giving errors. I also tried clone and transpose but nothing works, it keeps throwing the same error.</p>
<pre><code class="lang-auto">loss_fn = nn.CrossEntropyLoss()
loss = loss_fn(input, labels) + loss_fn(input.T, labels2)
</code></pre>
<p>So the problem is how can I transpose a input and use it in the loss function?<br>
Thanks a lot</p>

username: KFrank
cooked: <p>Hi Charles!</p>
<aside class="quote no-group" data-username="KawaiiNotHawaii" data-post="1" data-topic="140135" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kawaiinothawaii/48/45226_2.png" class="avatar"> KawaiiNotHawaii:</div>
<blockquote>
<pre><code class="lang-auto">loss_fn = nn.CrossEntropyLoss()
loss = loss_fn(input, labels) + loss_fn(input.T, labels2)
</code></pre>
</blockquote>
</aside>
<p>I am skeptical of your proposed use case.  However:</p>
<p>In its most common usage, <code>CrossEntropyLoss</code> takes integer class<br>
labels for its <code>target</code>.  These should be integers that run from zero<br>
to the number of classes you have, that is to say, the size of the<br>
<code>nClass</code> dimension of the <code>input</code>.  Note that transposing your <code>input</code><br>
can change which values are valid for your class labels.</p>
<p>Here is an example of transposing the <code>input</code> where the corresponding<br>
class labels are constructed to be valid:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt; x = 3
&gt;&gt;&gt; y = 5
&gt;&gt;&gt; input = torch.randn (x, y)
&gt;&gt;&gt; labels = torch.randint (y, (x,))
&gt;&gt;&gt; labels2 = torch.randint (x, (y,))
&gt;&gt;&gt; loss_fn = torch.nn.CrossEntropyLoss()
&gt;&gt;&gt; loss_fn (input, labels)
tensor(2.5805)
&gt;&gt;&gt; loss_fn (input.T, labels2)
tensor(2.9129)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: KawaiiNotHawaii
cooked: <p>I supposed it is the right way to implement this. But it turns out that I’ve tried the same implementation as yours and CUDA gives me an error.</p>
How does PyTorch LSTM dropout work?

username: Bjorn_Lindqvist
cooked: <p>I’m trying to port a model from Keras to PyTorch and I’m having trouble with the LSTM layers.</p>
<p>In Keras the dropout parameter specifies “Fraction of the units to drop for the linear transformation of the inputs.” Here I believe the linear transforms refers to the input, forget, cell and output gates respectively.</p>
<p>In PyTorch the dropout parameter appears to specify the dropout between LSTM layers. Hence the complaint when you add dropout to a one-layer lstm. There is nothing in-between to apply the dropout too.</p>
<p>What parameters should I give to the LSTM to get the same effect as the dropout parameter in Keras?</p>

username: JamesWright
cooked: <p>Hey <a class="mention" href="/u/bjorn_lindqvist">@Bjorn_Lindqvist</a> , were you able to figure this out? I am facing exactly same problem. Thanks!</p>
Tensor array question

username: MG_Song
cooked: <p>Hello,<br>
I’m here to ask a simple question of tensor. I would like to temporarily save a sub-array of tensor.<br>
Let’s suppose I have a  ([Batch, Channel, Width ,Height]) shape of tensor.<br>
How can I extract only the entire subarray of Channel?</p>
<p>For example<br>
The shape that I want= (Channel,)</p>

username: tsuijenk
cooked: <p>Hello,</p>
<p>In this case, you would do [:, channel_number, :, :].</p>
<p>Actually, maybe I am not understanding your question fully. But if you have some number of channels with a certain height and width, I don’t think you can reshape to 1D with just (channel,). Could you please elaborate more perhaps?</p>

username: MG_Song
cooked: <p>Thanks for the clarification.<br>
My purpose is to concatenate the entire tensors of channel with the half-size W and H array. (I will not use pooling operation) To be specific, if the shape of tensor is ([16(C), 112(W),112(H)]), then I firstly downscale only the H and W, and then want to combine with C. So it will be ([16,56,56]).</p>
<p>For this purpose,  should I flatten them individually and then reshape it ?? since the array[:, channel_number, :, :] will give me a specific channel tensor instead of whole.</p>
Filter out items from Dataset? "filter_pred" for any dataset

username: badranx
cooked: <p>Hello,</p>
<p>For some datasets in torchtext, I see the parameter <code>filter_pred</code>  which filter out part of the dataset.</p>
<p>I’m using torchtext.Datasets which gives an IterableDataset. This iterator you can pass to the Dataloader. Is it possible to simply apply a filter like filter_pred?</p>
<p>This is my solution, which I don’t like, if you have something better, please share.</p>
<pre><code class="lang-auto">class MyIterableDataset(IterableDataset):
    def __init__(self, iterableDataset, pred_filter=None):
        self.iterableDataset = iterableDataset
        self.pred_filter = pred_filter
    
    def __iter__(self):
        def it(myDataIter):
            for x in myDataIter:
                if self.pred_filter and not self.pred_filter(x):
                    continue
                yield x
        myiter = iter(it(self.iterableDataset))
        return myiter

</code></pre>

username: nivek
cooked: <p>Does this work for you?</p>
<pre><code class="lang-python">class MyIterableDataset(IterableDataset):

    def __init__(self, iterableDataset, pred_filter=None):
        self.iterableDataset = iterableDataset
        self.pred_filter = pred_filter

    def __iter__(self):
        return self

    def __next__(self):
        while True:
            item = next(self.iterableDataset)
            if self.pred_filter is None or self.pred_filter(item):
                return item
</code></pre>
AttributeError: Can't pickle local object 'AllInOneBlock.__init__.<locals>.<lambda>' 

username: QQR1
cooked: <p>Traceback (most recent call last):<br>
File “F:/QLing/QLing.Code/FastFlow/main.py”, line 31, in <br>
model = train(train_loader, test_loader)<br>
File “F:\QLing\QLing.Code\FastFlow\train.py”, line 146, in train<br>
save_model(model, c.modelname)<br>
File “F:\QLing\QLing.Code\FastFlow\model.py”, line 142, in save_model<br>
torch.save(model, os.path.join(MODEL_DIR, filename))<br>
File “D:\Programs\Python\Python37\lib\site-packages\torch\serialization.py”, line 379, in save<br>
_save(obj, opened_zipfile, pickle_module, pickle_protocol)<br>
File “D:\Programs\Python\Python37\lib\site-packages\torch\serialization.py”, line 484, in _save<br>
pickler.dump(obj)<br>
AttributeError: Can’t pickle local object ‘AllInOneBlock.<strong>init</strong>..’</p>
How can I take the opposite of gather

username: moefke
cooked: <p>Dear,</p>
<p>The Gather method gathers values along an axis specified by dim. I would like to take the output, so the part that is discarded then the values by chosen indexes. How should I do this?</p>
<p>Thanks</p>

username: tom
cooked: <p>I am not 100% sure I understand what you attempt to do, but would <code>scatter</code>ing a bool tensor <code>True</code> to the indices and then taking the complement (using the <code>~</code> unary operator) and applying this as a mask in advanced indexing work?</p>
<pre><code class="lang-auto">inp = torch.arange(25.).view(5, 5)
idx = torch.randint(4, (5, 1)) + torch.tensor([0, 1])
res = inp[~torch.zeros(inp.shape,dtype=torch.bool).scatter_(1, idx, torch.tensor(True).expand(idx.shape))].view(inp.size(0), inp.size(1) - idx.size(1))

print(f"{inp=}\n{idx=}\n{res=}")
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: moefke
cooked: <p>Thank you very much, this is the solution I needed.</p>
How do I remove forward hooks on a module without the hook handles?

username: ProGamerGov
cooked: <p>I can’t seem to figure out how to remove the hooks without the handles, despite being able to detect the hooks on the target module.</p>
<pre><code class="lang-auto">from typing import Tuple
import torch

model = torch.nn.Sequential(*[torch.nn.Identity()] * 4)

def forward_hook(module: torch.nn.Module, input: Tuple[torch.Tensor], output: torch.Tensor) -&gt; None:
    pass

def find_and_remove_hooks(m):
    print("Hooks")
    c = 0
    module_name = type(m).__name__
    print(m._forward_hooks)
    for k, v in m._forward_hooks.items():
        c+=1
        m._forward_hooks[v].remove() # Doesn't work
    print("All " + str(c) + " hooks found")

model[1].register_forward_hook(forward_hook)

find_and_remove_hooks(model[1])
</code></pre>

username: ProGamerGov
cooked: <p>Looks like using this code does nothing:</p>
<pre><code class="lang-auto">handle = torch.utils.hooks.RemovableHandle(module._forward_hooks)
handle.remove()
</code></pre>
<p>While, simply setting the value back to it’s default removes the hooks:</p>
<pre><code class="lang-auto">from collections import OrderedDict
from typing import Dict, Callable

module._forward_hooks: Dict[int, Callable] = OrderedDict()
</code></pre>
<p>All forward hooks can be removed like this:</p>
<pre><code class="lang-auto">from collections import OrderedDict
from typing import Dict, Callable
import torch

def remove_all_forward_hooks(model: torch.nn.Module) -&gt; None:
    for name, child in model._modules.items():
        if child is not None:
            if hasattr(child, "_forward_hooks"):
                child._forward_hooks: Dict[int, Callable] = OrderedDict()
            remove_all_forward_hooks(child)
</code></pre>
<p>And all the hooks can be removed like this, using the attributes from [here]:</p>
<pre><code class="lang-auto">from collections import OrderedDict
from typing import Dict, Callable
import torch

def remove_all_hooks(model: torch.nn.Module) -&gt; None:
    for name, child in model._modules.items():
        if child is not None:
            if hasattr(child, "_forward_hooks"):
                child._forward_hooks: Dict[int, Callable] = OrderedDict()
            elif hasattr(child, "_forward_pre_hooks"):
                child._forward_pre_hooks: Dict[int, Callable] = OrderedDict()
            elif hasattr(child, "_backward_hooks"):
                child._backward_hooks: Dict[int, Callable] = OrderedDict()
            remove_all_hooks(child)
</code></pre>
<p>This solution doesn’t seem to reset the global counter though as can be seen with the <code>find_hooks</code> function.</p>
Unable to properly visualize the DETR preprocessed images

username: Sayak_Paul
cooked: <p>I am trying to visualize the images preprocessed with the <a href="https://github.com/facebookresearch/detr" rel="noopener nofollow ugc">DETR utilities</a>. I have tried following their codebase and also this <a href="https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/detr_demo.ipynb" rel="noopener nofollow ugc">notebook</a>. Not sure what I am doing wrong but the visualizations tell me that something’s off:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c3bde237ed88ee04cc802be1d386ce621e9da4b9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c3bde237ed88ee04cc802be1d386ce621e9da4b9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c3bde237ed88ee04cc802be1d386ce621e9da4b9_2_512x500.jpeg" alt="image" data-base62-sha1="rVBVHhJ75kQV70jCXRNYAif4PXj" width="512" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c3bde237ed88ee04cc802be1d386ce621e9da4b9_2_512x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/c/3/c3bde237ed88ee04cc802be1d386ce621e9da4b9.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/3/c3bde237ed88ee04cc802be1d386ce621e9da4b9.jpeg 2x" data-dominant-color="8D7A57"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">572×558 90.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here’s <a href="https://colab.research.google.com/gist/sayakpaul/5a866d012fea13ba278e5777615a4f9b/scratchpad.ipynb" rel="noopener nofollow ugc">my Colab</a> to reproduce the issue.</p>
Which funciton will be lanuched mutiple times when I use init_process_group function?

username: Ardeal
cooked: <p>Hi,</p>
<p>I have 4 GPUs for training.<br>
<code>CUDA_VISIBLE_DEVICES=0,1,2,3 WORLD_SIZE=4 python -m torch.distributed.launch --nproc_per_node=4 --master_port 49611 train.py</code><br>
I know that spawn will initialize the function of its input parameter.<br>
when I  use the following code to initialize the training, which function will be launch multiple times?<br>
when will local_rank 0, 1, 2 or 3 be set respectively?</p>
<p>Is there any doc/link about how <code>init_process_group</code> function initialize functions?</p>
<pre><code class="lang-auto">def trainer():
    # code 
	if args.distributed:
		torch.cuda.set_device(args.local_rank)
		device = torch.device('cuda:{}'.format(args.local_rank))
		torch.distributed.init_process_group(backend='nccl', init_method='env://')
		args.world_size = torch.distributed.get_world_size()
		args.rank = torch.distributed.get_rank()
	else:
		_logger.info('Training with a single process on 1 GPU.')
	assert args.rank &gt;= 0
	```</code></pre>
Validation loss remains almost the same

username: hlopez
cooked: <p>Hi, I’m working on a task to predict breast cancer from mammograms in a maximum timelapse of 2 years.<br>
I have tried multiple models and hyperparameters but my validation loss does not improve, it remains almost the same since the beginning, always in the range of 0.69…</p>
<p>Both validation and training losses start at 0.69…, but at the moment that the training loss decreases to 0.68, the validation loss starts increasing, and the overfitting starts. When the loss is 0.69, the accuracy is at 0.50, which is the baseline because the data is equally balanced (50% cases, 50% controls).</p>
<p>That led me to think that the model was in a plateau or local minima since the beginning, and I tried to schedule the learning rate to be bigger in the first epochs, but it didn’t work.</p>
<p>Any suggestions or ideas about how to solve it? Maybe making the predictions using only the mammograms cannot solve the task?</p>
Meta-learning using second-order derivative

username: M_R
cooked: <p>Hi,<br>
Does PyTorch support the second-order derivative? I am trying to run the following code:</p>
<p>student_parameters = [param for param in Student.parameters()]<br>
teacher_params = [param for param in Teacher.parameters()]</p>
<p>psudo_labels = Teacher(inputs)</p>
<p>output =  Student(inputs)</p>
<p>err1 = torch.nn.MSELoss()(output,psudo_labels)</p>
<p>err1.backward(retain_graph=True, create_graph=True,inputs=student_parameters)</p>
<p>optimizer_student.step()</p>
<p>output_new =  Student(inputs2)</p>
<p>new_loss = torch.nn.MSELoss()(output_new, GT)</p>
<p>teacher_grad = grad(new_loss, teacher_params)</p>
<p>But I get the following Runtime error:</p>
<p>RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="M_R" data-post="1" data-topic="139884">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/bcef8e/48.png" class="avatar"> M_R:</div>
<blockquote>
<p>GT</p>
</blockquote>
</aside>
<p>You haven’t used the variable <code>GT</code> anywhere in your code so the derivative is equal to 0 by definition. That’s what the error code is stating “One of the differentiated Tensors appears to not have been used in the graph.” Are you using the correct variable here?</p>

username: M_R
cooked: <p>Sorry for not being clear. GT is a constant tensor that is defined somewhere beforehand. Think of it as a label or ground-truth.</p>

username: AlphaBetaGamma96
cooked: <p>oh ok, so I made a mistake on your issue. The reason why you’re getting that error is that <code>torch.autograd.grad</code> is taking the gradient of <code>new_loss</code> which solely depends on <code>Student</code> with respect to the parameters of <code>Teacher</code>. This is equal to zero by definition as <code>Student</code> doesn’t contain <code>teacher_parmas</code></p>

username: M_R
cooked: <p>Well that’s exactly my question.<br>
I do “optimizer_student.step()”, which uses gradients computed from “err1 = torch.nn.MSELoss()(output,psudo_labels)”, which depends on the teacher’s parameters. The reason I am getting this error is because the optimizer optimizer_student updates the parameters of the student using in-place operations and so the teachers’parameters are broken from the computation graph at that point. I wonder how I can achieve this goal.</p>

username: Ethel
cooked: <p>There is a lot of interesting literature on meta-learning with … The First-Order MAML ignores the second derivative part in red.<br>
<a href="https://www.rapidfs.run/" rel="noopener nofollow ugc">www rapidfs com activation</a></p>
Get the mean of a 2D tensor for each column

username: AlanSouthwark
cooked: <p>Hello, I want to know what is the most simple way to get the mean of the matrix along each column, namely my tensor only has two dimensions with shape (m X n).</p>
<p>For example, if I have a tensor object</p>
<pre><code class="lang-auto">T = torch.FloatTensor([[2.6, 5], [1.7, 6], [3.2, 7], [2.1, 8]])
</code></pre>
<p>I want some function that could return a tensor object as following</p>
<pre><code class="lang-auto">([2.4, 6.5])
</code></pre>
<p>I know I could do with for loop, but I wonder if there is a PyTorch built-in method that could do this for me, or is there some more elegant way for this purpose.</p>
<pre><code class="lang-auto">mean = torch.zeros(T.size(1))
for i in range(T.size(1)):
  mean[i] = torch.mean(T[:, i])
</code></pre>
<p>Thanks for any help.</p>

username: AlanSouthwark
cooked: <p>Anyway, I didn’t understand the argument “dim” correctly, with “dim” argument “torch.mean(T, dim=i)” do everything for me, but for getting the mean of each column, i=0 is the correct dimension.</p>
How does torch.jit trace submodules with "unsupported" input type

username: siahuat0727
cooked: <p>Hi, I have a model with two submodules A and B.<br>
A should not be traced (control flow depends on the input tensor) and B contains “unsupported” input types.</p>
<blockquote>
<p>Tracing is ideal for code that operates only on <code>Tensor</code> s and lists, dictionaries, and tuples of <code>Tensor</code> s. <a href="https://pytorch.org/docs/stable/generated/torch.jit.script.html#torch.jit.script" rel="noopener nofollow ugc">torch.jit.script — PyTorch 1.10.1 documentation</a></p>
</blockquote>
<p>How can I trace only on B? (or trace the whole model but skip submodule A? <code>torch.jit.ignore</code> does not work for this case)</p>
<p>I think it’s technically possible since <code>torch.jit.trace</code> can correctly trace submodules with “unsupported” input types. (see <code>comment1</code> below)</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn


class Foo(nn.Module):
    def __init__(self):
        super().__init__()
        self.identity = nn.Identity()

    def forward(self, x1, x2):
        out = self.identity(x1)
        if x2 is not None:
            out = out + self.identity(x2)
        return out


class Bar(nn.Module):
    def __init__(self):
        super().__init__()
        self.foo1 = Foo()
        self.foo2 = Foo()

    def forward(self, x):
        out = self.foo1(x, x)
        out = out + self.foo2(x, None)
        return out


bar = Bar()
data = torch.ones(42)

# comment1: This is ok
# bar = torch.jit.trace(bar, data)

# comment2: This is ok too
# bar.foo1 = torch.jit.trace(bar.foo1, (data, data))

# comment3: Fails
bar.foo2 = torch.jit.trace(bar.foo2, (data, None))
</code></pre>
<pre><code class="lang-auto">RuntimeError: Type 'Tuple[Tensor, NoneType]' cannot be traced. Only Tensors and (possibly nested) Lists, Dicts, and Tuples of Tensors can be traced
</code></pre>
<p>Any feedback would be greatly appreciated!</p>
<p>torch version: 1.10.1</p>

username: siahuat0727
cooked: <p>Well, I found a solution that worked for me.<br>
We can first <code>torch.jit.script</code> submodule A (maybe combine with <code>torch.jit.ignore</code> to make it scriptable, just to ensure that it will be skipped by <code>torch.jit.trace</code>), and then <code>torch.jit.trace</code> the whole model.</p>
<p>Still looking for a “solution” for the title.</p>
Is there an inverse of rnn.pad_sequence

username: dem123456789
cooked: <p>I encounter a situation where I need to store intemediate representations of the sequential dataset with variable length. I cannot directly store packed sequence as the order of batch data can be random. Therefore, I am wondering how we can inverse the operation of pad_sequence?</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="dem123456789" data-post="1" data-topic="140343">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/ce7236/48.png" class="avatar"> dem123456789:</div>
<blockquote>
<p>Therefore, I am wondering how we can inverse the operation of pad_sequence?</p>
</blockquote>
</aside>
<p>Not sure that I understand your problem, but inverse operation is unstacking with “unbind” op. There is no built-in way to strip variable-sized padding tails after that AFAIK.</p>
Constructing a tensor by taking mean over index list of another tensor

username: sajastu
cooked: <p>I’m looking for a faster method to do a specific operation (i.e., mean of specified indices) on a given tensor. To be more specific, let’s say I have a tensor <code>A</code> with the shape of <code>[B, N, D]</code>, with N being the batch size which is 1 in my case, N being the number of nodes, and D being the feature dimension. I also have a list <code>I</code> which is a list of lists, each of which contains index values. For instance, <code>A</code> could be of shape <code>[1, 2048, 768]</code> and I can be <code>I = [ [0, 1], [2], [3, 4, 5], ... ]</code>.</p>
<p>My goal is to create a resulting tensor <code>Z</code> which aggregates the mean values from <code>A</code> based off the indices provided in <code>I</code>. For now, I’m doing it as below, but it seems to be much time-consuming to produce the resulting Z:</p>
<pre><code class="lang-auto">idx_mapping = {k : v for k, v in zip(I, [s for s in range(len(I))])}

Z = torch.zeros(A.size(0), len(I), A.size(2))

for k, v in idx_mapping.items():
        
    # indices (List) which we want to take mean from tensor A
    mean_indices = k
    
    # mapped idx in the resulting vector Z
    embedded_into = v 
    
    # do mean on mean_indices of tensor A
    token_embed = torch.index_select(A, 1, torch.tensor(eval(f'[{mean_indices.replace("-", ",")}]')).mean(dim=1).unsqueeze(0))
        
    # replace the calculated mean in embedded_into's index of tensor Z
    Z = torch.cat((Z[:, :embedded_into, :], token_embed, Z[:, embedded_into+1: , :]), dim=1)
</code></pre>
<p>For reference, <code>idx_mapping</code> is a dictionary that maps the indices of tensor <code>A</code> to resulting tensor <code>Z</code>. I have constructed its keys from list <code>I</code> and values are incremented one by one. For instance, <code>idx_mapping</code> can be: <code>{'0': 0, '1-2': 1, '3-4-5': 2, ...}</code></p>
<p>Now the question is that what is the best (and possibly quickest) way to this operation in Pytorch?</p>

username: ptrblck
cooked: <p><code>scatter_add_</code> should work after creating the necessary indices:</p>
<pre><code class="lang-python">x = torch.randn(1, 6, 768)
I = [[0, 1], [2], [3, 4, 5]]

tmp = torch.arange(len(I))
idx = torch.repeat_interleave(tmp, torch.tensor([len(a) for a in I]))
idx = idx.unsqueeze(0).unsqueeze(2).expand_as(x)
res = torch.zeros(x.size(0), idx.max()+1, x.size(2))
res.scatter_add_(1, idx, x)
res = res / torch.tensor([len(a) for a in I])[None, :, None]

reference = torch.stack([x[:, i].mean(1) for i in I], dim=1)

print((res - reference).abs().max())
# &gt; tensor(0.)
</code></pre>
nn.Linear vs torch.matmul

username: Damian_Andrysiak
cooked: <p>Hi everyone!</p>
<p>I am wondering, why these outputs are different…</p>
<pre><code class="lang-auto">my_data = torch.tensor([1,2,3], dtype=torch.float32).repeat(1000, 1)
weights = torch.rand((3,2))
out = torch.matmul(my_data, weights)
print(out)

linear = nn.Linear(3, 2)
output_linear = linear(my_data)
print(output_linear)
</code></pre>
<p>out is not close to output_linear…</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30398bd9a9f5a9cec2448923e0c756fabc2513fe.png" data-download-href="https://discuss.pytorch.org/uploads/default/30398bd9a9f5a9cec2448923e0c756fabc2513fe" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30398bd9a9f5a9cec2448923e0c756fabc2513fe.png" alt="image" data-base62-sha1="6SCd4QKADNiCDIaxZUlLzaHRhfg" width="690" height="254" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1310×483 12.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I would like to recreate following schema and I think my torch.matmul multiplies it correctly… but dont know why nn.Linear gives my different results just…</p>

username: googlebot
cooked: <p>obv. you have to compare the module with something like matmul(my_data, linear.weights.T) + linear.bias</p>

username: Damian_Andrysiak
cooked: <p>hmmm,</p>
<pre><code class="lang-auto">out = torch.matmul(my_data, weights) + 1.
linear = nn.Linear(3, 2)
</code></pre>
<p>this way?</p>

username: tflahaul
cooked: <p>this way <a class="mention" href="/u/damian_andrysiak">@Damian_Andrysiak</a></p>
<pre><code class="lang-auto">&gt;&gt;&gt; linear = torch.nn.Linear(3, 3)
&gt;&gt;&gt; inputs = torch.rand(3, 3)
&gt;&gt;&gt; 
&gt;&gt;&gt; linear(inputs)
tensor([[-0.4848,  0.4174, -0.1633],
        [-0.4428,  0.5404, -0.1470],
        [-0.7272,  0.0770,  0.0321]], grad_fn=&lt;AddmmBackward0&gt;)
&gt;&gt;&gt; torch.mm(inputs, linear.weight.T).add(linear.bias)
tensor([[-0.4848,  0.4174, -0.1633],
        [-0.4428,  0.5404, -0.1470],
        [-0.7272,  0.0770,  0.0321]], grad_fn=&lt;AddBackward0&gt;)
&gt;&gt;&gt; 
</code></pre>
<p>You were using <code>weights</code> and 1 instead of the layer parameters.</p>

username: Damian_Andrysiak
cooked: <p>ok, works for me. Could you tell me please, does I do it right way? I mean about multiplication weights like on my paint:</p>
<pre><code class="lang-auto">       output = torch.matmul(x, self.weights_ih) + self.bias_ih
</code></pre>
<p>where:</p>
<pre><code class="lang-auto">        self.weights_ih = torch.rand((input_features, input_features), device=device)    
        self.bias_ih = torch.rand((input_features), device=device)
</code></pre>
Reinforcement Learning Model Not Learning

username: opfromthestart
cooked: <p>I am attempting to create an AI to play a game that I have created, that is similar to tic-tac-toe. I have chosen to have the AI make a choice about which move to make based on a set of scores that it gives, which is one for each play that can be made. I chose to not explicitly disallow it from trying to make a move when one has already been made but am trying to train it to learn not to play on tiles that have already been played.<br>
I have done some tests with differing losses, and it seems to work when I tell it to minimize error directly, as a score of 1 for available tiles and -1 for unavailable ones. When I attempt to have the loss based on the actual state of the board, and the moves that are made, the AI seems to behave randomly and not learn. I am unsure as to why the model does not learn. Are there any obvious problems with my code/logic, and how can I fix them?</p>
<p>Code is here below:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1CNWvv_9Xk_udTPGD4hCZNo5cFxnVRbjG?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20211221-060057-RC00_417582297" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1CNWvv_9Xk_udTPGD4hCZNo5cFxnVRbjG?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1CNWvv_9Xk_udTPGD4hCZNo5cFxnVRbjG?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Installation fail, cython library error?

username: Jonathan_Goodman
cooked: <p>I’m new to PyTorch, installed following instructions using pip3, ran the “tensor” demo OK, but failed on: “from torchvision import datasets”.  I’m running OS version 11.2.1 (Big Sur) and did not install the CUDA package because my macbook doesn’t have an NVIDIA graphics processor.  I would be grateful for hints on how to solve this.  Here’s what seems to be the relevant part of the error message:</p>
<p>ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torchvision/_C.cpython-37m-darwin.so, 2): Symbol not found: __ZN6caffe26detail36_typeMetaDataInstance_preallocated_0E<br>
Referenced from: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torchvision/_C.cpython-37m-darwin.so<br>
Expected in: flat namespace<br>
in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torchvision/_C.cpython-37m-darwin.so</p>
How to use different weights in forward and backward steps from update step in model.parameters

username: Babi
cooked: <p>Hi everyone!<br>
I want to use the same weights in the forward and backward steps in the training, but for the update step, i want to use different weights as old_weights in:</p>
<pre><code class="lang-auto">new_weights = old_weights - learning_rate * gradients
</code></pre>
<p>in fact, old_weights are different from the weights that are used in the forward and backward pass. How can i do this for model.parameters() in pytorch?</p>
<p>Thanks in advance</p>
What does happen when the parameters are sent to torch.optim.SGD() in the backward pass

username: Babi
cooked: <p>Hi<br>
I am a beginner in Pytorch. are these two codes the same? In fact, i am confused does the second code work correctly and each weight will update properly.</p>
<p>first:</p>
<pre><code class="lang-auto">model = models.resnet101(pretrained=False)
opt = optim.SGD(model.parameters(), lr=lr, momentum=0.8)
</code></pre>
<p>second:</p>
<pre><code class="lang-auto">model = models.resnet101(pretrained=False)

parameters = model.parameters()
weights = []
for name, param in model.named_parameters():
    if name.endswith('weight'):
        weights.append(param)
weight_id = list(map(id, weight_parameters))
other_parameters = list(filter(lambda param: id(param) not in weights_id, parameters))

opt = optim.SGD([{'params' : other_parameters},
            {'params' : weights}], lr=lr, momentum=0.9)
</code></pre>
torch.fx.proxy.TraceError:symbolically traced variables cannot be used as inputs to control flow

username: MartinZhang
cooked: <p>Today, I use<code> get_graph_node_names</code> to get the trian nodes and test nodes.</p>
<p>such as : <code>train_nodes, test_nodes = get_graph_node_names(module)</code></p>
<p>But while I’m running it</p>
<p>There a error about this :</p>
<p><code>torch.fx.proxy.TraceError:symbolically traced variables cannot be used as inputs to control flow</code>.</p>
<p>I know. This problem with the model，but the model can be normally forward</p>
<p>I can’t find other problem from this model.</p>
<p>please, help me, how to deal it. <img src="https://discuss.pytorch.org/images/emoji/apple/handshake.png?v=10" title=":handshake:" class="emoji" alt=":handshake:"></p>
<p>the code follow as :</p>
<pre><code class="lang-auto">from torch.fx import Tracer
import torch.nn as nn
import torch

lstm = nn.LSTM(300, 100, 1)
x = torch.randn(7, 64, 300)
h = torch.randn(1, 64, 100)
c = torch.randn(1, 64, 100)
lstm(x, (h, c))

nodes = Tracer().trace(lstm).nodes
</code></pre>
How to change the shape inside DataLoader?

username: Hongnan_G
cooked: <p>When training on a classification problem with multiclass and using <code>CrossEntropyLoss</code>, the targets should integer, and hence <code>targets.long()</code> is needed. But when training on multilabel or binary, we may invoke <code>BCEWithLogitsLoss</code>, which expects your targets to be in <code>float</code>.</p>
<p>I made changes in <code>Dataset</code> as follows:</p>
<pre><code class="lang-python">if CRITERION_PARAMS.train_criterion_name == "BCEWithLogitsLoss":
    # Make changes to reshape rather than in Trainer.
    y = torch.as_tensor(y, dtype=torch.float32).view(-1, 1)
else:
    y = torch.as_tensor(y, dtype=torch.long)
</code></pre>
<p>However, I soon realized that this does not work since <code>DataLoader</code> will (collate?) the shape wrongly. What I expect is the following:</p>
<pre><code class="lang-auto">original_y_shape -&gt; (batch_size, ) or (batch_size, 1)
modified_y_shape -&gt; (batch_size, 1) # to match shapes with BCEWithLogitsLoss
</code></pre>
<p>But if I change the shape inside <code>Dataset</code>, the <code>DataLoader</code> gives me <code>(batch_size, 1, 1)</code> instead.</p>
<p>One easy way is to change the shape in the training function directly. But I would like to keep the changes in <code>Dataset</code> unless there is a strong reason not to (speed issues).</p>

username: my3bikaht
cooked: <p>Starting torch 1.10 you can pass probabilities for each class into CrossEntropyLoss.<br>
Also getitem method in custom dataset should return single element, not the batch.</p>

username: Hongnan_G
cooked: <p>Yes, <code>getitem</code> returns a single element, but when I reshape the single tensor with <code>.view(-1, 1)</code>, the <code>DataLoader</code> somehow put an extra dimension 1 behind.</p>

username: my3bikaht
cooked: <p>Right, so, for example, your self.labels inside dataset have shape (10000,1)<br>
Then <strong>getitem</strong> will return single item (1), then you’re applying view(-1, 1), which transforms item to shape (1,1), then batch loader will return batch_num of elements with shape (1, 1)</p>

username: Hongnan_G
cooked: <p>So I actually do not need to use <code>view(-1, 1)</code> to ensure that the shape is correct to be passed in to <code>BCEWithLogitsLoss</code>?</p>

username: my3bikaht
cooked: <p>Or you can do flatten() instead, then both (x,1) and (x) will result in (x)</p>

username: Hongnan_G
cooked: <p>Yes, I read up the documentation, you are right:</p>
<pre><code>    By definition of collate function in DataLoader, it is mentioned in documentation that it prepends an extra dimension to the tensor as batch_size. Thus,
    if the input y is a tensor of shape [1,], then .view(-1, 1) will change the shape to [1, 1], and when we collate using DataLoader, say with batch_size = 4,
    then the collated y will be a tensor of shape [4, 1, 1] instead of [4, 1] since it prepends an extra dimension.</code></pre>
How to check existence of parameters?

username: The_duke
cooked: <p>Hi all,</p>
<p>As part of manually updating parameters, I am trying to check for existence of a parameter.</p>
<p>I have code that looks like this:</p>
<pre><code>try:
    with torch.nograd():
     model.linear_relu_stack[_i].weight=nn.Parameter(reshaped_params[_j])
except AttributeError:
    print("Weight doesn't exist,going to next")
</code></pre>
<p>Is there a more elegant way to check for whether or not a layer has parameters at all?</p>

username: tom
cooked: <p>Pythin hasattr or getattr with a default should work.</p>
<p>Best regards</p>
<p>Thomas</p>
Error during compilation on Linux

username: Calva
cooked: <p>Hello,</p>
<p>I am trying to build PyTorch from source because I need to install it with ROCm 4.5 on my system. I already installed ROCm successfully, and I successfully used PyTorch on the latest ROCm docker image. I am now trying to install it to my computer (without docker).</p>
<p>However I am unable to compile the source code, I have build errors. I tried different versions of PyTorch, the main branch, and release 1.10.1.</p>
<p>I downloaded the source code from <a href="https://github.com/pytorch/pytorch/releases/tag/v1.10.1" class="inline-onebox" rel="noopener nofollow ugc">Release PyTorch 1.10.1 Release, small bug fix release · pytorch/pytorch · GitHub</a><br>
Steps to reproduce:</p>
<pre><code class="lang-auto">cd ~/Download/pytorch-v1.10.1
python3 tools/amd_build/build_amd.py
PYTORCH_ROCM_ARCH="gfx900" python3 setup.py install
</code></pre>
<p>Here is the full output: <a href="https://pastebin.com/8yLaKZsz" class="inline-onebox" rel="noopener nofollow ugc">-- The CXX compiler identification is Clang 13.0.0-- The C compiler identifica - Pastebin.com</a><br>
I tried compiling using both gcc 11.2 and clang 13.0, resulting in the same error.<br>
My system is a fresh install Pop!_OS 21.10 with ROCm 4.5.0 installed.</p>
<p>Many thanks if anyone can point out what I’m doing wrong.</p>