akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pixelwise weights for MSELoss

abweiss: <p>I would like to use a weighted MSELoss function for image-to-image training.  I want to specify a weight for each pixel in the target.  Is there a quick/hacky way to do this, or do I need to write my own MSE loss function from scratch?</p>

smth: <p>you can do this:</p>
<pre><code class="lang-python">def weighted_mse_loss(input, target, weights):
    out = input - target
    out = out * weights.expand_as(out)
    # expand_as because weights are prob not defined for mini-batch
   loss = out.sum(0) # or sum over whatever dimensions
   return loss
</code></pre>

abweiss: <p>Oh, I see.  There’s no magic to the loss functions.  You just calculate whatever loss you want using predefined Torch functions and then call backward on the loss.  That’s super easy.  Thanks!</p>

liygcheng: <p>what if I want L2 Loss ?</p>
<p>just do it like this?</p>
<pre><code class="lang-auto">def weighted_mse_loss(input,target,weights):
    out = (input-target)**2
    out = out * weights.expand_as(out)
    loss = out.sum(0) # or sum over whatever dimensions
    return loss

</code></pre>
<p>right ?</p>
<hr>
<hr>

smth: <p><a class="mention" href="/u/liygcheng">@liygcheng</a> yes. that’s correct.</p>

Jordan_Campbell: <p>So long as all the computations are done on <code>Variables</code>, which will ensure the gradients can be computed.</p>

delhi_loafer: <p>Should the “weights” also be wrapped as a <code>Variable</code> in order for auto-grad to work?</p>

JindongJiang: <p>Same question, what is your thoughts now ?</p>

delhi_loafer: <p>yes, the “weights” should also be wrapped as a <code>Variable</code></p>

JindongJiang: <p>Thanks. Actually, variablization must be done if we need them operate at gpu(s).</p>

picolo: <p>Thanks for this!<br>
Can the Weights be learned as well? Would the model need to output the weights in that case? How would the code look?</p>

ptrblck: <p>If the weights are learned I would be concerned about the overall training, since the model could try to set them to zero to create a zero loss (or generally lower them).<br>
What is your use case to try to train the weights and how would you avoid a trained zero weight matrix?</p>

picolo: <p>I am trying an unsupervised approach to identify “important” areas of an image without explicitly pre-defining the weight-map. For the other question, I am not sure yet. I am thinking of a simple form like this - LW+(1/W) where L W is elementwise multiplication of learned weights and the MSE/MAE loss. and the second term prevents going too close to 0?</p>
Transformer tgt during prediction

Gawkie: <p>hello everyone, hope all is well</p>
<p>i am using transformers in audio processing where i try to generate instrumentals from a mixture audio track (music source seperation), during the training/validation it makes sense to pass the instrument track as the target and the mixture as the source, but during actual prediction, the target is still required which is where i am confused. i’ve read answers on the forums here but im still not quite sure as what to do.</p>
<p>thanks in advance for the help.</p>
Create a StackingRegressor similar to the one from sklearn

afsa: <p>I built my regression model, and now I want to make a staking regressor similar to the one presented in sklearn (<a href="https://scikit-learn.org/stable/auto_examples/ensemble/plot_stack_predictors.html" class="inline-onebox" rel="noopener nofollow ugc">Combine predictors using stacking — scikit-learn 1.2.1 documentation</a>).</p>
<p>I want to make the stacking of three models. Each model receives one matrix and produces a matrix with the same dimensions. So model_1 receives matrix_input and produces matrix_output_1. Model_2 receives matrix_input and produces matrix_output_2. And model_3 receives matrix_input and produces matrix_output_3.</p>
<p>I can produce a single final matrix by using a weighted sum to produce the final output. However, instead of a weighted sum, I want to create a single model that receives matrix_output_1, matrix_output_2, and matrix_output_3 and produces the matrix_output.</p>
<p>Does anyone have suggestions on how I can do it?</p>
Low validation accuracy and loss for multi-output classifier

m-dnsk: <p>Hi,</p>
<p>I’m trying to build multi-output image classifier based on MobileNetV2.<br>
First output has 2 classes and secondary has 7. I have balanced train and validate data (each image is labeled by one of value from both outputs):</p>
<ul>
<li>validate: 424 for each of 7 classes and 1484 for each of 2 classes</li>
<li>train: 1708 for each of 7 classes and 5978 for each of 2 classes<br>
I have two separate nn.Sequentail outputs, for loss I’m using cross_entropy and Adam as optimizer.<br>
During training, accuracy of each of outputs is increasing and total loss decreasing but every 25 epochs I’m validating data using saved checkpoints and unfortunately I got a problem with increasing loss and low accuracy score. For validation it looks like that:</li>
<li>After 25 epochs: loss - 2.31, first_class_acc - 0.79, second_class_acc - 0.19</li>
<li>After 50 epochs: loss - 2.13, first_class_acc - 0.85, second_class_acc - 0.25<br>
…</li>
<li>After 200 epochs: loss - 3.30, first_class_acc - 0.88, second_class_acc - 0.24<br>
…</li>
<li>After 300 epochs: loss - 4.54, first_class_acc - 0.88, second_class_acc - 0.24<br>
Do you have an idea what probably am I doing wrong?</li>
</ul>
Use and Abuse of .register_buffer( )

mbp28: <p>Hi,</p>
<p>I have some trouble understanding the use of register_buffer().<br>
I found just a little bit of explanation in the docs, mentioning “running_mean” in BatchNorm.</p>
<p>My questions are:</p>
<ol>
<li>When should I register a buffer? For what sort of Variables and for which not?</li>
<li>Could someone provide me with a simple example and code snippet of using register_buffer()?<br>
[3.] At the moment, I’m running some tests on an implementation of a custom gradient, which I subsequently modify. record the gradients before and after modification in two separate lists. Is that something I should consider register_buffer for to make the code cleaner? I guess not, if the buffer only holds one state at a time…</li>
</ol>
<p>Any help much appreciated.<br>
Many thanks,<br>
Max</p>

smth: <p>you use register_buffer when:</p>
<ul>
<li>you want a stateful part of your model that is not a parameter, but you want it in your state_dict</li>
<li><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py#L23-L24">https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py#L23-L24</a></li>
<li>registered buffers are Tensors (not Variables)</li>
</ul>

igreen: <aside class="quote no-group" data-username="mbp28" data-post="1" data-topic="4128">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mbp28/48/4265_2.png" class="avatar"> mbp28:</div>
<blockquote>
<p>running_mean</p>
</blockquote>
</aside>
<p>Hi, using the dataparallel to train the model in the multi-gpu mode, BN is conducted in device-wise manner. how running mean and running variance is estimated? also in device-wise manner? or one master and multi replica? Thanks</p>

lumin_liu: <p>A question about the register buffer. Can we delete the buffer after we register and how ?</p>

mingzhe_lv: <aside class="quote no-group" data-username="lumin_liu" data-post="4" data-topic="4128" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lumin_liu/48/10424_2.png" class="avatar"> lumin_liu:</div>
<blockquote>
<p>A question about the register buffer. Can we delete the buffer after we register and how ?</p>
</blockquote>
</aside>
<p>I have the same question.</p>

Saleh: <p>I have a question regarding multi-gpu training of a transformer model in which I have a register buffer variable for causal attention masking. in forward pass all 4 GPUs are being used (checked with nvidia-smi), however I get an error when it arrives to loss.backward(). Im getting this error : “RuntimeError: You are trying to call the hook of a dead Module!”. Can this issue be related to this register buffer or masking operation?</p>
<p>My code is working properly with 1 GPU or only on a CPU.</p>
Torchscript Loop: type mismatch

MissyDu: <p>This is  dump() result from a torchscript model.</p>
<pre><code class="lang-auto">%hidden0 : (Tensor, Tensor)?, %not_blank : bool, %symbols_added : int = prim::Loop(%348, %132, %hidden.7, %132, %151) 
  block0(%351 : int, %hidden0.15 : (Tensor, Tensor)?, %not_blank.7 : bool, %symbols_added.5 : int):
          .....
         %775 : bool = prim::If(%not_blank0)
         block0():
           %723 : bool = aten::lt(%symbols_added0.1, %722)
         -&gt; (%723)
        block1():
        -&gt; (%134)
  -&gt; (%775, %hidden3, %not_blank0, %symbols_added0.1
</code></pre>
<p>the return value %775 is a bool type, but the block of loop requires a int type.<br>
It 's allowed？</p>
Output of LSTM for linear classification

msabrii: <p>Hi everyone! i have a biLSTM model which I’m using to classify posts. It is a binary classification task. I am using batch first so the input to the lstm is of the shape [8x50x768], I then take the ‘output’ of the lstm layer which is of shape [8x50x40]. I then pass it through a linear layer and then a sigmoid function to map the output to a value between 0 and 1. However, the output after all this is a 3d tensor of shape [8x50x1] and I’m unsure how to use that to get a singular value for each item in the batch to compare to the labels which are a list of values of size of the batch.</p>
<p>I read online that you can use max pooling? but I’m not entirely sure if that is correct.I tried using that with torch.max(output, 1). And that leaves me with a tensor that’s [8x40] and after I pass that through the other layers I end with [8x1] which I then squeeze so I can compare with the labels. Not sure if any of this is “correct” though, so would appreciate if someone could explain that to me.</p>
<p>A side question I had was whether you should use the output part of the lstm output for binary classification or if you should use the h_n or c_n parts?</p>

vdw: <p>When you say <code>output</code> of the LSTM layer, I assume you have something like</p>
<pre><code class="lang-auto">output, (hidden, cell) = self.lstm(...)
</code></pre>
<p>in your code. <code>output</code> contains hidden states for all time steps (of the last layer in case you have <code>num_layer &gt; 1</code>). In principle you could average all hidden layers.</p>
<p>However, the default approach is usually to consider only the last hidden state with respect to the forward direction and the last hidden state with respect to the backward direction. You can then either sum/average these to hidden states or concatenate them. Depending on your choice, the input size for the linear layer is either <code>hidden_size</code> or <code>2*hidden_size</code>.</p>
<p>For an example, you can check out this <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/rnn.py" rel="noopener nofollow ugc">code for the GRU/LSTM classifier</a>. The relevant snippets are below the comments <code># Extract last hidden state</code> and <code># Handle directions</code></p>
Zscore calculation

klark: <p>Hello,</p>
<p>I’m trying to standard my loaded data before training the model using scipy.stats.zscore().<br>
I systematically get this message:</p>
<blockquote>
<p>/anaconda3/lib/python3.9/site-packages/scipy/stats/_stats_py.py:2899: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:143.)<br>
z[np.broadcast_to(isconst, z.shape)] = np.nan</p>
</blockquote>
<p>How can I avoid this warning ?<br>
I calculated it manually but I feel it’s better to use a predefined function scipy.stats.zscore().</p>
Why dynamic quantization requires input dimension >=2?

Chongxiao_Cao: <p>I got the error of <code>"The dimension of input tensor should be larger than or equal to 2"</code> when calling <code>torch.jit.trace()</code> on a dynamic quantized model. The error message is from <a href="https://github.com/pytorch/pytorch/blame/63bf7674fade22f2bb4b2f83ea0a3ec8af96abac/aten/src/ATen/native/quantized/cpu/qlinear_dynamic.cpp#L50" rel="noopener nofollow ugc">this line of source code of PyTorch</a>:</p>
<p>Why “input dimension &gt;=2” is a hard check here?</p>
<p>Example code:<br>
model architecture:</p>
<pre><code class="lang-auto">  (MLP_layer_0): Linear(
    (model): Sequential(
      (0): Linear(in_features=2, out_features=10, bias=True)
      (1): Sigmoid()
    )
  )
  (MLP_layer_1): Linear(
    (model): Linear(in_features=10, out_features=1, bias=True)
  )
</code></pre>
<p>Then calling dynamic quantization:</p>
<p><code>model = torch.quantization.quantize_dynamic(model)</code></p>
<p>Then calling torch.jit.trace() and hit above error:<br>
<code>model = torch.jit.trace(model, ...)</code></p>

supriyar: <p>this is the requirement of the backend, in this case fbgemm.<br>
The fbgemm library expects inputs to the linear op to be the following<br>
C(output) = A(input) x B(weight), where C, A, B are M x N, M x K, K x N</p>

Chongxiao_Cao: <p>thanks for quick response.</p>
<p>I’m not familiar with the backend CPP stuff.<br>
In my above step, do you suggest to change Linear layer shape, or adding extra dimension on my <code>example_inputs</code> tensor for <code>torch.jit.trace(model, example_inputs)</code> method?</p>
How to avoid deep copying iterable when using torchdata?

Vedant_Roy: <p>Let’s say I have a large in-memory array of byte buffers.<br>
If I use a datapipe like:</p>
<pre><code class="lang-auto">dp = IterableWrapper(very_big_array)
dp = dp.sharding_filter()
</code></pre>
<p>(I believe) this will deep clone the <code>very_big_array</code>, since the datapipe must be pickled and sent to each worker process.</p>
<p>Is there anyway to get around this?</p>

nivek: <p>Does it help if you use <code>IterableWrapper</code> with argument <code>deepcopy=False</code>? You can also have a custom DataPipe that holds a reference of <code>very_big_array</code> and read from it inside <code>__iter__</code> without copying the data.</p>
GPU memory footprints are different between 16GB and 32GB V100

f10w: <p>Today I encountered something very strange. Running the exact same training command on a 16GB V100 GPU and a 32GB V100 GPU, the former shows <code>max mem: 11229</code> while the latter shows <code>max mem: 20109</code>, where the command for printing was</p>
<pre><code class="lang-python">print(f'max mem: {torch.cuda.max_memory_allocated() / MB:.0f}')
</code></pre>
<p>Is this behavior normal?<br>
Thank you in advance for your replies!</p>

ptrblck: <p>Yes, this behavior might be expected as different libraries could pick different kernels using different internal workspaces.<br>
E.g. cuDNN would use its heuristics or profile different kernels (the behavior depends on <code>torch.backends.cudnn.benchmark</code> being <code>False</code> or <code>True</code>, respectively) to select the best kernel, while meeting the memory requirements.<br>
You could try to disable cuDNN and see if the change in memory usage is still visible.</p>

f10w: <p>Interesting. Thanks for the answer, <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
Detach().clone() vs data.clone()

alexeybelkov: <p>I compared x.detach().clone() and x.data.clone() for some random x with requires_grad=True in terms of computational time and detach().clone() seems a bit faster. But why <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20">?</p>

ptrblck: <p>I don’t see a difference using:</p>
<pre><code class="lang-python">x = torch.randn(1024, 1024, requires_grad=True)
%timeit x.data.clone()
%timeit x.detach().clone()
</code></pre>
<p>but note that the <code>.data</code> attribute is deprecated and using <code>.detach().clone()</code> is the right approach.</p>
Validation with DDP Pytorch

Anita: <p>Hi,</p>
<p>I implemented this validation loop for evaluating with DDP PyTorch based on the official tutorial <a href="https://github.com/pytorch/examples/blob/e4e8da8467d55d28920dbd137261d82255f68c71/imagenet/main.py" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at e4e8da8467d55d28920dbd137261d82255f68c71 · pytorch/examples · GitHub</a>.</p>
<p>I am using drop_last=True in the validation DistributedDataloader, and then adding a final condition to get a last validation run on the remaining dataset (when number of examples is not divisible by my world_size), data that was stripped off by the drop_last=True in the DistributedDataloader.</p>
<p>Can please anyone guide me, if this implementation is ok?<br>
Also is this implementation the same as if I would use drop_last=False in the validation DistributedDataloader?</p>
<p>I would really appreciate your feedback on this.</p>
<pre><code class="lang-auto">
def validate(valid_loader, model, loss_fn, optimizer, scaler, world_size,
            early_stopping, epoch, results_path, batch_size, num_workers,):
    """Validation implemented according to the official PyTorch guidelines
    here: https://github.com/pytorch/examples/blob/main/imagenet/main.py"""

    def run_validate(valid_loader):

        validation_loss = 0.0
        count = torch.zeros(1, dtype=torch.float32, device="cuda")

        with torch.no_grad():
            for i, (images, labels,  wsi_id) in enumerate(valid_loader):

                images = images.cuda(non_blocking=True)
                labels = labels.cuda(non_blocking=True)
                images = torch.squeeze(images)

                with torch.autocast(device_type="cuda", dtype=torch.float16):
                    logits, Y_prob, Y_hat, _ = model(images)
                    vloss = loss_fn(logits, labels)

                count += 1
                validation_loss += vloss

        return validation_loss, count

    model.eval()
    valid_loss, count = run_validate(valid_loader)

    # If number of examples is not divisible by the world_size, and drop_last=True on the
    # DistributedSampler, call run_validate again to evaluate the remaining samples.
    if len(valid_loader.sampler) * world_size &lt; len(valid_loader.dataset):
        aux_val_dataset = Subset(valid_loader.dataset, range(len(valid_loader.sampler) * world_size, len(valid_loader.dataset)))
        aux_val_loader = DataLoader(aux_val_dataset, batch_size=batch_size, shuffle=False, num_workers=num_workers, pin_memory=True)
        valid_loss_aux, count_aux = run_validate(aux_val_loader)
        # Update the metrics.
        valid_loss = torch.add(valid_loss, valid_loss_aux)
        count = count + count_aux

    dist.barrier()
    dist.all_reduce(valid_loss, dist.ReduceOp.SUM, async_op=False)
    dist.all_reduce(count, dist.ReduceOp.SUM, async_op=False)

    avg_val_loss = float((valid_loss / count).detach().cpu())
    print(f"Total validation loss = {avg_val_loss:.4f}")
</code></pre>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/how-do-i-validate-with-pytorch-distributeddataparallel/172269/8">here</a> with a follow up.</p>
Runtime Error: cuda runtime error (10)

Sasika_Amarasinghe: <pre data-code-wrap="Traceback"><code class="lang-plaintext">  File "train.py", line 26, in &lt;module&gt;
    MyModel = PretrainedModel(arch_name,hidden_layers, data_dir, epochs, device , learning_rate, manual_seed)
  File "/home/workspace/ImageClassifier/Model.py", line 62, in __init__
    self.flower_classifier = FlowerClassifier(self.input_layer_size, self.output_layer_size, self.hidden_layers)
  File "/home/workspace/ImageClassifier/flower_classifier.py", line 32, in __init__
    self.hidden_layers = nn.ModuleList([nn.Linear(input_size, hidden_layers[0])])
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 41, in __init__
    self.weight = Parameter(torch.Tensor(out_features, in_features))
RuntimeError: CUDA error (10): invalid device ordinal```

I got this error when trying to train a model. Does anyone know how to resolve this issue?</code></pre>

ptrblck: <p><code>RuntimeError: CUDA error (10): invalid device ordinal</code><br>
is raised if you try to specify a wrong device id as seen e.g. here:</p>
<pre><code class="lang-python">x = torch.randn(1).to("cuda:100")
# RuntimeError: CUDA error: invalid device ordinal
</code></pre>
<p>Check which GPU id is used and make sure it’s in <code>[0, nb_gpus-1]</code>.</p>
How does masking work to make neural networks use varying input lengths

Amin_Shn: <p>A well known solution to make Neural Networks (NNs) work with varying lengths of input data is to use padding and then masking the padded inputs (padded with zero, a common approach in natural language processing). It’s being said that this approach makes the NN skip the padded inputs. I want to know what’s the mechanism behind such skipping. For me NNs are all about matrix multiplication. So here are my questions:</p>
<ol>
<li>Is masking done by forcing some weights to zero?</li>
<li>How is it different than simply using the padded values that are zero without masking (zero times anything is zero again, hence I suppose the answer to my 1st question should be no, because there is no point in forcing the weights to zero when the input is zero itself)</li>
<li>I assume masking didn’t suddenly emerge out of thin air, so I want to know who invented the method? Are there any academic papers talking about it?</li>
</ol>
Comparing training Losses after each epoch on single-gpu system and multiple-gpu system

Peerzada_Danish: <p>I wanted to know when training a model on multiple gpu’s using <code>nn.DistributedDataParallel</code>, does the training loss get affected when compared to single gpu system, keeping all the parameters including batch size, learning rate same?</p>

Jamie_Donnelly: <p>Training a model with data parallelism, if done correctly, is mathematically equivalent to training in serial on one GPU. If the serial model and the replica models (in DDP) all begin with the same initial weights and use same training parameters, then there should be no discernible difference between the final models.</p>

Peerzada_Danish: <p>In my case, i am not able to replicate the results of single gpu system to 3-gpu system. The error seems to become stagnant after certain epochs unlike single-gpu system. Also after some epochs, <code>There appear to be 6 leaked semaphore objects ...</code> error appears unexpectedly. Can you help me out here?</p>

Peerzada_Danish: <p><code>72 semaphores</code> I guess.</p>

kumpera: <p>DDP doesn’t use semaphores, so it’s likely that your model code has some issues.</p>
<p>There are a few things that usually go wrong when doing DDP that needs to be accounted for. random draws during training and batchnorm are two common sources of problems.</p>

Peerzada_Danish: <p>The model works fine without DDP on a single gpu.</p>
RuntimeError nn.CrossEntropyLoss() shape issue

jdefriel: <p>I’m running into: Predictions and targets are expected to have the same shape, but got torch.Size([1013, 100]) and torch.Size([1013]). I am using class indices as the predictions of which there are 100 and the batch size is 1013. What am I missing? I’ve gotten this to work on other code and the example in PyTorch is very similar with targets as [5,3] and preds as [3].</p>

KFrank: <p>Hi John!</p>
<aside class="quote no-group" data-username="jdefriel" data-post="1" data-topic="172498" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jdefriel/48/56652_2.png" class="avatar"> jdefriel:</div>
<blockquote>
<p>Predictions and targets are expected to have the same shape, but got torch.Size([1013, 100]) and torch.Size([1013]).</p>
</blockquote>
</aside>
<p>The probable cause of your error is that your <code>target</code> is a <code>Float</code> tensor<br>
when it should be a <code>Long</code> tensor.</p>
<p><code>CrossEntropyLoss</code> supports both integer-class-index and floating-point<br>
probabilistic (“soft”) <code>target</code>s.  The former is required to have shape<br>
<code>[nBatch]</code> (no <code>nClass</code> dimension) and have values that run from <code>0</code> to<br>
<code>nClass - 1</code>, while the latter is required to have shape <code>[nBatch, nClass]</code><br>
with values that run from <code>0.0</code> to <code>1.0</code>.</p>
<p>If your <code>target</code> is a <code>Float</code> tensor, <code>CrossEntropyLoss</code> assumes that you<br>
are using a probabilistic <code>target</code> and requires it to have the same shape<br>
as your predictions (the <code>input</code> to <code>CrossEntropyLoss</code>), namely<br>
<code>[nBatch, nClass]</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

jdefriel: <p>Hi K. Frank,</p>
<p>Thanks for the reponse. I thought that same thing too, but after checking it in the debugger my prediction is HalfTensor and my target is LongTensor. I’m super confused. I went back and reran some old code that was fully functional solving a similar problem and the shapes agreed with what I’m doing now.</p>
<p>John</p>

jdefriel: <p>I ran through the debugger and it looks like it’s actually happening in my TorchMetrics accuracy function. I’ll dig in deep and get back to everyone.</p>
Clamp a Tensor dimension-wise

aliutkus: <p>Hi,</p>
<p>say <code>x</code> is a tensor with shape (M, N) while <code>min</code> and <code>max</code> have shape (M,).</p>
<p>I basically want to do:</p>
<pre><code class="lang-auto">for m in range(M):
    torch.clamp(x[m, :], min=min[m], max=max[m], out=x[m,:])
</code></pre>
<p>That would be applying <code>clamp</code> dimension-wise. Can I avoid the for loop ?</p>
<p>thanks</p>

aliutkus: <p>answering my own question:</p>
<pre><code class="lang-auto">torch.min(
   torch.max(x, min[:, None]),
   max[:, None])
</code></pre>
<p>less elegant than a call to <code>clamp</code> but works fine</p>

amitabe: <p>A more elegant way would be:</p>
<pre><code class="lang-auto">x.clamp(min=min_tensor.unsqueeze(-1), max=max_tensor.unsqueeze(-1))
</code></pre>
Status of named tensors

AlbertZeyer: <p>I think <a href="https://pytorch.org/docs/stable/named_tensor.html" rel="noopener nofollow ugc">Named Tensors</a> is a very promising concept. However, I wonder about its status in PyTorch. It has been around for quite a while (a few years, not sure since when exactly). But it still says:</p>
<blockquote>
<p>WARNING: The named tensor API is a prototype feature and subject to change.</p>
</blockquote>
<p>Why is this still the case? Why is it not stable by then? Has it actually ever changed? What are the plans?</p>
<p>And also, I just tried to use it, and for a very primitive op like <code>tensor.permute</code>, I got:</p>
<blockquote>
<p>RuntimeError: aten::permute is not yet supported with named tensors. Please drop names via <code>tensor = tensor.rename(None)</code>, call the op with an unnamed tensor, and set names on the result of the operation.</p>
</blockquote>
<p>(Feature request <a href="https://github.com/pytorch/pytorch/issues/94586" rel="noopener nofollow ugc">here</a>.)</p>
<p>So I wonder, how complete is this support really? What other low-level ops are not supported? When is support for that planned? I know the list <a href="https://pytorch.org/docs/stable/name_inference.html#name-inference-reference-doc" rel="noopener nofollow ugc">here</a> but I’m not sure how much percentage that really is, and what is actually missing.</p>
<p>When searching here for “named tensors”, I don’t see too much activity. <a href="https://discuss.pytorch.org/t/collating-named-tensors/78650">Here</a> is one, also stumbling upon many unsupported primitives, like <code>stack</code>, or <code>pin_memory</code>.</p>
<p>And also, when reading through public PyTorch code, I rarely see named tensors in use. Maybe because its support is very incomplete?</p>
<p>And also, the type information on functions like <code>torch.tensor</code> do not cover the <code>names</code> argument, so editors show warnings about unexpected arguments.</p>
<p>I would like to use it, however, now I’m reconsidering this, when I have to add ugly workaround code for every second op, function or module which does not properly support it.</p>

ptrblck: <p>Based on <a href="https://github.com/pytorch/pytorch/issues/60832">this response</a> from June 2021 it seems the development stopped. I don’t know what the current plan is and if the current implementation will be deprecated in favor of another approach or eventually dropped.</p>

remisphere: <p>You might want to take a look at the <a href="https://github.com/arogozhnikov/einops" rel="noopener nofollow ugc">einops</a> library,<br>
that implements this kind of semantics, but without the name propagation stuff (meaning it does not make the tensors track their dimensions’ names, only operations are aware of them).</p>
Convert Pandas DataFrame to List[Tensor[L, 4]] or Tensor[K, 5]

MikeTensor: <p>Hello!</p>
<p>I try to convert my Pandas DataFrame (BoundingBoxes) to a List of Tensors, or one single Tensor<br>
After conversion it should look like:</p>
<p><strong>(Tensor[K, 5] or List[Tensor[L, 4]]).</strong></p>
<p>As described at <a href="https://pytorch.org/vision/main/generated/torchvision.ops.roi_align.html#torchvision.ops.roi_align" rel="noopener nofollow ugc">roi_align</a></p>
<p><code>bboxes_tensor = torch.tensor([df.bbox], dtype=torch.float)</code> doesn’t work with <code>roi_align</code>.</p>
<p>Any idea how to get the conversion done?</p>
<p>Example of how the DataFrame Bboxes of a random image could look like.<br>
The amount of bboxes is varrying.</p>
<p>print(df.bbox)</p>
<p>0      [1056.16, 190.5, 1241.0, 374.0]<br>
1       [359.43, 179.3, 516.3, 270.97]<br>
2      [444.13, 163.92, 559.2, 247.04]<br>
3    [870.59, 115.22, 1101.06, 213.85]<br>
4      [724.49, 171.09, 790.5, 215.13]<br>
5      [534.25, 166.2, 596.08, 216.76]<br>
6      [709.81, 162.43, 765.7, 205.68]<br>
7     [562.02, 166.42, 605.57, 206.89]<br>
8      [600.19, 163.56, 633.57, 188.6]<br>
Name: bbox, dtype: object</p>

ptrblck: <p>I would guess <code>tensor = torch.from_numpy(df.bbox.to_numpy())</code> might work assuming your <code>pd.DataFrame</code> can be expressed as a numpy array.</p>

MikeTensor: <p>Thanks for your reply <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Unfortunately it doesn’t work:<br>
<code>TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.</code></p>
<p>But if somebody is facing the same issue; I found a (messy) workaround:</p>
<pre><code class="lang-auto">df['batch_index'] = 0
bboxes_batch = torch.tensor(df.batch_index, dtype=torch.float) # 1 is BBoxes
bboxes_tensor = torch.tensor(df.bbox, dtype=torch.float) # 1 is BBoxes
rois = torch.cat((bboxes_batch.unsqueeze(1), bboxes_tensor), 1)
</code></pre>
<p>This works for me and <code>roi_align</code></p>
<pre><code class="lang-auto">tensor([[   0.0000, 1056.1600,  190.5000, 1241.0000,  374.0000],
        [   0.0000,  359.4300,  179.3000,  516.3000,  270.9700],
        [   0.0000,  444.1300,  163.9200,  559.2000,  247.0400],
        [   0.0000,  870.5900,  115.2200, 1101.0601,  213.8500],
        [   0.0000,  724.4900,  171.0900,  790.5000,  215.1300],
        [   0.0000,  534.2500,  166.2000,  596.0800,  216.7600],
        [   0.0000,  709.8100,  162.4300,  765.7000,  205.6800],
        [   0.0000,  562.0200,  166.4200,  605.5700,  206.8900],
        [   0.0000,  600.1900,  163.5600,  633.5700,  188.6000]])

</code></pre>
<p>I add a coloumn of Zeros to the pd.Dataframe<br>
Make a tensor out of the pd.bboxes, a tensor out of the pseudo pd.batch_index and glue (cat) both tensors together.</p>
<p>Pretty sure this is not super elegant, but as for now it works.<br>
If somebody knows a smoother way feel free to commet.</p>
<p>Only works with <code>roi_align</code> if batch-size is one. Otherwise the first index has to be the batch_index (0, 1, …)</p>
Computational asymptotic complexity of Torch's main functions?

marc-gav: <p>Hello community,</p>
<p>I’m trying to optimize some PyTorch code and I’m interested in comparing the computational asymptotic complexity of different PyTorch functions. For example, I would like to compare the complexity of <code>torch.cat</code> vs <code>torch.stack</code>. I am talking about the most frequent functions that are present in let’s say 90% of torch use cases. Does anyone know of a chart or resource that already provides this information?</p>
<p>Alternatively, I would be interested in creating such a chart myself. Does anyone know of a platform where I could upload the chart so that others could access it and potentially update it in the future if there are changes to the PyTorch codebase?</p>
<p>Thank you in advance for any help or suggestions!</p>
Linear Layer accepts tensor, but gets tuple

setsailforfail: <p>I wanted to play around with RNN’s a bit, and was curious if LSTM could predict the next 2 number given a single number, e.g.</p>
<pre><code class="lang-auto">x = [1]
y = [2, 3]
</code></pre>
<p>For this, I tried the following:</p>
<pre><code class="lang-auto">def create_training_data(n):

    x = torch.randint(0, 200, size=(n, 1), dtype=torch.float32, requires_grad=True)
    y =  torch.cat([x+1, x+2], dim=1)

    return x, y 

x, y = create_training_data(1000)

model= nn.Sequential(
    nn.Linear(1, 10),
    nn.ReLU(),
    nn.LSTM(input_size=10, hidden_size=10),
    nn.Linear(10, 2)
)

loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
train = data_utils.TensorDataset(x, y)
train_dataloder = DataLoader(train, batch_size=64, shuffle=True)

for batch_idx, (x, y) in enumerate(train_dataloder):
    res = model(x)
</code></pre>
<p>However, I get the error:</p>
<p><code>TypeError: linear(): argument 'input' (position 1) must be Tensor, not tuple</code></p>
<p>I do not understand why it is a tuple though, it should clearly be a tensor. What am I doing wrong?</p>

ptrblck: <p><code>nn.LSTM</code> returns multiple outputs as described in <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM">the docs</a>, while the next <code>nn.Linear</code> layer expects a single tensor input.<br>
You could either write a custom <code>nn.Module</code> for the entire model or only for the <code>nn.LSTM</code> module returning the desired output tensor.</p>
How to obtain the shape of the output tensor from each step in a Pytorch pretrained model

krishnab: <p>So I am trying to use the Pytorch implementation of the VGG16 model. In other words, I downloaded the model and waits through the usual Pytorch API.</p>
<p>I need to make some changes at the bottom of the pre-trained model, to accommodate my own data. <strong>I was wondering if there is a way to “step through” the model with some random data, so that I can check the sizes of the input and output tensors for each layer?</strong> Now I could do a <code>print(model)</code> and see the summary information for each layer, and that is fine. But the summary info does not always match up exactly with the tensors coming into that layer–because the layer will automatically adjust itself for different batch sizes or other settings.</p>
<p>Hence, I wanted to see if I could just generate some data and step through a pretrained model. Is there a way to do this, using <code>pdb</code> or something? I guess part of my confusion is I could not find in the documentation the structure of these model objects.</p>
<p>Thanks.</p>

Mahmoud_Abdelkhalek: <p>You could step into the forward pass like this:</p>
<pre><code class="lang-auto">x = torch.randn((1,3,28,28))
y = pretrained_model(x)
</code></pre>
<p>Make sure you step into <code>y = pretrained_model(x)</code> using <code>pdb</code>. Eventually, you will reach the <code>forward</code> method of the pre-trained model’s class, after which you can probe the different tensor shapes.</p>
<p>Also, see  the VGG class’s <code>forward</code> method <a href="https://pytorch.org/docs/stable/_modules/torchvision/models/vgg.html" rel="nofollow noopener">here</a>.</p>
<p>Hope this helps.</p>

krishnab: <p><a class="mention" href="/u/mahmoud_abdelkhalek">@Mahmoud_Abdelkhalek</a> Ahh yes, that make sense. I get you now. That is great, I can just step into the model up till the forward() method, and then track from there. Thanks so much. Haha, that was easier than I thought. I was probably overthinking it, figuring that it would be harder than that. But what you suggested is easy enough. Thanks again :).</p>

Nazar_Puriy: <p>from torchinfo import summary<br>
summary(model_0, input_size=[1, 3, 64, 64])</p>
<p>This outputs a very nice summary of all that is happening in your model.</p>