akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The essence of learnable positional embedding? Does embedding improve outcomes better?

GoodManWEN: <p>Hi all, I was recently reading the bert source code from the hugging face project. I noticed that the so-called “learnable position encoding” seems to refer to a specific nn.Parameter layer when it comes to implementation.</p>
<pre><code class="lang-python">def __init__(self):
    super()
    positional_encoding = nn.Parameter()
def forward(self, x):
    x += positional_encoding
</code></pre>
<p>↑ Could be this feeling, then performed the learnable position encoding. Whether that means it’s that simple or not, I’m not sure I understand it correctly, I want to ask someone with experience.</p>
<p>In addition, I noticed a classic bert structure whose location is actually coded only once at the initial input. Does this mean that the subsequent bert layers, for each other, lose the ability to capture location information?</p>
<pre><code class="lang-python">BertModel(
  (embeddings): BertEmbeddings(
    (word_embeddings): Embedding(30522, 768, padding_idx=0)
    (position_embeddings): Embedding(512, 768)
    (token_type_embeddings): Embedding(2, 768)
    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0.1, inplace=False)
  )
  (encoder): BertEncoder(
    (layer): ModuleList(
      (0): BertLayer(...)
      ...
  (pooler): BertPooler(...)
</code></pre>
<p>Would I get better results if the results of the previous layer were re-positional encoded before the next bert layer?</p>
<p>Thanks!</p>

JuanFMontesinos: <p>Well I don’t know the case for bert specifically. The positional encoding is a kind of information you pass at the beginning. Once that’s done, subsequent layers can manage that info to make use of it in an optimal way. So yes, subsequent layers are aware of the position.</p>
<p>I don’t understand the question about the learnable one. Indeed, you just pass sum a signal which is nothing but learnable floats.</p>
Encoding a large dataset with sentence BERT + loading the saved embeddings to TensorDataset?

st-vincent1: <p>I need to encode a large (&gt;10M sentences) file with BERT (sentence embeddings), save it to a file, then load it into a torch.Dataset that will allow indexing in the training framework for my model.</p>
<p>I’m running into trouble in two places:</p>
<ol>
<li>Run out of GPU memory when encoding the file with BERT to create one massive tensor that will get saved to a pickle</li>
<li>Not sure how to load data from a <code>.pkl</code> file to a <code>TensorDataset</code> (ideally using a mechanism which avoids loading the entire <code>.pkl</code> into memory to create the Dataset).</li>
</ol>
<p>Is there anything I can do better? Alternative ways of saving the tensors, a different <code>Dataset</code>? Thanks!</p>
Multi class segmentation training with images where not all classes are annotated

toobeeh: <p>Hey there!<br>
I have a trained segmentation model for multiple classes (like cat, dog, mouse).<br>
Say, I get a new dataset with images where the classes cat and dog are annotated, but the others aren’t - so there may be an image that has a cat, a dog and a mouse, but only cat &amp; dog are annotated.<br>
I want to train the model further with this dataset.</p>
<p>From my understanding, the model which is already trained for all classes, will still predict the mouse but receive negative feedback because in the mask is no mouse annotated.<br>
This results in inaccurate training, if I got that right.</p>
<p>How can I solve this?</p>

knitemblazor: <p>Hi, yes i suppose you will have to use the trained model to make predictions and use the predictions as annotations and correct these annotations before training again.</p>

toobeeh: <p>Sorry for not being clear; I dont want to predict the images in the dataset but use them to train the model.</p>
<p>To be precise, I have a cross-trained DeepLabV3 model that I cross-trained with a dataset with multiple classes.<br>
Now I want to keep training the model with another dataset, but my problem is that in the images of the new dataset, not all classes that my model is trained for are annotated.<br>
I want to use this dataset anyway for training, without making results for the classes that are not annotated worse.</p>
<p>I hope it’s more clear now!</p>

knitemblazor: <p>As it is a supervised training i think the ground truth is required if you do not have them you will have to either annotate or go for an unsupervised approach using the features that you can obtain from your pretrained model and try some regional segmentation based on that.</p>
<p>I hope you find an answer by someone just trying to help.</p>
Request to join PyTorch slack

Jack47: <p>hey I’m pytorch developer from Sensetime, may you guys give <span class="mention">@sensetime.com</span>  domain permission to join pytorch slack? I’m leading a team to develop dtr heavily on pytorch for several months, want to join pytorch slack channels. Thanks</p>

Jack47: <p>I’m working on these two pull request, hopes to find useful dev tips on slack channles:</p>
<ol>
<li>dtr: <a href="https://github.com/pytorch/pytorch/pull/42056#issuecomment-1193470159" class="inline-onebox" rel="noopener nofollow ugc">[WIP] Dynamic Tensor Rematerialization(Checkpointing) by MarisaKirisame · Pull Request #42056 · pytorch/pytorch · GitHub</a>
</li>
<li>torch allocator: <a href="https://github.com/pytorch/pytorch/issues/82073#issuecomment-1193329735" class="inline-onebox" rel="noopener nofollow ugc">CUDACachingAllocator should be cuda memory merge/compact friendly · Issue #82073 · pytorch/pytorch · GitHub</a>
</li>
</ol>

smth: <p>invited you and allowed anyone from <a href="http://sensetime.com">sensetime.com</a> to join</p>

Jack47: <p>Great thanks, joined</p>
C++ Mask R-CNN example

kolkir: <p>I made C++ <a href="https://github.com/Kolkir/mlcpp/tree/master/mask_rcnn_pytorch" rel="nofollow noopener">implementation</a> of <a href="https://arxiv.org/abs/1703.06870" rel="nofollow noopener">Mask R-CNN</a> with PyTorch C++ frontend. The code is based on PyTorch implementations from <a href="https://github.com/multimodallearning/pytorch-mask-rcnn" rel="nofollow noopener">multimodallearning</a> and Keras implementation from <a href="https://github.com/matterport/Mask_RCNN" rel="nofollow noopener">Matterport</a> . Project was made for educational purposes and can be used as comprehensive example of PyTorch C++ frontend API. Besides regular API you will find how to: load data from MSCoco dataset, create custom layers, manage weights across language boundaries(Python to C++). Feel free to ask a questions.</p>

dambo: <p>Can not believe no-one commented on this. It is one of the most comprehensive examples of torchlib.</p>
<p>Thanks.</p>

gemm: <p>Hi,<br>
could you please tell me which weights do you use for the inference (demo) and where can one get them?<br>
Can we use the HDF5 file (.h5) as weights or do we need to convert the weights to a format accepted by c++ implementation? More info on this would be appreciated.</p>
<p>Thanks</p>

kolkir: <p>Hi, please find the instructions how to run demo on the page <a href="https://github.com/Kolkir/mlcpp/blob/master/mask_rcnn_pytorch/README.md" rel="nofollow noopener">https://github.com/Kolkir/mlcpp/blob/master/mask_rcnn_pytorch/README.md</a>, there is also a link to parameters.</p>

gemm: <p><a class="mention" href="/u/kolkir">@kolkir</a>  Thank you for the response. I tried that but got some issue with the compilation. Could you provide some inputs on how to convert the model to be read in C++? Was it using ‘jit’ torchvision?</p>

kolkir: <p>Please, take a look at this thread <a href="https://discuss.pytorch.org/t/how-to-copy-network-parameters-in-libtorch-c-api/32221" class="inline-onebox">How to copy network parameters in libtorch c++ API</a></p>

gemm: <p><a class="mention" href="/u/kolkir">@kolkir</a> okay, thank you for link.</p>

Pavel_Skribtsov: <p>Hi, Kirill - on your note that it’s only possible to train using cocoa dataset. Are you talking about format of data? Or in principle it’s only possible to train using a fixed, predefined number of datasets? What if I create my own dataset representing a particular segmentation problem and will convert it to cocoa format, will that work?</p>

kolkir: <p>Hi <a class="mention" href="/u/pavel_skribtsov">@Pavel_Skribtsov</a>, yes you can use your own data, just convert it into the MS Coco format.</p>

Mircea_Paul_Muresan: <p>Hi Kirill, what running time did you obtain for the C++ version?</p>

kolkir: <p>Hi, there is no visible performance difference, because python API of PyTorch uses the same internal C++ implementation. The use of C++ makes sense when you need to deploy your model to some specific device, or you can get some benefits from a data pipeline (for example to eliminate marshaling or use some system specific high-performance data-processing library)</p>
Apply 2D attention mask on 3D hidden_states

Almogg: <p>Hi,</p>
<p>I have a <code>hidden_state</code> tuple of tensors received from model’s <code>output</code> (hugging-face model), where each tensor has shape: <code>[batch_size, seq_len, hid_size].</code></p>
<p>I also have <code>attention_mask</code> tensor received from batch samples (from tokenizer) with shape: <code>[batch_size, seq_len]</code>.</p>
<p>I want to apply the mask on the hidden_states such that I will only extract hidden_states’ elements where mask is True. The extraction should not influence the hid_size dimension, only the 2 first dimensions. <strong>I am looking for something like tf.boolean_mask().</strong></p>
<p>I plan to flatten the 2 first dimensions together afterwards.</p>
<p>I have managed to do:</p>
<blockquote>
<p>torch.where(mask==1, hidden_states, mask)</p>
</blockquote>
<p>but it returns output from the same dimensions, with zeros where mask is False. I need to remove the zero elements without changing the dimensions.</p>
<p>I would appreciate any help!<br>
Thanks</p>

smth: <p>just try a regular indexing.<br>
something like <code>hidden_states[mask, :]</code></p>
<p>Alternatively, you can do two chained calls to <a href="https://pytorch.org/docs/stable/generated/torch.masked_select.html?highlight=masked_#torch.masked_select">masked_select</a></p>
Cuda fails to initializes

simines: <p>I have installed PyTorch using <code>conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch</code><br>
Installation is done with no error. However, when I import torch I get an error that cuda fails to initialize. You may not have gpu.<br>
Does anyone know why this happen?</p>

knitemblazor: <p>Hi,<br>
Is your machine a linux machine?.<br>
Does your machine have an nvidia gpu?<br>
If so try nvidia-smi if it fails you will have to install nvidia drivers.</p>
<p>For a cpu machine you can set cuda false and proceed or use torch version with just cpu support</p>

simines: <p>Thank you for your reply. My OS is Windows 10. I have NVIDIA gpu. I want to install the gpu version.  Can it be related to the fact that the gpu drivers may not be up to date?</p>

ptrblck: <p>That could be the case. On my Windows laptop I’m using an older <code>452.06</code> driver and can properly run the <code>1.12.0+cu116</code> binary as described <a href="https://discuss.pytorch.org/t/torch-cuda-is-available-is-false-on-windows-when-everything-installed/157472/3">here</a>.</p>

simines: <p>Thanks. My driver version is 391.33, and gpu Quadro p5000.<br>
Do I need to have administrative access to upgrade the drivers?</p>

simines: <p>By the way, I installed using conda not pip.</p>

ptrblck: <p>Yes, I guess you need admin permissions to update the driver.</p>
How can I improve the performance of an image classification model so that the loss on the test set will reduce

daunsi: <p>I’m working on an image classification task and the metric for evaluating the model performance is the log loss I have used pretrained model like efficient-net b1 and resnet18 to do fine tunning.<br>
I have as well perform data augmentation such as gaussianblur, but the least loss I have gotten on the test set is 0.4 which is greater than the bench mark</p>
<p>Please how can I improve the model performance?</p>
<p>Thanks in advance.</p>

knitemblazor: <p>Hi<br>
there are few benchmark datasets for image classification .<br>
I would suggest you to use them and evaluate your model and compare your model with the benchmark model for that dataset.</p>
<p>Try <a href="http://paperswithcode.com" rel="noopener nofollow ugc">paperswithcode.com</a><br>
Hope it helps.</p>
Non Emprical way to find the right resize of the images

bing: <p>Hi,<br>
I am trying to find at what size I should be resizing the image without training the network.<br>
I have images ranging from (500,500) to (4k,4k).<br>
Right now, I am starting from (448,448) and then eventually will keep adding (224,224) in the images till some satisfactory convergence is observed.<br>
But I am unable to find any logical deduction on how to reach to the preferred sizing without training the network first(which I don’t want to do because I have limited compute).<br>
Thanks.</p>

knitemblazor: <p>Hi, i found a recent approach named as bag of visual words. try using the approach to build a vocab out of the images and use these narrowed down images as nodes and work on a graph convoluion network.<br>
So, that you do not need to work on entire image with so many pixels</p>
How to use CrossEntropyLoss on a transformer model with variable sequence length and constant batch size = 1?

fkoe: <p>Hello,</p>
<p>I’m trying to train a model for predicting protein properties. The model takes as input a whole protein sequence (max_seq_len = 1000), creates an embedding vector for every sequence element and then uses a linear layer to create vector with 2 elements to classify each sequence element into 2 classes. To make use of a variable sequence length and also because of gpu memory limitation, the model takes in a single protein as a batch, i.e. batch_size is always 1. The output of the model is [batch_size, sequence_length, num_classes] (e.g. [1, 376, 2] or [1, 123, 2].). The target tensor has the shape [batch_size, sequence_length] with each element of the sequence_length dimension being an integer indicating the class, i.e. 0 or 1.<br>
How do I compute the loss correclty with CrossEntropyLoss in this case? Is CrossEntropyLoss the correct loss function for this case?</p>
<p>Thank you!</p>

ptrblck: <p>Yes, <code>nn.CrossEntropyLoss</code> sounds like the correct loss function for your multi-class sequence classification. To calculate the loss you would have to <code>permute</code> the model output as this loss function expects model output in the shape <code>[batch_size, nb_classes, *]</code> (where <code>*</code> would be the <code>seq_len</code> in your example) via <code>output = output.permute(0, 2, 1)</code>.</p>
<p>Since you are dealing with a two-class classification you could also check <code>nn.BCEWithLogitsLoss</code> but would need to make sure the model outputs a single logit for each sample in the sequence.</p>
Loading PyTorch checkpoint without extension

Tm4: <p>I downloaded one of the PyTorch checkpoints from <a href="https://github.com/google-research/scenic/tree/main/scenic/projects/vivit" rel="noopener nofollow ugc">ViViT</a> on github but it has no extension. When I want to load it I face this error:</p>
<p><code>unpickling stack underflow</code></p>
<p>Another time I add .pth extension to the file but I received the same error.  Also, I check the repository codes but it is not clear how they saved the checkpoints. How can I use and load this checkpoint.</p>

ptrblck: <p>It seems the repository uses <a href="https://github.com/deepmind/dmvr/tree/adf78b7a8edd4bb56c6ce1bffaa5003019761716"><code>dmvr</code></a> which is based on TensorFlow.<br>
The repository gives you some information about the checkpoints:</p>
<blockquote>
<p>The following table contains pretrained ViViT models trained on various datasets. Checkpoints are provided as Scenic checkpoints compatible with <a href="https://github.com/google/flax">Flax</a>, and also as <a href="https://www.tensorflow.org/guide/saved_model">Tensorflow SavedModels</a> for inference.</p>
</blockquote>
What formula is used for F.cross_entropy vs F.binary_cross_entropy vs F.with_logits

Mahdi_Amrollahi: <p>How can I know the difference between these three cross-entropies functions?<br>
How can I know the math formula of them?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a59573510ca516ff18ab8c1a7c47871580c3b64.png" data-download-href="https://discuss.pytorch.org/uploads/default/6a59573510ca516ff18ab8c1a7c47871580c3b64" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6a59573510ca516ff18ab8c1a7c47871580c3b64_2_656x500.png" alt="image" data-base62-sha1="faO1rHlTZ9dovwjaDgrbBPj5zms" width="656" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6a59573510ca516ff18ab8c1a7c47871580c3b64_2_656x500.png, https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a59573510ca516ff18ab8c1a7c47871580c3b64.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a59573510ca516ff18ab8c1a7c47871580c3b64.png 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">888×676 68.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.cross_entropy.html#torch.nn.functional.cross_entropy">docs</a> of these functional API calls should refer to the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss">class docs</a> which show you the used formula.</p>
Errors are reported when I tried to save model for C++ libtorch inference

Semmer: <p>Hi everyone,<br>
I have some issue in trying to save pytorch model for C++ libtorch inference by torch.jit.sctipt().<br>
First, the reported errors looks like</p>
<blockquote>
<p>Module ‘ConvModule’ has no attribute ‘padding_layer’</p>
</blockquote>
<p>And I found it was casued by uninitialized model variables. And those variables are initialized in its’ <strong>init</strong>() func. So I added <span class="mention">@torch.jit.export</span> key word for import <strong>init</strong>() to C++ model explicitly, and this issue seems fixed. Then it comes with another issue:</p>
<blockquote>
<p>Python type cannot be used as a value:<br>
File “/xxxx/conv_module.py”, line 87<br>
padding_mode=‘zeros’,<br>
order=(‘conv’, ‘norm’, ‘act’)):<br>
super(ConvModule, self).<strong>init</strong>()<br>
~~~~~~~~~~ &lt;— HERE</p>
</blockquote>
<p>I have no idea how to solve such issue. I did not  find any useful information online. Am I doing this right at first step? I’m sorry for not familiar with pytorch or torch.jit, so I just simply apply those solutions I found.<br>
I just know the torch.jit.sctipt would save model for C++ inference statically, so it cannot handle some if clause if the varibale type was unclear. Would it be the root cause which led to so many issues?</p>
<p>Please, any information would be appreciated, or just let me know if more info is necessary. Thank you.</p>

ptrblck: <p>Could you try to remove the input arguments to <code>super()</code> (it’s not needed in Python 3.x anymore) and call <code>super().__init__()</code>?</p>
PyTorch and torch_sparse were compiled with different CUDA versions

Mojtaba_z: <p>Hi. I want to run a python program but there is an error in pytorch:</p>
<blockquote>
<p>RuntimeError: Detected that PyTorch and torch_sparse were compiled with different CUDA versions. PyTorch has CUDA version 10.2 and torch_sparse has CUDA version 11.4. Please reinstall the torch_sparse that matches your PyTorch install.</p>
</blockquote>
<p>How to fix it?</p>

InnovArul: <p>From <code>torch-sparse</code> documentation, It seems there is an option to install the library for specific version of CUDA. Did you try that?</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pypi.org/project/torch-sparse/">
  <header class="source">
      <img src="https://pypi.org/static/images/favicon.6a76275d.ico" class="site-icon" width="32" height="30">

      <a href="https://pypi.org/project/torch-sparse/" target="_blank" rel="noopener">PyPI</a>
  </header>

  <article class="onebox-body">
    <img src="https://pypi.org/static/images/twitter.6fecba6f.jpg" class="thumbnail onebox-avatar" width="300" height="300">

<h3><a href="https://pypi.org/project/torch-sparse/" target="_blank" rel="noopener">torch-sparse</a></h3>

  <p>PyTorch Extension Library of Optimized Autograd Sparse Matrix Operations</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>For example, if you have torch 1.11, you could try:</p>
<pre><code class="lang-auto">pip install torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-1.11.0+cu102.html
</code></pre>

Mojtaba_z: <p>I tested it but it doesn’t work</p>

InnovArul: <p>Can you try a clean install or try to uninstall and then install?<br>
I am able to follow the procedure and install it for CUDA 11.3 &amp; Pytorch 1.11.</p>
<aside class="quote no-group" data-username="Mojtaba_z" data-post="3" data-topic="157433" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mojtaba_z/48/46521_2.png" class="avatar"> Mojtaba_z:</div>
<blockquote>
<p>I tested it but it doesn’t work</p>
</blockquote>
</aside>
<p>Also, In this case, it will be great if you could mention some more details such as the error message, the pytorch version, and how did you install torch (source / conda/ pip ) to know more context?</p>
When to use src_mask in nn.TransformerEncoderLayer

yipliu: <p>TransformerEncoderLayer</p>
<p><strong>scr_mask</strong>: is used to block specific positions from attention (feature tokens)</p>
<p><strong>src_key_padding_mask</strong> : is used to block attending to PAD tokens.</p>
<p>However, I’m still not sure if I need to use it or not in TransformerEncoderLayer. The following are my guesses：</p>
<ol>
<li>
<p>In the Language Model task: We need to generate the next word. The new word will be used to infer the feature word. we need to use src_mask</p>
</li>
<li>
<p>In the seq2seq task (Machine Translation): We need to generate a sequence. We do not need to use src_mask</p>
</li>
</ol>
Best practices for training model X on pre-trained embeddings of model Y?

st-vincent1: <p>I’d like to train a model <code>X(input, output)</code> which is a sequence-to-sequence model, but <code>input</code> is a list of pre-trained embedding vectors <code>(number_of_embeddings x embedding_dim)</code>, and <code>output</code> is a sentence. The input is originally a sentence, that gets embedded with an external BERT model <code>Y</code>.</p>
<p>I’m running into trouble with loading the embeddings.</p>
<p>So far I’ve tried the following:</p>
<ol>
<li>Embed all input with Y a priori, save to a pickle, then load as training dataset for model <code>X</code>;</li>
<li>Embed input samples/batches on the fly.</li>
</ol>
<p>Method 1. works for small datasets but as my input can reach tens of millions of sentences I run out of memory for storing the entire tensor (<code>embedding_dim</code> is 768). Method 2. works when samples are embedded on cpu (I’m using <a href="https://github.com/facebookresearch/fairseq" rel="noopener nofollow ugc">fairseq</a> where data pre-processing is done on cpu). However, this is pretty slow and is going to create a huge bottleneck. I tried doing this on GPU instead: raw text loaded into CPU, BERT model loaded into GPU, and then batch by batch samples moved to GPU, tokenized, embedded, and moved back to CPU. However, this throws the following error:</p>
<p><code>RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method</code></p>
<p>For BERT, I’m using <code>DistillBertTokenizer</code> and <code>DistillBertModel</code> from <code>huggingface/transformers</code>.<br>
Is there a better/more efficient way to do this? Thanks in advance!</p>
FCN with spatial encoding

Roua_Rouatbi: <p>I made images with spots inside of each. For each image, there is a specific distance.<br>
I want to classify my images based on these distances ( I have 10 classes = 10 distances, and thousands of images).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/0792ec57f14b01c57ecd9f6a1599609d68e2dfb7.png" alt="8" data-base62-sha1="1507G10HCCLmSzMpObAOPvJnH4r" width="186" height="244"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbb88ea4d0ad35fba78d68aa95df917fb9f4a20f.png" alt="12" data-base62-sha1="t4ckZ87VpsWiF14xW3lAROwrK4T" width="303" height="318"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dcacd2ac2a3dbac8b23aed9f036612a52cd73a40.png" alt="4" data-base62-sha1="vubjwCNmWt50QAGNzXUjQMIMzwQ" width="287" height="305"></p>
<p>I was thinking of a CNN with some spatial encoding that would then connect to an FCN with the distance function as a loss function. But i feel like it’s not that good of an idea. Are there perhaps some models that can already do this, or perhaps some clustering methods?</p>

knitemblazor: <p>Hi just a suggestion if only the relative spots are relevant you can use graph based approaches like graphconv networks from pytorch geometric</p>

Roua_Rouatbi: <p>Just to see if I understood what you meant correctly. I would have to first detect the spots positions and feed that to the graphconv and classify based on that?</p>

knitemblazor: <p>yes build a graph with spot positions as nodes  and feed the formed graph as input you can even use the relative distances as weight of edges. use this graph as input.</p>
<p>And for getting the spots you can use opencv based contour detection approaches</p>
Implementation of complex-valued activation functions

SantaTitular: <p>I’m working with complex-valued activation functions and, similarly to discussions at <a href="https://github.com/pytorch/pytorch/issues/47052" rel="noopener nofollow ugc">#47052</a>, I am interested in knowing what type of function does the ReLu (and others such as tanh, sigmoid, etc. <a href="https://complex-valued-neural-networks.readthedocs.io/en/latest/activations/etf.html" rel="noopener nofollow ugc">Complex Activation Functions</a>) function. I looked into the source code available on pytorch and did not find a concrete answer. Any ideas where I can find more information?</p>
Simple 3D Mask-RCNN Implementation

M2M: <p>Hello everyone,</p>
<p>I am wondering if there is a simple 3D Mask-RCNN code? Also, is there trained models that I can simply just import like ResNet. Please note that I need 3D not 2D.</p>
<p>Any other state-of-the-art 3D semantic segmentation/Instance segmentation models?</p>
<p>Thank you very much.</p>

KFrank: <p>Hi Rick!</p>
<aside class="quote no-group" data-username="M2M" data-post="1" data-topic="157189" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/m2m/48/33611_2.png" class="avatar"> M2M:</div>
<blockquote>
<p>I am wondering if there is a simple 3D Mask-RCNN code?</p>
</blockquote>
</aside>
<p>I am not aware of any pre-packaged (or pre-trained) 3D Mask-RCNN<br>
implementations.</p>
<aside class="quote no-group">
<blockquote>
<p>Any other state-of-the-art 3D semantic segmentation/Instance segmentation models?</p>
</blockquote>
</aside>
<p>A search will lead you to a number of pytorch 3D U-Net implementations.<br>
(I don’t know which, if any, of them are any good.)</p>
<p>Be aware that U-Net performs semantic (not instance) segmentation.<br>
(People do get good instance segmentation from U-Net in some use<br>
cases by post-processing (e.g., running connected components) the<br>
results of U-Net’s semantic segmentation.)</p>
<p>Note, for 3D images of any size, any such 3D model will quickly become<br>
resource hungry, both in terms of memory and computation time.</p>
<p>Best.</p>
<p>K. Frank</p>

M2M: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="157189">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>e, for 3D images of any size, any such 3D model will quickly become<br>
resource hungry, both in terms of memory and computation time.</p>
</blockquote>
</aside>
<p>Hi Frank,</p>
<p>Thank you very much. I have tried U-nets for Semantic segmentation and now I am looking for something different to U-net to see if it would make any difference. And you are absolutely correct about resource problem, it is the most difficult part dealing with 3D-Unets for me.</p>
Gradients of specific layer during backprop

demalenk: <p>Is it possible to access the gradient update of a specific layer during standard backprop?</p>
<p>I have tried setting layer.weight.retain_grad() for the layer of interest and then calling layer.weight.grad, but as I understand it this gives only the gradient of the layer parameters with respect to activations (dh_i / dtheta_i), but I am interested in getting the gradient of the loss with respect to layer activations (dL/dh_i). More specifically, if the update rule for the parameters of a given layer is:<br>
<strong>theta_i = theta_i - (dL / dh_i) * (dh_i / dtheta_i)</strong><br>
then is it possible to get the values of <strong>(dL / dh_i)</strong> that are backpropagated during training?</p>
Can I create a Generative Adversarial Network for text using the logits without argmax?

mlmaster: <p>There are two main problems when creating Generative Adversarial Networks for text:</p>
<ol>
<li>The discrete token values are not differentiable after applying argmax.</li>
<li>The language architecture may generate sequences with different lengths so the discriminator should be able to work with them either with padding or with some representation of the whole sequence.</li>
</ol>
<p>I was wondering if is it possible to adversarially train a model such as <a href="https://huggingface.co/docs/transformers/model_doc/t5" rel="noopener nofollow ugc">T5</a>. Its decoder produces a sequence with shape <code>[batch_size, seq_len, model_dim]</code> and then it is usually passed through a linear layer to get <code>[batch_size, seq_len, vocab_size]</code> logits. We can apply a softmax across the <code>vocab_size</code> dimension and then these probabilites can go to a discriminator. For the ground truth labels <code>[batch_size, seq_len]</code> we can generate one-hot vectors <code>[batch_size, seq_len, vocab_size]</code> and then apply the softmax as well. This will be sufficient for the discriminator to learn but since we do not apply argmax to the tokens from the generator, gradients should be able to reach it as well.</p>
<p>For the second problem, I was thinking of using a mean function to transform <code>[batch_size, seq_len, vocab_size]</code> probabilities to a “sequence representation” <code>[batch_size, vocab_size]</code>, but I am not sure if this makes sense.</p>
<p>So based on that, is <strong>1</strong> feasible and if not - why? What are some suggestions for solving <strong>2</strong>?</p>
Dataset with randomness for DDP

sqiangcao: <p>In my case, the dataset is sampled from all data. Therefore, each instantiated Dataset contains different data. And, the number of samples included in each process is difference, which can cause training hang at certain epochs.</p>
<pre><code class="lang-auto">local_rank: 0, num_samples: 5024
local_rank: 1, num_samples: 5026
</code></pre>
<p>And here is my code generating the dataloader:</p>
<pre><code class="lang-auto">def build_data_loader(cfg, args, phase):
    sampler = None

    if cfg.DDP.ENABLE: 
        print(f"local rank {args.local_rank} successfully build {phase} dataset")    
        num_tasks = dist.get_world_size()
        global_rank = dist.get_rank()
        with torch_distributed_zero_first(args.local_rank):
            dataset = build_dataset(cfg, phase)
    else:
        dataset = build_dataset(cfg, phase)
    
    if cfg.DDP.ENABLE:
        if phase == 'train':
            sampler = torch.utils.data.DistributedSampler(dataset, num_replicas=num_tasks, rank=global_rank)
        else:
            sampler = torch.utils.data.DistributedSampler(dataset, num_replicas=num_tasks, rank=global_rank)

    data_loader = data.DataLoader(
        dataset=dataset,
        batch_size=cfg.DATA_LOADER.BATCH_SIZE,
        shuffle=True if phase == 'train' and sampler == None else False,
        num_workers=cfg.DATA_LOADER.NUM_WORKERS,
        pin_memory=cfg.DATA_LOADER.PIN_MEMORY,
        sampler=sampler, 
        drop_last = phase =='train', 
  
    )

    return data_loader
</code></pre>

justanhduc: <p>Hey <a class="mention" href="/u/sqiangcao">@sqiangcao</a>. To avoid hanging, you may try to add <code>torch.distributed.barrier</code> at the end of each iteration or epoch to avoid imbalanced load between processes. Also, I am curious why there can be different number of samples as <code>DistributedSampler</code> uses padding to make the load balanced. You might wanna double check this.</p>

sqiangcao: <p>@ <a href="https://discuss.pytorch.org/u/justanhduc">justanhduc</a>, thank you for your answer. I found the length of the dataset in rank 0 and rank 1 is different. This may be the reason for hanging.<br>
So I try,</p>
<pre><code class="lang-auto">        ds_m = [None for _ in range(args.world_size)]
        dist.all_gather_object(ds_m, dataset)
        dataset = ds_m[0]
</code></pre>
<p>and the problem is solved. Is there a better way?</p>
Dataset vs DataLoader

Sarah125: <p>Consider the following code:</p>
<pre><code>         class Cifar10Data(Dataset):
                def __init__(self, data_dir, data_size = 0, transforms = None):
                        files = os.listdir(data_dir)
                        files = [os.path.join(data_dir,x) for x in files]
                        
                        if data_size &lt; 0 or data_size &gt; len(files):
                            assert("Data size should be between 0 to number of files in the dataset")
                        if data_size == 0:
                            data_size = len(files)
                        
                        self.data_size = data_size
                        self.files = np.random.choice(files, self.data_size)
                        self.transforms = transforms
                
                    def __len__(self):
                        return self.data_size
                
                    def __getitem__(self,index):
                        img_loc = self.files[index]
                        img = Image.open(img_loc)
                        img = preprocess(img)
                        #extract label from file_name
                        label_name = img_loc[:-4].split("_")[-1]
                        label = label_mapping[label_name]
                
                        img = img.astype(np.float32)
                
                        if self.transforms:
                            img = self.transforms(img)
                
                        return img, label
                
                
                
                
                trainset = Cifar10Data(data_dir = "cifar/train/", transforms=None)
                trainloader = DataLoader(trainset, batch_size=128, shuffle=True)
</code></pre>
<p>The Dataloader helps to facilitate the batching of the data and so on. It expects a a <code>Dataset</code> object as input.<br>
Since I can specify the batch_size to be 128, the data_size attribute of the class Cifar10Data is not useful anymore. I do not quite know what happens. What does  the data_size attribute do in this context?</p>

JuanFMontesinos: <p>It’s a parameter to select the amount of samples to use to construct the dataset class.</p>

Sarah125: <p>I see. So the batchsize depends on size of my dataset given by data_size.</p>
Block Diagonal in a 5D tensor

etc: <p>I have a vector A of [16,6,3,27,27] dimension and I want to create a block diagonal of [16,6, 81,81] efficiently using A matrix.</p>
<p><code>torch.</code> <code>block_diag</code> cannot be used because my vector has more than 2 dimensions.<br>
Do you know how can this be implemented?<br>
Thank you in advance.</p>

KFrank: <p>Hi etc!</p>
<aside class="quote no-group" data-username="etc" data-post="1" data-topic="157325" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/etc/48/50711_2.png" class="avatar"> etc:</div>
<blockquote>
<p>I have a vector A of [16,6,3,27,27] dimension and I want to create a block diagonal of [16,6, 81,81] efficiently using A matrix.</p>
</blockquote>
</aside>
<p>If you create the appropriate index tensors, you can do this with tensor<br>
indexing:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.10.2
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; A = torch.randn (16, 6, 3, 27, 27)
&gt;&gt;&gt;
&gt;&gt;&gt; B = torch.zeros (16, 6, 81, 81)
&gt;&gt;&gt;
&gt;&gt;&gt; i0 = torch.arange (16).expand (6, 3, 27, 27, 16).permute (4, 0, 1, 2, 3)
&gt;&gt;&gt; i1 = torch.arange (6).expand (16, 3, 27, 27, 6).permute (0, 4, 1, 2, 3)
&gt;&gt;&gt; i2 = torch.arange (81).expand (16, 6, 27, 81).transpose (2, 3).view (16, 6, 3, 27, 27)
&gt;&gt;&gt; i3 = i2.transpose (3, 4)
&gt;&gt;&gt;
&gt;&gt;&gt; B[i0, i1, i2, i3] = A
&gt;&gt;&gt;
&gt;&gt;&gt; torch.equal (A[0, 0, 0], B[0, 0, 0:27, 0:27])
True
&gt;&gt;&gt; torch.equal (A[7, 2, 1], B[7, 2, 27:54, 27:54])
True
&gt;&gt;&gt; torch.equal (A[11, 4, 2], B[11, 4, 54:81, 54:81])
True
&gt;&gt;&gt;
&gt;&gt;&gt; torch.all (B[:, :, 0:27, 27:81] == 0)
tensor(True)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Why transformer DataCollatorForSeq2Seq() Takes model as a parameter?

sandeep1: <p>Hi,</p>
<p>For traning of any seq2seq model say Bart :-</p>
<pre><code class="lang-auto">from transformers import DataCollatorForSeq2Seq
data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)

trainer = Seq2SeqTrainer(
    model,
    args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)
</code></pre>
<p>Why  DataCollatorForSeq2Seq Takes model as parameter when it is also provided to Seq2SeqTrainer.<br>
What exactly it does with the tokenizer and model as input?</p>
<p>I appreciate your help.</p>
<p>Thank You.</p>

ptrblck: <p>From the <a href="https://huggingface.co/docs/transformers/main_classes/data_collator#transformers.DataCollatorForSeq2Seq">docs</a>:</p>
<blockquote>
<p><strong>model</strong> (<a href="https://huggingface.co/docs/transformers/v4.20.1/en/main_classes/model#transformers.PreTrainedModel">PreTrainedModel</a>) — The model that is being trained. If set and has the <em>prepare_decoder_input_ids_from_labels</em>, use it to prepare the <em>decoder_input_ids</em><br>
This is useful when using <em>label_smoothing</em> to avoid calculating loss twice.</p>
</blockquote>
<p>I would assume it’s an optimization and apparently avoids some calculations.<br>
The <a href="https://discuss.huggingface.co/">HuggingFace discussion board</a> might be a better place for these HF-specific questions as you would find more experts there. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
How to apply same transform on a pair of picture?

Medivh: <p>I am working on stereo vision task, and I need to load a pair of picture at a time. But the vision.transform behave differently on two pictures. For example, RandomCrop get different range. Is there any easy way the apply the same transform on a pair of picture?</p>

ptrblck: <p>You could use the functional API from <code>torchvision.transforms</code>:</p>
<pre><code class="lang-auto">import torchvision.transforms.functional as TF

class YourDataset(Dataset):
    def __init__(self):
        self.image_left_paths = ...
        self.image_right_paths = ...

    def __getitem(self, index):
        image_left = # load image with index from self.left_image_paths
        image_right = # load image with index from self.right_image_paths
        # Resize
        resize = transforms.Resize(size=(520, 520))
        image_left = resize(image_left)
        image_right = resize(image_right)

        # Random crop
        i, j, h, w = transforms.RandomCrop.get_params(
            image_left, output_size=(512, 512))
        image_left = TF.crop(image_left, i, j, h, w)
        image_right = TF.crop(image_right, i, j, h, w)

        # Random horizontal flipping
        if random.random() &gt; 0.5:
            image_left = TF.hflip(image_left)
            image_right = TF.hflip(image_right)

        # Random vertical flipping
        if random.random() &gt; 0.5:
            image_left = TF.vflip(image_left)
            image_right = TF.vflip(image_right)

    image_left = TF.to_tensor(image_left)
    image_right = TF.to_tensor(image_right)
    return image_left, image_right

def __len__(self)
    return len(self.image_left_paths)
</code></pre>

Medivh: <p>It is really helpful. Thank you very much.</p>

Anton_Cheshkov: <p>Is <strong>get_params</strong> documented ?</p>

Mario_Galindo: <p>I think I have a simple solution:<br>
If the images are concatenated, the transformations are applied to all of them identically:</p>
<pre><code class="lang-auto">import torch
import torchvision.transforms as T

# Create two fake images (identical for test purposes):
image = torch.randn((3, 128, 128))
target = image.clone()

# This is the trick (concatenate the images):
both_images = torch.cat((image.unsqueeze(0), target.unsqueeze(0)),0)

# Apply the transformations to both images simultaneously:
transformed_images = T.RandomRotation(180)(both_images)

# Get the transformed images:
image_trans = transformed_images[0]
target_trans = transformed_images[1]

# Compare the transformed images:
torch.all(image_trans == target_trans).item()

&gt;&gt; True
</code></pre>
Weird behavior while evaluating using DDP

eslam_bakr: <p>Hello Everyone,</p>
<p>I am using DDP while training (4 v100 GPUs), and using distributed sampler for the training and while testing set the sampler to none.<br>
From my understanding setting the sampler to none while evaluating means the data will not be split into chunks and the four GPUs will run over the same data which means all of them will get the same results.<br>
But when I evaluate after each epoch the four GPUs give me different accuracy.<br>
Specifically, GPU 0 gives me different accuracy while the rest 3 GPUs are giving the same.</p>
<p>And after finishing the training when I load the weights then evaluate using exactly the same function, I got the same result across the 4 GPUs.</p>
<p>Thus my question is why GPU 0 gives a different accuracy while evaluating during the training. Despite I am using exactly the same function.</p>
<p>Thanks in advance.</p>

kumpera: <p>Hey,</p>
<p>For some reason, GPU 0 has different values for weights than the other ranks during training.</p>
<p>It works as you expected after loading because all devices load the same weights and don’t modify them any further, giving the results you expect.</p>
<p><a class="mention" href="/u/pritamdamania87">@pritamdamania87</a> <a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a> Do you have an idea on why rank 0 would give different results in this case?</p>

hendrikl: <p>Hi <a class="mention" href="/u/eslam_bakr">@eslam_bakr</a> , I ran into sort of the same issue. Do you already know why this is the case? My problem is that I get 4 completely different evaluation results… How do you ‘terminate’ after training?</p>

eslam_bakr: <p>I am not sure what is the root cause but for me I am using only one GPU for evaluation to overcome this issue, using the following code:</p>
<pre><code class="lang-auto">if not args.multiprocessing_distributed or (
                        args.multiprocessing_distributed and args.rank % ngpus_per_node == 0):
eval_function()
</code></pre>
Does pytorch works on radeon 600m gpu?

siddhartha045: <p>i am tryiing to work on pytorch but i have radeon 6800m graphics i not able to find any information about whether i can acess the gpu or not . Is it possible to install  Rocm or not ???</p>

AlphaBetaGamma96: <p>You can get GPU support via using ROCm. Just go to <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">getting started</a> and select the <code>ROCm</code> option rather than NVIDIA.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e5e0e9195b2e2c7f6c29ad662d14f0c6087f981.png" data-download-href="https://discuss.pytorch.org/uploads/default/2e5e0e9195b2e2c7f6c29ad662d14f0c6087f981" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e5e0e9195b2e2c7f6c29ad662d14f0c6087f981_2_690x268.png" alt="image" data-base62-sha1="6CbtJJkuYZCqkEd2HHje13lxzbz" width="690" height="268" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e5e0e9195b2e2c7f6c29ad662d14f0c6087f981_2_690x268.png, https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e5e0e9195b2e2c7f6c29ad662d14f0c6087f981.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e5e0e9195b2e2c7f6c29ad662d14f0c6087f981.png 2x" data-dominant-color="F4DAD6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">811×315 24.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
torch.fx.proxy.Proxy error

Arthur_Logan: <p>Hello. When I run the following code</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision
from torchvision import transforms
from torchvision.models.feature_extraction import create_feature_extractor

def n_alphabet(n, r, c):
    return nn.Sequential(
        nn.Linear(n, r*c), 
        nn.Unflatten(1, [1, r, c]), 
        nn.Conv2d(1, 8, 3, padding=1), 
        nn.Tanh(), 
        nn.Conv2d(8, 1, 3, padding=1),
        transforms.RandomAffine(degrees=30, translate=(0.3, 0.3), scale=(0.75, 1.25)), 
        nn.Conv2d(1, 8, 3, padding=1), 
        nn.Tanh(), 
        nn.Flatten(), 
        nn.Linear(8*r*c, n), 
        nn.Softmax(dim=1)
    )

x = n_alphabet(4, 4, 4)

net = create_feature_extractor(x, return_nodes={'4':'alphabet', '10':'latent'})
</code></pre>
<p>I get the following error:</p>
<pre><code class="lang-auto">TypeError: Unexpected type &lt;class 'torch.fx.proxy.Proxy'&gt;
</code></pre>
<p>I know it has to do with the <code>RandomAffine</code> term in the sequence. I don’t understand why though. Why is this error occurring, and what can I do to correct it?</p>
<p>Thanks</p>

AlphaBetaGamma96: <p>I think when using <code>nn.Sequential</code> all its modules must be of type <code>nn.Module</code> so when you use an object which is of a different type it throws the error. I think you’ll have to write your class out explicitly as an <code>nn.Module</code> object.</p>
Creation of custom CUDA kernels in Pytorch

Ruslan_Allayarov: <p>Hello everybody!<br>
We’re working on novel software for micromagnetic simulations.<br>
And we wanna use Pytorch for all stuff.<br>
My question is simple - is it possible to create a custom CUDA Kernel? I mean, if we wanna make some nontrivial tensor logic at only 1 kernel, how we can do it?</p>
<p>Thanks in advance,<br>
Ruslan</p>

ptrblck: <p>Yes, that’s possible and you would use a custom CUDAExtension for these use cases e.g. as explained <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">in this tutorial</a>.</p>

Ruslan_Allayarov: <p>Thanks a lot!<br>
We’ll try to use it.</p>