akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is it possible to write pure C++/CUDA mixed program without interfacing with python?

chjz1024: <p>Hi,</p>
<p>Maybe this is a strange question, but I’m using libtorch as a scientific computing library to directly interface with hardware. The ease-of-use c++ frontend really saved my life in many cases, especially when I need to use cuda to accelerate the program. However, it becomes very hard when I want to manually write some cuda processing codes to improve the performance.</p>
<p><a href="https://pytorch.org/cppdocs/notes/tensor_basics.html" class="inline-onebox" rel="noopener nofollow ugc">Tensor Basics — PyTorch master documentation</a> is a good implication about what I’d like to do, but I don’t even know how to call those cuda kernels inside my c++ code or even compile them together. I’m a newbie in CUDA programming so and advice will be of great help!</p>
<p>The most related question is from <a href="https://discuss.pytorch.org/t/libtorch-with-pure-cuda-codes-cannot-compiled/126373" class="inline-onebox">Libtorch with pure cuda codes cannot compiled</a> but it no solution is provided. My platform is ubuntu2004.</p>

ptrblck: <p>I think one valid approach would be to add custom C++ ops as described <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html">in this tutorial</a>. If I’m not mistaken, <code>torchvison</code> also uses this approach and you can take a look at their usage <a href="https://github.com/pytorch/vision/tree/460b37d252a5c9d49f508e38a14a83a9077ba68d/torchvision/csrc/ops/cuda">in these files</a>.</p>
Bi boolean statements

arya47: <p>Lets say i have a tensor X<br>
now i want to change all elements in the range (r1,r2) to y</p>
<p>i’m trying to do<br>
X[r1&lt;X and X &lt; r2] = y</p>
<p>But i getting<br>
RuntimeError: Boolean value of Tensor with more than one value is ambiguous</p>
<p>Is there a way to do this operation?</p>
<p>Thanks</p>

arya47: <p>Solved</p>
<p>X[torch.logical_and(condition, condition)] = y</p>
Loss decreasing but accuracy *on training set* not improving

carbocation: <p>Trying to ask this in a fairly architecture agnostic way. Here is my loss plot for a single-label two-class binary cross entropy problem:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/67171e6cf29a23247aedd08b20585724d25c8c6d.png" alt="image" data-base62-sha1="eHYI6tTI0xaBix7k8Jyj3Spmzet" width="372" height="248"></p>
<p>My interpretation is that I’m overfitting fairly badly. But, overfitting <em>is my goal</em> for the moment. My expectation is that if I were to apply this model <em>to the training data</em>, the model’s predictions should look good, since it has memorized the training data. However, I am not actually seeing this; instead, my model performs terribly on training data, which is surprising.</p>
<p>Am I right to be surprised that my model is performing badly on the very training data that it appears to be overfit to?</p>

my3bikaht: <p>Doesn’t look like overfitting. More like some mixup in data/labels.</p>

carbocation: <p>That is an interesting insight. Is your thought that this mixup is only happening in my training loop, so that the labels are mixed up but fixed and therefore learnable (explaining the improvement in loss during training)? And then when I am no longer in the training loop, I evaluate the model in a way where the labels are not mixed up, which is why the results don’t hold up?</p>

my3bikaht: <p>I think there’s something wrong with labels during training loop because BCELoss = -ln(1-x) if label==0 and -ln(x) if label==1. It can decrease only if your predictions are getting closer to labels you provided.</p>
<p>If bce loss is dropping to .1 it means average distance between labels and predictions is about 0.095 (solve -ln(x)=.1 for label==1 or -ln(1-x)=.1 for label==0)</p>

carbocation: <p>This makes perfect sense, thank you. I’ll dig into this and see if I can identify the problem in my training loop.</p>

carbocation: <p>After following up on your suggestion to investigate the training loop, I think I narrowed down the issue to a mal-alignment between X and Y in my dataloader. I’m still not 100% sure of the root cause, but it seems to be due to an error in my mental model of when <code>pandas</code> may reorder data. When I manually join X and Y before converting them to tensors, I can overfit as expected (same appearance as the above plot) but reassuringly I can also reproduce the overfit values outside of the training loop.</p>
Check which optimization flags have been used for compilation

mgiessing: <p>Hi, is it possible to check how a pytorch package has been compiled?</p>
<p>I’m specifically interested in the optimization intrinsics (e.g. if SSE or AVX for x86, VSX or MMA for ppc64le etc. has been enabled)</p>
<p>Thanks!</p>

ptrblck: <p>You could get some information about lib support via <code>print(torch.__config__.show())</code>.</p>

mgiessing: <p>Perfect - exactly what I was looking for. Thanks!</p>
How to increase Mask R-CNN recall

Bernd: <p>Hello PyTorchers<br>
I am currently training a PyTorch Mask R-CNN based on ResNet50 backbone.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/vision/blob/main/torchvision/models/detection/mask_rcnn.py">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/detection/mask_rcnn.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/main/torchvision/models/detection/mask_rcnn.py" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/main/torchvision/models/detection/mask_rcnn.py</a></h4>


    <pre><code class="lang-py">from collections import OrderedDict

from torch import nn
from torchvision.ops import MultiScaleRoIAlign

from ..._internally_replaced_utils import load_state_dict_from_url
from ...ops import misc as misc_nn_ops
from ..resnet import resnet50
from ._utils import overwrite_eps
from .backbone_utils import _resnet_fpn_extractor, _validate_trainable_layers
from .faster_rcnn import FasterRCNN

__all__ = [
    "MaskRCNN",
    "maskrcnn_resnet50_fpn",
]


class MaskRCNN(FasterRCNN):
    """
</code></pre>


  This file has been truncated. <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/detection/mask_rcnn.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>During my training session I noticed that the precision is very good, going up to 94%, but the recall only up to 79%. For my use case it is very important that I detect all objects, while the actual classification is less important.<br>
There are tons of parameters in the MaskRCNN base class, and I wonder if any of them could be changed to move toward a higher recall in favor of some precision.</p>
<p>I tried playing around with box_fg_iou_thresh and box_bg_iou_thresh, but without any noticable change in recall or precision.<br>
Has anyone a suggestion which parameters could be used to increase recall?</p>
<p>Regards,<br>
Bernd</p>
Create a custom dnn which incorporate pretrained alexnet and linear regression model

nikiguo93: <p>hi I use the activation feature from pretrained alexnet to train linear regression model. And now I need to create a custom deep neural network which would incoporate the pretrained alexnet and linear regression model and also a fully connected layer. Can anybody tell me how to realize that?</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html">finetuning tutorial</a> might be a good starter, as it explains how to modify pre-trained <code>torchvision</code> models for a custom task and fine tune them.</p>
Does xgboost accept pytorch tensors?

Bibhabasu_Mohapatra: <p>does xgboost accept pytorch tensors?       I wanted add xgboost regressor as last layer in FC layer!!<br>
Has anybody done it yet ?<br>
My base model is efficientnet_b2.</p>

ptrblck: <p>I don’t think <code>xgboost</code> will directly accept tensors, but would expect numpy or cupy arrays, so you could transform your tensor to one of these types.</p>
Using dataloader for both train and test sets

dararipe: <p>Hi everyone!</p>
<p>I’m trying to train an LSTM network to compare to baseline models that I have trained in the past. The problem is that I have a test set that is way too big to load into Cuda all at once for calculating my model’s loss.</p>
<p>However, I would like to use this same test set as I would like to keep it consistent with my baseline models. And I don’t really want to remove elements from my train set either for the same reason.</p>
<p>Does anyone have tips on how to do that? I’m trying to iterate through both train and test DataLoaders at once, but no success so far…</p>

ptrblck: <p>Usually you wouldn’t load the entire data into the host RAM or the GPU unless you are working with tiny datasets and are willing to use device memory for the latter case.<br>
Check the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">data loading tutorial</a> to see how to lazily load the samples.</p>
Doubt in RNN Code

121810401016_DUKKA_H: <pre><code class="lang-auto">type or paste code here
# parameters 
BATCH_SIZE = 64
N_STEPS = 28
N_INPUTS = 28
N_NEURONS = 150
N_OUTPUTS = 10
N_EPHOCS = 10
class ImageRNN(nn.Module):
    def __init__(self, batch_size, n_steps, n_inputs, n_neurons, n_outputs):
        super(ImageRNN, self).__init__()
        
        self.n_neurons = n_neurons
        self.batch_size = batch_size
        self.n_steps = n_steps
        self.n_inputs = n_inputs
        self.n_outputs = n_outputs
        
        self.basic_rnn = nn.RNN(self.n_inputs, self.n_neurons) 
        
        self.FC = nn.Linear(self.n_neurons, self.n_outputs)
        
    def init_hidden(self,):
        # (num_layers, batch_size, n_neurons)
        return (torch.zeros(1, self.batch_size, self.n_neurons))
        
    def forward(self, X):
        # transforms X to dimensions: n_steps X batch_size X n_inputs
        X = X.permute(1, 0, 2) 
        
        self.batch_size = X.size(1)
        self.hidden = self.init_hidden()
        
        # lstm_out =&gt; n_steps, batch_size, n_neurons (hidden states for each time step)
        # self.hidden =&gt; 1, batch_size, n_neurons (final state from each lstm_out)
        lstm_out, self.hidden = self.basic_rnn(X, self.hidden)      
        out = self.FC(self.hidden)
        
        return out.view(-1, self.n_outputs) # batch_size X n_output

dataiter = iter(trainloader)
images, labels = dataiter.next()
model = ImageRNN(BATCH_SIZE, N_STEPS, N_INPUTS, N_NEURONS, N_OUTPUTS)
logits = model(images.view(-1, 28,28))

print(logits.size())
print(logits[0:10])
</code></pre>
<p>Can anyone explain the working of of 1,0,2 in line X = X.permute(1, 0, 2) and what is logits,why am I getting the size of logits as 64,10.<br>
I’m B.Tech final year student. Im trying to explore RNN for image classification. Can anyone help me in understanding the working of X = X.permute(1, 0, 2) in the above code.</p>

ptrblck: <p><code>nn.RNN</code> will use <code>dim1</code> as the batch dimension by default as shown in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.RNN.html?highlight=rnn#torch.nn.RNN">docs</a> (you can change it via <code>batch_first=True</code>). The <code>permute</code> operation moves the axes and swaps <code>dim0</code> with <code>dim1</code> so that the following layers will get an input in the shape <code>[batch_size, ...]</code>.</p>
Model = Net() what does this line mean while loading a model

Afaq_Saeed: <p>I dont know much about pytorch but does any one know what this line means</p>

ptrblck: <p><code>model = Net()</code> will create an object of the <code>Net</code> class, which is most likely derived from <code>nn.Module</code> and assign this object to the <code>model</code> variable.<br>
The <code>model</code> variable can then be used to execute the forward pass by passing an input tensor to it (<code>output = model(input)</code>), to load a <code>state_dict</code> (<code>model.load_state_dict(state_dict)</code>) etc.</p>

Afaq_Saeed: <p>So it cannot be used for training</p>

ptrblck: <p>It can be used for training as it’s just the model object.<br>
I would recommend to take a look at <a href="https://pytorch.org/tutorials/">the tutorials</a> and check its usage.</p>
How to convert a single-GPU PyTorch script to a multi-GPU multi-node PyTorch script with DDP?

Olivier-CR: <p>How to convert a single-GPU PyTorch script to a multi-GPU multi-node PyTorch script with DDP?<br>
I read <a href="https://pytorch.org/tutorials/beginner/dist_overview.html" rel="noopener nofollow ugc">this</a> 10 times already but honestly it’s not really helpful ; what I need is a place that lists the modification needed to convert a single-GPU code to a multi-node multi-GPU code.</p>
<p>Is there a place in the doc that explains how to distribute a PyTorch training script over multiple machines?</p>

Syzygianinfern0: <p>Check out these resources. They helped me understand how to do it. I agree that the docs as of now are not to the point.</p>
<ul>
<li><a href="https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed data parallel training in Pytorch</a></li>
<li><a href="https://github.com/GoldenRaven/Pytorch_DistributedParallel_GPU_test" class="inline-onebox" rel="noopener nofollow ugc">GitHub - GoldenRaven/Pytorch_DistributedParallel_GPU_test: Pytorch distributed data parallel test of GPU on MNIST</a></li>
<li><a href="https://towardsdatascience.com/how-to-convert-a-pytorch-dataparallel-project-to-use-distributeddataparallel-b84632eed0f6" rel="noopener nofollow ugc">https://towardsdatascience.com/how-to-convert-a-pytorch-dataparallel-project-to-use-distributeddataparallel-b84632eed0f6</a></li>
<li><a href="https://medium.com/analytics-vidhya/distributed-training-in-pytorch-part-1-distributed-data-parallel-ae5c645e74cb" class="inline-onebox" rel="noopener nofollow ugc">Distributed Training in PyTorch (Distributed Data Parallel) | by Praneet Bomma | Analytics Vidhya | Medium</a></li>
</ul>

cbalioglu: <p>Hey <a class="mention" href="/u/olivier-cr">@Olivier-CR</a>, do you mind opening an issue in the PyTorch repository for this? There is definitely room for improvement in our documentation and this would help us to prioritize it in the near future. Thanks!</p>
Doubt about the difference between weight and pos_weight in binary_cross_entropy_loss

Dhorka: <p>Hi,</p>
<p>I have a doubt related to the weighting on binary cross entropy. I was checking the two available functions: <code>torch.nn.functional.</code> <code>binary_cross_entropy</code> and <code>torch.nn.functional.</code> <code>binary_cross_entropy_with_logits</code>. As far as I understood the only difference between both functions is that the second one integrates the sigmoid. However, I found another difference. I observed that binary_cross_entropy_with_logits has two parameters to weight the loss: weight and pos_weight. Whereas the binary_cross_entropy just has the weight parameter.</p>
<p>I would like to understand in which scenarios is useful each of the weight parameters, and why the binary_cross_entropy just has one parameter to weight the loss.</p>
<p>Thanks!</p>
DDP script hangs forever, doesn't run

Olivier-CR: <p>Hi,</p>
<p>I have a well working single-GPU script that I (believe) I correctly adapted to use DDP</p>
<p>I tried to use it in a single-node, 4-GPU EC2 with 2 different techniques, both hang forever (1min+) with CPU and GPU idle. What is wrong? How to use DDP?</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --use_env train.py \
    --gpu-count 4 \
    --dataset . \
    --cache tmp \
    --height 604 \
    --width 960 \
    --checkpoint-dir . \
    --batch 10 \
    --workers 24 \
    --log-freq 20 \
    --prefetch 2 \
    --bucket $bucket \
    --eval-size 10 \
    --iterations 20 \
    --class-list a2d2_images/camera_lidar_semantic/class_list.json
</code></pre>
<p>hangs</p>
<pre><code class="lang-auto">python /home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/distributed/launch.py \
    train.py \
    --gpu-count 4 \
    --dataset . \
    --cache tmp \
    --height 604 \
    --width 960 \
    --checkpoint-dir . \
    --batch 10 \
    --workers 24 \
    --log-freq 20 \
    --prefetch 2 \
    --bucket $bucket \
    --eval-size 10 \
    --iterations 20 \
    --class-list a2d2_images/camera_lidar_semantic/class_list.json
</code></pre>
<p>hangs too.</p>
<p>I strongly suggest the PyTorch team to work on improve the distribution experience. As model and dataset scale, this is a feature people will use more and more</p>

cbalioglu: <p>It is very hard to root cause your problem by just looking at the commands you have run. I suggest checking out our debugging tools described <a href="https://pytorch.org/docs/stable/distributed.html#debugging-torch-distributed-applications" rel="noopener nofollow ugc">here</a>. They might give you more context about the problem.</p>
<p>Also make sure that you read our documentation on <a href="https://pytorch.org/docs/stable/elastic/run.html#usage" rel="noopener nofollow ugc"><code>torchrun</code></a> which is the officially recommended way to start distributed jobs.</p>

yselivonchyk: <p>DDP default timeout is 30min. Adjusting it through <code>init_process_group</code> would not resolve the problem but would allow to fail faster</p>
How could I create a module with learnable parameters when one set of parameters is from Dataset class

Khabbab_Zakaria: <p>So, I have a model with some parameters that I need to train. And also, there are some parameters in the  class Dataset(torch.utils.data.Dataset) which do some preprocessing. I need to train them as well with the model’s parameters. So, can you please let me know if what I am doing is correct:<br>
params = list(model.parameters())<br>
params.extend(list(Dataset.parameters()))<br>
opt = torch.optim.Adam(params,lr=1e-4)</p>
<p>One more question. As Dataset class will generate both the train_ds and val_ds, I only need to train the parameters of the Dataset class when getting train_ds. And to get val_ds, I need to use the trained parameters of the Dataset. So,  how do I create the train_ds and val_ds? Should I initially create both of them and then train the model with tarin_ds, and then create them again and use the val_ds for testing?</p>
Can DDP be equivalent to a single process (GPU) update if the batch size is divided by the number of GPUs? Could there be a situation where it is slower than a single GPU?

tlim: <p>Q1: Equivalence</p>
<p>From my understanding, DDP spawns multiple unique processes for updating gradients. Each process in DDP gathers all the gradients from all other processes and <strong>averaging</strong> it via an all-reduction operation.</p>
<p>For example, if a researcher has published a paper stating to train using SGD, LR=0.1, batch = 64 for 100 epochs. Would it be equivalent to me doing DDP with SGD, LR=0.1, <strong>batch = 16</strong> for 100 epochs if distributed to 4 GPUs?  I want to know whether is it equivalent so that I scale experiments done in a single GPU to a distributed setting for faster training.  It seems to me that it is equal unless I have misunderstood that all gradients from every process are averaged by every individual process in parallel.</p>
<p>I see an exact example here: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L151" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at master · pytorch/examples · GitHub</a></p>
<p>Q2:</p>
<p>If I am using a workstation with more than 1 GPU, is there any particular situation where I should opt not to use DPP and just use 1 GPU for training?</p>

cbalioglu: <p>Regarding your first question. That is correct. DDP should give you the same result as if your training was run on a single process with a single GPU. As you described, DDP by default averages your gradients across all nodes, meaning except any deviations due to floating point arithmetic, mathematically the outputs should be identical (assuming you scale your hyperparameters -i.e. batch size- accordingly).</p>
<p>There is no definitive answer for your second question. Overall the best approach would be to simply measure the speed and the convergence rate of your training for a few epochs. One particular case where it might not be worth to use more than one GPU is if your batch size is fairly small and can already fit into a single GPU. Obviously in such case it makes not much sense to use a multi-GPU setting.</p>
Implementing register_grad_sampler for an unsupported layer

anibadde: <p>I’m running into a torchdp.dp_model_inspector.IncompatibleModuleException when trying to train a private version of the following models <a href="https://github.com/anibadde/transformers/blob/cde0c750af2fae6848ed3ee8be381b4f1230ecd0/src/transformers/models/vit/modeling_vit.py#L479" class="inline-onebox" rel="noopener nofollow ugc">transformers/modeling_vit.py at cde0c750af2fae6848ed3ee8be381b4f1230ecd0 · anibadde/transformers · GitHub</a><br>
and attempted to implement a register_grad_sampler for the nn.Parameter layer to resolve this issue. Are there docs on how to correctly implement this as even after implementing this I’m still running into the same error.</p>
Reconstruct model from inlined_graph

robeast: <p>Hello, probably a very obscure question:</p>
<p>assuming we have a string which encodes a valid <code>inlined_graph</code> IR representation of a (JIT scripted) model, is there a way to re-assemble the original model in form of, e.g., an <code>nn.Module</code> or <code>RecursiveScriptModule</code> or any similar complete object?</p>
<p>Thanks a lot,<br>
Best regards,<br>
RB</p>
PyTorch 0.3 Build from source with CUDA 10 - CMake Error: - CUDA_cublas_device_LIBRARY (ADVANCED)

Arda_Tumay: <p>I am trying to build PyTorch 0.3 from the source with CUDA 10. During the build process below error is printed.</p>
<p>The command I used to build <code>CUDA_HOME=/usr/local/cuda-10.0 python setup.py install</code></p>
<pre><code class="lang-auto">CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
CUDA_cublas_device_LIBRARY (ADVANCED)
    linked by target "THC" in directory /home/atumay/research/pytorch/torch/lib/THC
</code></pre>
<p>Is this because PyTorch 0.3 is not compatible with CUDA 10?</p>

ptrblck: <p>Yes, most likely CUDA10 is not supported in PyTorch <code>0.3</code> as the latter was released in ~Oct 2017 and was using CUDA7.5 back then.<br>
I would not recommend to use such ancient PyTorch releases and instead to update to the latest release.</p>

Arda_Tumay: <p>Okay, thank you I’ll consider switching to the latest version.</p>
What is the python equivalent of at::max_out

Maxwell_Redacted: <p>I have a some code written in  C++ which I am rewriting to have in python,<br>
I wanted to check whether the two are compatible:</p>
<p>at::max_out(tensor1, tensor2, tensor3)</p>
<p>and</p>
<p>torch.max(tensor1, tensor2, out = tensor3)</p>
<p>I have found the documentation slightly lacking on the matter</p>

ptrblck: <p>Yes, <code>max_out</code> is defined as:</p>
<pre><code class="lang-auto">Tensor&amp; max_out(const Tensor&amp; self, const Tensor&amp; other, Tensor&amp; result) {
  return at::maximum_out(result, self, other);
}
</code></pre>
<p>so the last input tensor should be the <code>out</code> tensor.</p>

Maxwell_Redacted: <p>awesome, thanks for the help</p>
Resize vs accuracy

my3bikaht: <p>Stuck with something that doesn’t fit common sense.<br>
I have set of spectrograms with shape 64x192 (1:3 ratio).</p>
<p>Option 1: first I do upscale (cv2 bilinear) to 224x224 and feed images to resnet50, as a result getting 90%+ top1 accuracy.</p>
<p>Option 2: keep 64x192, and replace head of resnet50 (conv2d(kernel=7), maxpool) with strided convolution, stride=3. New head’s output, before bottleneck layers is 64x64x64 (CxWxH), which is similar to resnet (64x56x56). But, whatever kernel size I use, 3/5/7, new models’ ability to generalize is worse, top1 accuracy is getting stuck at about 80%+.</p>
<p>It doesn’t really make sense, since bilinear upscaling doesn’t introduce new information, yet it should make data even worse due to bilinear algorithm. The only reason I consider somehow possible is that bilinear upscaling smoothing values of pixels, thus prevents overfitting, but still…<br>
Any ideas or something from your previous experience?</p>
How to run my pytorch fork on jupyter notebook

Bibhabasu_Mohapatra: <p>I wanted to contribute to pytorch and have forked pytorch and made some  changes regarding issue and commited to my forked pytorch,</p>
<p>Now I wanted to check the features I added by using my fork as torch,     I tried the following steps<br>
==&gt;!git clone --recursive <a href="https://github.com/bibhabasumohapatra/pytorch.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - bibhabasumohapatra/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
==&gt;cd pytorch<br>
==&gt;!git submodule sync<br>
==&gt;!git submodule update --init --recursive --jobs 0<br>
==&gt; !python setup.py install<br>
but I don’t know how to continue…after that</p>
<p>can anybody guide me how to test my forked version (on colab would be better)?</p>

ptrblck: <p>After you’ve built the local branch of your PR the current pip/conda environment would use this source build and you could execute the corresponding tests or test the PR in any other way.<br>
I would probably not work in a Jupyter notebook to build PyTorch or what is the reason to trigger the build through the notebook?</p>
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same?

mrswjung: <p>I am trying to make a model for <strong>pixel rendering</strong>.</p>
<p>this is how the <strong>model</strong> looks like.</p>
<p>class RGB_to_PenTile_to_RGB(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.conv1 = nn.Conv2d(3, 64, kernel_size=7, padding=2, padding_mode='replicate')
    self.conv2 = nn.Conv2d(64, 64, kernel_size=1, padding=2, stride=2, padding_mode='replicate')
    self.conv3 = nn.Conv2d(64, 32, kernel_size=1, padding=0, stride=1)
    self.conv4 = nn.Conv2d(32, 32, kernel_size=1, padding=0, stride=1)
    self.conv5 = nn.Conv2d(32, 1, kernel_size=6, padding=2, padding_mode='replicate')
    self.conv6 = nn.Conv2d(32, 1, kernel_size=6, padding=2, stride=2, padding_mode='replicate')
    self.conv7 = nn.Conv2d(32, 1, kernel_size=6, padding=2, stride=2, padding_mode='replicate')
    self.conv8 = nn.Conv2d(32, 1, kernel_size=6, padding=2, stride=2, padding_mode='replicate')
    self.conv9 = nn.Conv2d(32, 1, kernel_size=6, padding=2, stride=2, padding_mode='replicate')
    
def forward(self, x):
    x = F.relu(self.conv1(x))
    x = F.relu(self.conv2(x))
    x = F.relu(self.conv3(x))
    x = F.relu(self.conv4(x))
    
    g = self.conv5(x)
    r1 = self.conv6(x)
    r2 = self.conv7(x)
    b1 = self.conv8(x)
    b2 = self.conv9(x) 
    
    r1 = F.conv_transpose2d(r1, mask_R1, stride=4, padding=0) *This is where the error occurs.*
    r2 = F.conv_transpose2d(r2, mask_R2, stride=4, padding=0)
    g = F.conv_transpose2d(g, mask_G, stride=2, padding=0)
    b1 = F.conv_transpose2d(b1, mask_B1, stride=4, padding=0)
    b2 = F.conv_transpose2d(b2, mask_B2, stride=4, padding=0)

    r, b = torch.add(r1,r2), torch.add(b1,b2)

    print('b',b)
    r = F.conv2d(r, weights_RB, None, 1, 3)
    g = F.conv2d(g, weights_G, None, 1, 1)
    b = F.conv2d(b, weights_RB, None, 1, 3)

    outputs = torch.cat((r,g,b), dim=1)
 
    return outputs
</code></pre>
<p>and the filter looks like.<br>
mask_R1 = torch.tensor([<br>
[0, 0, 0, 0],<br>
[0, 1, 0, 0],<br>
[0, 0, 0, 0],<br>
[0, 0, 0, 0],<br>
])<br>
mask_R1 = mask_R1.view(1,1,4,4).float()</p>
<p>but it keeps showing this error.</p>
<p><strong>RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same</strong></p>
<p>So my question is…</p>
<p>For training, how can I use a fixed weight convolution layer inside or outside the model?</p>

Anand_Krish: <p>Load the  <code>mask_R1</code> onto the GPU as well.</p>

mrswjung: <p>I did this</p>
<p>mask_R1 = mask_R1.to(device)<br>
r1 = F.conv_transpose2d(r1, mask_R1, stride=4, padding=0)</p>
<p>but it says</p>
<p><strong>UnboundLocalError: local variable ‘mask_R1’ referenced before assignment</strong></p>

Anand_Krish: <p>The <code>mask_R1</code> tensor in not inside the neural network module. Try make it a member of your network module.</p>
How to extract individual weights after per channel static quantization?

babak_hss: <p>I quantized my CNN network with the static quantization module and the <strong>per channel quantization</strong> . Here is a part of my quantized network:</p>
<pre><code class="lang-auto">Sequential(
  (0): QuantizedConv2d(2, 2, kernel_size=(1, 8), stride=(1, 2), scale=0.43247514963150024, zero_point=62, padding=(0, 3), groups=2)
  (1): QuantizedConvReLU2d(2, 32, kernel_size=(1, 1), stride=(1, 1), scale=0.0933830738067627, zero_point=0)
  (2): Identity()
  (3): Identity()
  (4): Dropout(p=0.5, inplace=False)
)
</code></pre>
<p>In the end, I tried to extract the weights and quantization parameters of each convolution kernel.</p>
<pre><code class="lang-auto">In   [518]: layers[0].weight()
Out[518]: 
tensor([[[[ 0.5521, -0.4270, -0.9423, -0.8687, -0.4932, -0.3313, -0.3755,
            0.0221]]],

        [[[-0.4360, -0.6763, -0.7154, -0.5980, -0.6372, -0.0447, -0.1733,
           -0.2962]]]], size=(2, 1, 1, 8), dtype=torch.qint8,
       quantization_scheme=torch.per_channel_affine,
       scale=tensor([0.0074, 0.0056], dtype=torch.float64),
       zero_point=tensor([0, 0]), axis=0)
</code></pre>
<p>I tried to read the weights, but I got this error:</p>
<pre><code class="lang-auto">In [562]: layers[0].weight().data[0,0,0,0]
Traceback (most recent call last):

  File "&lt;ipython-input-562-742a141c2263&gt;", line 1, in &lt;module&gt;
    layers[0].weight().data[0,0,0,0]

RuntimeError: Setting strides is possible only on uniformly quantized tensor
</code></pre>
<p>Also, I can get a single scale for the whole layer[0], but I do not know what it is related to compared to the per-channel scale (q_per_channel_scales)?</p>
<pre><code class="lang-auto">In [567]: layers[0].scale

Out[567]: 0.43247514963150024
</code></pre>

babak_hss: <p>OK. It seems like we can use<br>
<code>layer[0].weight().int_repr().data[i,j,l,m]</code><br>
to get the INT8 representation of the weight entries.</p>
<p>Also,<br>
<code>layer[0].weight().dequantize()</code><br>
gives the tensor of the weights in FP format to have element-wise access to its contents.</p>

Forceless: <p>Hi,<br>
I tried adopted your advice and it works well<br>
But I have another question<br>
How can I change the respondent value to the model<br>
I change the value directly like what I do in normal models  that will not affect the value of the model’s value<br>
like:</p>
<pre><code class="lang-python">number.int_repr()[0]=number.int_repr()[0]*2
number.int_repr()[0]
# Don't have any change
number.dequantize()=number.dequantize()*2
number.dequantize()
# Don't change too
</code></pre>
<p>Thank you</p>
Support for layernorm on onnx

geekgirldecodes: <p>When I use torch.nn.LayerNorm in my model and perform a conversion to ONNX model representation, I observe that the (layer_norm) mapping is missing and it’s represented as a number of smaller ops performing the math for layer norm. I also checked ONNX operators.md list and there is no LayerNorm type listed there. Does this mean there is no support for layernorm type?</p>
How to use EmbeddingBag with uneven 3D data?

Abhishaike_Mahajan: <p>I have a dataset of B x T x C, where B is batches, T is timestep (uneven), and C is characters (uneven). I would like to use EmbeddingBag to get a mean-embedding of each timestep of characters.</p>
<p>For example, lets say I have three datapoints in my batch:</p>
<ol>
<li>[[], [0, 4], [1, 1], [5]]</li>
</ol>
<ul>
<li>This has 4 time steps, and 0, 2, 2, 1, respectively, characters in each timestep.</li>
</ul>
<ol start="2">
<li>[[1], [2, 3]]</li>
</ol>
<ul>
<li>This has 2 time steps, and 1, 2, respectively, characters in each timestep.</li>
</ul>
<ol start="3">
<li>[[2, 4, 5], []]</li>
</ol>
<ul>
<li>This has 2 timesteps, and 3, 1, respectively, characters in each timestep</li>
</ul>
<p>So let’s init that:<br>
<code>all_tensors = [[[], [0, 4], [1, 1], [5]], [[1], [2,3]], [[2, 4, 5], []]]</code></p>
<p>And I know this is what I want my embedder to look like:<br>
<code>embedder = torch.nn.EmbeddingBag(num_embeddings = 6, embedding_dim = 2, mode = 'mean')</code></p>
<p>And…this is where I am stuck. Is there a good tutorial for how this problem should be approached?</p>
<p>Edit: Think I got a bit closer…</p>
<pre><code class="lang-auto">def pad_array(base_input):
    for index1, datapoint in enumerate(base_input):
        base_input[index1] = torch.LongTensor(np.asarray([np.pad(a, (0, 5 - len(a)),  'constant', 
                                                                 constant_values=0) for a in datapoint]))
    return base_input

all_tensors = [[[], [0, 4], [1, 1], [5]], [[1], [2,3]], [[2, 4, 5], []]]
paddedchar_tensors = pad_array(all_tensors)
paddedchar_tensors = rnn_utils.pad_sequence(padded_tensors, batch_first=True)
</code></pre>
<p>This gives me <code>paddedcode_tensors</code> as:</p>
<pre><code class="lang-auto">tensor([[[0, 0, 0, 0, 0],
         [0, 4, 0, 0, 0],
         [1, 1, 0, 0, 0],
         [5, 0, 0, 0, 0]],

        [[1, 0, 0, 0, 0],
         [2, 3, 0, 0, 0],
         [0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0]],

        [[2, 4, 5, 0, 0],
         [0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0]]])
</code></pre>
<p>But once again, I am stuck, running this through the EmbeddingBag gives me this error: <code>ValueError: input has to be 1D or 2D Tensor, but got Tensor of dimension 3</code></p>

Vincent_Kennedy: <p><a class="mention" href="/u/abhishaike_mahajan">@Abhishaike_Mahajan</a> did you ever find a solution to this problem? I’m looking for the solution for a similar application I’m working on and would really appreciate any pointers if you’ve had success. Thank you!</p>
AMD RDNA 2 and Direct ML

Stanley_C: <p>Does anyone know if Pytorch will support RDNA 2 GPUs? From the documentation, it seems that Pytorch relys on ROCm to run, yet some people have been saying that AMD has abandoned ROCm for RDNA, and is instead focusing on software for their server compute card line up, CDNA.<br>
Also, will Pytorch support DirectML? I’ve read that tensorflow support its, and it allows support for APUs, AMD and Nvidia GPUs. I’ve also it reads its runs faster. I’m not sure if I understand the concepts of ROCm and DirectML correctly, so correct me if my understanding is wrong.</p>

Pedro_Lima: <p>Unfortunally I don’t see Pytorch adopting Directml, with regards to ROCm I’ve been following this for over a year now and people still don’t have support for RDNA 1, maybe this will change with RDNA 2, but I doubt, it Basically Cuda, Rocm and Directml are APIs that provide fast matrix multiplication on a given platform, I like directml because on Windows at least is hardware agnostic, 'cuz it runs on top of Directx, when I heard about Rocm a few years ago I thought that this would finally be the platform to rule them all, but it runs only on Linux and in older hardware, CUDA have been very consistent that’s why I think the PyTorch team will not try to add another runtime for its framework, OpenCL would be the perfect candidate to a heterogeneous computing platform, but it didn’t work out well at the end, plaidml is something that you should look into, unfortunally PyTorch is a CUDA only framework at the moment.</p>

Pedro_Lima: <p>There is now support for PyTorch with directml: <a href="https://devblogs.microsoft.com/windowsai/introducing-pytorch-directml-train-your-machine-learning-models-on-any-gpu/" class="inline-onebox" rel="noopener nofollow ugc">Introducing PyTorch-DirectML: Train your machine learning models on any GPU - Windows AI Platform</a></p>
How to sample data from the proximity of existing data?

Ravi_Raja: <p>I have data for xor as below -</p>
<div class="md-table">
<table>
<thead>
<tr>
<th style="text-align:center">x</th>
<th style="text-align:center">y</th>
<th style="text-align:center">z</th>
<th style="text-align:center">x ^ y ^ z</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center">0</td>
<td style="text-align:center">0</td>
<td style="text-align:center">1</td>
<td style="text-align:center">1</td>
</tr>
<tr>
<td style="text-align:center">0</td>
<td style="text-align:center">1</td>
<td style="text-align:center">0</td>
<td style="text-align:center">1</td>
</tr>
<tr>
<td style="text-align:center">1</td>
<td style="text-align:center">0</td>
<td style="text-align:center">0</td>
<td style="text-align:center">1</td>
</tr>
<tr>
<td style="text-align:center">1</td>
<td style="text-align:center">1</td>
<td style="text-align:center">1</td>
<td style="text-align:center">1</td>
</tr>
</tbody>
</table>
</div><p>Kept only the ones that make the xor of all three equal to 1.</p>
<p>I want to generate synthetic data around the already available data within some range uniformly at random. The above table can be thought of as seed data. An example of expected table will be as follows:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th style="text-align:center">x</th>
<th style="text-align:center">y</th>
<th style="text-align:center">z</th>
<th style="text-align:center">x ^ y ^ z</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center">0.1</td>
<td style="text-align:center">0.3</td>
<td style="text-align:center">0.8</td>
<td style="text-align:center">0.9</td>
</tr>
<tr>
<td style="text-align:center">0.25</td>
<td style="text-align:center">0.87</td>
<td style="text-align:center">0.03</td>
<td style="text-align:center">0.99</td>
</tr>
<tr>
<td style="text-align:center">0.79</td>
<td style="text-align:center">0.09</td>
<td style="text-align:center">0.28</td>
<td style="text-align:center">0.82</td>
</tr>
<tr>
<td style="text-align:center">0.97</td>
<td style="text-align:center">0.76</td>
<td style="text-align:center">0.91</td>
<td style="text-align:center">0.89</td>
</tr>
</tbody>
</table>
</div><p>Above table is sampled with a range of 0 to 0.3 for 0 value and with range 0.7 to 1 for value 1.</p>
<p>I want to achieve this using pytorch.</p>

OrielBanne: <p>why do you want to achieve this with pytorch? why not use some random generator every time you want a new synthetic table?</p>

Ravi_Raja: <p>Yeah… actually that would also work.</p>

OrielBanne: <p>so I guess we should close this topic <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Ravi_Raja: <p>Hey, I got my solution. we can close this topic. Thanks.</p>
Accessing weights in Conv2d

AreTor: <p>How can I zero out the weights of a <code>Conv2d</code> module? I am specifically not able to access the weights.</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">conv = nn.Conv2d(3, 16, 3, 1, 1)
with torch.no_grad():
    conv.weight.zero_()
    
print(conv.weight)
</code></pre>

AreTor: <p>I am sorry, I have not specified that I need the solution in C++ (I thought the <code>C++</code> tag was enough).<br>
Anyway, I managed to solve the problem:</p>
<pre><code class="lang-auto">    auto conv = torch::nn::Conv2d(
        torch::nn::Conv2dOptions(3, 16, 3)
    );
    conv-&gt;requires_grad_(false);
    conv-&gt;weight.zero_();
</code></pre>

ptrblck: <p>Ah, sorry I missed to check the tag. Yes, it should generally be sufficient but well… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>