akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to add new embeedings dynamically

Satish_1542: <ul>
<li>I am building binary classification model using neural network.</li>
<li>All features are categorical variables, hence I am using  nn.embedding layer. Features are [‘action’, ‘subtype_action’, ‘user’] . I have written model at the end the query.</li>
<li>I need to update this model every day and each day new users will come.  Let us say maximum number of users are 50K for lifetime of a model and each day 1K new users join.</li>
<li>In that case can I add embedding layer of (Max_num_users, embedding_size_for_users) from day one. Or else is there any better way to increase embedding layer 1K each day.</li>
</ul>
<p>Below is my model</p>
<p>BinaryClassification(<br>
(emb_layers): ModuleList(<br>
(0): Embedding(4, 2)<br>
(1): Embedding(18, 2)<br>
(2): Embedding(1000, 8)<br>
)<br>
(linear_layers): ModuleList(<br>
(0): Linear(in_features=14, out_features=8, bias=True)<br>
)<br>
(output_layer): Linear(in_features=8, out_features=1, bias=False</p>
Change data shape

Roua_Rouatbi: <p>I’m trying to change my data shape from:<br>
[array([[[<br>
…,<br>
, dtype=float32),<br>
array([[[<br>
…,<br>
, dtype=float32),<br>
…]</p>
<p>to array([[[[ … ]]],<br>
[[[ … ]]],<br>
…]]]])</p>
<p>This is how I’m loading the images</p>
<pre><code class="lang-auto">import torchvision.transforms as T
preprocess = T.Compose([
   T.ToPILImage(),
   T.ToTensor(),
   #T.Resize(32),
   T.Normalize(
       mean=[0.485, 0.456, 0.406],
       std=[0.229, 0.224, 0.225]
   ),
   T.Grayscale(num_output_channels=1)
   
])

def NormalizeData(data):
    return (data - np.min(data)) / (np.max(data) - np.min(data))

import glob
image_list = []
for filename in glob.glob(path+'*.tif'): 
    im=cv2.imread(filename)
    im=preprocess(im)
    im= np.array(im)
    im = NormalizeData(im)
    image_list.append(im)
</code></pre>
<p>The next step is to load this into a data loader</p>
<pre><code class="lang-auto">imgset_test =image_list.astype(np.float64)
imgset_test = imgset_test/ (2**14)

imgset_test = imgset_test.reshape(imgset_test.shape[0],1,32,32)
imgset_loader  = DataLoader(dataset=imgset_test,  batch_size=batch_size, shuffle=False)
</code></pre>
<p>imgset_test shape should be (number of images,1,32,32)<br>
my images are already greyscale now and 32x32 pixels</p>

ptrblck: <p>Could you describe what issue you are hitting using the approach?<br>
I guess one of the posted operation fails or would you like to check if your approach sounds reasonable (it does)?</p>

Roua_Rouatbi: <p>Just wanted to check. turns out there was a small error I made and everything works now! thanks a lot.</p>
Very slow CUDA initialization with libtorch built with Bazel

pierre_falez: <p>Hi,</p>
<p>We try to add Pytorch as a dependency of our Bazel repo.<br>
When I try to compile very simple programs (addition of two tensor in Cuda) with the Bazel build system,<br>
the first call to a CUDA function take lots of times (about 10 minutes). then all work normally. Behavior is the same with more complex models executed with TorchScript. During initialization, GPU are not under heavy load and VRAM is about 300Mb.</p>
<p>When CUDA is not involved, execution is normal. I tested my program only by adding a binary to the Pytorch repo directly to avoid any interference</p>
<p>Here is my environment:<br>
Cuda 11.3<br>
CuDNN 8<br>
Pytorch (commit e35bf564611ac00886f6d745b0406d759e054fef)<br>
Bazel 5.0.0 / GCC-8</p>
<p>Tested on a RTX 3090 and a GTX 1650: same behavior</p>
<p>Thanks,<br>
Pierre Falez</p>

ptrblck: <p>I guess CUDA is JIT compiling the application as the binary wasn’t built for the needed architectures.<br>
Make sure to build for all devices you are using if you don’t want to JIT compile the code in the first call.</p>

pierre_falez: <p>Thank you for your help !<br>
Issue solved by adding other -gencode to nvcc options</p>
<p>Pierre</p>
Once CUDA is activated, is everything running on GPU?

ljjsalt: <p>For computations like <code>torch.cos and torch.cat</code>.</p>
<p>During the execution of these functions, is this possible that the operated data moved between CPU and GPU?</p>

AlphaBetaGamma96: <p>you can move between devices with the <code>.to(device)</code> method.</p>
<pre><code class="lang-auto">x = torch.randn(1) #on CPU 
x.to(torch.device('cuda')) # on GPU
</code></pre>

ptrblck: <p>NIT: you would have to reassign the tensor, since <code>to()</code> is not an inplace operation on tensors.</p>

AlphaBetaGamma96: <p>Ah thanks for the correction! I assume that doesn’t apply when moving a <code>net</code> to GPU ? i.e. <code>net.to('cuda')</code> is ok?</p>

ptrblck: <p>Yes, that’s correct. The <code>to()</code> or <code>cuda()/cpu()</code> calls on <code>nn.Module</code>s will be applied recursively to all registered submodules, buffers, and parameters, so you don’t need to reassign the module.</p>
LSTM loss is stagnated

AfonsoSalgadoSousa: <p>Hi there. I know this is a recurrent issue, but I could not find a fitting solution for my problem in other threads.<br>
The task is a Seq2Seq task.<br>
I have the following model:</p>
<pre><code class="lang-auto">StackedResidualLSTM(
  (encoder): RecurrentEncoder(
    (embed_tokens): Embedding(32100, 128, padding_idx=0)
    (dropout): Dropout(p=0, inplace=False)
    (rnn): LSTM(128, 128, num_layers=2, batch_first=True)
  )
  (decoder): RecurrentDecoder(
    (embed_tokens): Embedding(32100, 128, padding_idx=0)
    (dropout_in_module): Dropout(p=0, inplace=False)
    (dropout_out_module): Dropout(p=0, inplace=False)
    (rnn): LSTM(128, 128, num_layers=2, batch_first=True)
    (dropout): Dropout(p=0, inplace=False)
    (fc_out): Linear(in_features=128, out_features=32100, bias=True)
  )
)
</code></pre>
<p>My forward function is the following:</p>
<pre><code class="lang-auto">def forward(self, input_ids, tgt, return_loss=True, **kwargs):
    state = self.encoder(input_ids, **kwargs)
    prev_output_tokens = self._shift_right(tgt)
    decoder_out, _ = self.decoder(prev_output_tokens, state=state, **kwargs)
    if return_loss:
        return self.loss(decoder_out, tgt, ignore_index=kwargs.get("ignore_index", -100))
    else:
        return decoder_out
</code></pre>
<p>Previous Output Tokens are the targets shifted right one character, starting with eos_token.<br>
The loss is a CrossEntropy with ignore index = pad_token.<br>
The optimizer is a SGD with lr=1e-3, momentum=0.9, weight decay=1e-2<br>
Do you have any insights on what may hinder the proper training of the model? Thanks in advance for any help you can provide.</p>
KLDivLoss for binary classification

meshghi: <p>Hi, I have two outputs from two models of form logit, and I’m using <code>BCEWithLogitsLoss</code> for loss#1 for each output, and <code>KLDivLoss</code> for loss#2 between the two outputs. The following code for <code>KLDivLoss</code> gives me the following error. I was wondering what the problem is? Thanks!</p>
<p>The code:</p>
<pre><code class="lang-auto">loss = KLDivLoss(torch.sigmoid(output1), torch.sigmoid(output2))
</code></pre>
<p>The error:</p>
<pre><code class="lang-auto">if size_average and reduce:
RuntimeError: Boolean value of Tensor with more than one value is ambiguous
</code></pre>

ptrblck: <p>You would need to create the <code>criterion</code> object first before calling it, while both tensors are used as input arguments to the <code>KLDivLoss.__init__</code> method in your code.</p>
<pre><code class="lang-auto">loss = nn.KLDivLoss()(input, target)
# or 
criterion = nn.KLDivLoss()
loss = criterion(input, target)
</code></pre>
<p>should work.<br>
Also note this section of the docs:</p>
<blockquote>
<p>As with <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss"><code>NLLLoss</code></a>, the input given is expected to contain <em>log-probabilities</em> and is not restricted to a 2D Tensor. The targets are interpreted as <em>probabilities</em> by default, but could be considered as <em>log-probabilities</em> with <code>log_target</code> set to <code>True</code>.</p>
</blockquote>
<p>as you were using <code>torch.sigmoid</code> for both tensors in your code.</p>

meshghi: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>! For the second part of your answer, my understanding is that I should use <code>LogSigmoid(output1)</code> and <code>LogSigmoid(output2)</code> instead of <code>sigmoid(output1)</code> and <code>sigmoid(output2)</code>. Am I right?</p>

ptrblck: <p>The targets should contain probabilities by default unless you use <code>log_target=True</code>.</p>
How to initialize the parameter of nn.LazyLinear since I have to use dataparallel to push the model onto?

Jo-w: <p>Hi there,</p>
<p>I’m currently using a LazyLinear layer in my network. I need to push the network to 2 GPUs, so I need to use DataParallel (model).<br>
It seems like I cannot initialize the LazyLinear layer before I input dummy data into the model. But if I:<br>
input dummy data<br>
 → use dataparallel<br>
 → input real data to train the model,<br>
Another error occurred.</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
</code></pre>
<p>Could anyone tell me how to use both lazylinear and dataparallel?</p>
<p>Thank you!</p>
How to access latest torchvision.models (e.g. ViT)?

maflobra: <p>I have seen in the <a href="https://pytorch.org/vision/master/models.html" rel="noopener nofollow ugc">official torchvision docs</a> that recently vision transformers and the ConvNeXt model families have been added to the PyTorch model zoo. However, even after upgrading to latest torchvision version 0.11.3 (via <code>pip</code>) these new models are not available:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torchvision; torchvision.__version__
'0.11.3+cu102'
&gt;&gt;&gt; import torchvision.models as models
&gt;&gt;&gt; model = models.resnext50_32x4d()  # previous models work fine
&gt;&gt;&gt; model = models.vit_b_16()  # vision transformers don't work
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
AttributeError: module 'torchvision.models' has no attribute 'vit_b_16'
</code></pre>
<p>Any ideas how I can access these latest model additions in PyTorch?</p>

ptrblck: <p>The linked docs show the <code>main</code> branch which is currently at <code>0.13.0a0+d785158</code> so you might need to update <code>torchvision</code> to the latest nightly release.</p>

maflobra: <p>Thanks a lot, that did the trick.<br>
Do you have any ideas when 0.13 will officially be released?</p>

suraj.pt: <p>v0.12 will be released officially this week, and includes ViT and ConvNeXt!</p>

maflobra: <p>Thank you very much!</p>
Merged images UNET

Nurkhan_Laiyk: <p>Hello, I was trying to do an image segmentation and after testing saved the three images ( as an example).<br>
The problem is that 3 images are merged, while I want to save each one separately.<br>
def save_test(loader, model, folder=‘/kaggle/working/saved_test/’,device=‘cuda’):<br>
model.eval()<br>
for idx, x in enumerate(loader):<br>
x = x.to(device=device)<br>
with torch.no_grad():<br>
preds = torch.sigmoid(model(x))<br>
preds = (preds &gt; 0.5).float()<br>
torchvision.utils.save_image(<br>
preds, f"{folder}/test_{idx}.png"<br>
)<br>
<span class="hashtag-raw">#torchvision</span>.utils.save_image(y.unsqueeze(1), f"{folder}{idx}.png")</p>
<pre><code>model.train()
</code></pre>
<p>Could you check, please?<br>
Does anyone know what is the problem?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef5aa788281fb515b986b71415974b6c41c3192c.png" data-download-href="https://discuss.pytorch.org/uploads/default/ef5aa788281fb515b986b71415974b6c41c3192c" title="Снимок экрана (1361)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef5aa788281fb515b986b71415974b6c41c3192c_2_690x167.png" alt="Снимок экрана (1361)" data-base62-sha1="y9qlbgcJGTC4eVbqJDXvTKHK3us" width="690" height="167" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef5aa788281fb515b986b71415974b6c41c3192c_2_690x167.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef5aa788281fb515b986b71415974b6c41c3192c_2_1035x250.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef5aa788281fb515b986b71415974b6c41c3192c_2_1380x334.png 2x" data-dominant-color="020202"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Снимок экрана (1361)</span><span class="informations">1861×453 1.99 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>torchvision.utils.save_image</code> creates a grid if a batch of images is passed to it.<br>
Call this method on each images separately to create separate images.</p>
Exporting model having torch.histc operation to onnx giving error

ammar_naich: <p>Hi,<br>
I am trying to export the model which has a torch.histc() operation in it to onnx but it gives an error.<br>
<code>RuntimeError: Exporting the operator histc to ONNX opset version 14 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.</code><br>
I have tried to change with a different opset version but it still gives the same error.<br>
Will really appreciate if any one can help, Thanks</p>

ptrblck: <p>Based on the error message it seems that <code>histc</code> is not a supported op in ONNX, so feel free to create a feature request on GitHub and mention in the feature request if you would be interested in implementing it.</p>
Train GAN on array?

ljeonjko: <p>Hi!</p>
<p>Is there a way to train a GAN without images, but rather numpy data/torch tensors? My data would be the equivalent of 5-6+ channel images, and I don’t know if such a format exists and whether torchvision would support the extension. So, is it possible to follow the generic DCGAN tutorial for example, but exchange images for arrays/tensors?</p>
<p>Thanks!</p>

ptrblck: <p>The model itself is expecting tensors as its input. Assuming you can create the input tensors without running into errors from any transforms or image libs, it should work.<br>
I don’t know if the DCGAN architecture is suitable for your format, but you could certainly try to train the model.</p>
AttributeError: 'Tensor' object has no attribute '__array_interface__'

Ashley_Gao: <p>Does anyone happen to know how to solve this?</p>
<pre><code class="lang-auto">AttributeError:
'Tensor' object has no attribute '__array_interface__'
</code></pre>
<p>I wrote a custom function to extract exactly one class of class “category” (an int) from the dataset usps, and my code is:</p>
<pre><code class="lang-auto">dataset_tgt = datasets.USPS(root='./data',
                              train=True,
                              transform=transform,
                              download=True)
dataset_tgt.data, dataset_tgt.targets = get_particular_class(dataset, category, 'usps')

dataloader_tgt = torch.utils.data.DataLoader(dataset_tgt, batch_size=batch_size,
                                         shuffle=True)
</code></pre>
<p>and the get_particular_class function is defined as</p>
<pre><code class="lang-auto">def get_particular_class(dataset, category, order):

    print('getting class {} in dataset {}'.format(category, order))
    try:
        targets = dataset.targets
    except:
        targets = dataset.labels

    data = dataset.data

    new_targets = []
    new_data = []
    for target, sample in zip(targets, data):
        if target == category:
            new_targets.append(target)
            if order == 'svhn':
                new_data.append(sample.transpose(2,1,0))
            else:
                new_data.append(sample)
    return new_data, new_targets
</code></pre>
<p>And the dataloader is able to load the updated dataset_tgt (USPS) just fine. However, in training loop, it gives this error.</p>
<pre><code class="lang-auto">generate_fake_mnist_usps.py 134 &lt;module&gt;
for i, (data, data_tgt) in enumerate(zip(dataloader, cycle(dataloader_tgt)), 0):

dataloader.py 435 __next__
data = self._next_data()

dataloader.py 475 _next_data
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration

fetch.py 44 fetch
data = [self.dataset[idx] for idx in possibly_batched_index]

fetch.py 44 &lt;listcomp&gt;
data = [self.dataset[idx] for idx in possibly_batched_index]

usps.py 80 __getitem__
img = Image.fromarray(img, mode='L')

Image.py 2739 fromarray
arr = obj.__array_interface__

AttributeError:
'Tensor' object has no attribute '__array_interface__'
</code></pre>

ptrblck: <p>It seems <code>Image.fromarray</code> receives a tensor now instead of a <code>np.array</code>:</p>
<pre><code class="lang-python">img = torch.randn(3, 24, 24)
Image.fromarray(img)
# &gt; AttributeError: 'Tensor' object has no attribute '__array_interface__'
</code></pre>
<p>so I guess your <code>get_particular_class</code> manipulates the internal <code>data</code> structure and converts them into tensors?</p>
PyTorch ARM64 Size

soulven90: <p>I was trying to use PyTorch on AWS Lambda and I have an interesting observation whereby the torch package on a x86_64 OS docker image is 1.7G in size while being only 207MB in size on a arm64 OS docker image.</p>
<p>What would cause such a huge difference in file fize?</p>

ptrblck: <p>I’m not sure which containers you are comparing but I cannot find an ARM64 container in <a href="https://hub.docker.com/r/pytorch/pytorch/tags">docker hub</a>.<br>
Based on the size difference I would guess the x86 container ships with a CUDA toolkit (and other additional libs).</p>

soulven90: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="145883">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I’m not sure which containers you are comparing but I cannot find an ARM64 container in <a href="https://hub.docker.com/r/pytorch/pytorch/tags" rel="noopener nofollow ugc">docker hub</a>.</p>
</blockquote>
</aside>
<p>Oh I am using the one here <a href="https://hub.docker.com/r/mlupin/docker-lambda" class="inline-onebox" rel="noopener nofollow ugc">Docker</a></p>

ptrblck: <p>Ah OK. I don’t know how these images are built.</p>
Convert the below PIL based library function to Pytorch in training Loop

KURUVILLA_ABRAHAM: <p>Hi Team,</p>
<p>I am using the below PIL function to find the most dominant color in a image in training loop of my Pytorch script,</p>
<p>How could I convert the function to pytorch or a Autograd function <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<h1>
<a name="program-1" class="anchor" href="#program-1"></a>Program</h1>
<p>def get_dominant_color(img):<br>
width, height = img.size<br>
# The RGB values we will “snap” to<br>
colors = [255, 223, 191, 159, 127, 95, 63, 31, 0]</p>
<pre><code>original_color_count = {}
color_count = {}
# Loop through every pixel in the image and modify it
for w in range(width):
    for h in range(height):
        current_color = img.getpixel((w, h))

        if current_color in original_color_count:
            original_color_count[current_color] += 1
        else:
            original_color_count[current_color] = 1

        r, g, b = current_color
        r_set = False
        g_set = False
        b_set = False

        #  Loop through our allowed values and find the closest value to snap to
        for i in range(len(colors)):
            color_one = colors[i]
            color_two = colors[i + 1]

            if not r_set:
                if color_one &gt;= r &gt;= color_two:
                    distance_one = color_one - r
                    distance_two = r - color_two
                    r = color_one if distance_one &lt;= distance_two else color_two
                    r_set = True

            if not g_set:
                if color_one &gt;= g &gt;= color_two:
                    distance_one = color_one - g
                    distance_two = g - color_two
                    g = color_one if distance_one &lt;= distance_two else color_two
                    g_set = True

            if not b_set:
                if color_one &gt;= b &gt;= color_two:
                    distance_one = color_one - b
                    distance_two = b - color_two
                    b = color_one if distance_one &lt;= distance_two else color_two
                    b_set = True

            if all((r_set, g_set, b_set)):
                break

        # Set our new pixel back on the image to see the difference
        new_rgb = (r, g, b)
        img.putpixel((w, h), new_rgb)

        if new_rgb in color_count:
            color_count[new_rgb] += 1
        else:
            color_count[new_rgb] = 1


# Count and sort the colors
all_colors = color_count.items()
all_colors = sorted(all_colors, key=lambda tup: tup[1], reverse=True)
all_colors = all_colors[1:]

filtered_colors = [color for color in all_colors]

if len(filtered_colors) !=0:
    dominant_color = filtered_colors[0][0]
    dominant_color = torch.tensor(dominant_color,dtype=torch.float32)
    dominant_color = config.Generate_color(dominant_color)
else:
    dominant_color = torch.tensor([255,0,0],dtype=torch.float32)
    dominant_color = config.Generate_color(dominant_color)
return dominant_color</code></pre>
Unfold function takes too much GPU memory

Toby: <p>Hi,</p>
<p>I am trying to implement the HaloNets, I used the halo attention from <a href="https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/halo_attn.py" rel="noopener nofollow ugc">timm library</a>.</p>
<p>The HalloAttn object is expected to replace the convolution layer. However when I test with the input size <code>[2, 512, 256, 256]</code> with two modules:</p>
<ul>
<li>a Halo layer with <code>789,888</code> parameters, attention with 8 heads, block size 8, halo size 3.</li>
<li>a convolutional layer with <code>2,359,808</code> parameters, <code>nn.Conv2d(512,512,3,1,0)</code>
</li>
</ul>
<p>With <code>torch.no_grad()</code>, the HaloAttn took <code>1.44 GB</code> for inference and the Convolution took <code>0.21 GB</code>. The conv cost less even though the number of parameters is three times HalloAttn. Therefore, I dug a little deeper and test GPU memory at each module in HaloAttn, the result shows the unfold function (code snip below) cost 0.4GB.</p>
<pre><code class="lang-auto">kv = kv.unfold(2, self.win_size, self.block_size).unfold(3, self.win_size, self.block_size).reshape(
            B * self.num_heads, self.dim_head_qk + self.dim_head_v, num_blocks, -1).permute(0, 2, 3, 1)
</code></pre>
<p>Is there any alternative to unfold with more memory-friendly? Or can someone let me know why this happens?</p>

ptrblck: <aside class="quote no-group" data-username="Toby" data-post="1" data-topic="145950">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/toby/48/29648_2.png" class="avatar"> Toby:</div>
<blockquote>
<p>Or can someone let me know why this happens?</p>
</blockquote>
</aside>
<p><code>unfold</code> explicitly creates patches/windows of the input tensor, which could increase the memory usage depending on the stride and window size.<br>
I guess the <code>unfold</code> approach is used for a “manual” conv implementation using a <code>matmul</code>?<br>
If so, then a higher memory usage would be expected as conv layers could use other kernels, which do not work on the unfolded input.</p>
Indexing into a recursively generated "tensor of tensors"

ironicoxidant: <p>I would like to be able to fill in elements of a tensor recursively, and then compute the gradient w.r.t. an element for backpropagation. Currently, my code (using Python looks something like this:</p>
<pre><code class="lang-auto">def function(length: int, idx: Tensor):
    _dummy: Tensor = torch.empty(0)
    x: List[Tensor] = [_dummy for _ in range(length)]
    x[0] = torch.zeros(batch_size)
    for i in range(1, length):
        x[i] = some_fn(x[i - 1])
    return tensor([x[i] for i in idx.tolist()])  # Slow step!
</code></pre>
<p>However, this is slower on GPU than on CPU, and I suspect that this is due to the use of Python lists needing to synchronize between CPU/GPU (given that RNN training works just fine). Given that I already know the length of my array and the indices I need in advance, I’d like to be able to do something more akin to this:</p>
<pre><code class="lang-auto">def vectorized_function(length: int, idx: Tensor):
    t: Tensor = torch.empty(batch_size, length)
    t[:, 0] = 0
    for i in range(1, length):
        t[:, i] = some_fn(t[:, i - 1])
    return t.index_select(idx)
</code></pre>
<p>However, as I understand it, this won’t work because the gradient of <code>t</code> cannot be computed with in-place operations. Is there any way around this?</p>

ptrblck: <aside class="quote no-group" data-username="ironicoxidant" data-post="1" data-topic="145918">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/9e8a1a/48.png" class="avatar"> ironicoxidant:</div>
<blockquote>
<p>However, as I understand it, this won’t work because the gradient of <code>t</code> cannot be computed with in-place operations. Is there any way around this?</p>
</blockquote>
</aside>
<p>Are you getting an “inplace error” or is this just a concern?<br>
If inplace operations are disallowed in your code, PyTorch will raise an error, so you could still try to execute your code given that <code>t</code> is a newly initialized tensor.</p>

ironicoxidant: <p>I’m an idiot, I think something else was broken the first time I tried it and after refactoring it works. Thank you so much!</p>
Are layer weights are transposed and stored for RNN?

KUSHAL_JAIN: <p>The weight (<code>W</code>) matrices for <code>nn.Linear</code> layer are stored as <code>W.T</code>. That is if we have 100 neurons in the input layer and 200 neurons in the next layer, our layer definition would be <code>nn.Linear(in_features=100, out_features=200)</code> and I would expect the weight matrix to be of shape <code>(100,200)</code> since for every neuron in the first layer we have 200 connections in the second layer and th e weights are “propagted” in this direction. However the weights are transposed and then stored for efficiency during backprop.<br>
Is this behavior only restricted for <code>nn.Linear</code> layers or is it implemented in all <code>nn</code> modules. I specifically want to know if the internal weight matrices are transposed for an <code>RNN</code> layer. I can see that the <code>weight_ih</code> that is input to hidden matrix is transposed while storing but I cannot be sure about the <code>weight_hh</code> since it’s a square matrix. I need to know since I am updating weight manually for each connection and transposed matrices might imply that I am updating the wrong connections. Basically I want to know “which neuron led the neruon in subsequent layer to fire”.</p>
Effectively switch tensors in nn.Linear out

PLarsen79: <p>I switch tensors inside running networks out, during training but in between training passes.</p>
<p>Doing it manually on weights and biases could be done like this:</p>
<pre><code class="lang-auto">with torch.no_grad():
    linear1.weight.data[...] = torch.Tensor([[-0.1], [0.2]])
</code></pre>
<p>But what I am looking for is an efficient way to replace “one neuron” and store the replaced tensor in a storage for later use.</p>
<p>Is there a way to have a storage and in the nn.Linear just point to the tensors I want to currently use? If not possible, what is an efficient way to copy tensors back and forth between storage and active?</p>

AlphaBetaGamma96: <p>you could use something along the lines of  <code>copy_()</code></p>
<pre><code class="lang-auto">weight = linear1.weight.clone()
new_weight = change_neuron(weight)
linear1.weight.copy_(new_weight)
</code></pre>
<p>The change neuron function could be just done via indexing/slicing of the Tensor. I.e.,</p>
<pre><code class="lang-auto">weight[0,0] = 4 #overwrite value with 4
</code></pre>
<p>p.s. don’t use the <code>.data</code> attribute it’s deprecated and can lead to unexpected behavior.</p>

PLarsen79: <p>Thx, kind of straight forward then. I was hoping for something similar to memory pointers or references, but havn’t found anything in the docs</p>
What number of epoch should I select as a starter given my image dataset size?

Mona_Jalal: <p>I want to do fine-tuning based on Inception V3 pre-trained on imagenet and while I am aware that the number of epochs is a hyperparameter, I would like to know what is a rule of thumb for selecting a starter number of epochs?<br>
number of 512x512 jpeg images in each subfolder based on label:</p>
<p>-train<br>
–label_0:  134,889<br>
–label_1:  32,882</p>
<p>-val<br>
–label_0: 42,553<br>
–label_1: 8,799</p>
<p>-test<br>
–label_0: 45,230<br>
–label_1: 11,051</p>
<p>Should I start with 20 epochs or is it very low? Any suggestion is really appreciated.</p>
Compiling latests version with CUDA 10.0

charlielito: <p>Hi! I have a system that has CUDA 10.0 (and can’t be changed easily) and want to update our PyTorch version to one of the latest version (v1.10.2). The problem is that when compiling Pytorch, there is a hard requirement for CUDA &gt;= 10.2 at <a href="https://github.com/pytorch/pytorch/blob/master/cmake/public/cuda.cmake#L49" class="inline-onebox" rel="noopener nofollow ugc">pytorch/cuda.cmake at master · pytorch/pytorch · GitHub</a><br>
I am wondering if that’s indeed a hard requirement or if it could be compiled with CUDA 10.0 without any problems.</p>

ptrblck: <p>Since a fatal error is raised, I would recommend to keep the check. Note that you are checking the cmake files which would be used in a local source build. If you want to install the pip wheel or conda binaries for <code>1.10.2</code>, both would ship with their own CUDA runtime and do not use the local CUDA toolkit unless you build custom CUDA extensions.</p>
Best technique for mean pooling middle tokens embeddings for transformers

Santosh_Gupta: <p>A common technique for certain nlp tasks are to mean pool sentences or entities which span several tokens. So in some cases, the mean of a slice of the final context embeddings is calculated. This is easy to do for a single row, but less obvious to calculate in a batch.</p>
<p>Say that I want to take the mean pool of a couple entities, each spanning a few tokens.</p>
<p>I have <code>last_hidden_state</code> which is a 512 tokens x 768 embedding size tensor.</p>
<p>I also have <code>mean_slices</code> which is a n x 2 sized tensor, where n is the number of entities within the 512 tokens. Each row contains two numbers, they are the start and end token index for each entity.</p>
<p>Since there are no ragged tensors in pytorch, I came up with a workaround where I would calculate the cumsum of the embeddings along the axis of the 512 tokens, and then for each cumsum of an end entity index, I would subtract out the cumsum before the start entity index, and then divide by the length of the slices.</p>
<p>Here’s how it would look like in pytorch code</p>
<pre><code class="lang-auto">padded = torch.nn.functional.pad(
    last_hidden_state.cumsum(dim=0), (0, 0, 1, 0)
)

pools = torch.diff(
    padded[mean_slices], dim=1
).squeeze()/torch.diff(mean_slices, dim=1)
</code></pre>
<p>However, I am wondering if this may not work as intended, as gradients will be passed to all tokens, not just the ones being averaged…I think.</p>

wu_ao: <p>I tested on pytorch, using code like this</p>
<pre><code class="lang-python">last_hidden_state = torch.randn(5, 2, requires_grad=True)
mean_slices = torch.tensor([[0, 3], [3,4]])

padded = torch.nn.functional.pad(
    last_hidden_state.cumsum(dim=0), (0, 0, 1, 0)
)

pools = torch.diff(
    padded[mean_slices], dim=1
).squeeze()/torch.diff(mean_slices, dim=1)

pools.sum().sum().backward()
#outputs
"""
tensor([[0.33, 0.33],
        [0.33, 0.33],
        [0.33, 0.33],
        [1.00, 1.00],
        [0.00, 0.00]])
"""
#when I do this without using cumsum
last_hidden_state.grad.data.zero_()  # zero gradient
span1 = last_hidden_state[0:3, ...].mean(0, keepdim=True)
span2 = last_hidden_state[3:4, ...].mean(0, keepdim=True)
spans = torch.cat([span1, span2], dim=0)
spans.sum().sum().backward()
#outputs
"""
tensor([[0.33, 0.33],
        [0.33, 0.33],
        [0.33, 0.33],
        [1.00, 1.00],
        [0.00, 0.00]])
"""
</code></pre>
<p>so I think their output are exactly the same, tokens will receive 1/span_len gradient of its belonging span<br>
, and other tokens outside any span will not receive gradient.<br>
Your solution works fine, feel free to use it.</p>
A question on @register_grad_sampler of a Conv2D-related customized module

Lei_Jiang1: <p>Dear all,</p>
<p>I have a customized module named MyMod. It is actually a Conv2D, but the weights of Conv2D is actually the product of the trainable weights and a const dictionary.</p>
<pre><code class="lang-auto">class MyMod(nn.Module):
    def __init__(self, ...., ):
        super(MyMod, self).__init__()

        self.dct = nn.Parameter(a const dict, requires_grad=False) #dict size torch.Size([1, 9, 3, 3])
        self.weight = torch.Tensor(...) #requires_grad=True #weight size torch.Size([32, 32, 9, 1, 1])

    def forward(self, x):
        filt = torch.sum(self.weight * self.dct, dim=2) # filt size torch.Size([32, 32, 3, 3])
        x = F.conv2d(x, filt, stride=1, padding=1, dilation=1, groups=1)
        return x
</code></pre>
<p>I want to write a register_grad_sampler  function for it, but I have no clue. I just copied the register_grad_sampler for Conv2d here. Could you please give me some hints to modify it to support MyMod? Thanks!</p>
<pre><code class="lang-auto">@register_grad_sampler([MyMod])
def compute_conv_grad_sample(module, activations, backprops):
    n = activations.shape[0]
    activations = unfold2d(
        activations,
        kernel_size=module.kernel_size, # 3
        padding=module.padding,           # 1
        stride=module.stride,                   # 1
        dilation=module.dilation,              # 1
    )
    backprops = backprops.reshape(n, -1, activations.shape[-1])
    # n=batch_sz; o=num_out_channels; p=(num_in_channels/groups)*kernel_sz
    grad_sample = torch.einsum("noq,npq-&gt;nop", backprops, activations)
    # rearrange the above tensor and extract diagonals.
    grad_sample = grad_sample.view(
         n,
         module.groups,
         -1,
         module.groups,
         int(module.in_channels / module.groups),
         np.prod(1),
     )
     grad_sample = torch.einsum("ngrg...-&gt;ngr...", grad_sample).contiguous()
     shape = [n] + list(module.weight.shape)
    
     ret = {module.weight: grad_sample.view(shape)}
     if module.bias is not None:
         ret[module.bias] = torch.sum(backprops, dim=2)
        
     return ret
</code></pre>

pstock: <p>Hey Lei Jiang,</p>
<p>Thanks for your interest! The simplest approach would be the following:</p>
<ul>
<li>Wrap the <code>filt</code> computation into a <code>nn.Module</code> and compute a custom grad sample for this module</li>
<li>Then, use the standard <code>Conv2D</code> module on the output of that layer. Hence, you’ll need to only write one grad sampler for your custom approach (<code>filt</code>).</li>
</ul>
<p>This should make things easier, do not hesitate to grad check your computations on simple examples.</p>
<p>Hope this helps,<br>
Pierre</p>
How to speed up pytorch in for loop

fengqs: <p>As is know,for loop in python is always in low speed.And in my model ,i design a matrix to index,so i use clusters to make one image in different clusters.And use their index like(1,2,3,4) to express it.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa88d2ea4d5c7b5bef450fdce3c006f37fc6b83a.png" data-download-href="https://discuss.pytorch.org/uploads/default/aa88d2ea4d5c7b5bef450fdce3c006f37fc6b83a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aa88d2ea4d5c7b5bef450fdce3c006f37fc6b83a_2_690x204.png" alt="image" data-base62-sha1="okCjsgcWssbRsWKHlTnpuNYYu8i" width="690" height="204" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aa88d2ea4d5c7b5bef450fdce3c006f37fc6b83a_2_690x204.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aa88d2ea4d5c7b5bef450fdce3c006f37fc6b83a_2_1035x306.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa88d2ea4d5c7b5bef450fdce3c006f37fc6b83a.png 2x" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1059×314 31.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It like this and now the only idea i have is to make multi-process to handle it to speed up?Any one has better ideas? Thanks a lot!</p>
How to distribute training across ALL GPUs

Laya1: <p>I have an object detection task. I am using <code>torch.nn.DataParallel</code> over the model and using <code>torch.device('cuda')</code> as the device at the time of training. I see that only one of the GPUs does most of the processing, and the other two are given only 1-1.5 GB of data. Here is the code for setting up the training:</p>
<pre><code class="lang-auto">model = torch.nn.DataParallel(model)
params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=1e-3, momentum=0.9, weight_decay=0.005)
num_epochs = 20
model, best_model = train_model(model=model,
                                optimizer=optimizer,
                                data_loader=data_loader_train,
                                device=torch.device('cuda'),
                                num_epochs=num_epochs)
</code></pre>
<p>I am using a batch size of 32. If I increase the batch size, then then the GPU runs out of memory. If I use a batch size of 32, then the workload is very unevenly distributed. Is there any way to evenly distribute the workload so that I can increase the batch size?</p>

pritamdamania87: <p><a class="mention" href="/u/laya1">@Laya1</a> Would recommend using <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch 1.10 documentation</a> instead of <code>torch.nn.DataParallel</code>. You can find a tutorial for it here: <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.10.1+cu102 documentation</a></p>

Laya1: <p>Thank you for the reply.<br>
I tried to use DDP, but I cannot figure out the configurations for <code>os.environ</code>. I have one computer with 3 GPUs, so I used the following settings based on a few posts online:</p>
<pre><code class="lang-auto">s.environ['MASTER_ADDR'] = '127.0.0.1'
with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
    s.bind(('', 0))
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    temp = str(s.getsockname()[1])
os.environ['MASTER_PORT'] = temp
torch.distributed.init_process_group(backend='nccl', world_size=3, rank=0)
model = DDP(model)
</code></pre>
<p>The compiler goes into some sort of infinite loop at <code>init_process_group</code>. I am not sure what is happening here and where I have made a mistake.<br>
After this, I switched to <code>DataParallel</code> which at least executed, but with different loads on different GPUs. I just want to use all 3 GPUs - whichever way possible. Thanks</p>
Loading data to torch::tensor from GPU Address

pepinu: <p>Hello,</p>
<p>I’ve got a function that provides an address/a pointer to the currently buffered data within the GPU memory. Is there an easy way to load/move data to a torch::tensor object without duplicating it on GPU?</p>
<p>I know I can use <a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1ad7fb2a7759ef8c9443b489ddde494787.html" rel="noopener nofollow ugc">torch::from_blob();</a> to create a new tensor and just read through the bytes, however is there a more convenient/efficient way of doing this?</p>
<p>Best Regards</p>

deepfailure: <p>Any news on this?</p>
<p>I believe am trying to do the same: given a pointer to GPU I want to build a tensor (without copying anything) to do an inference in a model. My steps are:</p>
<ol>
<li>Get the pointer.</li>
<li>new_tensor = torch:from_blob(pointer)</li>
<li>model.inference(new_tensor)</li>
</ol>
<p>Is this efficient? Do we copy a tensor in the step 2?</p>
Strided Convolution

Jay_Timbadia: <pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/mnt/sda1/luck/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 433, in __init__
    super(Conv2d, self).__init__(
  File "/mnt/sda1/luck/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 94, in __init__
    raise ValueError("padding='same' is not supported for strided convolutions")
ValueError: padding='same' is not supported for strided convolutions
</code></pre>
<p>But tensorflow do support that</p>
<pre><code class="lang-auto">teacher = tf.keras.Sequential(
    [
        tf.keras.Input(shape=(32, 32, 3)),

        tf.keras.layers.Conv2D(256, (3, 3), strides=(2, 2), padding='same'),

        tf.keras.layers.LeakyReLU(alpha=0.2),
        
        tf.keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(1, 1), padding='same'),

        tf.keras.layers.Conv2D(512, (3, 3), strides=(2,2)),

        tf.keras.layers.LeakyReLU(alpha=0.2),

        tf.keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(1, 1), padding='same'),

        tf.keras.layers.Flatten(), 

        tf.keras.layers.Dense(10)
    ],
    name='teacher_model'
)
</code></pre>
<p>This is the summary</p>
<pre><code class="lang-auto">Model: "teacher_model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 16, 16, 256)       7168      
_________________________________________________________________
leaky_re_lu (LeakyReLU)      (None, 16, 16, 256)       0         
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 16, 16, 256)       0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 7, 7, 512)         1180160   
_________________________________________________________________
leaky_re_lu_1 (LeakyReLU)    (None, 7, 7, 512)         0         
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 512)         0         
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0         
_________________________________________________________________
dense (Dense)                (None, 10)                250890    
=================================================================
Total params: 1,438,218
Trainable params: 1,438,218
Non-trainable params: 0
_____________________________
</code></pre>
<p>The confusion is why first conv gives 16 instead of 15, and other sencond convlution is working correctly as per the formula.</p>
<p>Also, pytorch does not support strided conv padding same, so I could not even compare.</p>
<p>Confused here please help?</p>
Question about pytorch Nan

ZiQing_Li: <ol>
<li>Can you list what operations will cause Nan in forward and backward pass (e.g. inf-inf)?</li>
<li>How to detect Nan and avoid it (e.g. add if judgment in the program or detect_anomaly)?</li>
</ol>

huahuanZ: <ol>
<li>Quite lots of operations can produce <code>nan</code>. In summary, if we can’t tell the result with the operation, that would probably produce <code>nan</code>. e.g.</li>
</ol>
<pre><code class="lang-auto">inf - inf -&gt; nan
(-inf) - (-inf) -&gt; nan
inf / inf -&gt; nan
0./0. -&gt; nan
# inf plus inf is still inf
inf + inf -&gt; inf
# inf plus any non-infinite number is still inf
inf + 1 -&gt; inf
# inf divived by zero is still inf
inf / 0 -&gt; inf
</code></pre>
<ol start="2">
<li>Enable detect_anomaly() is OK in debugging, but this would lead to performance degration. torch.isnan() can be used to tell whether a tensor is nan.<br>
Avoiding nan is a rather complex topic. nan could be caused by problematic datas, numerical error (usually in mixed precison training) or code-related bugs. To avoid nan, I think you should first find out why nan occurs.</li>
</ol>
How to know which image is being processed inside the network

z7s: <p>Hello folks</p>
<p>I’m quite new to torch, sorry if this question looks stupid. I’m now working on a modified version of CNN. I have successfully implemented CNN from scratch with the torch framework. I want to introduce some additional information to the convolution and pooling layer. For example, inside the code of convolution layer, I want to use the transparency or depth of a given pixel to compute a result.</p>
<p>It seems that adding an extra channel could be complicated since I’m modifying the code of existing pytorch models. The existing code looks grayscale or RGB-compatible only. Then I thought can I just know which image I’m currently processing and read the extra information from disk directly. Like, the convolution layer is currently processing 001.jpg, so I can load 001.csv. The question is, how can I know which file the network is dealing with?</p>
<p>I’m open for other options as well. Thank you pytorch folks a lot.</p>

ptrblck: <p>Based on the description I think writing a custom <code>Dataset</code> as described e.g. <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a> and returning the sample with the additional information in the <code>Dataset.__getitem__</code> would be the cleanest and easiest way.</p>
AMP twice as slow when using a different GPU

youi: <p>I have an NVIDIA A5000 RTX and an NVIDIA Titan RTX card.</p>
<p>When using PyTorch’s native amp, an epoch takes around 40 minutes on the Titan RTX as opposed to 2 hours on the A5000 RTX. Nothing else is changed. I run the same scripts merely changing CUDA_VISIBLE_DEVICES from 0 to 1 using the different GPU.</p>
<p>This seems very strange to me. What am I missing?</p>

ptrblck: <p>Could you post a minimal, executable code snippet as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>