akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
If I have images saved in many folders, is there a way to begin training without loading all the file paths first?

Vivek_Reddy: <p>Hi,</p>
<p>I have around a million images I want to train spread across around 50 folders.  Do I have to load in all the filenames of every image.  Right now, I have my Dataset set up so it does an os.walk across all the directories and retrieves a list of all filenames in the <strong>init</strong> function.  This takes several hours.  Is there a way to just start on one folder and start loading image paths for training while simultaneously loading the batch size of images and starting training?  Below is my code.<br>
Thank you!</p>
<blockquote>
<p>class Custom_Dataset(Dataset):</p>
<pre><code>  def  __init__( self, transform=None, config=None):  
        self.samples = []
        for crop_path in config.dataset_crops_path_list:
            for root, subdirs, filenames in os.walk(crop_path):
                file_name_list = [f for f in filenames]    #MAJOR TIME SINK
             
                for filename in file_name_list:
                        self.samples.append(os.path.join(root, filename))
                
    self.targets = []
    for ii in tqdm(self.samples):
        image_label = os.path.basename(ii)
        self.targets.append(image_label)
    
    self.transform = transform
    self.output_shape = config.input_sz

def __len__(self):
    return len(self.samples)

def __getitem__(self, idx): 
    if torch.is_tensor(idx):
        idx = idx.tolist()
    image_path = self.samples[idx]
    image = io.imread(image_path)
    image_label = self.targets[idx]

    non_transformed_image = transform.resize(image.copy(), (self.output_shape, self.output_shape), preserve_range=True)

    if self.transform: 
        image = self.transform(image)
        return image, image_label
    return non_transformed_image, image_label
</code></pre>
</blockquote>

ptrblck: <aside class="quote no-group" data-username="Vivek_Reddy" data-post="1" data-topic="142459">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vivek_reddy/48/28909_2.png" class="avatar"> Vivek_Reddy:</div>
<blockquote>
<p>Right now, I have my Dataset set up so it does an os.walk across all the directories and retrieves a list of all filenames in the <strong>init</strong> function. This takes several hours.</p>
</blockquote>
</aside>
<p>This seems quite unexpected. I’ve just tested <code>os.walk</code> on a full <code>ImageNet</code> (~1000 folders) and it finishes in ~2s.</p>

Vivek_Reddy: <p>Wow.  Imagenet is 14 million images it seems.  Perhaps it is the list comprehension that is slowing it down.  Did you use a list comprehension for each folder and append it to a list like I did?</p>

ptrblck: <p>I used these lines as your comment indicated these are the bottleneck:</p>
<pre><code class="lang-python">for root, subdirs, filenames in os.walk(path):
    file_name_list = [f for f in filenames]    #MAJOR TIME SINK
</code></pre>
Torch.sigmoid function gradient issue after first epoch (Trying to backward through the graph a second time)

Elocin: <p>Hi, I’m running the following code for an optimization problem. (The loss function here is just a simplified example). The loss depends on the values of the weights tensor, which is passed through a sigmoid layer and division to make sure: (1) Each entry is between 0, 1 (2) The vector sums to 1.</p>
<pre><code class="lang-auto">def test_weights(epochs):
  weights = torch.rand(64, requires_grad= True)
  optimizer = torch.optim.SGD([weights], lr=1e-2, momentum=0.9)
  
  for e in range(epochs):
    optimizer.zero_grad()

    weights = torch.sigmoid(weights.clone())
    weights = (weights / weights.sum()).clone()

    error = weights[1] + weights[2]
    error.backward()
    optimizer.step()
  
    print(weights[0:5])
  return weights
</code></pre>
<p>However, after printing the first epoch, I got the error:</p>
<blockquote>
<p>Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
</blockquote>
<p>Is there any way to solve this issue?</p>

AlphaBetaGamma96: <p>Try modifying your <code>loss.backward()</code> call with <code>loss.backward(retain_graph=True)</code></p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="Elocin" data-post="1" data-topic="141984">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elocin/48/45887_2.png" class="avatar"> Elocin:</div>
<blockquote>
<pre><code class="lang-auto">    error = weights[1] + weights[2]
</code></pre>
</blockquote>
</aside>
<p>Just a question, why do you have your <code>weights</code> Tensor be of size 64 but only define the error of the first two? (granted they are normalised so there will be gradient for all gradients). I’m not 100% sure but doing <code>weights[1]</code> might be in-place so check your Tensor does indeed have a <code>grad_fn</code>.  Otherwise, your gradient will be zero!</p>
<p>Also, you might want to reduce your learning rate and momentum constant, you’re likely converging directly into a local minima (or perhaps even diverging) and getting stuck. Try lr=1e-4 with/without momentum =0.9.</p>
How do you train nn.LSTM with multidimensional sequence data?

mscorp: <p>If i wanted to train nn.LSTM on sequence data, say from a csv that had multiple columns, like this:</p>
<p>timestamp  columna columnb columnc columnd<br>
10000, 1, 2, 3, 4<br>
10001, 1, 2.1, 3, 4<br>
…<br>
100## 1, 2, 3, 4, label</p>
<p>where the sequence of values in the columns represents the change over time lets say 5 seconds, and the whole sequence represents the behavior i wish to learn and make predictions on, how would i do that? most of the examples i see for LSTM all show a single value’s changes over time.  How does one present multiple dimensions to the LSTM (how does one format it, and how does one train it)</p>
<p>Thank you</p>
Link errors for extension with -rdc=true

Andrei_Pokrovsky: <p>Hello,</p>
<p>Trying to figure out a linker issue, specifically a runtime .so loading error<br>
undefined symbol: __cudaRegisterLinkedBinary<br>
on a function compiled and linked with -rdc=true (separate compilation).<br>
My setup is<br>
NVCC_FLAGS := -arch sm_61 -gencode=arch=compute_61,code=compute_61 <br>
-std=c++11 -O3 --use_fast_math -x cu -Xcompiler -fPIC -lineinfo -rdc=true # -ptx<br>
NVCC_LINK_FLAGS := -arch sm_61 -lcudadevrt -lcudart -Xcompiler -fPIC -shared</p>
<pre><code>$(NVCC_COMPILE) build/file.o src/file.cu $(NVCC_FLAGS) $(INCLUDE_FLAGS)
$(NVCC_LINK) -o build/file_dlink.so build/file.o $(NVCC_LINK_FLAGS)
</code></pre>
<p>Then in build_ffi.py:</p>
<p>extra_compile_args=["-std=c99"],<br>
extra_link_args=["-L/usr/local/cuda-9.0/targets/x86_64-linux/lib/ -lcudadevrt -lcudart"],</p>
<p>extra_objects += [osp.join(abs_path, ‘build/file_dlink.so’)]<br>
extra_objects += [osp.join(abs_path, ‘build/file.o’)]<br>
extra_objects += glob.glob(’/usr/local/cuda/lib64/*.a’)</p>
<p>The final extension .so file doesn’t have the reference to __cudaRegisterLinkedBinary</p>
<p>Did anyone have any experience with separate compilation combined with create_extension?</p>

Andrei_Pokrovsky: <p>Never mind i figured it out. This setup works, the link error was coming from a different file with similar name that didn’t need -rdc=true which I missed. Hopefully this is useful for future reference.</p>

cougarmilk: <p>Hello Andrei,<br>
If you still have it handy, could you share the boilerplate of the setup you’ve mentioned above?<br>
I think the community can really benefit form this given that there aren’t any specific examples that popup when it comes to extending to PyTorch using dynamic parallelism.</p>
<p>You can also dm meat twitter  <span class="mention">@KnockturnalNed</span></p>
About using with torch.no_grad()

raesol: <p>I am new for pytorch and I had a few confusions so bear with me. I  know we don’t do back propagation during the evaluation stage and for that we can use torch.no_grad(). However,I was just wondering if it’s a must to use torch.no_grad()? That is, is it okay not to use torch.no_grad() as long as we don’t do loss.backward() and optimizer.step() in evaluation step like the following:<br>
criterion = torch.nn.CrossEntropyLoss()<br>
for epoch in range(n_epochs):<br>
model.train()<br>
for x,target in data:<br>
prediction=model(x)<br>
optimizer.zero_grad()<br>
loss=criterion(prediction,target)<br>
loss.backward()<br>
optimzer.step()</p>
<h2>
<a name="as-can-see-from-above-we-do-back-propagation-followed-by-step-1" class="anchor" href="#as-can-see-from-above-we-do-back-propagation-followed-by-step-1"></a>as can see from above, we do back propagation followed by step.</h2>
<p>model.eval()<br>
for x,target in data:<br>
prediction = model(x)<br>
loss=criterion(prediction,target)<br>
VS<br>
model.eval()<br>
for x, target in data:<br>
with torch.no_grad():<br>
prediction=model(x)<br>
loss=criterion(prediction,target)<br>
Since we aren’t doing back propagation and updating each parameter of the model, do we necessarily have to write with torch.no_grad() and if so why is it so? thanks for your help in advance.</p>

ptrblck: <p>No, you don’t need to use the <code>no_grad()</code> wrapper during the evaluation phase. While it would reduce the memory usage by avoiding to store intermediate tensors needed to calculate the gradients, it’s a performance improvement and won’t change the functionality otherwise.</p>
Batched range select/assign

LeanderK: <p>I have a bunch of computations like this:</p>
<p>start_coords has shape [b,2,2] with start and end ranges for each image</p>
<pre><code class="lang-auto">img_available = torch.zeros_like(t)
img_available[:,start_coords[:,0,0]:start_coords[:,0,1],start_coords[:,1,0]:start_coords[:,1,1]]=1
</code></pre>
<p>or the other way around:</p>
<pre><code class="lang-auto">selected = imgs[:, start_coords[:,0,0]:start_coords[:,0,1],start_coords[:,1,0]:start_coords[:,1,1]]
</code></pre>
<p>so for each image <code>i</code> in batch, i need <code>imgs[i, start_coords[i,0,0]:start_coords[i,0,1],start_coords[i,1,0]:start_coords[i,1,1]]</code></p>
<p>but this is not working. I need some way to assign/get values using batched-ranges. The length is always the same.</p>
<p>This is inside a loop, so it needs to reasonably fast!</p>
Best practice for modifying intermediate activations of CNNs

ChrisLiu2: <p>For my project I plan to add RNNs between every CNN block. I can just hack any CNN network, but this will take a lot of time. I wish to take advantage of the pretrained torch vision models, and hopefully do so with minimal hacking. Preferably I can reuse the code for all the provided pretrained models. This might be difficult though, since the models provided by torch vision all have slightly different construction.</p>
<p>I looked around and the closest thing that I found are forward hooks. I can register the RNNs as hooks after each layer’s output activation. However it was noted that it introduces global states and should only be used for debugging and profiling. What exactly does this mean? Would this be a recommended practice? If not, what are your suggested alternatives?</p>
Replace features with features of K neighbors without using for loop

Yimiao_Xu: <p>In a KNN-GCN model, I am trying to replace orginal features with features of K-nearest neighbors. The size of input should be (B,N,D), with B: the batch size, N: the number of channels, D: the dimention of features. The size of output should be (B,N,K,D), with K: the parameter of KNN.</p>
<p>I first tried to do this with repeat and gather. However, this would form a tensor of size (B,N,N,D), which is a waste of GPU memory as N is large. The code is:</p>
<pre><code class="lang-auto">def knn(x, k):
    inner = -2 * torch.matmul(x.transpose(2, 1), x)
    xx = torch.sum(x ** 2, dim=1, keepdim=True)
    pairwise_distance = -xx - inner - xx.transpose(2, 1)
    idx = pairwise_distance.topk(k=k, dim=-1)[1]  # (batch_size, num_points, k)
    return idx

top_idx = knn(x.permute(0, 2, 1), K)
top_idx = top_idx.unsqueeze(3).repeat(1, 1, 1, D)
output = torch.gather(x.unsqueeze(1).repeat(1, N, 1, 1), 2, top_idx)
</code></pre>
<p>The returned value of function knn() is a tensor of size (B,N,K).</p>
<p>To avoid generating the tensor of size (B,N,N,D), I used a for loop which repeat K times. The code is:</p>
<pre><code class="lang-auto">top_idx = knn(x.permute(0, 2, 1), K)
top_idx = top_idx.unsqueeze(3).repeat(1, 1, 1, D)
output=torch.empty(size=(B,N,1,D))
for i in range(K):
    temp=torch.gather(x,1,top_idx[:,:,i,:].view(B,N,D)).view(B,N,1,D)
    output=torch.cat((output,temp),dim=2)
output=output[:,:,1:,:]
</code></pre>
<p>Is there any way that I can replace features with features of K neighbors without using for loop? The size of input is (B,N,D). The size of KNN index is (B,N,K). The size of output should be (B,N,K,D).</p>
Which trust level is required for DiscourseHub

theofpa: <p>When I’m trying to connect to the PyTorch discourse using the official mobile app of discourse for iOS (DiscourseHub), I’m getting the following error message:</p>
<pre><code class="lang-auto">Authorize application access
Sorry, you do not have the required trust level to access the user API
</code></pre>
<p>Which trust level is required for me to be able to follow discussions in my mobile?</p>

ptrblck: <p>Trust Level 3 is currently needed to generate user API keys.</p>

theofpa: <p>Would you consider changing this to level 0, like other projects do? Examples: tensorflow, huggingface. That would make the forum more accessible.</p>

ptrblck: <p>No, I don’t think Level 0 would be a good option due to the spam bots we are already dealing with. We can discuss using Level 1 or 2 maybe.</p>

theofpa: <p>I understand the problem with the spam bots, we have a trade-off situation here between making the community more accessible vs requiring less maintenance. <a href="https://forum.bondhome.io/t/discourse-app-api-access/299/6" rel="noopener nofollow ugc">An other community</a> seems to have had the same problem. You might argue that TL1 is relatively easy to be reached, but I’d propose to allow the user api usage for it.</p>

theofpa: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you enable trust level 1 access to the user API, please?</p>
Different weight initialization for every element of the batch

yuri: <p>customized weight initialization can be achieved by model.apply(inialize_weight()) such as inialize_weight() is the build method for this procedure.<br>
however, at the first batch, kernels applied to different batch elements will be identical.<br>
what if we want to initialize different kernels values for each element within the batch.<br>
convnets kernels in PyTorch has the (dim_out, dim_in, kernel_width, kernel_height size) one can see that batch_size does not get involved in the kernel size hence we can confirm that the same kernels will be initially applied to every element of the batch.<br>
a way to bypass this, is to set batch_size to 1, is there another way to achieve that for &gt;1 batch_size?</p>
Synchronization in Libtorch

Hatem: <p>Continuing the discussion from <a href="https://discuss.pytorch.org/t/synchronize-cuda-calls-in-libtorch/77996/2">Synchronize CUDA calls in Libtorch</a>:</p>
<pre><code class="lang-auto">// demo example 
timer.start();
auto input_gpu = input_cpu.to(at::kCUDA);
std::vector&lt;torch::jit::IValue&gt; jit_input;
jit_input.push_back(input_gpu);
auto output = model-&gt;forward(jit_input).toTensor().to(at::kCPU);
timer.stop();
</code></pre>
<p>Do  I have to do synchronization here<br>
or the conversion  to CPU will do the synchronization.</p>
<p>Thank you.</p>
Sending Layers of nn.ModuleList() and tensors to GPU

Aleks1: <p>Hi,<br>
I am trying to run over 6000 custom layers initialized with nn.ModuleList() on GPUs. I can use 80-120 GPUs on cluster. What is the best way to distribute the layers on the GPU? Should I do it in <strong>init</strong> method or in forward? Do the layers run asynchronously when I execute them in a for loop? I don’t know how to execute over 6000 layers more efficiently with pytorch.</p>
<p>Thank you very much for your help</p>

Aleks1: <p>I am initializing the layers in the following way:</p>
<blockquote>
<p>if torch.cuda.is_available():<br>
self.nr_gpus = torch.cuda.device_count()<br>
print(f’Number of GPUs available: {self.nr_gpus}')</p>
<pre><code>    # initialize the Oligo Kernel Layers
    self.CMKN_layers: nn.ModuleList = nn.ModuleList()

    for gene_nr in range(self.num_genes):
        self.CMKN_layers.append(CONLayer(in_channels, self.anchor_points[gene_nr], filter_size, padding=padding,
                                subsampling=stride, kernel_func=kernel_func, kernel_args=kernel_args,
                                kernel_args_trainable=kernel_args_trainable, **kwargs))
</code></pre>
</blockquote>
<p>and my forward method looks like this:</p>
<pre><code>def forward(self, genes: Tuple[torch.Tensor]) -&gt; torch.Tensor:

    x_out_temp: List[torch.Tensor] = []

    gpu_nr: int = 0
    for idx, CMKN_layer in enumerate(self.CMKN_layers):
        gene = genes[idx].cuda(gpu_nr)
        CMKN_layer = CMKN_layer.cuda(gpu_nr)
        x_out: torch.Tensor = CMKN_layer(genes[idx])

        if gpu_nr == self.nr_gpus - 1:
            gpu_nr = 0
        else:
            gpu_nr += 1

        x_out = x_out.to('cpu')
        x_out_temp.append(x_out.view(x_out.size(0), -1))

    x_out = torch.cat(x_out_temp, dim=1)
    x_out = self.fc(x_out)

    return self.classifier(x_out)
</code></pre>
<p>I am trying to share my model on different GPUs. But I’m not sure if this is properly done in this way. I want to decrease the time needed for training.</p>
<p>I am grateful for any advises to improve this.</p>
Torch model copy with computational graph

sesale: <p>Hi All,<br>
I’d like to create a copy of my model. However, I realized that I cannot use copy.deepcopy() since I’d like to have a “stateless” version of the model, meaning I would like to keep the computational graph and be able to compute the derivatives of the new model’s parameters w.r.t the original parameters.<br>
So, the original module or any of its submodules have state (e.g. batch norm), this will be copied too, but further updates (e.g. during inner loop training) will cause these to diverge without changing the state of the original module.<br>
Does anyone know how to achieve this?<br>
Thanks!</p>
Error in pytorch dataloader

srya_Srta: <p>Hi ,</p>
<p>I am newbie in pytorch, I am getting this error frequently, after some percentage of epoch, is it due to ram or multiprocessing  issue im not able to understand<br>
i have kept no of workers as 4 .<br>
batch size is 32 , with each file around 2sec(63kb).</p>
<p>File “/root/surya/installs_anaconda/anaconda3/envs/pytc/lib/python3.7/site-packages/torch/utils/data/dataloader.py”, line 872, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/root/surya/installs_anaconda/anaconda3/envs/pytc/lib/python3.7/multiprocessing/queues.py”, line 105, in get<br>
raise Empty<br>
_queue.Empty</p>
<p>During handling of the above exception, another exception occurred:</p>
<p>Traceback (most recent call last):<br>
File “multihead_torch.py”, line 101, in <br>
for batch in pbar:<br>
File “/root/surya/installs_anaconda/anaconda3/envs/pytc/lib/python3.7/site-packages/tqdm/std.py”, line 1170, in <strong>iter</strong><br>
for obj in iterable:<br>
File “/root/surya/installs_anaconda/anaconda3/envs/pytc/lib/python3.7/site-packages/torch/utils/data/dataloader.py”, line 435, in <strong>next</strong><br>
data = self._next_data()<br>
File “/root/surya/installs_anaconda/anaconda3/envs/pytc/lib/python3.7/site-packages/torch/utils/data/dataloader.py”, line 1068, in _next_data<br>
idx, data = self._get_data()<br>
File “/root/surya/installs_anaconda/anaconda3/envs/pytc/lib/python3.7/site-packages/torch/utils/data/dataloader.py”, line 1034, in _get_data<br>
success, data = self._try_get_data()<br>
File “/root/surya/installs_anaconda/anaconda3/envs/pytc/lib/python3.7/site-packages/torch/utils/data/dataloader.py”, line 880, in _try_get_data<br>
if self._workers_status[worker_id] and not w.is_alive():<br>
File “/root/surya/installs_anaconda/anaconda3/envs/pytc/lib/python3.7/multiprocessing/process.py”, line 156, in is_alive<br>
returncode = self._popen.poll()<br>
File “/root/surya/installs_anaconda/anaconda3/envs/pytc/lib/python3.7/multiprocessing/popen_fork.py”, line 28, in poll<br>
pid, sts = os.waitpid(self.pid, flag)<br>
File “/root/surya/installs_anaconda/anaconda3/envs/pytc/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py”, line 66, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 16353) is kill</p>
Pytorch can't handle tokens > vocab_size but it is not the case with TF

Abir_ELTAIEF: <p>I built a tokenizer from an old one “bert-base-uncased” with the same tokenization algorithm (on a specific dataset “not English language…”), but with a vocab_size of 150000 (that largely exceeds the vocab_size of the original bert (30522). When I used PyTorch to fine tune the model (with this new tokenizer), it gives me an error (index out of range), and the model crashes (when it encounters a token with an id &gt; 30522, it can’t handle it).</p>
<p>Using Tensorflow, it works very well (with this vocab_size = 150000)! and does not return any errors. I could fine-tune it with tensorflow and get satisfactory results “not very very well” but up to a certain degree, satisfactory results (for the test set), but not with Pytorch (even by rebuilding a new tokenizer with vocab_size = 30522 (like original bert): the accuracy goes from 37 to 100, then to 50, then to 25, over epochs and the predictions get worse…Does anyone have an explanation for this?</p>
Batchwise greater value

cbd: <p>I have tensor of size [3,1,2,2] ( Consider 3 batch of size (1,2,2) } . I want number which is greater than 0.1 in each batch. Below code gives total count greater than 0.1 which is 8. I want array of [4,3,1] corresponding to value greater than 0.1 in each batch.</p>
<pre><code class="lang-auto">a=torch.FloatTensor([[[[0.1807, 0.5755],
          [0.2828, 0.4056]]],


        [[[0.4920, 0.0308],
          [0.8676, 0.2128]]],


        [[[0.0993, 0.9040],
          [0.0488, 0.0790]]]]) 
print(a,a.shape)
b=torch.sum(a&gt;0.1)
print(b)</code></pre>

ptrblck: <p>What should the output tensor contain for the values smaller than your threshold? If you would like to fill it with default value, you could initialize the result tensor with these default values and use <code>scatter</code> to fill in the values meeting your threshold (<code>torch.where</code> might also work).</p>

cbd: <p>I just want count value of tensor for each batch which is greater than 0.1.</p>

ptrblck: <p>Your code would already do this (i.e. return <code>8</code>), so I don’t understand this requirement:</p>
<aside class="quote no-group" data-username="cbd" data-post="1" data-topic="142354">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<p>I want array of [4,3,1] corresponding to value greater than 0.1 in each batch.</p>
</blockquote>
</aside>
<p>Could you describe it a bit more with an example, if possible?</p>

cbd: <p>In below tensor a[0] has 4 value which is &gt;0.1, a[1] has 3 value which is &gt;0.1 and a[2] has 1 value which is &gt;0.1. The output is giving sum of (4+3+1=8). I want tensor of [4,3,1].</p>
<pre><code class="lang-auto">a=torch.FloatTensor([[[[0.1807, 0.5755],
          [0.2828, 0.4056]]],


        [[[0.4920, 0.0308],
          [0.8676, 0.2128]]],


        [[[0.0993, 0.9040],
          [0.0488, 0.0790]]]]) 
</code></pre>

ptrblck: <p>A tensor in the shape <code>[4, 3, 1]</code> has 3 dimensions and contains 12 elements (<code>4*3*1</code>).<br>
Print this random tensor in the same shape to see how the values are stored: <code>print(torch.randn(4, 3, 1))</code>.<br>
Since your output has 8 values, you won’t be able to use a tensor with 12 elements to store them (unless you want to use default values for the other indices as described in the previous post).</p>

cbd: <p>Sorry i mean i need value [4,3,1] (size of tensor 1x3).</p>

ptrblck: <p>Ah, it seems I’ve misunderstood the question and you want to get the number of elements <code>&gt; 0.1</code> in <code>dim=[2, 3]</code>?<br>
If so, this should work: <code>b = torch.sum(a&gt;0.1, dim=[2, 3])</code>.</p>
How can I use sklearn.inverse transform on my predictions

sparshgarg23: <p>I am currently training a model in pytorch for regression purposes(the model consists of some dense layers).</p>
<p>I have used sklearn’s standard scaler on the data which consists of an image and the ground truth regression targets.<br>
How can I use the scaler’s inverse transform on the predicted output from the model during evaluation stage.<br>
The training code is shown below</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
  model.train()
  for batch_idx,(img,gt_pose) in enumerate(train_loader):
    img=img.to(device)
    gt_pose=gt_pose.to(device)
    pred_pose=model(img)
    train_loss=loss(pred_pose,gt_pose)
    train_loss=train_loss/accum_iter
    train_loss.backward()
    if((batch_idx+1)%accum_iter==0) or (batch_idx+1==len(train_loader)):
      optimizer.step()
      optimizer.zero_grad()
    
    print ('Epoch: %03d/%03d | Batch %04d/%04d | Training loss: %.4f' 
                   %(epoch, num_epochs, batch_idx, 
                     len(train_loader), train_loss))
    train_losses.append(train_loss.item())
  model.eval()
  for batch_idx,(img_val,gt_pose_val) in enumerate(val_loader):
    img_val=img_val.to(device)
    gt_pose_val=gt_pose.to(device)
    pred_pose_val=model(img_val)
    loss_val=loss(pred_pose_val,gt_pose_val)
    print ('Epoch: %03d/%03d | Batch %04d/%04d | Validation loss: %.4f' 
                   %(epoch, num_epochs, batch_idx, 
                     len(train_loader), loss_val))
    valid_losses.append(loss_val.item())

</code></pre>
<p>Model evaluation</p>
<pre><code class="lang-auto">sc=StandardScaler()
img=img.to(device)#same transformations applied on image as during training
torch.manual_seed(123)
with torch.no_grad():
  pred=model(img)
  pred=pred.detach().cpu().numpy()
#sc.inverse_transform(pred) gives an error

</code></pre>
<p>if i do  sc.inverse_transform(pred),i get an error of the form scalar needs to be fit on the data.</p>

ptrblck: <p>You would need to <code>fit</code> the <code>StandardScaler</code> before using <code>transform</code> or <code>inverse_transform</code>, which is missing in the posted code snippet. Either reuse the original scaler object or store the <code>sc.mean_</code> and <code>sc.scale_</code> and undo the normalization manually.</p>

sparshgarg23: <p>I went with storing the mean and scale,and then used them to undo the normalization,thanks</p>
How can I preprocess video data while preserving all frames?

larcane: <p>I want to batch video data for action counting task. say, I want to get (Batch, Channel, Frame, Height, Width)<br>
But video data frames are not same each other and It looks like not to possible to apply frame subsampling… because action counting task needs to fed all video frames!</p>
<p>Is there better way to batch video data preserving video frames??</p>

thecho7: <p>To best of my knowledge, it is impossible.<br>
However, if you construct them into a graph, please see <a href="https://pytorch-geometric.readthedocs.io/en/latest/" class="inline-onebox" rel="noopener nofollow ugc">PyG Documentation — pytorch_geometric 2.0.4 documentation</a></p>

larcane: <p>Thank you!! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I’ll try this!</p>
Libtorch dlls delayed loading

lubian64: <p>Pytorch Release ver. 1.10.0<br>
Microsoft Visual Studio Community 2019<br>
The application we are developing has to load the pytorch dlls in a delayed way so that the end user is not forced to install them if he does not need them.<br>
The problem is that we can’t link the pytorch dlls in a delayed way (/DELAYLOAD).<br>
There are two different problems:</p>
<ol>
<li>linker error<br>
If I add the c10.dll or torch_cpu.dll in the project properties → linker → input → “Delay Loaded Dlls” I have the following linker errors:</li>
</ol>
<p>fatal error LNK1194: cannot delay-load ‘c10.dll’ due to import of data symbol '"__declspec (dllimport) struct std :: atomic &lt;struct c10 :: impl :: DeviceGuardImplInterface const *&gt; * c10 :: impl :: device_guard_impl_registry "(_<em>imp</em>? device_guard_impl_registry @ impl @ c10 @@ 3PAU? $ atomic @ PEBUDeviceGuardImplInterface @ impl @ c10 @@@ std @@ A) '; link without /DELAYLOAD:c10.dll</p>
<p>fatal error LNK1194: cannot delay-load ‘torch_cpu.dll’ due to import of data symbol ‘"__declspec(dllimport) struct torch::enumtype::kValid const torch::kValid" (_<em>imp</em>?kValid@torch@<span class="mention">@3U0enumtype</span>@1@B)’; link without /DELAYLOAD:torch_cpu.dll</p>
<ol start="2">
<li>anomaly in runtime<br>
If I add the dll torch_cuda_cu.dll in the project properties → linker → input → “Delay Loaded Dlls” it happens that the CPU is always used and the GPU is ignored. The function torch::cuda::is_available() returns always false.</li>
</ol>
<p>Thanks for your help</p>
Default value of set_to_none in Adam's zero_grad

kmkurn: <p>Hi,</p>
<p>Is there a reason why the default value of <code>set_to_none</code> in <code>Adam.zero_grad()</code> isn’t changed to <code>True</code>? The current default results in parameter update even though they aren’t used to compute the loss. I suppose this also holds true for other similar optimisers. Setting it to <code>True</code> seems like what you would expect, no?</p>

ptrblck: <p>I think the <code>set_to_none</code> attribute is set to <code>False</code> by default for backward compatibility reasons.<br>
Note that it’s a performance improvement (avoiding an accumulation kernel to add the new gradients to zeroes) and wasn’t introduces to dodge the side effect of parameter updates with a zero gradient for optimizers with momentum.</p>
Pytorch tensor indexing consumes much more time when first called

Akira_Ju: <p>Hi,</p>
<p>I found a time-consuming hot spot in my code.<br>
It is a indexing operation over a tensor.  It is time-consuming when first called.<br>
To simplify the question, I have done a simple experiment as follows.</p>
<pre><code class="lang-auto">import torch
import time


def func_1():
    a = torch.rand(10)
    indice = torch.tensor((1, 2, 3))
    for i in range(5):
        torch.cuda.synchronize()
        st = time.time()

        b = a[indice]

        torch.cuda.synchronize()
        ed = time.time()
        print((ed - st) * 1000)


def func_2():
    a = torch.rand(10).cuda()
    indice = torch.tensor((1, 2, 3)).cuda()
    for i in range(5):
        torch.cuda.synchronize()
        st = time.time()

        b = a[indice]

        torch.cuda.synchronize()
        ed = time.time()
        print((ed - st) * 1000)


if __name__ == '__main__':
    for i in range(2):
        func_1()
        time.sleep(5)
        print("----")
    print("###############")
    for i in range(2):
        func_2()
        time.sleep(5)
        print("----")

</code></pre>
<p>The code produced output as</p>
<pre><code class="lang-auto">0.1461505889892578
0.01430511474609375
0.0095367431640625
0.008821487426757812
0.00858306884765625
----
0.10752677917480469
0.04220008850097656
0.03361701965332031
0.0324249267578125
0.03170967102050781
----
###############
3.1435489654541016
0.10132789611816406
0.053882598876953125
0.048160552978515625
0.04696846008300781
----
0.14138221740722656
0.06222724914550781
0.051975250244140625
0.0476837158203125
0.04649162292480469
----

</code></pre>
<p>It is obvious the indexing operation need more time when first called. And if you called it and then turn to do other things, it will cost more when you are back. It seems like a cup of coffee, which need heating continuously…</p>
<p>My environment: pytorch 1.9.0 + py3.8_cuda11.1_cudnn8.0.5_0, device: NVIDIA GeForce RTX 3090.</p>
<p>I need help to explain this phenomenon and then optimize my code…</p>
<p>Thanks for your attention.</p>

ptrblck: <p>I cannot reproduce the issue and get quite noisy results for the GPU workload e.g.:</p>
<pre><code class="lang-python">0.0476837158203125
0.11849403381347656
0.06389617919921875
0.053882598876953125
0.05459785461425781
----
0.07605552673339844
0.15306472778320312
0.10085105895996094
0.08416175842285156
0.08344650268554688
</code></pre>
<p>Use <code>torch.utils.benchmark</code> to profile specific operations, as it would add warmup iterations and stabilize the timing using multiple iterations. Alternatively, use a profiler to check the actual kernel runtimes and check if your GPU goes into IDLE mode if it’s not used (which is why warmup iters are used during profiling).</p>
Load data with ImageFolder

dre: <p>hi im tyring to load my data with ImageFolder  but i have problem to load it bicuse my data has two subfolder one of them for class and the other is for diffrent angle of my photos  like 0,30,90</p>
<p>image_datasets = datasets.ImageFolder(root= PATH_OF_DATA, transform=data_transforms)</p>
<h1>
<a name="comment-2-using-the-image-datasets-and-the-transforms-define-the-dataloaders-1" class="anchor" href="#comment-2-using-the-image-datasets-and-the-transforms-define-the-dataloaders-1"></a>Comment 2: Using the image datasets and the transforms, define the dataloaders</h1>
<p>dataloaders = torch.utils.data.DataLoader(image_datasets, batch_size=32, shuffle=False, num_workers=2)</p>

ptrblck: <p>If your data uses a different structure than the expected one in <code>ImageFolder</code>, you could write a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">in this tutorial</a>.</p>
Pack_padded_sequence with inconsistent (repeated) padding

m_h: <p>Following <a href="https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pad_packed_sequence.html#torch.nn.utils.rnn.pad_packed_sequence" rel="noopener nofollow ugc">the example from the docs</a> I am trying to solve a problem where the padding is inconsistent rather than at the end of the tensor for each batch (in other words, no pun intended, I have a left-censored and right-censored problem across my batches):</p>
<pre><code class="lang-auto"> # Data structure example from docs
seq = torch.tensor([[1,2,0], [3,0,0], [4,5,6]])
 # Data structure of my problem
inconsistent_seq = torch.tensor([[1,2,0], [0,3,0], [0,5,6]])

lens = ...?
packed = pack_padded_sequence(seq, lens, batch_first=True, enforce_sorted=False)
</code></pre>
<p>How can I solve the problem of masking these padded 0’s when running them through an LSTM using (preferably) PyTorch functionality?</p>

AbdulsalamBande: <p>If I have understood your problem right, you mean that some data points are padded at the left while others are at the right. Usually the padded 0’s are to make the length of all the data points the same (usually added at the end like above). So if you have any inconsistent paddings(left or right), you might need a separate character instead of 0.</p>

m_h: <p>That is correctly understood. The position in the time series holds information whereas I can not just pad with 0’s in the end. It would not make much sense to use any other character than 0 as I am not looking to impute but rather mask all the instances that I don’t want to throw into the network and impact the weights.</p>
Import a variable from a function within a class from one file to another file

danielmanu93: <p>Hi All, I have 2 files “utils.py” and “trainer.py”. The trainer.py has a class “Trainer” with a function “tnr” defined under the class as shown in the image below. I want to access the highlighted variables “edge_logits” and “nodes_logits” in the last line of the trainer.py and use them in the “reconstruction_scores” function in the utils.py file but I have tried all possible options but getting error. As can be seen, I have already imported utils in the trainer.py file and so when I import trainer in the utils.py, it will throw a error as a result of circular imports. Any ideas to help access the “nodes_logits” and “edges_logits” from the trainer.py and use them in the “reconstruction_scores” function in the utils.py are welcomed. Below are the images of the trainer.py and utils.py.</p>
<p>trainer.py<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/3/13fd26aeb95428e1e2db187b978308b93ad64eae.jpeg" alt="1" data-base62-sha1="2QPrDbckM7GTg3uSfshmix4rQcK" width="658" height="471"></p>
<p>utils.py<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/576e980b368dfc9ccd044deee40f3bab6b5acc6f.jpeg" alt="2" data-base62-sha1="ctsu9nOZURHjRImELhWY7QotL7F" width="601" height="191"></p>

ptrblck: <p>You should be able to import <code>MolecularMetrics</code> and use it in <code>trainer.py</code> via:</p>
<pre><code class="lang-python">...
edges_logits, nodes_logits = self.G(z)
MolecularMetrics.reconstruction_score(edges_logits, nodes_logits)
</code></pre>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>

danielmanu93: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>. Thanks for the reply but I want to do the opposite of what you are thinking. I want to import from the trainer.py and use <code>edge_logits</code> and <code>nodes_logits</code> in the <code>utils.py</code> as highlighted under the <code>reconstruction_scores</code> function in the utils image. How do I go about it without incurring circular import errors.<br>
I have already tried importing Trainer class in the <code>utils.py</code> by <code>from trainer import Trainer</code> and using <code>Trainer.tnr.nodes_logits</code> and <code>Trainer.tnr.edges_logits</code> in the reconstruction_scores function, but I get circular import error since I have also imported <code>utils.py</code> in the <code>trainer.py</code>.</p>
<p><code>ImportError: cannot import name 'Trainer' from 'trainer'</code></p>

ptrblck: <p>You won’t be able to access <code>edges_logits</code> or <code>nodes_logits</code> from <code>Trainer.tnr</code> as they are not attributes but local variables.<br>
Maybe <code>return</code> these tensors and call the <code>Trainer</code> class from <code>MolecularMetrics</code>.<br>
To avoid circular imports, check e.g <a href="https://stackoverflow.com/questions/22187279/python-circular-importing">this post</a>.</p>
Reverse the Tensor channel

cbd: <p>For below tensor i want to reverse the channel.<br>
<strong>Note:</strong> Sequence of tensor value should  remain as it is just channel should be reverse and tensor could be of variable length channel and it is known. Here channel is 3. It could be 6 or 7. Input -Output should be</p>
<pre><code class="lang-auto">in=torch.tensor([[[[1., 2.],
          [3., 4.]]],


        [[[11., 12.],
          [13., 14.]]],


        [[[21., 22.],
          [23., 24.]]]])
out=torch.tensor( [[[21., 22.],
          [23., 24.]]]],


        [[[11., 12.],
          [13., 14.]]],


       [[[[1., 2.],
          [3., 4.]]])

</code></pre>

thecho7: <p><code>torch.flip</code> might be the answer you are finding.</p>
<pre><code class="lang-auto">in = torch.rand(a, b, c)
out = torch.flip(in, [0, 1])
</code></pre>
Performances : Embedding layers vs pre-trained embedding

JohnCwok: <p>Hello,</p>
<p>I read quite a lot about the importance of word embedding in the context of NLP, but i’ve never seen the following issue beeing adresed : Are pre-trained embeddings (word2vec, GloVe etc…) performing better or worse than an embedding layer trained along with the model ?<br>
I intuitively would think that an embedding layer trained along with the model should perform better since it’s trained task-specifically by the network during back-propagation (and maybe generalize less easily ?) but didn’t really find much info on this.</p>
<p>I’m aware that there is probably no clear answer to this, and that this is definitely case dependant, but i’d emjoy hearing your feedback on this matter.</p>
<p>Cheers !</p>

ptrblck: <p>I’m not an expert on NLP, but I would assume pretrained embeddings might be treated similar to pretrained kernels in a CNN.<br>
If you are dealing with data from the same or similar domain (e.g. English texts / “natural” images), the pretrained parameters might just work and probably even better than training from scratch, e.g. if you are working with a small dataset.</p>
<p>That being said, if your data comes from another domain (e.g. source code / medical images), the pretrained layers might not work pretty well and you would need to finetune them or train form scratch.<br>
In the case of NLP and embeddings I would assume this use case to fail pretty badly, since embeddings are used as a lookup table for each word index.<br>
If the words you are dealing with are not in the dictionary, you might get an “unknown token” for a lot of words, which might make the embedding useless.</p>

vdw: <p>To add a bit more details to <a class="mention" href="/u/ptrblck">@ptrblck</a>’s answer from an NLP point of view: As you’ve rightly pointed out, there’s no clear answer, and the decision is generally task dependent. It essentially boils down to the semantic of word embeddings, and if they are meaningful in the context of your task.</p>
<p>The most straightforward example is sentiment analysis. In Word2Vec and Glove, to word vectors are similar if the respective words are often used in similar contexts. That means, the words like <code>ugly</code> and <code>beautiful</code> may have similar word vectors since both words are used to describe appearances. These word embeddings generally do not capture the “polarity” of words. That’s not just a guess, but I’ve noticed this first hand when I <a href="https://textminingonline.com/training-word2vec-model-on-english-wikipedia-by-gensim" rel="nofollow noopener">trained a Word2Vec model</a> myself, and checked the most similar words of adjectives.</p>

James_McCaffrey: <p>I worked on a project where we compared GloVe embeddings vs. an Embedding layer for one specific NLP classification problem. We got inconclusive results. Briefly, both approaches gave similar error and accuracy results. In the end, we used an Embedding layer because it was simpler to work with (and the extra training time was manageable).</p>
Pytorch cuda memory leak

user29: <p>When I do multiplication matrices and does not assign result any variable it increase gpu memory usage</p>
<p><a href="http://joxi.ru/J2bwnJxHqXlDKA" class="onebox" target="_blank" rel="noopener nofollow ugc">http://joxi.ru/J2bwnJxHqXlDKA</a></p>
<p>I can find this tensors in gc objects</p>
<p><a href="http://joxi.ru/GrqREVlHzQw9am" class="onebox" target="_blank" rel="noopener nofollow ugc">http://joxi.ru/GrqREVlHzQw9am</a></p>
<p>but can’t remove its</p>
<p>Same question was here but without answer <a href="https://stackoverflow.com/questions/66947763/gpu-memory-usage-accumulating-when-calling-function-even-though-tensor-isnt-sto" class="inline-onebox" rel="noopener nofollow ugc">python - GPU memory usage accumulating when calling function even though tensor isn't stored - Stack Overflow</a></p>
<p>Why it happen and how find and remove this objects</p>

user29: <p>It is not pytorch, it is IPython. IPython create variables like ___{number of row}. If remove this variable with all references (use %xdel) memory free</p>
Access GPU partitions in MIG

Caipi: <p>Hello,</p>
<p>I have been given access to a GPU cluster where the GPUs (2x NIVIDIA A100 80GB) are partitioned using <a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/" rel="noopener nofollow ugc">MIG</a> to partition their GPUs into sub-elements…</p>
<p>Unfortunately, the I cannot find an example which can show me how to access the part via a given UUID of the sub element (MIG-11c29e81-e611-50b5-b5ef-609c0a0fe58b)… Or rather how to tell torch to use that?</p>
<p>device(“cuda:0”) would not be enough, it only describes the GPU the partition is placed on…</p>

ptrblck: <p>You could use <code>CUDA_VISIBLE_DEVICES</code> to specify the desired MIG instance.</p>

Caipi: <p>This variable does not exist… StackOverflow suggests to export like “export CUDA_VISIBLE_DEVICES=0” but what would then CUDA_VISIBLE_DEVICES give me? I already know all ids?</p>

ptrblck: <p>You have to define this env variable as given in the <a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/#cuda-gi">user guide</a>:</p>
<pre><code class="lang-python">$ nvidia-smi -L
GPU 0: A100-SXM4-40GB (UUID: GPU-e86cb44c-6756-fd30-cd4a-1e6da3caf9b0)
  MIG 3g.20gb Device 0: (UUID: MIG-GPU-e86cb44c-6756-fd30-cd4a-1e6da3caf9b0/1/0)
  MIG 3g.20gb Device 1: (UUID: MIG-GPU-e86cb44c-6756-fd30-cd4a-1e6da3caf9b0/2/0)


$ CUDA_VISIBLE_DEVICES=MIG-GPU-e86cb44c-6756-fd30-cd4a-1e6da3caf9b0/1/0 ./BlackScholes &amp;
$ CUDA_VISIBLE_DEVICES=MIG-GPU-e86cb44c-6756-fd30-cd4a-1e6da3caf9b0/2/0 ./BlackScholes &amp;
</code></pre>
A large number of indexing operations result in very slow back propagation in pytorch

shelvey_jiang: <p>when I use the following code to train my model in pytorch, I find the backward operation is extremely slow.</p>
<pre><code class="lang-auto">def sampling_prob(prob, label, num_neg):
    label = label - 1  
    batch, pred_len, l_m, = prob.shape[0], prob.shape[1], prob.shape[2] - 1
    init_label = np.zeros_like(label)
    init_prob = torch.zeros(size=(batch, pred_len, num_neg + batch))  # torch.Size([32, 12, 42])

    for step in range(pred_len):
        _label = np.linspace(0, batch - 1, batch)
        _prob = torch.zeros(size=(batch, num_neg + batch))

        batch_step_label = label[:, step]

        random_ig = random.sample(range(1, l_m + 1), num_neg)  
        while len([lab for lab in batch_step_label if lab in random_ig]) != 0:  
            random_ig = random.sample(range(1, l_m + 1), num_neg)

        global global_seed
        random.seed(global_seed)
        global_seed += 1

        for k in range(batch):
            for i in range(num_neg + batch):
                if i &lt; batch:
                    _prob[k, i] = prob[k, step, batch_step_label[i]]
                else:
                    _prob[k, i] = prob[k, step, random_ig[i - batch]]

        init_label[:, step] = _label
        init_prob[:, step, :] = _prob

prob = self.model(train_input, train_m1, self.mat2s, train_m2t, train_len)
prob_sample, label_sample = sampling_prob(prob, train_label, self.num_neg)
loss_train = F.cross_entropy(prob_sample, label_sample)
loss_train.backward()
optimizer.step()
optimizer.zero_grad()
</code></pre>
<p>Indeed, I want to sample some data from the the output of the model, i.e.,prob, to calculate the loss and optimize the parameters. But the speed of the backward operation is extremely slow. I am wondering this is because a lot of indexing operations. What would be the reason for that?  How can I solve this problem?</p>

googlebot: <p>This records all _prob tensor “versions”, the framework doesn’t know that writes are non-overlapping and intermediate states are discardable. Then you have a long chain of autograd nodes to be processed during backward().</p>
<p>Workaround is to use torch.gather to set all values at once. Another option is to rewrite this with torch.where &amp; torch.cat, similarly avoiding assignments to a buffer.</p>