akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Trying to apply custom kernels in convolution. Getting size errors

s_mkb: <p>Hi there,</p>
<p>I want to add one convolutional layer within my network with my own, custom weights. The remaining convolutional layers are not custom, and generated with Conv2d. The custom weights are generated with nn.functional.conv2d.</p>
<p>Basically, my data will have 64 channels when it enters my custom convolutions. I have 8 3x3 kernels I want to apply to EACH of the 64 channels, so that my out_channels is 512, but I’ve been having issues with the tensor size.</p>
<pre><code class="lang-auto">def tile(self, a, dim, n_tile):
        init_dim = a.size(dim)
        repeat_idx = [1] * a.dim()
        repeat_idx[dim] = n_tile
        a = a.repeat(*(repeat_idx))
        order_index = torch.LongTensor(np.concatenate([init_dim * np.arange(n_tile) + i for i in range(init_dim)]))
        return torch.index_select(a, dim, order_index)

weights = torch.tensor([[
                        [[1, 0., -1.],
                         [1., 0., -1.],
                         [1., 0., -1.]],
                        [[1., 1., 1.],
                         [0., 0., 0.],
                         [-1., -1., -1.]],
                        [[1., 1., 1.],
                         [-2., -2., -2.],
                         [1., 1., 1.]],
                        [[0., 0., 0.],
                         [0.,1., 0.],
                         [0., 0., 0.]],
                        [[1., 0., -1.],
                         [-2., 0., 2.],
                         [1., 0., -1.]],
                        [[1., 0., -1.],
                         [-2., 0., 2.],
                         [1., 0., -1.]],
                         [[1., -2., 1.],
                         [0., 0., 0.],
                         [-1., 2., 1.]],
                        [[1., -2., 1.],
                         [-2., 4., -2.],
                         [1., -2., 1.]],
                         ]])
        weights = self.tile(weights,0,64)
        weights = weights.view(64,8,3,3)
        weights = weights.double()
        print(weights)
        print(weights.size())
</code></pre>
<p>A snippet of my forward function looks like this:</p>
<pre><code class="lang-auto">encode2 = self.batch2(func.relu(self.conv2(x_in)))
        encode3 = self.batch3(torch.tanh(self.conv3(encode2)))
        encode4 = torch.tanh(func.conv2d(input=encode3, weight=weights, stride=2, groups=64))
        encode5 = self.batch6(torch.tanh(self.conv5(encode4)))
</code></pre>
<p>I am getting the following output:</p>
<pre><code class="lang-auto">torch.Size([64, 8, 3, 3]) # size of weights 

ERROR message:
encode4 = torch.tanh(func.conv2d(input=encode3, weight=weights, stride=2, groups=64))
RuntimeError: Given groups=64, weight of size [64, 8, 3, 3], expected input[20, 64, 25, 15] to have 512 channels, but got 64 channels instead
</code></pre>
<p>20 is my batch size. I’ve tried a bunch of things and I’m unsure how to resolve this.</p>
How does PyTorch use word embeddings to represent sentences for sentiment analysis?

kodlekereanup: <p>I’ve done a lot of reading and I’ve seen that word2vec or GloVe embeddings are used in sentiment analysis tasks to retain semantic value of words. But one thing I still don’t understand is, how does a word embedding represent a sentence?</p>
<p>For example, for a twitter sentiment analysis project I want to classify a tweet into positive/negative/neutral.</p>
<pre><code class="lang-auto">Raw Input: The weather does not look good tonight.
</code></pre>
<pre><code class="lang-auto">Pre-processed Input before converting to vectors:  weather not good tonight
</code></pre>
<p>Just assume some random vectors for each of those words above.</p>
<p>weather: [34, 56…768] etc.</p>
<p>Now, how do I represent the sentence “weather not good tonight” using those word embeddings? I read somewhere that we need to average the word vectors for each of the words in the sentence and that’ll do the job.</p>
<p>But then how will it capture the order of the words in training? Also multiple inputs can lead to the same mean, how will the LSTM handle this?</p>

vdw: <p>I’m not sure if I understand your questions, but a sentence is a sequence of words. So using word embeddings, a sentence is a sequence of embeddings (i.e., vectors). Thus a sentence is represented as a 2-dim tensor of shape <code>(num_words, embed_dim)</code>.</p>
<p>In practice, you also of the dimension for the batch to handle multiple sentences in parallel. So the final representation is then a 3-dim tensor if shape <code>(batch_size, num_words, embed_dim)</code>.</p>
<p>Note, in many documentations, you will probably see <code>seq_len</code> instead of <code>num_words</code> since LSTM are not only used for sentences.</p>
Teacher Forcing for LSTM in model.eval() and model.train() mode

francescoalongi: <p>Hello,<br>
I was wondering whether the statements <code>model.eval()</code> and <code>model.train()</code> affect the pytorch <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">LSTM</a> module in any way.<br>
As a default, following the theory of a seq2seq model, when the model is set to training mode, it should apply teacher forcing, while instead, it should not apply teacher forcing when the model is set to evaluation mode. If so, why it is not clearly specified in the docs? Is there anything obvious which I am not seeing?</p>
<p>Thanks a lot</p>

vdw: <p>Teacher forcing is not intrinsic to the model but how you use the model. So <code>eval()</code> and <code>train()</code> has nothing to do with. Sure, if you make inferences, you should always set a model to <code>eval()</code>.</p>
<p>If you check the <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="noopener nofollow ugc">PyTorch Seq2Seq tutorial</a>, teacher forcing is only a thing in the <code>train()</code> method, but not in the <code>evaluate()</code> method for making inferences. Here, you simply have no ground truth, so the prediction of the next word depends on the last predicted word.</p>
Normal(): optimization step brings in nan into the model

sureshj: <p>I have a neural network representing the policy for an RL agent.<br>
Sometimes the training process fails with <code>nan</code> values in the model parameters.</p>
<p>I realized that it happens when the policy is iterated over a very similar <code>loc</code> for <code>Normal</code> when the <code>scale</code> is already very small. I have reproduced the error below.</p>
<ol>
<li>It happens when the training process is iterated even after the values for <code>loc</code> and <code>scale</code> are well converged. Is there any way to detect this and stop the iteration? But things could get complicated if we have an array of loc and scale.</li>
<li>Is <code>sigmoid</code> a good choice for an activation function whose output goes as <code>scale</code> for <code>Normal</code> ?</li>
</ol>
<pre><code class="lang-auto">import torch as th

loc = th.tensor(0.5, requires_grad=True)
scale = th.tensor(-45., requires_grad=True)

optimizer = th.optim.Adam((loc, scale))

for i in range(10):
    N = th.distributions.Normal(loc, scale.sigmoid())
    loss = -N.log_prob(th.tensor(0.5))
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
Alternating dataloader

erap129: <p>Hello,<br>
Is there a pre-existing dataloader somewhere that accepts N datasets and loads samples from a different dataset each time (say, in a round-robin fashion).<br>
I need this for some form of naïve supervised transfer learning method I am trying via a multi-headed network with a common feature extraction module. The output of each batch will go into head <em>n</em>, where <em>n</em> is the <em>n</em>’th dataset.</p>
Training process keep getting killed

tagynedlrb: <aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/eriklindernoren/PyTorch-YOLOv3" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/7977505?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/eriklindernoren/PyTorch-YOLOv3" target="_blank" rel="noopener nofollow ugc">eriklindernoren/PyTorch-YOLOv3</a></h3>


  <p><span class="label1">Minimal PyTorch implementation of YOLOv3. Contribute to eriklindernoren/PyTorch-YOLOv3 development by creating an account on GitHub.</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<p>
Here’s where I got the code, and run custom training.</p>
<p>I ran my code “python3 train.py --model_def config/yolov3-custom.cfg --data_config config/custom.data”,<br>
but the process is keep getting killed.<br>
Mac OS, python 3.7 is my current spac(no GPU, only by CPU)<br>
According to following article(<a href="https://stackoverflow.com/questions/64772335/pytorch-w-parallelnative-cpp206" class="inline-onebox" rel="noopener nofollow ugc">parallel processing - Pytorch : W ParallelNative.cpp:206 - Stack Overflow</a>),<br>
I’ve handled my multiprocessing error issue, and edited my [n_cpu = 0].<br>
Also, I even resized my data 1/100, which is only 8KB each, and put only 3 of 'em.<br>
I don’t get what’s wrong with it…</p>
<p>---- mAP 0.0<br>
Training Epoch 4: 100%|███████████████████████████| 1/1 [00:09&lt;00:00, 9.19s/it]</p>
<p>---- Evaluating Model ----<br>
Detecting objects: 0%| | 0/1 [00:00&lt;?, ?it/s]<br>
zsh: killed python3 train.py --model_def config/yolov3-custom.cfg --data_config</p>

ptrblck: <p>Could you try to run it via <code>gdb</code> and see, if you would get any proper error?</p>
<pre><code class="lang-python">gdb --args python train.py --model....
...
run
...
bt
</code></pre>

tagynedlrb: <p>I got trouble trying using gdb, but fortunately I solved my problem!<br>
I still don’t know what was the main problem, but I guess I didn’t insert plenty of datasets…<br>
or else just by my mistake on valid.txt &amp; train.txt data…Thanks for your help!</p>
Tensor assignment with cascaded masking

TinfoilHat0: <p>I’ve a tensor T, and two boolean masks b1, and b2.</p>
<p>I’d like to do the assignment T[b1][b2] = c but this doesn’t seem to be working. I mean it doesn’t change the values of T.</p>
<p>Why’s that, and how can I do the assignment properly?</p>
Use dataloader to load images from single class

SiddharthSingi: <p>Hi,<br>
I am looking to build a Gan model using pytorch which requires me to load images of different classes seprerately. I have used the <code>torchvision.datasets.imagefolder</code> to load my dataset. I have images of horses and zebras. Now how can I load the images from only one class of the data. That is, I want to be able to specify to my data loader whether I want images of horses or zebras?</p>

ptrblck: <p>You could extract all indices for the desired class (e.g. horses) and create a <code>torch.utils.data.Subset</code> using these indices and the dataset before passing it to the <code>DataLoader</code>.<br>
Alternatively, you could also pass these indices to a <code>RandomSubsetSampler</code> and assign it to the <code>DataLoader</code>.</p>

SiddharthSingi: <p>Hi,<br>
Thank you for the solution <a class="mention" href="/u/ptrblck">@ptrblck</a> . Using <code>torch.utils.data.Subset</code> did the trick for me.</p>
<p>Here is how I managed to do it.</p>
<pre><code class="lang-auto">import torchvision.datasets as dset
from torch.utils.data import DataLoader, Subset

gan_dataset = dset.ImageFolder(root=dataroot, transform=transforms.Compose([
                                            transforms.Resize(256),
                                            transforms.ToTensor(), 
                                            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]))

# the ImageFolder class has the following variables which can be used to find 
# the range of indices for Subset:

print(gan_dataset.class_to_idx)
# gan_dataset.imgs is a list of tuples of (file_path, class_index) for all items in the dataset
print(gan_dataset.imgs)

horse_dset = Subset(gan_dataset, range(260, 1327))
zebra_dset = Subset(gan_dataset, range(1327, 2670))

horse_loader = DataLoader(horse_dset, batch_size=5, shuffle = True)
zebra_loader = DataLoader(zebra_dset, batch_size=5, shuffle = True)

</code></pre>
Learning training / test errors and their accuracy

Yasar_Niyazoglu: <p>I trained a neural network. I did this to learn about my training and testing achievements as well as my error rates, do you think it is correct? can i trust my own code?  I want to get an idea about bias and variance.</p>
<p><em><strong>Sorry for my english I use google translation <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></strong></em>*</p>
<p>class ANN(nn.Module):<br>
def <strong>init</strong>(self,input_dim,hidden_dim,output_dim):<br>
super(ANN,self).<strong>init</strong>()</p>
<pre><code>    self.fc1=nn.Linear(input_dim,hidden_dim)
    self.aktivasyon1=nn.SELU()
    self.fc2=nn.Linear(hidden_dim,hidden_dim)
    self.aktivasyon2=nn.SELU()
    self.fc3=nn.Linear(hidden_dim,hidden_dim)
    self.aktivasyon3=nn.SELU()
    self.fc4=nn.Linear(hidden_dim,output_dim)
    
def forward(self,x):
    x=self.fc1(x)
    x=self.aktivasyon1(x)
    x=self.fc2(x)
    x=self.aktivasyon2(x)
    x=self.fc3(x)
    x=self.aktivasyon3(x)
    x=self.fc4(x)
    return x
</code></pre>
<p>input_dim=28*28<br>
hidden_dim=150<br>
output_dim=10</p>
<p>model=ANN(input_dim,hidden_dim,output_dim).to(device)</p>
<p>optimizer=torch.optim.SGD(model.parameters(),lr=0.001)</p>
<p>error=nn.CrossEntropyLoss()<br>
loss_list = []</p>
<p>for epoch in range(num_epochs):<br>
<span class="hashtag">#lr</span>.step()<br>
for i, (images,labels) in enumerate(train_l):<br>
images=images.to(device)<br>
labels=labels.to(device)</p>
<pre><code>    images=Variable(images.view(-1,28*28))
    labels=Variable(labels).to(device)
    
    optimizer.zero_grad()
    out=model(images.float())
    loss=error(out,labels)
    loss.backward()
    optimizer.step()
</code></pre>
<hr>
<hr>
<p>def check_accuracy(loader, model):</p>
<pre><code>total=0
correct=0
correct_error=0
model.eval()
with torch.no_grad():
        
    for images,labels in loader:
        images=images.to(device) 
        labels=labels.to(device)
        images=Variable(images.view(-1,28*28))
           
        out=model(images.float())
        y_pred=torch.max(out.data,1)[1]
        total+=len(labels)  
        correct += (y_pred == labels).sum()
        
    accuracy = 100 * correct / float(total)
    error= 100 - accuracy

    print('Accuracy: {}% Error: {}%'.format(accuracy,error))
model.train()
</code></pre>
<p>print(“Train Result:”)<br>
*****check_accuracy(train_l, model)<br>
print("----------------------------------------------------")<br>
print(“Test Result:”)<br>
*****check_accuracy(test_l, model)</p>
<h2>–&gt;Train Result:<br>
Accuracy: 89% Error: 11%</h2>
<p>–&gt;Test Result:<br>
Accuracy: 88% Error: 12%</p>

Dwight_Foster: <p>It looks like you can trust your model. Your code works fine and if the model is doing well on the test set that means it is not overfitting.</p>

Yasar_Niyazoglu: <p>thank you my friend. these views made me happy</p>
Has PyTorch (as of 1.8) dropped Python 2 support?

lithuak: <p>Hi!</p>
<p>While working on minor contribution I’ve stumbled across some Python2-specific code and I’m curious if Python 2 support has been already dropped?</p>
<p>In this case I’ll push a PR to remove this old code.</p>
<p>Thanks!</p>

ptrblck: <p>Yes, Python 2 support was removed in 2020, which is also the EOL of Python 2, as described <a href="https://discuss.pytorch.org/t/dropping-python-2-support-on-january-1st-2020/56730">here</a>.</p>

lithuak: <p>Thanks you very much, <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
How to implement "Transfer Learning" on "YOLOv5"

praveen_sekar: <p>Hi,<br>
Please advise me, how to implement “Transfer Learning” for YOLOv5(<a href="https://pytorch.org/hub/ultralytics_yolov5/" class="inline-onebox" rel="noopener nofollow ugc">YOLOv5 | PyTorch</a>) instead of resnet18 used in the below tutorial.</p>
<p>Source - Transfer Learning(<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 1.8.0 documentation</a>)</p>
<p>Thanks.</p>
Where can I get FX Graph Mode Quantization documents?

Jonson: <p>The following FX pages are 404.</p>
<ul>
<li><a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_quant_guide_tutorial.html" rel="noopener nofollow ugc">User Guide on Using FX Graph Mode Quantization</a></li>
<li><a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static_tutorial.html" rel="noopener nofollow ugc">FX Graph Mode Post Training Static Quantization</a></li>
<li><a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_dynamic_tutorial.html" rel="noopener nofollow ugc">FX Graph Mode Post Training Dynamic Quantization</a></li>
</ul>

ptrblck: <p>Based on a comment from the <a href="https://github.com/pytorch/tutorials/tree/3e11c1f1dd542fed421cc6e90235d7aa667edcf7/prototype_source">source</a> it seems they are intentionally not shows, since they are prototypes:</p>
<blockquote>
<p><strong>Prototype features</strong> are not available as part of binary distributions like PyPI or Conda (except maybe behind run-time flags). To test these features we would, depending on the feature, recommend building from master or using the nightly wheelss that are made available on <a href="http://pytorch.org">pytorch.org</a>.</p>
</blockquote>
<blockquote>
<p>These are intentionally left out of the <a href="http://pytorch.org/tutorials" class="inline-onebox">Welcome to PyTorch Tutorials — PyTorch Tutorials 2.1.1+cu121 documentation</a> build and will not show up on the website.</p>
</blockquote>
Different seeds but weights are still init'ed same for different processes in DDP

sarv_ghotra: <p>I am setting different seed for each process (distributed training setting) but the models weights are still same for all the processes. Is it an expected behavior?</p>
<p>PS: I was under the impression that since weight init depends on seed so, different seed will lead to different weights init for each process.</p>

sio277: <p>Yes. At the time of DDP wrapping, parameters and buffers (i.e., model.state_dict()) on rank0 are broadcasted to all other ranks. So although weights on different ranks were initialized from different seeds, they will have the same starting point (rank0’s state) after DDP construction.</p>

wayi: <p>sio277 already gave the right answer. This behavior is expected.</p>
<p>DDP must keep the weights on different processes synced at the beginning, so operating the same initial weights and synced gradients can lead to the same updated weights, and this makes the distributed training equivalent to the sequential version. Otherwise distributed training will be mathematically wrong.</p>
BatchNorm1d calls torch.batch_norm, which does not look existing in either scripts or documents

sekigh: <p>I analyze the behavior of BatchNorm1d by scripts and debugger and come across following issues;</p>
<ol>
<li>
<p>BatchNorm1d goes to def <strong>call</strong> in nn.Module, which executes self.forward() in nn.Module. Then it  goes to def forward() in nn.Module.batchnorm then it calls Functional.batch_norm(). Then it goes to torch.batch_norm.</p>
</li>
<li>
<p>I tried to see the contents of torch.batch_norm but I can not locate scripts nor documents. Also torch.batch_norm looks running without going that corresponding scripts and without  any  error when debug and execution.  Is there anyone to explain this situation ? Where can I see the scripts ? Does it run properly or any issues behind this case ?</p>
</li>
</ol>
<p>sekigh</p>

sekigh: <p>One additional information. I pointed a cursor at torch.batch_norm() on eclipse debugger and a message " builtin_function_or method:&lt;built-in method batch_norm of type object at xxxxx." shows up. Where do I see document for it and what are returned as a result ?<br>
sekigh</p>

king_wang: <p>I met the same problem with u. wait for answer</p>

dkwasny: <p>I guess the final call is calling some precompiled C++ code? That python file with type annotations looks a lot like a C++ header file, so my guess is that it kinda works like one, though I do not know the details of   communication between the C++ code and the python API. Wouldn’t mind getting some link describing it briefly though.</p>

sekigh: <p>Thank you for your reply. I can not locate C++code file you mentioned. Could you point at locations of that python file and its associated C++ file if possible.  I would like to look into them. Thank you in advance.</p>

fkemeth: <p>I am also wondering the same. If you build from source, running</p>
<pre><code class="lang-auto">grep -R "batch_norm" torch/csrc/
</code></pre>
<p>in the pytorch repo returned some results. But I also couldn’t find the actual equation that does the normalization.</p>

ptrblck: <p>The CPU implementation of the normalization can be found <a href="https://github.com/pytorch/pytorch/blob/700109eb630b79fd65cb93becb7f2d14f93bdb5c/aten/src/ATen/native/Normalization.cpp#L238-L260">here</a>. The source file also includes the backward pass, stats updates, etc.</p>

guoxiwuruo: <p>thanks for your answer, how can i debug the cpu code？</p>

ptrblck: <p>You could use a debugger such as <code>gdb</code>.</p>

guoxiwuruo: <p>step from torch.batch_norm into the cpu source code?</p>

ptrblck: <p>Yes, you should be able to set breakpoints via <a href="https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_28.html#SEC29"><code>break</code></a>.</p>
Build integration for PyTorch in Glow

jr.r: <p>Hi, I am trying to integrate pytorch in glow, but I got the following errors:</p>
<p>In file included from glow/torch_glow/src/TorchGlowBackend.cpp:2:<br>
glow/torch_glow/src/TorchGlowBackend.h:40:23: error: only virtual member functions can be marked ‘override’<br>
bool is_available() override;</p>
<p>glow/torch_glow/src/TorchGlowBackend.cpp:161:21: error: no matching constructor for initialization of ‘torch::jit::backend’<br>
static auto cls = torch::jit::backend(“glow”, preprocess);</p>
<p>Thank you</p>

jfix: <p>Hi, I imagine this is due to the build environment/compiler (and its version) you’re using. I would guess fixing that would be simple (mark <code>is_available</code> as virtual). If you try out a fix and it works feel free to submit a PR to fix it, thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Performance comparison of nn.Module vs jit.ScriptModule

shivakanthsujit: <p>I am working on code that subclasses the models using jit.ScriptModule but still uses them in native python and doesnt convert it to C++ using the trace function. In this case, is there any performance benefit from using the ScriptModule or can I change all the models back to nn.Module without any degradation in speed if I am running python scripts only?</p>
Sync inside forward() in Module?

111480: <p>I found Synchronize() on the nvprof even though I did not sync after creating a non-default stream as non-blocking.<br>
Does this happen inside the forward() function of the module?<br>
If so, let me know exactly where it happens on the code.</p>

ptrblck: <p>It depends on your model implementations and the used operations.<br>
To isolate it further you could use nvtx markers and check where the synchronization is happening (via <code>torch.cuda.nvtx.range_push</code> and <code>torch.cuda.nvtx.range_pop</code>).</p>

111480: <p>I am using the model provided by Torchvision.<br>
Is it possible only with NVTX?</p>
<p>I use nvprof and I could know whether the cudaDeviceSynchronize(), cudaStreamSynchronize() function was called, but it’s not known who is calling synchronize().</p>
Share the CUDA context created by pytorch

k-ye: <p>Hi,</p>
<p>Our program runs both Pytorch and another library (<a href="https://github.com/taichi-dev/taichi" rel="noopener nofollow ugc"><code>taichi</code></a>) in the same process.  Since both packages use CUDA, this has caused a problem when running the program on a GPU device that’s set to <code>EXCLUSIVE_PROCESS</code> mode. We are thinking about changing <code>taichi</code> to re-use the CUDA context created by pytorch, but wonder if this is a good practice? (Even if pytorch creates one CUDA context per device throughout the execution lifetime, that seems like an implementation detail) If not, is there any recommendation for how to workaround the problem, assuming we have only a single GPU?</p>
<p>Thanks</p>

StoicLD: <p>May be you could use the CUDA driver API cuCtxGetCurrent, it shall get the current cuda context.</p>

k-ye: <p>Thanks for your suggestion <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Yeah I’ve tried that and it actually worked (on a simple test case). What I mainly wanted to know is if PyTorch guarantees that the CUDA context is not destroyed throughout the lifetime of the process. Without this guarantee, I think it’s better not to assume that the context is sharable.</p>
How can I use multiple GPUs on WSL2?

Naruki_Ichihara: <p>Hello,</p>
<p>I try to use multiple GPUs (RTX 2080Ti *2) with torch.distributed and pytorch-lightning on WSL2 (windows subsystem for linux).<br>
But I receiving following error:</p>
<pre><code class="lang-auto">NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:518, unhandled system error, NCCL version 2.4.8
</code></pre>
<p>I can run same process using single GPU.</p>
<p>Thanks for your great help.</p>

ebarsoum: <p>Is SLI enabled? What the output of nvidia-smi?</p>

Naruki_Ichihara: <p>Thank you for your reply.<br>
Since I use CUDA on WSL (<a href="https://docs.nvidia.com/cuda/wsl-user-guide/index.html#getting-started" rel="nofollow noopener">https://docs.nvidia.com/cuda/wsl-user-guide/index.html#getting-started</a>), Ubuntu (on WSL) cannot use the “nvidia-smi” utility.<br>
nvidia-smi on windows shell output following:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.41       Driver Version: 455.41       CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208... WDDM  | 00000000:1A:00.0  On |                  N/A |
| 41%   50C    P0    59W / 250W |    662MiB / 11264MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208... WDDM  | 00000000:68:00.0 Off |                  N/A |
| 27%   35C    P8    20W / 250W |    216MiB / 11264MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
</code></pre>
<p>Does this output mean SLI is enabled?</p>

phamvanvung: <p>I have the same question, it would be great if we could have the answer for this?</p>

solarflarefx: <p><a class="mention" href="/u/naruki_ichihara">@Naruki_Ichihara</a> did you ever figure this out?</p>
Effective Model Tuning

Niranjan_Rao: <p>Hello, I am training a Unet model for Image Segmentation. I wanted to know what is the best way to carry out tests. I want to evaluate the effect of pre-filtering images. data augmentation, adding an extra layer etc.</p>
<p>I am training my model for 100 epochs. What is the most efficient to evaluate models? Do I have to retrain the entire model and compare the validation scores? This is really time consuming, is there a smarter way to do this?</p>