akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
AttributeError: module 'pyexpat.model' has no attribute 'eval'

danielmanu93: <p>Hi All,</p>
<p>I have an <code>evaluate</code> function that performs testing after training the model. The function runs very well without any errors however I want to link the function to a button so that when the button is pressed, the testing can be done. I have created the button and fed the function to the command argument of the button but when I press the button, the <code>model.eval()</code> throws this error <code>module 'pyexpat.model' has no attribute 'eval'</code>. Below is my code for demostration:</p>
<pre><code class="lang-auto">def evaluate(model, dataloader, data_min, data_max,  label_max, label_min):    
    model.eval()
    for i in range(1):
        data, label =iter(dataloader).next()
        pred = model(data)

def main(args):
    global data_max, data_min, label_max, label_min, model, dataloader_valid
        data_min = -20
        data_max = 38
        label_min = 1500
        label_max = 4500

        dataset_valid = torch.load(args.val_anno)
        dataloader_valid = torch.utils.data.DataLoader(
        dataset_valid, batch_size=args.batch_size, shuffle=True)
        model = network.model_dict[args.model]()
        chkpnt = torch.load(args.resume, map_location='cpu')
        model.load_state_dict(chkpnt['model'])

        evaluate(model, dataloader_valid, data_min, data_max, label_min, label_max)

def parse_args():
    import argparse
    parser = argparse.ArgumentParser(description='Testing')
    args = parser.parse_args()
    return args

def combineFunc(*funcs):
       def combinedFunc(*args, **kwargs):
            for f in funcs:
                f(*args, **kwargs)
       return combinedFunc

model_button = customtkinter.CTkButton(master=root, image = Inv, text="", corner_radius=1, fg_color="white", 
                        command = combineFunc(lambda: evaluate(model, dataloader_valid, data_min, data_max, label_min, label_max), 
                                            lambda: args, 
                                           lambda: main(args)))
</code></pre>
11.3 binaries compatibel with Cuda 11.7?

atbolsh: <p>Hello all. I have Cuda 11.7, with the 515 driver, on my system. I wanted to see if I could get a functional version of pytorch quick-and-dirty (no downgrading or building from source), so I used this command, not really expecting it to work:</p>
<p>conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch</p>
<p>Surprisingly, these binaries seem to work, at least for some basic things I’ve tested so far. Will this work in general? Or did conda install a local 11.3 cuda toolkit? Either way - should I be very worried about unstable behavior? Or is it probably fine?</p>

ptrblck: <p>The binaries ship and will use their own CUDA runtime and your local CUDA toolkit will be used if you are building PyTorch from source or e.g. a CUDA extension.<br>
To run the binaries you would only need to install the NVIDIA driver.</p>
Swapping labels

rjsdebug: <p>Hello,<br>
I am trying to load mnist data with a slight twist. I want to change the labels of some small percent of the training data to a different class for some experiments. Is there a way to easily do this with pytorch dataloader?</p>

ptrblck: <p>Depending on your use case you could use several different approaches:</p>
<ul>
<li>manipulate the internal <code>.targets</code>  attribute and change some labels</li>
<li>derive a custom <code>Dataset</code> from <code>MNIST</code> and change some labels in the <code>__getitem__</code> on the fly</li>
<li>manipulate some labels of the current batch in the training loop</li>
<li>use a custom <code>collate_fn</code> to manipulate some targets (this seems unnecessarily complicated)</li>
</ul>

rjsdebug: <p>Thank you, changing targets is likely the solution for me. Does pytorch allow some sort of image hashing, for me to remember which images’ labels I changed even after a shuffle of the dataset?<br>
EDIT: I can just hash the tensor of the image in the dataset, correct? And then for lookup I need to use the same tensor object from the dataset? Unless there is a way to hash based on tensor value</p>
Tensor access with tuples causes error for sizes less than 32

arrowtea: <p>The following code does not work if N &lt; 32:</p>
<pre><code class="lang-auto">N = 31
a = torch.ones(N, N)
indices = [(i, i) for i in range(N)]
print(a[indices])
</code></pre>
<p>The error it outputs is:</p>
<pre><code class="lang-auto">IndexError: too many indices for tensor of dimension 2
</code></pre>
<p>But if N &gt;= 32, it works. I suspect this might be a bug in PyTorch itself?</p>
<p>My PyTorch version is 1.11.0.</p>
<p>I found a similar topic discussing it here: <a href="https://stackoverflow.com/questions/71176095/pytorch-tensor-indexing-error-for-sizes-m-32" class="inline-onebox" rel="noopener nofollow ugc">python - Pytorch tensor indexing error for sizes M &lt; 32? - Stack Overflow</a></p>
First inferences are faster then normal at TorchScript model

vodan37: <p>3080 and V100<br>
Ubuntu 18.04<br>
driver 510.47.03 / 470.42.01 (V100)<br>
cuda_11.6<br>
torch 1.11.0+cu113</p>
<p>I have a model which i converted to TorchScript and save:</p>
<p>traced_script_module = torch.jit.trace(model, example)<br>
traced_script_module.save(“traced_resnet_model.pt”)</p>
<p>then I load this model and try to inference (no matter rand or real image):</p>
<p>model = torch.jit.load(weights).eval().half()<br>
for i in range(20):<br>
t1 = time.time()<br>
# pred = model(im)<br>
pred = model(torch.rand((1, 3, 640, 640), dtype=torch.half, device=‘cuda:0’))<br>
print(time.time() - t1)<br>
Times:<br>
0.13088488578796387<br>
1.0498087406158447<br>
0.0052030086517333984<br>
0.0051364898681640625<br>
0.0077664852142333984<br>
0.010211706161499023<br>
0.010109663009643555<br>
0.010195255279541016<br>
0.010144472122192383<br>
0.010176897048950195<br>
0.010180473327636719<br>
…<br>
As expected, we see at 1st and 2nd iterations some trash (warmup). But after it we see some iterations are faster then others:<br>
0.0052030086517333984<br>
0.0051364898681640625<br>
0.0077664852142333984</p>
<p>How explain it? Is there any way how to avoid it?</p>

ptrblck: <p>CUDA operations are executed asynchronously so you would need to synchronize the code manually via <code>torch.cuda.synchronize()</code> before starting and stopping the timers.</p>

vodan37: <p>Thanks a lot! Time is 10 ms now.</p>
Deadlock when freeing memory

dfalbel: <p>Hello,</p>
<p>I’m seeing a deadlock when trying to delete tensors from inside a  <code>c10::FreeMemoryCallback</code>, if it ends up executed from a thread that is not the main thread.</p>
<p>I’m using <code>REGISTER_FREE_MEMORY_CALLBACK</code>  to register a callback that calls the R garbage collector whenever LibTorch’s allocator needs more memory. That works fine when this is called from the main thread. When the callback is executed from the main thread everything works fine and memory is correctly released.</p>
<p>However, in some situations we call LibTorch from a different thread, because we need the main thread to be free to execute arbitrary R functions (eg backward hooks), when this happens, we call the R GC from the main thread and in turn start deleting allocated Tensors from the main thread.</p>
<p>This is causing a deadlock with a backtrace like this in the main thread:</p>
<pre><code class="lang-auto">#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff51620f4 in __GI___pthread_mutex_lock (mutex=0x555561b739a0) at ../nptl/pthread_mutex_lock.c:115
#2  0x00007fffe6b1de13 in c10::cuda::CUDACachingAllocator::raw_delete(void*) ()
   from /home/dfalbel/torch/lantern/build/libtorch/lib/libc10_cuda.so
#3  0x00007fffe92e3a4c in c10::TensorImpl::release_resources() ()
   from /home/dfalbel/torch/lantern/build/libtorch/lib/libc10.so
#4  0x00007fffe751c64d in c10::intrusive_ptr&lt;c10::TensorImpl, c10::UndefinedTensorImpl&gt;::reset_() ()
   from /home/dfalbel/torch/inst/lib/liblantern.so
#5  0x00007fffe7518c46 in c10::intrusive_ptr&lt;c10::TensorImpl, c10::UndefinedTensorImpl&gt;::~intrusive_ptr() ()
   from /home/dfalbel/torch/inst/lib/liblantern.so
</code></pre>
<p>And here is the thread that called R gc:</p>
<pre><code class="lang-auto">#8  0x00007fffe7eaf7e3 in call_r_gc(bool) () at autograd.cpp:408
#9  0x00007fffe763624f in c10::GarbageCollectorCallback::Execute() () from /home/dfalbel/torch/inst/lib/liblantern.so
#10 0x00007fffe6b3948b in c10::cuda::CUDACachingAllocator::DeviceCachingAllocator::malloc(int, unsigned long, CUstream_st*) () from /home/dfalbel/torch/lantern/build/libtorch/lib/libc10_cuda.so
#11 0x00007fffe6b3acdc in c10::cuda::CUDACachingAllocator::THCCachingAllocator::malloc(void**, int, unsigned long, CUstr
</code></pre>
<p>I know this is not a detailed report and that it’s hard to answer precisely but any hint is appreciated regarding what could be causing the lock.</p>
<p>Thank you very much!</p>
How to use WeightedRandomSampler with KFold?

Arohan_Ajit: <p>I used the following code for weighted random sampling in my dataset:</p>
<pre><code class="lang-auto">     class_weights = [250,859]
     sample_weights = [0] * len(data)
     for idx, (inputs,label) in enumerate(data.imgs):
         class_weight = class_weights[label]
         sample_weights[idx] = class_weight
</code></pre>
<p>However when using stratified kfold, how can I use WeightedRandomSampler with the train and test idx provided by each split?</p>
<p>My implementation(which probably is incorrect since it used all the data for sampling as opposed to idx in the split):</p>
<pre><code class="lang-auto">splits = 2
kfold = StratifiedKFold(n_splits=splits,shuffle=True,random_state=2)
for fold,(train_idx,test_idx) in enumerate(kfold.split(data,data.targets)):
    print(f'FOLD {fold}')
    print('--------------------------------')

    print(train_idx)
    class_weights = [250,859]
    sample_weights = [0] * len(data)
    for idx, (inputs,label) in enumerate(data.imgs):
        class_weight = class_weights[label]
        sample_weights[idx] = class_weight

    train_sampler = WeightedRandomSampler(sample_weights, num_samples=len(sample_weights), replacement=True)

    test_sampler = SubsetRandomSampler(test_idx)
    loaders = {
    'train': torch.utils.data.DataLoader(data, batch_size=16, sampler=train_sampler),
    'test': torch.utils.data.DataLoader(data, batch_size=16, sampler=test_sampler)
    }
</code></pre>

Asma_Bouzidi: <p>I have the same problem did you find a solution, please?</p>
How to get indices from tensor where the scores (values) satisfies a condition after using torch.topk

deshwal.mahesh: <p>I have a Tensor of shape <code>NxN</code> which is basically the similarity or inner product of two tensors. I want to Get all the values which are above any threshold, say 0.5.</p>
<p>The result I’m looking for is something like: <strong>For each index <code>i</code> , get all the values which are closer than <code>0.5</code></strong>. Obviously for each index <code>i</code>, there’ll be a minimum 1 element (self similarity is 1.0) and can be a maximum of <code>N</code> elements (<code>similarities</code> is <code>NxN</code> matrix with the diagonal elements as <code>1</code></p>
<p>How could I go this?</p>
<pre><code class="lang-auto">x = torch.randn((9052, 512))
similarities = x @ x.T
scores, indices = torch.topk(similarities, x.shape[0]) # topk == shape[0] means get all the values, sorted
</code></pre>
<p>I have tried</p>
<pre><code class="lang-auto">mask = torch.ones(scores.size()[0])
mask = 1 - mask.diag()
sim_vec = torch.nonzero((scores &gt;= 0.5)*mask)
</code></pre>
<p>Gives me a tensor of  shape: <code>torch.Size([51147416, 2])</code></p>
<p>I’ve also tried</p>
<pre><code class="lang-auto">(scores &gt; 0.5 ).nonzero()
</code></pre>
<p>It too gives me a tensor of  shape: <code>torch.Size([51147416, 2])</code></p>

harpone: <aside class="quote no-group" data-username="deshwal.mahesh" data-post="1" data-topic="154082">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deshwal.mahesh/48/46943_2.png" class="avatar"> deshwal.mahesh:</div>
<blockquote>
<p>It too gives me a tensor of shape: <code>torch.Size([51147416, 2])</code></p>
</blockquote>
</aside>
<p>well that sounds about right… <code>scores</code> is a 9052x9052 matrix, about half of all elements match that condition. Maybe you want something else?</p>

deshwal.mahesh: <p>So how do I interpret the results in this case? Only thing I could think of was to loop through each element and save in a list but that’d be time consuming.</p>

harpone: <p>I think you could do something like this:</p>
<pre><code class="lang-auto">scores_flat = scores.flatten()
idx = (scores_flat &gt; 0.5 ).nonzero()
scores_match = torch.gather(scores, dim=0, index=idx)
</code></pre>
<p>So then <code>scores_match</code> would be the <code>scores</code> that match the condition <code>scores &gt; 0.5</code>. Haven’t tried this so may need some experimentation. Hope this is what you want!</p>
Change mask in RandomHorizontalFlip Transform

Armin_Dashti: <p>I’m learning object detection from this <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">tutorial</a>. in a part of that, The tutorial uses a transform to RandomHorizontalFlip for data augmentation.<br>
It uses this code:</p>
<pre><code class="lang-auto">def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)
</code></pre>
<p><code>img, target = self.transforms(img, target)</code></p>
<p>I got an error because the transform just gets an image but the tutorial pass two-argument (img, target)<br>
I remove the target and pass only img. It seems OK.<br>
But the dataset is for object detection and every image needs a mask. so if I use RandomHorizontalFlip then I need to change the mask. Right?<br>
How can I solve the problem?<br>
Thanks.</p>

InnovArul: <p>Pl. see if this answer helps with your question:</p>
<aside class="quote quote-modified" data-post="3" data-topic="153985">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/transforms-helper-function-has-no-attribute-totensor/153985/3">"transforms" helper function has no attribute ToTensor</a> 
  </div>
  <blockquote>
    I think that the tutorial was written based on an old version (v0.8.2) of Torchvision source code. 
You can see ToTensor() method in the file: 

Further, if you look at the <a href="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/torchvision_finetuning_instance_segmentation.ipynb">Google Colab file</a>, you can notice that somewhere down the line, they checkout a specific Github tag (v0.8.2) before executing the code. 
# Download TorchVision repo to use some files from
# references/detection
git clone https://github.com/pytorch/vision.git
cd vision
git checkout v0.8.2

cp references/detection/utils.py ../
…
  </blockquote>
</aside>


Armin_Dashti: <p>Thanks.<br>
I hope this tutorial will be corrected.</p>
Error when indexing: TypeError: only integer tensors of a single element can be converted to an index

papoo13: <p>I would like to find the indexes of certain targets and create a new data and target for my CIFAR10 dataset. This is how I’ve done it but I get the error.</p>
<p>class _CIFAR_Split(data.Dataset):</p>
<pre><code>def __init__(self, root, targets, split='train', transform=None,  download=False):
    self.root = os.path.expanduser(root)
    self.transform = transform
    self.split = split

    if self.split == 'train':
        # pdb.set_trace()
        self._data = torchvision.datasets.CIFAR10(root=self.root, train=True, download=download, transform=None)
        # inputs to full_like aren't accepted. First input must be a torch.tensor
        t = torch.Tensor(self._data.targets)
        idx = torch.full_like(t, 0, dtype=bool)
        # idx is a tensor in boolean type
        # targets = [0,1]
        for target in targets:
            idx = (idx | (self._data.targets == target))
        self._data.targets = self._data.targets[idx]
        self._data.data = self._data.data[idx]
</code></pre>
<p>I get the error in like <code>self._data.targets = self._data.targets[idx]</code>.<br>
I tried running in two environments but I get the same error.<br>
First environment: Python 3.6.9, Pytorch 1.4.0, torchvision 0.5<br>
Second environment: python 3.10, pytorch 1.11.0, torchvision 0.12</p>
<p>Is there a way to make this work? I think one possibility is to use np.where() but I would like to know if I can do any type conversions so this works. Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

harpone: <p>If you have a bool index, you’d get only lots of the first and second items in your targets… I suggest using an int dtype index.</p>
<p>Also please note that the loop above does nothing but just picks the last element of the loop, i.e. <code>target = 1</code>.</p>
Unexpected type for torch.norm()

promach: <p>For <a href="https://gist.github.com/buttercutter/b6f526c56e20f029d68e6f9041c3f5c0/8031304d01029090919d4c8cee3af9df480ab6f0#file-gdas-py-L823" rel="noopener nofollow ugc">torch.norm(Lval)</a> , how to get around <code>Unexpected type(s):(int)Possible type(s):(Tensor)(Tensor)(Tensor)(Tensor)</code> properly without upsetting the back-propagation mechanism ?</p>

harpone: <p><code>Lval</code> seems to be an <code>int</code> dtype. If you want to apply a normalization, first cast to float by e.g. <code>torch.norm(Lval.float())</code>.</p>

promach: <p>I just resolved the issue with <a href="https://github.com/buttercutter/gdas/commit/74a468c6e77de06a7aaa3038551136edb77daeba" rel="noopener nofollow ugc">this commit</a></p>
Undefined reference to torchaudio::sox_io::load_audio_file

pi-null-mezon: <p>I need to port Python code (simple 1D CNN that process audio records) to C++. So it is evident that I need to load audio records to torch::Tensor in C++ like it was done In Python (with <em>torchaudio.load()</em>). So I have build torchaudio.so from sources and link my C++ app to it. But at a compile time error appears ‘undefined reference to torchaudio::sox_io::load_audio_file…’. So the question is where implementation of  <em>torchaudio::sox_io::load_audio_file</em> resides? Please, help me</p>

pi-null-mezon: <p>For those, who will search the same: <a href="https://github.com/pi-null-mezon/torchaudio_cpp" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pi-null-mezon/torchaudio_cpp: How to make torchadio work in your C++ application</a></p>
Multi-label classification

se7en.dreams: <p>Hello,</p>
<p>I have a question regarding my classifier. Let me first explain my problem.</p>
<p><strong>Input</strong><br>
It consists of two things: text and 3D object features</p>
<p><strong>Text features</strong> are represented as: <code>10 x 256</code>, where <code>10</code> is the number of phrases in a given sentence and each phrase is represented using a <code>256</code> dimensional vector.</p>
<p><strong>Object features</strong> are represented as: <code>M x 128</code>, where <code>M</code> is the number of objects in a given 3D scene, each represented using a <code>128</code> dimensional vector.</p>
<p>I then combine each text feature with the complete object features to create the fused features of shape <code>10 x M x 128</code>, where <code>128</code> is the dimension of each fused feature.</p>
<p><strong>Objective</strong><br>
I want to learn text-to-objects mapping, where each phrase in <code>10 x M x 128</code> can be mapped to one or more objects from <code>M</code> objects. Essentially, I want to train an <code>M</code> number of classifiers, one for each object in <code>M</code>.</p>
<p><strong>Output</strong><br>
For the final input <code>10 x M x 128</code>, I expect the output to be <code>10 x M</code>, i.e., for each of the <code>10</code> text phrases, we get an <code>M</code> dimensional vector. Values in <code>M</code> will be <code>1</code> for the all objects that were classified to belong to that phrase and <code>0</code> otherwise.</p>
<p><strong>Model</strong><br>
Now when <code>batch_size</code> comes into picture, let’s say, input with <code>batch_size 14</code> will be represented as <code>14 x 10 x M x 128</code>. If I reshape it to <code>140 x M x 128</code> and then pass it through the following model and finally squeeze the output from <code>140 x 1 x M</code> to <code>140 x M</code> and resize to <code>14 x 10 x M</code>. Does it make sense? Is there a better way to do this?</p>
<pre><code class="lang-auto">hidden_size = 128
M = 128

classifier = nn.Sequential(
    nn.Conv1d(M, hidden_size, 1),
    nn.ReLU(),
    nn.BatchNorm1d(hidden_size),
    nn.Conv1d(hidden_size, hidden_size, 1),
    nn.ReLU(),
    nn.BatchNorm1d(hidden_size),
    nn.Conv1d(hidden_size, M, 1),
    nn.ReLU(),
    nn.BatchNorm1d(M),
    nn.Conv1d(M, 1, 1)
)
</code></pre>
<p>Thank you for your help in advance!</p>
How Generate sinwave function using pytorch

Praveen_Megavath: <p>sinwave function:<br>
y = A . sin(Bx) + c<br>
and have to add gaussian noise to it +n<br>
Updated equation : y = A . sin(Bx) + c + n</p>

J_Johnson: <pre><code class="lang-auto">x=torch.arange(10)

x_wave=torch.sin(x)
</code></pre>
<p>And how fancy you want to get after that is up to you.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.sin.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.sin.html</a></p>

Praveen_Megavath: <p>can you mention the detailed solution for this i did not understand</p>

harpone: <pre><code class="lang-auto">x = torch.linspace(0, 1, 100)
y = a* torch.sin(b * x) + c + d * torch.randn_like(x)
</code></pre>
<p>where a, b, c, d are scalars. d would be the (Gaussian) noise standard deviation.</p>

Praveen_Megavath: <p>for this same sin function how build an AI, training and predicting test data</p>
<p>Thanks in Advance</p>
Extremely huge loss value

promach: <p>Why when I changed the <a href="https://www.diffchecker.com/YlPSUQaw" rel="noopener nofollow ugc">coding for train_architecture()</a> , its return value <code>lval</code> increases from value around 2 to value above 10000 ?</p>

promach: <p>the above issue is resolved with <a href="https://github.com/buttercutter/gdas/commit/7cf7f7b17a8a60949c87f672c389124f3fc2b45c" rel="noopener nofollow ugc">line 765 of this git commit</a></p>
Self.init_weights() with Dynamic STD

HCH: <p>Hi,</p>
<p>I want to run my NN with different standard deviation to see what is the best value to have the best performance. I have a loop to pass different values for STD to my network. When I try to pass the value   I get the error: TypeError: init_weights() missing 1 required positional argument: ‘STD’</p>
<p>I simplyfied my code here:</p>
<blockquote>
<p><strong>A.</strong> instantiate the network and pass the STD value:<br>
Mynet = Net_simple(STD   = 0.01)</p>
<p><strong>B.</strong> My net work<br>
class Net_simple(torch.nn.Module):<br>
def <strong>init</strong>(self, STD):<br>
super(Net_simple, self).<strong>init</strong>()</p>
<pre><code>    self.nn = torch.nn.Sequential(torch.nn.Linear(100, 1),torch.nn.LeakyReLU())        
    self.float()
    self.apply(self.init_weights(STD))

def forward(self, x):
    x = x
    o1 = self.nn(x)
    return o1

def init_weights(self, m,STD):
    if isinstance(m, torch.nn.Linear):
        print('initiating weight..'+  m.__class__.__name__)
        torch.nn.init.normal(m.weight, STD)
        m.bias.data.fill_(0.01)
</code></pre>
</blockquote>
<blockquote>
<p>Blockquote</p>
</blockquote>
Can StyleGAN2-ADA accepts custom images dtype and ranges?

ltronchin: <p>Hi!<br>
I would like to train a StyleGAN2-ADA on a custom medical dataset of CT and MR images. I’m using the latest repository provided by the authors (<a href="https://github.com/NVlabs/stylegan3" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVlabs/stylegan3: Official PyTorch implementation of StyleGAN3</a>) but I see inspecting the code that the network requires images falling in the range [0 255] and with “dtype” uint8. You can see the line of code “assert image.dtype == np.uint8” in row 92 of the dataset.py script under the training folder of the repository.</p>
<p>I do not understand if I need to use images processed with these two constraints or if I can only comment on this line in the code</p>
<p>Do you have any experience with that problem?</p>
<p>Thank you!</p>
Conficts with pytorch torchvision and torchaudio

QuentinC: <p>Hi all !</p>
<p>I’m trying to install pytorch using the following command as recommended to use mps</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio -c pytorch-nightly
</code></pre>
<p>and the command returns this error :</p>
<pre><code class="lang-auto">PackagesNotFoundError: The following packages are not available from current channels:

  - torchaudio
</code></pre>
<p>I don’t need torchaudio. So I tried to run :</p>
<pre><code class="lang-auto">conda install pytorch torchvision -c pytorch-nightly
</code></pre>
<p>And I have this error :</p>
<pre><code class="lang-auto">UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -&gt; Available versions

Package pytorch conflicts for:
pytorch
torchvision -&gt; pytorch==1.13.0.dev20220607
</code></pre>
<p>What do I have to do to fix this issue ?</p>
<p>Thanks</p>

ptrblck: <p>Maybe uninstall all previous versions first and then rerun the install command.<br>
Alternatively, create a new virtual environment and try to install PyTorch and <code>torchvision</code> there.<br>
It also seems you are not using any command from the <a href="https://pytorch.org/get-started/locally/">website</a> so you might want to add e.g. the missing <code>cudatoolkit</code> in case this would be the next issue.</p>

QuentinC: <p>Hi ptrblck,</p>
<p>Thank you for your help but I already use a clean environment and I don’t understand your response because I have followed the website’s commands step by step.<br>
The exception is that I didn’t install Anaconda as the website says because it was already install on my computer. Is the Anaconda version important for the new version of pytorch ?</p>
<p>Moreover, why do I need the cudatoolkit if the M1 doen’t use cuda but mps ?</p>
<p>Thanks</p>

ptrblck: <p>Oh sorry I missed the <code>mps</code> part. I believe you would need to install the nightly binaries for the mps support and I don’t know if all sublibraries are already available (PyTorch should be there at least).</p>

QuentinC: <p>No problem,</p>
<p>Yes, that’s what I’m trying to install. It seems that the module was update during the week end because it seems to work now.</p>
<p>Thank you for your help.</p>
Torchscript model predict result is not right

wdongdongde: <pre><code class="lang-auto">import torch
example = torch.from_numpy(input0_data).to(dtype=torch.float)# torch.rand(1, 10080).cuda()
model = mv_clf.model
script_module = torch.jit.trace(model, example)
torch.jit.save(script_module, "model1.pt")
</code></pre>
<p>I use tsai to rain a pytorch model, and save it ,and use the code above to transform the model to torch_script, but when i use triton to inference with the model ,the result is very different from the result using the pkl model. is there any problem with the code saving torch script model??</p>
Why program still use CPU to run my GPU libtorch program?

Fuzhiyuan: <p>I try to run my project with GPU Version in release mode, but my task manager shows that it seems to run on CPU? But this occasion has not occurred in Debug version.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a59ac86478a8cff36121ad1204fa50842146d5f4.png" data-download-href="https://discuss.pytorch.org/uploads/default/a59ac86478a8cff36121ad1204fa50842146d5f4" title="企业微信截图_16551027845570"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a59ac86478a8cff36121ad1204fa50842146d5f4_2_554x500.png" alt="企业微信截图_16551027845570" data-base62-sha1="nD0oZVC5WnptHpCqgqSFfd4cLQw" width="554" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a59ac86478a8cff36121ad1204fa50842146d5f4_2_554x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a59ac86478a8cff36121ad1204fa50842146d5f4_2_831x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/5/a59ac86478a8cff36121ad1204fa50842146d5f4.png 2x" data-dominant-color="E2E6EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">企业微信截图_16551027845570</span><span class="informations">894×806 29.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>From the above picture, we can see that GPU utilization is 0, and most of the tensors appear to be deployed in memory instead of the GPU memory.</p>

ptrblck: <p>I don’t know any details about your workload but note that the CPU will still be used even if the actual kernel execution is done on the GPU. In this case the CPU would still be at least responsible for the kernel launches and usually also for the data loading and processing.</p>
<p>To properly see the GPU utilization in the Windows task manager you would have to select the “compute” tab (in case that’s not already done) as the default view apparently does not show the utilized compute resources (we’ve had similar discussions about it e.g. <a href="https://discuss.pytorch.org/t/it-seems-pytorch-doesnt-use-gpu/74673/6">here</a>).</p>

Fuzhiyuan: <p>Thanks for your reply!<br>
I have checked task manager(change to Cuda and compute), and it still does not show any difference(that means Cuda and compute are zero <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20">   ). But I inspect nvidia-smi which proves my module does be deployed in GPU.<br>
Last, I want to know when can I use FP16 in libtorch?</p>
Disabling data augmentation on training set for inference

TinfoilHat0: <p>I’m using  some data augmentation on my training set during the training. After the training ends, I’d like to measure my train accuracy using the original, non-augmented training set.</p>
<p>To do so, do I have to initialize dataset again without augmentation, or is there a better way to do it?</p>

ptrblck: <p>Re-creating a new dataset without any transformations sounds like a proper approach assuming you are lazily loading the samples (which would then be cheap).<br>
Otherwise, you could also try to set the internal <code>.transform</code> attribute to <code>False</code> assuming <code>self.transform</code> is used to apply the transformations on the samples (check the <code>Dataset</code> source to make sure that’s indeed the case).</p>
Even with the same seed, different random numbers are generated by different devices?

chausies: <pre data-code-wrap="python3"><code class="lang-nohighlight">import torch
cuda = torch.device("cuda")
torch.manual_seed(10)
torch.randn(3)
# always yields tensor([-0.6014, -1.0122, -0.3023])
torch.manual_seed(10)
torch.randn(3, device=cuda)
# always yields tensor([-0.1029,  1.6810, -0.2708], device='cuda:0')
</code></pre>
<p>As you can see, the random numbers yielded are different depending on the device used, even though I use the same seed for them. Isn’t this a problem for reproducibility? Depending on whether the machine is using a particular device, the random numbers will be different. It should be the same across any set of devices, right?</p>

ptrblck: <aside class="quote no-group" data-username="chausies" data-post="1" data-topic="154054">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chausies/48/2889_2.png" class="avatar"> chausies:</div>
<blockquote>
<p>It should be the same across any set of devices, right?</p>
</blockquote>
</aside>
<p>No, the pseudorandom number generator implementations might differ on different devices and if different software stacks are used, so the reproducibility (assuming you are following <a href="https://pytorch.org/docs/stable/notes/randomness.html">these docs</a>) should be visible by rerunning the same script on a specific system.</p>
With Conv2d, If the padding is bigger than one kernel dimension, will the kernel still go there?

Sam_Lerman: <p>Will a convolutional kernel go to regions of a image that are nothing but padding? Say, all zeroes?</p>
<p>To be clear, I don’t mean the general case when an edge is padding. In that case, the kernel will process that edge. I mean, if the kernel’s stride takes it completely out of the real image and into solely padding.</p>

ptrblck: <p>Yes, this seems to be the case based on this example which uses a padding value equal to the kernel size:</p>
<pre><code class="lang-python">conv = nn.Conv2d(1, 1, kernel_size=3, padding=3, bias=False)
x = torch.randn(1, 1, 1, 1)
out = conv(x)
print(out)
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">tensor([[[[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
          [ 0.0000,  0.1960, -0.1705,  0.3167,  0.0000],
          [ 0.0000, -0.0529, -0.2177,  0.2176,  0.0000],
          [ 0.0000, -0.2484, -0.2235, -0.2436,  0.0000],
          [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000]]]],
       grad_fn=&lt;ConvolutionBackward0&gt;)
</code></pre>
<p>As the can see the padding region was used as indicated by the zeros.</p>
Breaking Computation Graph

LoiseauNicolas: <p>Hello everyone,<br>
I have probably a problem which is probably not a problem… I’m trying to implement a custom nn.Module loss function.</p>
<pre><code class="lang-auto"># Here I want to compute repeatability between two sets of points
def repeatability(kp1, kp2, threshold=3):
    if 0 in (len(kp1_), len(kp2_)):
        return 0
    dist = torch.cdist(kp1_, kp2_)
    r_0 = torch.sum(dist.min(dim=0).values &lt;= tau) # this function break the computation graph
    r_1 = torch.sum(dist.min(dim=1).values &lt;= tau)
    rep = torch.div(r_0 + r_1,  len(kp1_) + len(kp2_))
    return rep
  k0 = torch.randn(50, 3)
  k1 = torch.randn(50, 3)
  repeatability_kp(k0, k1)
</code></pre>
<p>Is there a problem for the training step if I break the computation graph during loss computation ?<br>
Can i just return like</p>
<pre><code class="lang-auto">return torch.tensor(rep, requires_grad=True)
</code></pre>
<p>Thank you for your response.</p>

ptrblck: <p>Based on your code snippet I don’t think it would be a problem to re-wrap <code>rep</code> into a new tensor, as it shouldn’t be attached to a computation graph in the first place (so you are not detaching it at all).</p>
RTX 2080 ti/ driver / cuda / cudnn / pytorch dependencies?

jaehwanee0411: <p>Hi,</p>
<p>I’m currently working on getting environment in my ubuntu 18.04 with a GeForce RTX 2080 Ti.</p>
<p>I’ve tried numerous time to install cuda 11.x versions with driver 470.129.06 that can go with pytorch 1.7.1 or higher.</p>
<p>‘The following packages have unmet dependencies:’ error is keep poping and cannot install cuda appropriate.</p>
<p>For anyone who have installed “drivers / cuda / cudnn / pytorch 1.7.1” with 2080 in 18.04 before, can give me some advice?</p>
<p>Thank you in advance.</p>

ptrblck: <p>To install the CUDA toolkit, I would recommend to stick to <a href="https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#runfile">these docs</a> which use the local installers and will walk you through the installation of the CUDA toolkit including the NVIDIA driver.<br>
Note that you would not need to install the CUDA toolkit to run the PyTorch binaries, but only the NVIDIA driver (which might be easier), unless you plan to build PyTorch from source or any CUDA extension.</p>
Label smoothing results in very large loss values, is this normal?

catqaq: <p>Without label smoothing, loss is very small such as 0.3418：<br>
loss = F.cross_entropy(preds, labels, label_smoothing=0)</p>
<p><strong>With label smoothing, loss becomes very large</strong>(2604166656.0000), but the training process is normal：<br>
loss = F.cross_entropy(preds, labels, label_smoothing=0.1)<br>
So, is there anything wrong?</p>

ptrblck: <p>The value looks indeed quite large, but could be expected. Could you calculate the loss manually using loss smoothing and the plain cross entropy loss formula and compare it to the results?</p>
Changing original label of MNISTto new label

Praveen_Megavath: <p>import torch<br>
from torch.utils.data import Dataset<br>
from torchvision import datasets<br>
from torchvision.transforms import ToTensor<br>
import matplotlib.pyplot as plt<br>
import numpy as np</p>
<p>training_data = datasets.MNIST(root = “data”,train = True,download = True,transform = ToTensor())<br>
test_data = datasets.MNIST(root = “data”,train = False,download = True,transform = ToTensor())</p>
<p>labels_map = {0: “Label : 0”,</p>
<pre><code>          1: "Label : 1",

          2: "Label : 2",

          3: "Label : 3",

          4: "Label : 4",

          5: "Label : 5",

          6: "Label : 6",

          7: "Label : 7",

          8: "Label : 8",

          9: "Label : 9",

          }
</code></pre>
<p>figure = plt.figure(figsize=(8,8))</p>
<p>cols, rows = 3, 3</p>
<p>for i in range(1,cols*rows + 1):</p>
<p>sample_idx = torch.randint(len(training_data),size=(1,)).item()</p>
<p>img,label = training_data[sample_idx]</p>
<p>A = np.zeros(10)</p>
<p>A[label] = 1</p>
<p>print(A)</p>
<p>figure.add_subplot(rows, cols, i)</p>
<p>plt.title(labels_map[label])</p>
<p>plt.axis(“Off”)</p>
<p>plt.imshow(img.squeeze(),cmap = “gray”)</p>
<p>plt.show()</p>
<p>for this how to change label<br>
if you select number ‘3’ image should come of containing as ‘1’ all other images should be ‘0’<br>
if an original label is same as the selected number than change into ‘1’<br>
if an original label is not same as the selected number than change into ‘0’</p>
Methods for synthesizing images in Pytorch

eastwood_gale: <p>Main Q: I want to know are there any methods to synthesize images to illicit the maximum value of CNN last layer (if we dense the last layer to 1, it only gives a number) in Pytorch?</p>
<p>What did I try? I wrote a TensorFlow version, I used the random images to input to a pre-trained CNN, and then gave one value. I did the SGD to change the random image to an image that will elicit the largest CNN final layer value (maybe not the largest, maybe to a certain value I defined say 10). And the results will give me a certain image that looks good, but when I wrote the same Pytorch implement version, it generated an image that will have lots of noise around. I wonder are there any methods or codes to generate images that will elicit the largest response (or certain response) of pre-trained CNN. Thank you all!</p>
How do I apply Gradient accumulation in GANs?

Andrew_Legend: <p>I am aware that people have answered such a question once or twice here however I still do not get how to apply gradient accumulation work while training a generator and a discriminator in an alternate way where I first update the Generator and then <strong>with the same data in the batch</strong> I update the Discriminator</p>
<pre><code class="lang-auto"># Train Generator

optimizer_G.zero_grad()
g_loss= Criterion(discriminator(fake_data), real_label)
g_loss.backward() 
optimizer_G.step() 

# Train Discriminator

optimizer_D.zero_grad()
d_loss= Criterion(discriminator(fake_data), fake_label)+Criterion(discriminator(real_data), real_label)
d_loss.backward() 
optimizer_G.step() 
</code></pre>
<p>Is it possible for someone to provide some kind of coded example of how to implement this please? Thank you in advance</p>

IneedMrmeeseeks: <p>I am not the expert, but I wonder why would you want to accumulate gradient?<br> I’ve learnt that I need to set the gradients zero after each epoch is done.</p>

Tianyi_Zhou1: <p>sometimes we need large batch size, but there isn’t enough gpu memory</p>

Andrew_Legend: <p>I am fully aware of the purpose of Gradient accumulation but not fully sure <strong>how</strong> to implement it in my situation</p>
Performance is good on the validation set, poor on the test set

yipliu: <p>Hi everyone,</p>
<p>As the technology becomes more and more advanced nowadays, there are a variety of hyperparameter automatic search scripts.</p>
<p>I used these scripts to get the model hyperparameters and while the performance was very good on the validation set, it degraded severely on the test set.</p>
<p>In addition, I have <strong>set seed</strong> and <em>torch.backends.cudnn.deterministic==True</em>, but the performance is still difficult to replicate using the Optimal hyperparameters。</p>
<p>Some common tricks I also used (EarlyStop, lr_shedule)</p>
<p>What is everyone’s opinion on this phenomenon?</p>
How to load large model with multiple GPU cards?

Jiheng_Yang: <p>This might be a simple question, but bugged me the whole afternoon.</p>
<p>I was trying to use a pretained m2m 12B model for language processing task (44G model file). I have 8 Tesla-V100 GPU cards, each of which has 32GB graphics memory. The program OOMed at:</p>
<pre><code class="lang-auto">model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100-12B-avg-5-ckpt")
</code></pre>
<p>Error being:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 31.75 GiB total capacity; 30.49 GiB already allocated; 177.75 MiB free; 30.50 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
</code></pre>
<p>I know the problem is one single GPU card’s memory is not big enough to load the whole model, but how can I leverage all my 8 cards memory to load the model and do predictions/generations? There must be someway to do this, otherwise if we have models that 's really huge, we eventually can’t have a GPU card with enough memory to load the model. I would really appreciate if someone can point me some directions or show me the path. Thanks in advance!</p>
<p>Thanks so much for the help!</p>

ptrblck: <p>You could load the model on the CPU first (using your RAM) and push parts of it to specific GPUs to shard the model. This would of course also need changes to the <code>forward</code> pass as you would need to push the intermediate activations to the corresponding GPU using this naive model sharding approach, so I would expect to find some model sharding / pipeline parallel scripts in the repository (unless the authors trained the model on e.g. an A100 80GB).</p>

Jiheng_Yang: <p>Thank you for your kindly response, and it seems like a complicated issue. One thing I keep thinking is, for instance, if I want to use Facebook’s OPT model or OpenAPI’s GPT3 model, which is pre-trained and huge like several hundred G in size. How can I find a single GPU card with that much GPU memory to do question answering tasks or sentence prediction? So the only way to do this is to use CPU RAM or single GPU card with huge memory?</p>
<p>Really appreciate your help!</p>

ptrblck: <p>These large models usu usually a parallelism approach, such as model parallel, tensor parallel, pipeline parallel etc. e.g. via Megatron, DeepSpeed etc. and come with scripts to load them onto compute clusters.</p>

Jiheng_Yang: <p>Thanks, I’ll look them up and see whether they can solve my problem.</p>
<p>Thank you!</p>

cbalioglu: <p><a class="mention" href="/u/jiheng_yang">@Jiheng_Yang</a> you might also want to check out Deferred Module Initialization in torchdistX:<br>
<a href="https://pytorch.org/torchdistx/latest/deferred_init.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/torchdistx/latest/deferred_init.html</a></p>
Getting tensor values of a dataset

TinfoilHat0: <p>Is there a direct way to get tensors of a dataset? To do so, I’m currently wrapping the dataset with a dataloader, iterating over it and extracting tensors for data and the label.</p>
<p>I was wondering whether there’s a more straightforward way of doing this.</p>

Andrei_Cristea: <p>You should be able to just iterate over the dataset directly, without requiring a dataloader.</p>
<pre><code class="lang-auto">class MyDataset(torch.utils.data.Dataset):
    def __init__(self, n):
        self.data = list(range(n))
    def __len__(self):
        return len(self.data)
    def __getitem__(self, idx):
        return self.data[idx]

my_dataset = MyDataset(100)
for i in my_dataset:
    print(i, end=",")

Output:
0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,
</code></pre>
LibTorch CUDA huge initial VRAM allocation

abstanton: <p>Hi. I’m using libtorch for inference. I’m running into an issue where I need to run two instances of a program, but due to libtorch’s massive initial memory allocation, I can only fit one instance on my GPU.</p>
<p>Inspecting VRAM usage, the first call to:</p>
<pre><code class="lang-auto">torch::Tensor ones = torch::ones({1, 1, 1, 1}).cuda();
</code></pre>
<p>increases VRAM usage by 1.6GB. Is this normal, or is there a way to get around it? This seems way too high to be reasonable.<br>
Cheers</p>
Multitask learning with multiple uneven datasets

Torphix: <p>Hi,</p>
<p>I want to train a backbone model coupled with several output heads for different tasks.<br>
Each task has a different dataset with varying lengths.</p>
<p>My current approach is to train the model for 10 epochs on each dataset swapping out the head for each specific task.<br>
However this most likely means that the model weights of the backbone trained on the first task is ‘forgotten’ with enough downstream tasks (my guess, perhaps I am mistaken?)</p>
<p>My thoughts on better ways to approach the problem are as follows if someone could point me in the right direction I would greatly appreciate it:</p>
<ol>
<li>
<p>Crop all datasets to the length of the shortest dataset thereby removing the problem of uneven datasets and optimize on the shortened datasets finetuning the head on the full dataset for the specific task at the end. (Pros: Easy to implement and quick, Cons: Smaller data distribution to work with)</p>
</li>
<li>
<p>Use multiprocessing to train the model on the different datasets at the same time (not sure if this would work ie: update the backbone model weights on each different task effectively updating the backbone model once each step on a different task thereby ensuring that the models ‘memory’ contains a mixture of each task)</p>
</li>
</ol>
<p>Thank you, let me know if I didn’t explain clearly enough and I will try to do better</p>

Torphix: <p>Approach I ended up taking was to concat the datasets together and organize datapoints that had the same tasks togther for easy seperation later.<br>
ie: in my collate fn I sorted the datapoints so all binary classification datapoints where together and all regression datapoints where togther, Then at training time I iterated over the sharded batch feeding each shard to the backbone model and the respective head model finally backpropagating with the total loss</p>