Custom Loss function has: element 0 of tensors does not require grad and does not have a grad_fn

Noanemo: <p>Hello everyone. I’m having a slight issue with implementing a custom loss function. So have I looked at the other posts related to my similar problem and have double checked the loss function itself of detaching operations but I think I have got rid of all of them and have switched them up to torch functions. However, the error still exists. I am still fairly new to this so any help/advice would be appreciated. Thank you all in advance.<br>
Below is the custom loss function itself.</p>
<pre><code class="lang-auto">class Regress_Loss(torch.nn.Module):
    def __init__(self):
        super(Regress_Loss,self).__init__()
        
    def loss(self,img_inter,labels_test,temp,batch_size,device):
        
        
        img_sim = -torch.sqrt(torch.sum(torch.square((img_inter.unsqueeze(2)-img_inter.squeeze()).squeeze()),2))
        dist = (labels_test.unsqueeze(2)-labels_test.squeeze()).squeeze()
        
        
        A = img_sim
        m = A.size(0)
        s0,s1 = A.stride()
        out = A.as_strided((m-1, m), (s0+s1, s1)).unfold(0, m-1, 1)
        img_sim_num = out


        processed_dist = dist.abs().flatten()
        processed_sim = (-torch.exp(img_sim/temp)).flatten()
        processed_sim_num = (-torch.exp(img_sim_num/temp)).flatten()
        (sorted_dist, indices_d) = processed_dist.sort()

        total_den = torch.sum(processed_sim)

        index = 0
        in_index = 0
        subtractor = 0
        denom = torch.zeros(dist.size(0)*dist.size(0)-dist.size(0)).to(device)
        
        
        for distances in sorted_dist:

            if (distances != 0 or index &gt; np.sqrt(list(sorted_dist.size())[0])-1):
                subtractor = subtractor + processed_sim[indices_d[index]]
                
                inter = total_den - subtractor + processed_sim[indices_d[index]]
                
                
                denom[in_index] = inter
                in_index = in_index + 1
            index = index + 1   
        
        
        loss = -torch.div(torch.sum(torch.log(torch.div(processed_sim_num,denom))),
                     torch.mul(batch_size,batch_size-1))
        
    
        return loss
</code></pre>

ptrblck: <p>Could you post random tensor definitions for all input arguments to make the code snippet executable?</p>

Noanemo: <pre><code class="lang-auto">temp = 2
batch_size  = 256
device = "cuda" if.cuda.is_available() else "cpu"
img_inter = torch.rand(1,256,512) 
labels = torch.rand(1,256)

</code></pre>
<p>Hopefully this is what you meant. For more context I am using the AgeDB dataset which is just a database of images. Thanks again</p>

ptrblck: <p>I cannot reproduce the issue using:</p>
<pre><code class="lang-python">class Regress_Loss(torch.nn.Module):
    def __init__(self):
        super(Regress_Loss,self).__init__()
        
    def forward(self,img_inter,labels_test,temp,batch_size,device):
        
        
        img_sim = -torch.sqrt(torch.sum(torch.square((img_inter.unsqueeze(2)-img_inter.squeeze()).squeeze()),2))
        dist = (labels_test.unsqueeze(2)-labels_test.squeeze()).squeeze()
        
        
        A = img_sim
        m = A.size(0)
        s0,s1 = A.stride()
        out = A.as_strided((m-1, m), (s0+s1, s1)).unfold(0, m-1, 1)
        img_sim_num = out


        processed_dist = dist.abs().flatten()
        processed_sim = (-torch.exp(img_sim/temp)).flatten()
        processed_sim_num = (-torch.exp(img_sim_num/temp)).flatten()
        (sorted_dist, indices_d) = processed_dist.sort()

        total_den = torch.sum(processed_sim)

        index = 0
        in_index = 0
        subtractor = 0
        denom = torch.zeros(dist.size(0)*dist.size(0)-dist.size(0)).to(device)
        
        
        for distances in sorted_dist:

            if (distances != 0 or index &gt; np.sqrt(list(sorted_dist.size())[0])-1):
                subtractor = subtractor + processed_sim[indices_d[index]]
                
                inter = total_den - subtractor + processed_sim[indices_d[index]]
                
                
                denom[in_index] = inter
                in_index = in_index + 1
            index = index + 1   
        
        
        loss = -torch.div(torch.sum(torch.log(torch.div(processed_sim_num,denom))),
                     torch.mul(batch_size,batch_size-1))
        
    
        return loss

temp = 2
batch_size  = 256
device = "cuda" if torch.cuda.is_available() else "cpu"
img_inter = torch.rand(1,256,512).to(device).requires_grad_()
labels = torch.rand(1,256).to(device)

criterion = Regress_Loss()
loss = criterion(img_inter, labels, temp, batch_size, device)
loss.backward()

print(img_inter.grad)
</code></pre>
<p>Note that I needed to replace the <code>loss</code> function with <code>forward</code> as this method is expected to be defined in an <code>nn.Module</code>.<br>
Also, I set the <code>requires_grad</code> attribute of the <code>img_inter</code> to <code>True</code> as I assume this would be the model output.</p>
<p>I see NaN gradients, but this might also be expected e.g. if <code>torch.sqrt</code> received a negative input.</p>

Noanemo: <p>Sorry for the late response but I was trying different things trying to fix my error but are still having troubles. I’m just going to post my entire code. If the loss function itself is not the issue it might be something else. Thank you so much for trying to help.</p>
<p>Also I did try it with changing the bits to make it similar to how you tested it, however the model parameters do not seem to change if I set img_inter to True for its requires_grad.</p>
<pre><code class="lang-auto">import torch
import gc
with torch.cuda.device('cuda:3'):
    torch.cuda.empty_cache()

import os
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from torchvision import transforms
from PIL import Image, ImageOps
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
import torchvision.models as models
from torch.autograd import Variable

class AgeDBDataset(Dataset):
    def __init__(self, image, label, transform=None):
        self.image = image
        self.label = label
        self.transform = transform
        
    def __len__(self):
        return(len(self.image))
        
    def __getitem__(self, index):
        if self.transform:
            tran_image = self.transform(image[index])
        else:
            tran_image = self.image[index]
        return(tran_image, self.label[index])   

def DataOrganizer(directory): 
    y = []
    
    dataset = os.scandir(directory)

    for title in dataset: 
        label = title.name.split('_')
        y.append(label[2])

    y = np.array(y)
    y = y.astype('float32')
    y = torch.from_numpy(y)
    
    X = torch.zeros((len(y),3,224,224))
    count = 0
    for img in os.listdir(directory):
        trans = transforms.Compose([transforms.ToTensor(),
                                    transforms.Resize((224,224))])
        pic = Image.open(directory+img) 
        #pic = ImageOps.grayscale(pic)
        pic = trans(pic)
        
        if pic.size(0) == 1:
            pic = torch.cat((pic,pic,pic),0)
    
        
        X[count] = (pic)
        count = count + 1
    
    return(X, y)    

class Regress_Loss(torch.nn.Module):
    def __init__(self):
        super(Regress_Loss,self).__init__()
        
    def contrastive_loss(self,img_inter,labels_test,temp,batch_size,device):
        
        
        img_sim = -torch.sqrt(torch.sum(torch.square((img_inter.unsqueeze(2)-img_inter.squeeze()).squeeze()),2))
        dist = (labels_test.unsqueeze(2)-labels_test.squeeze()).squeeze()
        
        
        A = img_sim
        m = A.size(0)
        s0,s1 = A.stride()
        out = A.as_strided((m-1, m), (s0+s1, s1)).unfold(0, m-1, 1)
        img_sim_num = out


        processed_dist = dist.abs().flatten()
        processed_sim = (-torch.exp(img_sim/temp)).flatten()
        processed_sim_num = (-torch.exp(img_sim_num/temp)).flatten()
        (sorted_dist, indices_d) = processed_dist.sort()

        total_den = torch.sum(processed_sim)

        index = 0
        in_index = 0
        subtractor = 0
        denom = torch.zeros(dist.size(0)*dist.size(0)-dist.size(0)).to(device)
        
        
        for distances in sorted_dist:

            if (distances != 0 or index &gt; np.sqrt(list(sorted_dist.size())[0])-1):
                subtractor = subtractor + processed_sim[indices_d[index]]
                
                inter = total_den - subtractor + processed_sim[indices_d[index]]
                
                
                denom[in_index] = inter
                in_index = in_index + 1
            index = index + 1   
        
        
        loss = -torch.div(torch.sum(torch.log(torch.div(processed_sim_num,denom))),
                     torch.mul(batch_size,batch_size-1))
        
        
        return loss

def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

temp = 2
batch_size = 256
linearepoch = 100
encoderepoch = 400
device = "cuda:3" if torch.cuda.is_available() else "cpu"

transform_0 = transforms.Compose([transforms.RandomResizedCrop((224,224)),
                                  transforms.RandomVerticalFlip(0.5),
                                  transforms.RandomHorizontalFlip(0.5)])

transform_1 = transforms.Compose([transforms.ColorJitter()])

images, labels = DataOrganizer(r"/home/parkn1/Documents/AgeDB/")

images_0 = transform_0(images)
images_1 = transform_1(images)

aug_img = torch.cat((images_0, images_1),0)
aug_lab = torch.cat((labels,labels), 0)

dataset = AgeDBDataset(aug_img, aug_lab, transform=None)
train_set, validation_set, test_set = torch.utils.data.random_split(dataset, [250*2,125*2,125*2])

train_loader = DataLoader(dataset=train_set, batch_size=batch_size, shuffle = True)
validation_loader = DataLoader(dataset=validation_set, batch_size=batch_size, shuffle = True)
test_loader = DataLoader(dataset=test_set, batch_size=batch_size, shuffle = True)

resnet18 = models.resnet18(weights=True)

resnet18.to(device)
loss_class = Regress_Loss().to(device)
lr = 0.1


optimizer = torch.optim.SGD(params = resnet18.parameters(), lr = lr)
lambda1 = lambda epoch: epoch / 10
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, encoderepoch)

for epoch in range(encoderepoch):
    for (inputs, labels) in train_loader:
        inputs, labels = inputs.to(device), labels.to(device)
        resnet18.train()
        
        layer = resnet18._modules.get('avgpool')
        activation = {}

        

        resnet18.avgpool.register_forward_hook(get_activation('avgpool'))
        
        output = resnet18(inputs.squeeze())



        img_inter = activation['avgpool'].squeeze(dim = 3).squeeze(dim = 2).unsqueeze(dim = 0)
        
        print(img_inter.size())
        print(labels.unsqueeze(0).size())
        
        loss = loss_class.contrastive_loss(img_inter,
                                           labels.unsqueeze(0)
                                           ,temp,batch_size,device)

      optimizer.zero_grad() 
        a = list(resnet18.parameters())[0].clone()
        loss.backward()
        optimizer.step()
        b = list(resnet18.parameters())[0].clone()
        print(torch.equal(a.data,b.data))
        #validation where it should go
        scheduler.step()
    print(loss)
</code></pre>

ptrblck: <p>Your code is not executable as it uses an undefined dataset.<br>
However, are you still running into the original error (<code>element 0 tensor does not require grad...</code>) or are you just seeing no changes of the parameters at all?<br>
In the latter case could you check the <code>.grad</code> attribute of a parameter before and after the first <code>backward()</code> call?<br>
It should show <code>None</code> before and a valid gradient afterwards.</p>

Noanemo: <p>I am still running into the element 0 tensor does not require error. I have tried setting some parts of the code to Variable() or setting the requires_grad = True, that solves the error but then the parameters do not change.</p>

ptrblck: <p>Recreating tensors and setting their <code>.requires_grad</code> attribute to <code>True</code> is not the right approach as it won’t fix the issue, but just hide it.<br>
Could you post a minimal and executable code snippet reproducing the issue?</p>

Noanemo: <p>Here is some code that I reduced down that reproduces the error</p>
<pre><code class="lang-auto">import os
import torch
import torch.nn as nn
import numpy as np
from torchvision import transforms
from PIL import Image, ImageOps
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
import torchvision.models as models
from torch.autograd import Variable

class AgeDBDataset(Dataset):
    def __init__(self, image, label, transform=None):
        self.image = image
        self.label = label
        self.transform = transform
        
    def __len__(self):
        return(len(self.image))
        
    def __getitem__(self, index):
        if self.transform:
            tran_image = self.transform(image[index])
        else:
            tran_image = self.image[index]
        return(tran_image, self.label[index])

class Regress_Loss(torch.nn.Module):
    def __init__(self):
        super(Regress_Loss,self).__init__()
        
    def forward(self,img_inter,labels_test,temp,batch_size,device):
        
        
        img_sim = -torch.sqrt(torch.sum(torch.square((img_inter.unsqueeze(1)-img_inter.squeeze())),2))
        dist = (labels_test.unsqueeze(1)-labels_test.squeeze()).squeeze()
        
        m = img_sim.size(0)
        img_sim_num = img_sim.flatten()[1:].view(m-1, m+1)[:,:-1].reshape(m, m-1)


        processed_dist = dist.abs().flatten()
        processed_sim = (-torch.exp(img_sim/temp)).flatten()
        processed_sim_num = (-torch.exp(img_sim_num/temp)).flatten()
        (sorted_dist, indices_d) = processed_dist.sort()
        total_den = torch.sum(processed_sim)

        index = 0
        in_index = 0
        subtractor = 0
        denom = torch.zeros(dist.size(0)*dist.size(0)-dist.size(0)).to(device)
        
        for distances in sorted_dist:

            if (distances != 0 or index &gt; np.sqrt(list(sorted_dist.size())[0])-1):
                subtractor = subtractor + processed_sim[indices_d[index]]
                
                inter = total_den - subtractor + processed_sim[indices_d[index]]
                
                
                denom[in_index] = inter
                in_index = in_index + 1
                
            index = index + 1  
        
        loss = -torch.div(torch.sum(torch.log(torch.div(processed_sim_num,denom))),
                     torch.mul(batch_size*2,batch_size*2-1))
        
        
        return loss   

def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

temp = 2
batch_size = 256
linearepoch = 100
encoderepoch = 400
device = "cuda:0" if torch.cuda.is_available() else "cpu"

transform_0 = transforms.Compose([transforms.RandomResizedCrop((224,224)),
                                  transforms.RandomVerticalFlip(0.5),
                                  transforms.RandomHorizontalFlip(0.5)])

transform_1 = transforms.Compose([transforms.ColorJitter()])

images = torch.rand(512, 3, 224, 224)
labels = torch.rand(512)

dataset = AgeDBDataset(images, labels, transform=None)

train_set, validation_set, test_set = torch.utils.data.random_split(dataset, [256,128,128])

train_loader = DataLoader(dataset=train_set, batch_size=batch_size, shuffle = True)
validation_loader = DataLoader(dataset=validation_set, batch_size=batch_size, shuffle = True)
test_loader = DataLoader(dataset=test_set, batch_size=batch_size, shuffle = True)

resnet18 = models.resnet18(weights=True)

resnet18.to(device)
loss_class = Regress_Loss().to(device)
lr = 0.1


optimizer = torch.optim.SGD(params = resnet18.parameters(), lr = lr)
lambda1 = lambda epoch: epoch / 10
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, encoderepoch)

for epoch in range(encoderepoch):
    for (inputs, labels) in train_loader:
        aug_img, aug_lab = inputs.to(device), labels.to(device)         
        
       
        layer = resnet18._modules.get('avgpool')
        resnet18.eval()
        activation = {}
        resnet18.avgpool.register_forward_hook(get_activation('avgpool'))
        output = resnet18(aug_img.squeeze())
        img_inter = activation['avgpool'].squeeze(dim = 3).squeeze(dim = 2)
        
        
        criterion = Regress_Loss()
        loss = criterion(img_inter,aug_lab,temp,batch_size,device)
    

        print(loss)
        optimizer.zero_grad() 
        a = list(resnet18.parameters())[0].clone()
        loss.backward()
        optimizer.step()
        b = list(resnet18.parameters())[0].clone()
        print(torch.equal(a.data,b.data))
        #validation where it should go
        scheduler.step()
    print(loss)
    
</code></pre>

ptrblck: <p>Thanks for the code! In <code>get_activation</code> you are explicitly detaching the intermediate activation.<br>
Remove the <code>detach()</code> call and the code should work.</p>

Noanemo: <p>After getting rid of the .detach(). Now I’m getting this error:</p>
<p>GET was unable to find an engine to execute this computation</p>

ptrblck: <p>You might be running out of memory so check the memory usage and reduce the batch size of possible.</p>

Noanemo: <p>Thank you so much for your continued help, but reducing the batch size does not seem to help with alleviating the error.<br>
If it helps I also receive this before the explicit error message:</p>
<p><code>Could not load library libcudnn_cnn_train.so.8. Error: /usr/local/anaconda3/bin/../lib/./libcudnn_ops_train.so.8: undefined symbol: _ZN5cudnn3ops26JoinInternalPriorityStreamEP12cudnnContexti, version libcudnn_ops_infer.so.8</code></p>

ptrblck: <p>This seems to be the actual error then.<br>
Could you explain how you’ve installed the PyTorch binaries as it seems some cuDNN libraries are mismatching. E.g. are you preloading a system-wide cuDNN installation?</p>

Noanemo: <p>I am slightly new to this so I do not know much. But, the machine I’m running it on was pre-installed with cuDNN and Pytorch. So I would not know about that. Is there a way in which I could check and remedy this? Thank you.</p>

ptrblck: <p>I assume you can easily reproduce the issue?<br>
If so, could you post the output of <code>python -m torch.utils.collect_env</code>?<br>
Also, could you rerun the failing script with <code>LD_DEBUG=libs python script.py &gt; out.tmp</code>, <code>grep</code> the output for <code>libcudnn</code>, and post the result here?</p>

Noanemo: <p>Yes every time I run it the error pops up.</p>
<p>Here is the output of <code>python -m torch.utils.collect_env</code>:</p>
<pre><code class="lang-auto">PyTorch version: 2.0.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Rocky Linux 8.8 (Green Obsidian) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18)
Clang version: Could not collect
CMake version: version 3.26.4
Libc version: glibc-2.28

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:56:21)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-4.18.0-477.15.1.el8_8.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA RTX A4000
GPU 1: NVIDIA RTX A4000
GPU 2: NVIDIA RTX A4000
GPU 3: NVIDIA RTX A4000

Nvidia driver version: 520.61.05
cuDNN version: Probably one of the following:
/usr/lib64/libcudnn.so.8.9.2
/usr/lib64/libcudnn_adv_infer.so.8.9.2
/usr/lib64/libcudnn_adv_train.so.8.9.2
/usr/lib64/libcudnn_cnn_infer.so.8.9.2
/usr/lib64/libcudnn_cnn_train.so.8.9.2
/usr/lib64/libcudnn_ops_infer.so.8.9.2
/usr/lib64/libcudnn_ops_train.so.8.9.2
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              48
On-line CPU(s) list: 0-47
Thread(s) per core:  2
Core(s) per socket:  12
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz
Stepping:            7
CPU MHz:             2400.000
CPU max MHz:         3500.0000
CPU min MHz:         1000.0000
BogoMIPS:            4800.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            16896K
NUMA node0 CPU(s):   0-11,24-35
NUMA node1 CPU(s):   12-23,36-47
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[pip3] torchvision==0.15.2
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] magma                     2.5.4                hc72dce7_4    conda-forge
[conda] mkl                       2022.2.1         h84fe81f_16997    conda-forge
[conda] mkl-include               2023.1.0         h84fe81f_48680    conda-forge
[conda] numpy                     1.21.5                   pypi_0    pypi
[conda] pytorch                   1.12.1          cuda112py39hb0b7ed5_201    conda-forge
[conda] pytorch-cuda              11.8                 h7e8668a_5    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.12.1               py39_cu113    pytorch
[conda] torchvision               0.14.1          cuda112py39h1a1de93_0    conda-forge
</code></pre>
<p>Below is the LD_DEBUG=libs python script.py &gt; out.tmp:</p>
<pre><code class="lang-auto">1468137:	find library=libpthread.so.0 [0]; searching
   1468137:	 search path=/usr/local/anaconda3/bin/../lib/glibc-hwcaps/x86-64-v4:/usr/local/anaconda3/bin/../lib/glibc-hwcaps/x86-64-v3:/usr/local/anaconda3/bin/../lib/glibc-hwcaps/x86-64-v2:/usr/local/anaconda3/bin/../lib/tls/haswell/avx512_1/x86_64:/usr/local/anaconda3/bin/../lib/tls/haswell/avx512_1:/usr/local/anaconda3/bin/../lib/tls/haswell/x86_64:/usr/local/anaconda3/bin/../lib/tls/haswell:/usr/local/anaconda3/bin/../lib/tls/avx512_1/x86_64:/usr/local/anaconda3/bin/../lib/tls/avx512_1:/usr/local/anaconda3/bin/../lib/tls/x86_64:/usr/local/anaconda3/bin/../lib/tls:/usr/local/anaconda3/bin/../lib/haswell/avx512_1/x86_64:/usr/local/anaconda3/bin/../lib/haswell/avx512_1:/usr/local/anaconda3/bin/../lib/haswell/x86_64:/usr/local/anaconda3/bin/../lib/haswell:/usr/local/anaconda3/bin/../lib/avx512_1/x86_64:/usr/local/anaconda3/bin/../lib/avx512_1:/usr/local/anaconda3/bin/../lib/x86_64:/usr/local/anaconda3/bin/../lib		(RPATH from file python)
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/glibc-hwcaps/x86-64-v4/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/glibc-hwcaps/x86-64-v3/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/glibc-hwcaps/x86-64-v2/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/tls/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/tls/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/tls/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/tls/haswell/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/tls/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/tls/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/tls/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/tls/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/haswell/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/libpthread.so.0
   1468137:	 search path=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/glibc-hwcaps/x86-64-v4:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/glibc-hwcaps/x86-64-v3:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/glibc-hwcaps/x86-64-v2:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/haswell/avx512_1/x86_64:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/haswell/avx512_1:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/haswell/x86_64:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/haswell:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/avx512_1/x86_64:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/avx512_1:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/x86_64:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/haswell/avx512_1/x86_64:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/haswell/avx512_1:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/haswell/x86_64:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/haswell:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/avx512_1/x86_64:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/avx512_1:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/x86_64:/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8:/usr/lib/wsl/lib/glibc-hwcaps/x86-64-v4:/usr/lib/wsl/lib/glibc-hwcaps/x86-64-v3:/usr/lib/wsl/lib/glibc-hwcaps/x86-64-v2:/usr/lib/wsl/lib/tls/haswell/avx512_1/x86_64:/usr/lib/wsl/lib/tls/haswell/avx512_1:/usr/lib/wsl/lib/tls/haswell/x86_64:/usr/lib/wsl/lib/tls/haswell:/usr/lib/wsl/lib/tls/avx512_1/x86_64:/usr/lib/wsl/lib/tls/avx512_1:/usr/lib/wsl/lib/tls/x86_64:/usr/lib/wsl/lib/tls:/usr/lib/wsl/lib/haswell/avx512_1/x86_64:/usr/lib/wsl/lib/haswell/avx512_1:/usr/lib/wsl/lib/haswell/x86_64:/usr/lib/wsl/lib/haswell:/usr/lib/wsl/lib/avx512_1/x86_64:/usr/lib/wsl/lib/avx512_1:/usr/lib/wsl/lib/x86_64:/usr/lib/wsl/lib:/usr/local/cuda/lib64/glibc-hwcaps/x86-64-v4:/usr/local/cuda/lib64/glibc-hwcaps/x86-64-v3:/usr/local/cuda/lib64/glibc-hwcaps/x86-64-v2:/usr/local/cuda/lib64/tls/haswell/avx512_1/x86_64:/usr/local/cuda/lib64/tls/haswell/avx512_1:/usr/local/cuda/lib64/tls/haswell/x86_64:/usr/local/cuda/lib64/tls/haswell:/usr/local/cuda/lib64/tls/avx512_1/x86_64:/usr/local/cuda/lib64/tls/avx512_1:/usr/local/cuda/lib64/tls/x86_64:/usr/local/cuda/lib64/tls:/usr/local/cuda/lib64/haswell/avx512_1/x86_64:/usr/local/cuda/lib64/haswell/avx512_1:/usr/local/cuda/lib64/haswell/x86_64:/usr/local/cuda/lib64/haswell:/usr/local/cuda/lib64/avx512_1/x86_64:/usr/local/cuda/lib64/avx512_1:/usr/local/cuda/lib64/x86_64:/usr/local/cuda/lib64:/usr/local/lib64/glibc-hwcaps/x86-64-v4:/usr/local/lib64/glibc-hwcaps/x86-64-v3:/usr/local/lib64/glibc-hwcaps/x86-64-v2:/usr/local/lib64/tls/haswell/avx512_1/x86_64:/usr/local/lib64/tls/haswell/avx512_1:/usr/local/lib64/tls/haswell/x86_64:/usr/local/lib64/tls/haswell:/usr/local/lib64/tls/avx512_1/x86_64:/usr/local/lib64/tls/avx512_1:/usr/local/lib64/tls/x86_64:/usr/local/lib64/tls:/usr/local/lib64/haswell/avx512_1/x86_64:/usr/local/lib64/haswell/avx512_1:/usr/local/lib64/haswell/x86_64:/usr/local/lib64/haswell:/usr/local/lib64/avx512_1/x86_64:/usr/local/lib64/avx512_1:/usr/local/lib64/x86_64:/usr/local/lib64:/usr/local/lib/glibc-hwcaps/x86-64-v4:/usr/local/lib/glibc-hwcaps/x86-64-v3:/usr/local/lib/glibc-hwcaps/x86-64-v2:/usr/local/lib/tls/haswell/avx512_1/x86_64:/usr/local/lib/tls/haswell/avx512_1:/usr/local/lib/tls/haswell/x86_64:/usr/local/lib/tls/haswell:/usr/local/lib/tls/avx512_1/x86_64:/usr/local/lib/tls/avx512_1:/usr/local/lib/tls/x86_64:/usr/local/lib/tls:/usr/local/lib/haswell/avx512_1/x86_64:/usr/local/lib/haswell/avx512_1:/usr/local/lib/haswell/x86_64:/usr/local/lib/haswell:/usr/local/lib/avx512_1/x86_64:/usr/local/lib/avx512_1:/usr/local/lib/x86_64:/usr/local/lib		(LD_LIBRARY_PATH)
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/glibc-hwcaps/x86-64-v4/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/glibc-hwcaps/x86-64-v3/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/glibc-hwcaps/x86-64-v2/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/haswell/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/tls/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/haswell/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/glibc-hwcaps/x86-64-v4/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/glibc-hwcaps/x86-64-v3/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/glibc-hwcaps/x86-64-v2/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/tls/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/tls/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/tls/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/tls/haswell/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/tls/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/tls/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/tls/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/tls/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/haswell/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib/wsl/lib/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/glibc-hwcaps/x86-64-v4/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/glibc-hwcaps/x86-64-v3/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/glibc-hwcaps/x86-64-v2/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/tls/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/tls/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/tls/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/tls/haswell/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/tls/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/tls/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/tls/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/tls/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/haswell/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/cuda/lib64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/glibc-hwcaps/x86-64-v4/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/glibc-hwcaps/x86-64-v3/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/glibc-hwcaps/x86-64-v2/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/tls/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/tls/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/tls/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/tls/haswell/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/tls/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/tls/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/tls/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/tls/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/haswell/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/glibc-hwcaps/x86-64-v4/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/glibc-hwcaps/x86-64-v3/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/glibc-hwcaps/x86-64-v2/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/tls/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/tls/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/tls/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/tls/haswell/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/tls/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/tls/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/tls/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/tls/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/haswell/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/local/lib/libpthread.so.0
   1468137:	 search path=/usr/lib64/glibc-hwcaps/x86-64-v4:/usr/lib64/glibc-hwcaps/x86-64-v3:/usr/lib64/glibc-hwcaps/x86-64-v2:/usr/lib64/tls/haswell/avx512_1/x86_64:/usr/lib64/tls/haswell/avx512_1:/usr/lib64/tls/haswell/x86_64:/usr/lib64/tls/haswell:/usr/lib64/tls/avx512_1/x86_64:/usr/lib64/tls/avx512_1:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/haswell/avx512_1/x86_64:/usr/lib64/haswell/avx512_1:/usr/lib64/haswell/x86_64:/usr/lib64/haswell:/usr/lib64/avx512_1/x86_64:/usr/lib64/avx512_1:/usr/lib64/x86_64:/usr/lib64		(system search path)
   1468137:	  trying file=/usr/lib64/glibc-hwcaps/x86-64-v4/libpthread.so.0
   1468137:	  trying file=/usr/lib64/glibc-hwcaps/x86-64-v3/libpthread.so.0
   1468137:	  trying file=/usr/lib64/glibc-hwcaps/x86-64-v2/libpthread.so.0
   1468137:	  trying file=/usr/lib64/tls/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib64/tls/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib64/tls/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib64/tls/haswell/libpthread.so.0
   1468137:	  trying file=/usr/lib64/tls/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib64/tls/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib64/tls/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib64/tls/libpthread.so.0
   1468137:	  trying file=/usr/lib64/haswell/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib64/haswell/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib64/haswell/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib64/haswell/libpthread.so.0
   1468137:	  trying file=/usr/lib64/avx512_1/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib64/avx512_1/libpthread.so.0
   1468137:	  trying file=/usr/lib64/x86_64/libpthread.so.0
   1468137:	  trying file=/usr/lib64/libpthread.so.0
   1468137:	
   1468137:	find library=libdl.so.2 [0]; searching
   1468137:	 search path=/usr/local/anaconda3/bin/../lib		(RPATH from file python)
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/libdl.so.2
   1468137:	 search path=/usr/local/cuda/lib64:/usr/local/lib64:/usr/local/lib		(LD_LIBRARY_PATH)
   1468137:	  trying file=/usr/local/cuda/lib64/libdl.so.2
   1468137:	  trying file=/usr/local/lib64/libdl.so.2
   1468137:	  trying file=/usr/local/lib/libdl.so.2
   1468137:	 search path=/usr/lib64/tls:/usr/lib64		(system search path)
   1468137:	  trying file=/usr/lib64/tls/libdl.so.2
   1468137:	  trying file=/usr/lib64/libdl.so.2
   1468137:	
   1468137:	find library=libutil.so.1 [0]; searching
   1468137:	 search path=/usr/local/anaconda3/bin/../lib		(RPATH from file python)
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/libutil.so.1
   1468137:	 search path=/usr/local/cuda/lib64:/usr/local/lib64:/usr/local/lib		(LD_LIBRARY_PATH)
   1468137:	  trying file=/usr/local/cuda/lib64/libutil.so.1
   1468137:	  trying file=/usr/local/lib64/libutil.so.1
   1468137:	  trying file=/usr/local/lib/libutil.so.1
   1468137:	 search path=/usr/lib64/tls:/usr/lib64		(system search path)
   1468137:	  trying file=/usr/lib64/tls/libutil.so.1
   1468137:	  trying file=/usr/lib64/libutil.so.1
   1468137:	
   1468137:	find library=librt.so.1 [0]; searching
   1468137:	 search path=/usr/local/anaconda3/bin/../lib		(RPATH from file python)
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/librt.so.1
   1468137:	 search path=/usr/local/cuda/lib64:/usr/local/lib64:/usr/local/lib		(LD_LIBRARY_PATH)
   1468137:	  trying file=/usr/local/cuda/lib64/librt.so.1
   1468137:	  trying file=/usr/local/lib64/librt.so.1
   1468137:	  trying file=/usr/local/lib/librt.so.1
   1468137:	 search path=/usr/lib64/tls:/usr/lib64		(system search path)
   1468137:	  trying file=/usr/lib64/tls/librt.so.1
   1468137:	  trying file=/usr/lib64/librt.so.1
   1468137:	
   1468137:	find library=libm.so.6 [0]; searching
   1468137:	 search path=/usr/local/anaconda3/bin/../lib		(RPATH from file python)
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/libm.so.6
   1468137:	 search path=/usr/local/cuda/lib64:/usr/local/lib64:/usr/local/lib		(LD_LIBRARY_PATH)
   1468137:	  trying file=/usr/local/cuda/lib64/libm.so.6
   1468137:	  trying file=/usr/local/lib64/libm.so.6
   1468137:	  trying file=/usr/local/lib/libm.so.6
   1468137:	 search path=/usr/lib64/tls:/usr/lib64		(system search path)
   1468137:	  trying file=/usr/lib64/tls/libm.so.6
   1468137:	  trying file=/usr/lib64/libm.so.6
   1468137:	
   1468137:	find library=libc.so.6 [0]; searching
   1468137:	 search path=/usr/local/anaconda3/bin/../lib		(RPATH from file python)
   1468137:	  trying file=/usr/local/anaconda3/bin/../lib/libc.so.6
   1468137:	 search path=/usr/local/cuda/lib64:/usr/local/lib64:/usr/local/lib		(LD_LIBRARY_PATH)
   1468137:	  trying file=/usr/local/cuda/lib64/libc.so.6
   1468137:	  trying file=/usr/local/lib64/libc.so.6
   1468137:	  trying file=/usr/local/lib/libc.so.6
   1468137:	 search path=/usr/lib64/tls:/usr/lib64		(system search path)
   1468137:	  trying file=/usr/lib64/tls/libc.so.6
   1468137:	  trying file=/usr/lib64/libc.so.6
   1468137:	
   1468137:	
   1468137:	calling init: /usr/lib64/libpthread.so.0
   1468137:	
   1468137:	
   1468137:	calling init: /usr/lib64/libc.so.6
   1468137:	
   1468137:	
   1468137:	calling init: /usr/lib64/libm.so.6
   1468137:	
   1468137:	
   1468137:	calling init: /usr/lib64/librt.so.1
   1468137:	
   1468137:	
   1468137:	calling init: /usr/lib64/libutil.so.1
   1468137:	
   1468137:	
   1468137:	calling init: /usr/lib64/libdl.so.2
   1468137:	
   1468137:	
   1468137:	initialize program: python
   1468137:	
   1468137:	
   1468137:	transferring control: python
   1468137:	
   1468137:	
   1468137:	calling init: /usr/local/anaconda3/lib/python3.9/lib-dynload/_heapq.cpython-39-x86_64-linux-gnu.so
   1468137:	
Traceback (most recent call last):
  File "/home/parkn1/Working.ipynb", line 261, in &lt;module&gt;
    "scrolled": true
NameError: name 'true' is not defined
   1468137:	
   1468137:	calling fini: python [0]
   1468137:	
   1468137:	
   1468137:	calling fini: /usr/lib64/libdl.so.2 [0]
   1468137:	
   1468137:	
   1468137:	calling fini: /usr/lib64/libutil.so.1 [0]
   1468137:	
   1468137:	
   1468137:	calling fini: /usr/lib64/librt.so.1 [0]
   1468137:	
   1468137:	
   1468137:	calling fini: /usr/lib64/libpthread.so.0 [0]
   1468137:	
   1468137:	
   1468137:	calling fini: /usr/lib64/libm.so.6 [0]
   1468137:	
   1468137:	
   1468137:	calling fini: /usr/local/anaconda3/lib/python3.9/lib-dynload/_heapq.cpython-39-x86_64-linux-gnu.so [0]
   1468137:	
</code></pre>
<p>Finally, I’m not sure what you meant by grep the libcudnn output.<br>
Thank you and I apologies for my lack of knowledge.</p>

ptrblck: <p>You are mixing multiple PyTorch installations:</p>
<pre><code class="lang-auto">[pip3] torch==2.0.1
...
[conda] pytorch                   1.12.1          cuda112py39hb0b7ed5_201    conda-forge
</code></pre>
<p>Uninstall both and reinstall only the one you want to use to avoid library conflicts.</p>

Noanemo: <p>Thank you so much for your extended help.</p>