Specified kernel cache directory could not be created with custom loss function

Andrew_Hollis: <p>Hi all! I am very new to PyTorch, and I am trying to implement a custom loss function, but when I try to evaluate the function on cuda I get this error:</p>
<p>UserWarning: Specified kernel cache directory could not be created! This disables kernel caching. Specified directory is /people/holl433/.cache/torch/kernels. This warning will appear only once per process. (Triggered internally at  /opt/conda/conda-bld/pytorch_1646756402876/work/aten/src/ATen/native/cuda/jit_utils.cpp:860.)<br>
ll_loss = torch.digamma(a_ans) - torch.digamma(a_zero)</p>
<p>Here is a script to reproduce the error:</p>
<p>import torch<br>
from torch import nn<br>
from torch import cuda</p>
<p>class BMLoss(nn.Module):<br>
def <strong>init</strong>(self, coeff, prior=1.):<br>
super(BMLoss, self).<strong>init</strong>()<br>
self.prior = prior<br>
self.coeff = coeff</p>
<pre><code>def forward(self, logits, targets):
    '''
    Compute loss: kl - evi
    
    '''
    alphas = torch.exp(logits) 
    betas = torch.ones_like(logits) * self.prior
    
    # compute log-likelihood loss: psi(alpha_target) - psi(alpha_zero)
    a_ans = torch.gather(alphas, -1, targets.unsqueeze(-1)).squeeze(-1)
    a_zero = torch.sum(alphas, -1)
    ll_loss = torch.digamma(a_ans) - torch.digamma(a_zero)
    
    # compute kl loss: loss1 + loss2
    #       loss1 = log_gamma(alpha_zero) - \sum_k log_gamma(alpha_zero)
    #       loss2 = sum_k (alpha_k - beta_k) (digamma(alpha_k) - digamma(alpha_zero) )
    loss1 = torch.lgamma(a_zero) - torch.sum(torch.lgamma(alphas), -1)
    
    loss2 = torch.sum(
        (alphas - betas) * (torch.digamma(alphas) - torch.digamma(a_zero.unsqueeze(-1))),
    -1)
    kl_loss = loss1 + loss2
    
    loss = ((self.coeff*kl_loss - ll_loss)).mean()
    
    return loss 
</code></pre>
<p>device = torch.device(“cuda:0”) if cuda.is_available() else torch.device(“cpu”)<br>
bm_loss=BMLoss(coeff=0.01)<br>
bm_loss=bm_loss.to(device)<br>
inputs=torch.randn(4,6).to(device,dtype=torch.float)<br>
targets=torch.tensor([0,1,0,3]).to(device,dtype=torch.long)<br>
loss=bm_loss(inputs,targets)</p>
<p>I don’t get the warning message if I run the loss function on a cpu, only when I try to run it with cuda do I get an issue. Did I miss something in my implementation of the loss function that is causing the warning to pop up? Do I need to do something to make the loss function cuda compatible? I am using version 1.11.0 of PyTorch and version 11.3.1 of cudatoolkit, and version 11.4 of cuda.</p>

ptrblck: <p>Could you check if your Python process has write permissions to <code>/people/holl433/.cache/torch/kernels</code>? If not, allow the process to write cache files to this folder or change the path via <code>PYTORCH_KERNEL_CACHE_PATH</code> to a directory which your process has access to.<br>
In any case, this is triggered internally and will just JIT the kernel in the next execution again instead of using the cached kernel.</p>

Andrew_Hollis: <p>Thanks for the speedy reply! Do I set the value of PYTORCH_KERNEL_CACHE_PATH in my script? Or do I need to access and change it some other way? Also, am I correct in thinking that whether or not the cached kernel is used does not change the outcome of the learning process it just effects the speed?</p>

ptrblck: <p>Sorry for not being clear enough. <code>PYTORCH_KERNEL_CACHE_PATH</code> is an environment variable and you can set it in linux via:</p>
<pre><code class="lang-python">export PYTORCH_KERNEL_CACHE_PATH=/your/desired/path
</code></pre>
<p>in the terminal where you would run your Python script.</p>
<p>Yes, you are correct. If the cache isn’t accessible it would mean that the next script execution would need to recompile the kernel using NVRTC (NVIDIA runtime compiler) and would add a small overhead. The actual training workload will not change.</p>

Andrew_Hollis: <p>No problem, I am very new. Thanks for the clarification. Setting the environment variable to an accessible directory worked perfectly. Thank you again for the help!</p>