akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
An keras training loss nan or inf

hedinghong: <p>Can i ask for help with the tensorflow or keras question?<br>
i used this repo:<br>
<a href="https://github.com/yu4u/noise2noise" class="inline-onebox" rel="noopener nofollow ugc">GitHub - yu4u/noise2noise: An unofficial and partial Keras implementation of "Noise2Noise: Learning Image Restoration without Clean Data"</a>  which is keras framework(keras=2.1.6 and tensorflow-gpu==1.5.0)<br>
but get the loss of inf or nan from the 2th or 3th step<br>
do anyone have some idea about this?<br>
the training data is from the guidance of the repo, i think it’s not the problem of the data.</p>

hedinghong: <p>If i use cpu, it works ok</p>

ptrblck: <p>I would recommend asking Keras/Tensorflow-specific questions in their discussion board as you would find the experts there <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

hedinghong: <p>thank you for reply, i will look back to this if i have time and i turned to another model using torch</p>
Creating custom dataset of tilapia length and width measurement using mask r-cnn

Orekih: <p>Hello I am new in this field and I am wondering if someone can help me identifying if I am in right track. I am creating a code for custom dataset, I am using VIA annotation tool which I am using COCO format.</p>
<pre><code class="lang-auto">class MaskRCnnDataset(torch.utils.data.Dataset):
  def __init__(self,image_root, anno_file, transform = None):
    self.image_root = image_root
    self.anno_file = anno_file
    self.transform = transform

    #load the annotations
    self.coco = COCO(self.anno_file)

    #get the category ids
    self.category_ids = self.coco.getCatIds()

    #get the image id for each category
    self.image_ids = []
    for category_id in self.category_ids:
      image_ids = self.coco.getImgIds(catIds = [category_id])
      self.image_ids += image_ids

  def __getitem__(self, idx):
    #get the image id
    image_id = self.image_ids(idx)

    #lets load image
    imageInfo = self.coco.loadImgs(image_id)
    imagePath = os.path.join(self.image_root, imageInfo['file_name'])
    image = Image.open(imagePath).convert('RGB')

    #get the annotations from annotation_file
    annotation_ids = self.coco.getAnnIds(imageIds = image_id, catIds = self.category_ids, iscrowd = None)
    annontations = self.coco.loadAnns(annotation_ids)

    #extracting boxes and labels
    boxes = []
    masks = []
    labels = []
    lengths = []
    widths = []
      for ann in annotations:
        x, y, w, h = ann['bbox']
        box = [x, y, x + w, y + h]
        boxes.append(box)
        mask = self.coco.annToMask(ann)
        masks.append(mask)
        label = ann['category_id']
        labels.append(label)
        length = ann['length']
        lengths.append(length)
        width = ann['width']
        widths.append(width)

    #after extracting apply transforms
    if self.transform is not None:
      image = self.transform(image)
    
    #lets converts bboxes to tensor
    bboxes = torch.as_tensor(bboxes, dtype=torch.int32)

    #lets create the target dict
    targets = {}
    targets['boxes'] = bboxes
    target['labels'] = torch.as_tensor(labels, dtype=torch.int64)
    target['masks'] = masks
    target['image_id'] = torch.tensor([img_id])
    target['area'] = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
    target['iscrowd'] = torch.zeros((len(anns),), dtype=torch.int64)
    target['lengths'] = lengths
    target['widths'] = widths

    return image, target
  
  def __len__(self):
    return len(self.image_ids)
</code></pre>
What is the recommended way to calculating multi head attention scores?

Kenny_Stryker: <p>Which of them is correct?</p>
<pre><code class="lang-auto">embeddings = nn.Embedding(vocab, dim_size)
mha = nn.MultiHeadAttention(dim_size, num_heads)

embeds = embeddings(inp)
mha_out = mha(embeds, embeds, embeds)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">embeddings = nn.Embedding(vocab, dim_size)
query_linear = nn.Linear(dim_size, dim_size)
key_linear = nn.Linear(dim_size, dim_size)
value_linear = nn.Linear(dim_size, dim_size)
mha = nn.MultiHeadAttention(dim_size, num_heads)

embeds = embeddings(inp)
q = query_linear(embeds)
k = key_linear(embeds)
v = value_linear(embeds)
mha_out = mha(q,k,v)
</code></pre>
<p>I’ve been looking around Internet and original transformer paper. But I have been getting conflicting answers. I’d appreciate any help.</p>
Torch.cuda.is_available() fails with segmentation fault on ROCm

TheFrieber: <p>Hello, i got an issue when i try to run torch.cuda.is_available(), it crashes with Segmentation fault (core dumped) message. I am not very experienced with linux so i don’t really know the cause of this.</p>
<p>Specs:</p>
<blockquote>
<p>GPU: Amd rx580 8gb<br>
CPU: AMD ryzen 5 2600<br>
OS: Ubuntu 22.04 dual boot (Windows 10)<br>
Total of 24GB Ram</p>
</blockquote>
<p>pip list:</p>
<pre><code class="lang-auto">torch                  2.0.1+rocm5.4.2
torchaudio             2.0.2+rocm5.4.2
torchvision            0.15.2+rocm5.4.2
</code></pre>
<p>ROCm Version: 5.5.0</p>
<blockquote>
<p>apt show rocm-libs -a:</p>
</blockquote>
<pre><code class="lang-auto">Package: rocm-libs
Version: 5.5.0.50500-63~22.04
Priority: optional
Section: devel
Maintainer: ROCm Libs Support &lt;rocm-libs.support@amd.com&gt;
Installed-Size: 13.3 kB
Depends: hipblas (= 0.54.0.50500-63~22.04), hipfft (= 1.0.11.50500-63~22.04), hipsolver (= 1.7.0.50500-63~22.04), hipsparse (= 2.3.6.50500-63~22.04), miopen-hip (= 2.19.0.50500-63~22.04), rccl (= 2.15.5.50500-63~22.04), rocalution (= 2.1.8.50500-63~22.04), rocblas (= 2.47.0.50500-63~22.04), rocfft (= 1.0.21.50500-63~22.04), rocrand (= 2.10.16.50500-63~22.04), rocsolver (= 3.21.0.50500-63~22.04), rocsparse (= 2.5.1.50500-63~22.04), rocm-core (= 5.5.0.50500-63~22.04), hipblas-dev (= 0.54.0.50500-63~22.04), hipcub-dev (= 2.10.12.50500-63~22.04), hipfft-dev (= 1.0.11.50500-63~22.04), hipsolver-dev (= 1.7.0.50500-63~22.04), hipsparse-dev (= 2.3.6.50500-63~22.04), miopen-hip-dev (= 2.19.0.50500-63~22.04), rccl-dev (= 2.15.5.50500-63~22.04), rocalution-dev (= 2.1.8.50500-63~22.04), rocblas-dev (= 2.47.0.50500-63~22.04), rocfft-dev (= 1.0.21.50500-63~22.04), rocprim-dev (= 2.10.9.50500-63~22.04), rocrand-dev (= 2.10.16.50500-63~22.04), rocsolver-dev (= 3.21.0.50500-63~22.04), rocsparse-dev (= 2.5.1.50500-63~22.04), rocthrust-dev (= 2.10.9.50500-63~22.04), rocwmma-dev (= 0.7.0.50500-63~22.04)
Homepage: https://github.com/RadeonOpenCompute/ROCm
Download-Size: 1,004 B
APT-Manual-Installed: yes
APT-Sources: https://repo.radeon.com/rocm/apt/5.5 jammy/main amd64 Packages
Description: Radeon Open Compute (ROCm) Runtime software stack
</code></pre>
<p>test.py script: (Followed the verification tab on pytorch)</p>
<pre><code class="lang-auto">import torch
torch.cuda.is_available()
</code></pre>
<p>Python version: 3.10.6</p>
<p>Thanks for help in advance</p>

eqy: <p>It looks like this issue may be related: <a href="https://github.com/RadeonOpenCompute/ROCm/issues/1930" class="inline-onebox" rel="noopener nofollow ugc">rocm/pytorch:latest Segmentation fault · Issue #1930 · RadeonOpenCompute/ROCm · GitHub</a></p>
<p>Specifically, this solution: <a href="https://github.com/RadeonOpenCompute/ROCm/issues/1930#issuecomment-1465319536" class="inline-onebox" rel="noopener nofollow ugc">rocm/pytorch:latest Segmentation fault · Issue #1930 · RadeonOpenCompute/ROCm · GitHub</a> which suggests that you might need to rebuild PyTorch for your specific GPU architecture.</p>
Libtorch + CUDA kernels

umbriel: <p>I’m writing a module using libtorch with lots of point-wise operations (physics engine).<br>
It runs pretty slow on CUDA and as I understand I’ve got to use custom kernels.<br>
Is there an ‘official’ way to do it?<br>
I saw this <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">tutorial</a>  but the info relates only to python front end.</p>

eqy: <p>Is there a reason you must use <code>libtorch</code>? The typical recommendation for PyTorch 2.0+ is to <code>torch.compile</code> for workloads that have many pointwise operations amenable to e.g., operator fusion: <a href="https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">torch.compile Tutorial — PyTorch Tutorials 2.0.1+cu117 documentation</a></p>
What is the simplest way to change class_to_idx attribute?

ptorres: <p>Hello everyone, i’m defining my data and dataloader in the currently way:</p>
<pre><code class="lang-auto">	data = {
		'train':
		datasets.ImageFolder(root=train_path, transform=image_transforms['train'], ),
		'valid':
		datasets.ImageFolder(root=valid_path, transform=image_transforms['valid']),
	}
	
	dataloaders = {
		'train': DataLoader(data['train'], batch_size=batch_size, shuffle=True),
		'val': DataLoader(data['valid'], batch_size=batch_size, shuffle=True)
	}
</code></pre>
<p>In my problem i have four classes (A, AB, AC, ABC). When I print <code>data['train'].class_to_idx</code> i get the following output:</p>
<pre><code class="lang-auto">{'A': 0, 'AB': 1, 'ABC': 2, 'AC': 3}
</code></pre>
<p>All that i wanna to do is change index of AC to 2, and ABC to 3. and get the following result</p>
<pre><code class="lang-auto">{'A': 0, 'AB': 1, 'ABC': 3, 'AC': 2}
</code></pre>
<p>What is the simplest way to do that, and what other changes i have to do along my code?</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

ptrblck: <p>I would recommend to create a custom <code>Dataset</code> and maybe just reuse some parts of the <code>ImageFolder</code> dataset from <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py">here</a>.<br>
Changing this attribute after the <code>ImageFolder</code> was created seems to be wrong, since <code>class_to_idx</code> will be used to create the dataset as seen <a href="https://github.com/pytorch/vision/blob/e92b5155d7d78363826a822ccacf7c046d19245a/torchvision/datasets/folder.py#L95">here</a>.</p>

Nikki: <p>Would you like to share your solution?</p>

vanlyvu: <p><a class="mention" href="/u/nikki">@Nikki</a><br>
you can create custom <code>CustomImageFolder</code> loader class and override <code>find_classes</code> function like this:</p>
<pre><code class="lang-python">from typing import Tuple, List, Dict

from torchvision.datasets import ImageFolder

def classes_to_idx() -&gt; dict:
    return load_json_file("class_indexes.json")


class CustomImageFolder(ImageFolder):
    def find_classes(self, directory: str) -&gt; Tuple[List[str], Dict[str, int]]:
        """
        Override this method to load from setting file instead of scanning directory
        """
        classes = list(classes_to_idx().keys())
        classes_to_idx = classes_to_idx()
        return classes, classes_to_idx


</code></pre>
<p>Here is sample <code>class_indexes.json</code> json file mapping class to index (“1”, “2”, …“13” is class folder name; 0, 1, …, 12 is index of class):</p>
<pre><code class="lang-json">{
    "1": 0,
    "2": 1,
    "3": 2,
    "4": 3,
    "5": 4,
    "6": 5,
    "7": 6,
    "8": 7,
    "9": 8,
    "10": 9,
    "11": 10,
    "12": 11,
    "13": 12
}
</code></pre>
How to prevent DistributedDataParallel with nccl backend from creating duplicate processes?

biock: <p>When I run the official demo of PyTorch <code>DistributedDataParallel</code> training (“elastic_ddp.py” from <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#initialize-ddp-with-torch-distributed-run-torchrun" rel="noopener nofollow ugc">Initialize DDP with torch.distributed.run/torchrun</a>), I found redundant model copies were created on GPU:0.</p>
<p>Are there any solutions to avoid creating extra copies? Thanks!</p>
<hr>
<p>Environment:</p>
<ul>
<li>PyTorch: 1.12.1</li>
<li>Python: 3.9.7</li>
<li>cuda: 11.6</li>
</ul>
<p>Run the script:<br>
<code>torchrun --nnodes=1 --nproc_per_node=4 --rdzv_id=100 --rdzv_backend=c10d --rdzv_endpoint=localhost:29400 elastic_ddp.py</code></p>
<p><code>nvidia-smi</code> output (redundant copies were found on GPU:0, PID=2754936/2754937/2754938):</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A   2754935      C   ...opt/miniconda3/bin/python      955MiB |
|    0   N/A  N/A   2754936      C   ...opt/miniconda3/bin/python      653MiB |
|    0   N/A  N/A   2754937      C   ...opt/miniconda3/bin/python      653MiB |
|    0   N/A  N/A   2754938      C   ...opt/miniconda3/bin/python      653MiB |
|    1   N/A  N/A   2754936      C   ...opt/miniconda3/bin/python      911MiB |
|    2   N/A  N/A   2754937      C   ...opt/miniconda3/bin/python     1023MiB |
|    3   N/A  N/A   2754938      C   ...opt/miniconda3/bin/python     1007MiB |</code></pre>

ptrblck: <p>You could set the device via <code>torch.cuda.set_device(rank)</code> and use <code>torch.cuda.current_device</code> or the <code>rank</code> to move the model and data to the corresponding device.</p>
Is it possible to set learning rates for each channel of the weight in a conv layer?

txfs1926: <p>Hello,<br>
I wonder if it is possible to set learning rates for each channel of the weight in a certain conv layer? I wrote an example but it would raise an error.</p>
<pre><code class="lang-auto"># e.g.
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv = nn.Conv2d(3, 8, kernel_size=1, padding=0, stride=1, bias=False)

    def forward(self, x):
        return self.conv(x)

model = Model()
print(model.conv.weight.shape) # torch.Size([8, 3, 1, 1])

lr = [0.01* i for i in range(1, model.conv.weight.shape[0] + 1)] # [0.01, 0.02, ..., 0.08]

torch.optim.Adam(([{'params': p, 'lr': l} for p,l in zip(model.conv.weight, lr)])
# ValueError: can't optimize a non-leaf Tensor
</code></pre>
<p>Thanks!</p>

zeakey: <p>A possible solution is to split the weight parameter in a layer (e.g. nn.Conv2d) into multiple parameters,<br>
and then concat the outputs together.</p>
<p>For example, for a layer with 2 input channels and 4 output channels, a pseudo code works as bellow:</p>
<pre><code class="lang-auto">class MyConv(nn.Module):

    def __init__(self, channel_in, channel_out, kernel_size):
        self.weight1 = Parameter(torch.zeros(1, channel_in, kernel_size, kernel_size))
        self.weight2 = Parameter(torch.zeros(1, channel_in, kernel_size, kernel_size))
        ...
        self.weight4 = Parameter(torch.zeros(1, channel_in, kernel_size, kernel_size))

        ...
    def forward(self, input):
        output1 = torch.nn.functional.conv2d(input, self.weight1, ...)
        ...
        output4 = torch.nn.functional.conv2d(input, self.weight1)
        return torch.cat((output1, ..., output4), dim=1)
</code></pre>
<p>And then you can apply channel-wise learning rate tuning by adjusting the learning rates of parameter1, …, parameter4.</p>

Zheng_Perry: <p>Compared with splitting the weight, I prefer to modify “sgd.py” with a few lines. For example, you can change input settings into <code>lr: Union[float, Tensor],</code> and change this line: <code>torch._foreach_add_(device_params, device_grads, alpha=-lr)</code> by add the judgement sentence and the index of ‘lr’.</p>
RuntimeError: DataLoader worker (pid(s) 48916, 48917) exited unexpectedly

Killpit: <p>I have encountered the error when I try to run the <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="noopener nofollow ugc">tutorial</a> with tensorboard. Initially, it told me that tensorboard was not found and I installed tensorboard along with TensorFlow. Even though I had not encountered an issue, now I have a runtime error and couldn’t derive a solution to this problem, I’ve created a file in GitHub and you can reach the file from this <a href="https://github.com/Killpit/PyTorch-Projects/blob/main/tensor_board.py" rel="noopener nofollow ugc">link</a></p>
Implementing K-fold cross validation in pytorch

Mahran-xo: <p>is there a guide on how to implement k-fold cross validation in pytorch in clean and optimized way?</p>

ptrblck: <p>You could take a look at <a href="https://github.com/skorch-dev/skorch">skorch</a>, which is a scikit-learn compatible neural network library for PyTorch allowing you to use the k-fold CV methods from scikit-learn directly.</p>

Mahran-xo: <p>i looked at the documentation it is kind of unorganized , they do not provide a snippet explaining the k-fold strategy</p>
Iterating all layers in network

Zihan_LI: <p>Is there any way to recursively iterate over all layers in a nn.Module instance including sublayers in nn.Sequential module</p>
<p>I’ve tried .modules() and .children(), both of them seem not be able to unfold nn.Sequential module. It requires me to write some recursive function call to achieve this. I wonder is there any elegant way to do it.</p>

ptrblck: <p>It seems to work for me:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.seq1 = nn.Sequential(
            nn.Sequential(
                nn.Linear(1, 2),
                nn.ReLU(),
                nn.Linear(2, 3)
            ),
            nn.ReLU(),
            nn.Linear(3, 4)
        )
        self.out = nn.Linear(4, 5)
        
model = MyModel()
for name, module in model.named_modules():
    print(name, type(module))

#  &lt;class '__main__.MyModel'&gt;
# seq1 &lt;class 'torch.nn.modules.container.Sequential'&gt;
# seq1.0 &lt;class 'torch.nn.modules.container.Sequential'&gt;
# seq1.0.0 &lt;class 'torch.nn.modules.linear.Linear'&gt;
# seq1.0.1 &lt;class 'torch.nn.modules.activation.ReLU'&gt;
# seq1.0.2 &lt;class 'torch.nn.modules.linear.Linear'&gt;
# seq1.1 &lt;class 'torch.nn.modules.activation.ReLU'&gt;
# seq1.2 &lt;class 'torch.nn.modules.linear.Linear'&gt;
# out &lt;class 'torch.nn.modules.linear.Linear'&gt;
</code></pre>
Why can't my torch._C._TensorBase recognize my GPU?

Brian_Bakerman: <p>Quick context:<br>
I’m running Whisper and Pyannote Audio diarization in a google cloud VM instance group. Whisper is able to detect my GPU just fine, but Pyannote cannot. torch.cuda.is_available() returns true.</p>
<p>Below describes my setup, which should make the issue reproducible. I have tried to use the development branch, torch 2.0.0, upgraded cuda, etc. Note: Pyannote worked for me in the past, not sure what happened.</p>
<p>Running the following txt file:<br>
torch==1.11.0<br>
torchaudio==0.11.0<br>
torchvision==0.12.0<br>
pyannote.audio</p>
<p>Docker file:<br>
FROM nvidia/cuda:11.2.1-runtime-ubuntu20.04</p>
<p>RUN apt-get update &amp;&amp; apt-get install -y --no-install-recommends<br>
python3.9<br>
python3-pip<br>
&amp;&amp;<br>
apt-get clean &amp;&amp;<br>
rm -rf /var/lib/apt/lists/*</p>
<p>Followed these directions for starting the VM with GPU:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus?hl=ko">
  <header class="source">
      <img src="https://www.gstatic.com/devrel-devsite/prod/v693c51561ffcdff9e6200b7e4cd9c2bdcde516af6b265b56d58924d489006b90/cloud/images/favicons/onecloud/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus?hl=ko" target="_blank" rel="noopener nofollow ugc">Google Cloud</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://cloud.google.com/_static/cloud/images/social-icon-google-cloud-1200-630.png?hl=ko" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus?hl=ko" target="_blank" rel="noopener nofollow ugc">GPU 가속기로 인스턴스 실행  |  컨테이너 최적화 OS  |  Google Cloud</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Pytorch c++ BLSTM

scottrs: <p>i am trying to solve a BLSTM model in c++ using pytorch. I copied the code from here</p>
<aside class="quote quote-modified" data-post="1" data-topic="46054">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/koro/48/16213_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-use-bidirectional-lstm-in-c-api-correctly/46054">How to use Bidirectional LSTM in C++ API correctly?</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    Dear Community, 
I’ve been trying to test out a bidirectional LSTM in Pytorch’s C++ API, but I can’t get it to work. 
If I define a LSTM-module with the option bidirectional(true), does it automatically create a reverse version 
of the input sequence and feed that into to reverse LSTM? Does it also automatically concatenate the output tensors into one, that can be directly fed into a dense layer, and how to format it to feed it into one? 
My Code: 
        auto bilstm = torch::nn::LSTM(torch::nn…
  </blockquote>
</aside>

<p>however i am getting an error that output is not a member of ltsm1 (which apparently is a tuple of at::Tensor).</p>
<p>i assume this code must have worked at some point but it doesn’t now.  any thoughts?</p>
<p>here is the exactly error</p>
<p>|Error (active)|E0135|class std::tuple&lt;at::Tensor, std::tuple&lt;at::Tensor, at::Tensor&gt;&gt; has no member output|testpyt|W:\bb\git\testpyt\srstst.cpp|84||</p>
Torch onnx tensorrt question

aifartist: <p>When converting from diffusers, to onnx and then to TRT I notice warnings like:</p>
<blockquote>
<p>[W] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.&gt;L&gt;&gt;<br>
[W] onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clamped</p>
</blockquote>
<p>Is this because the torch to onnx conversion is converting float32 bit weight to int64 and then the TRT engine build tries to convert those to int32 leading to the error?  Considering these values are only going to be converted again to fp16 this seems like a poor way to do it.</p>
<p>Going from float32 to float16 directly should result in far less clamping being needed than float32 → int64 → int32 → float16.<br>
I’m wonder if this might be the reason the Stable Diffusion images rendered by TensorRT have an obvious lower quality than normal pytorch inference at fp16.</p>
<p>Even though tensorRT doesn’t support int64 wouldn’t another viable approach be to convert from int64 directly to float16 without the intermediate int32?</p>
Index_select where the indices are 2D rather than 1D?

TheMountain: <p>Hi!</p>
<p>I’m trying to make my own nn.Module where the input and outputs are both 2D grayscale images.  Each 2D output pixel (ignoring batches for now) will copy a pixel in the input 2D image, according to a provided lookup tensor of 2D co-ordinates.  So, for example, this snippet would rotate the corner pixels (the ones with values 40 to 43) of the input 4x4 image clockwise:</p>
<pre><code class="lang-auto">        input_image = torch.Tensor([
            [40,  1,  2, 41],
            [ 3,  4,  5,  6],
            [ 7,  8,  9,  1],
            [42,  2,  3, 43]
        ])

        coords_to_look_up = torch.Tensor([
            [[3, 0], [0, 1], [0, 2], [0, 0]],
            [[1, 0], [1, 1], [1, 2], [1, 3]],
            [[2, 0], [2, 1], [2, 2], [2, 3]],
            [[3, 3], [3, 1], [3, 2], [0, 3]],
        ]).long()

        result_i_want = torch.Tensor([
            [42,  1,  2, 40],
            [ 3,  4,  5,  6],
            [ 7,  8,  9,  1],
            [43,  2,  3, 41]
        ])
</code></pre>
<p>What operation would I perform on input_image and coords_to_look_up to produce result_i_want?  I found index_select, but it seems to only deal with 1D cases.</p>
<p>Thanks for any help. <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Direct indexing should work:</p>
<pre><code class="lang-python">input_image[coords_to_look_up[:, :, 0], coords_to_look_up[:, :, 1]]
</code></pre>

TheMountain: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="180328">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>input_image[coords_to_look_up[:, :, 0], coords_to_look_up[:, :, 1]]</code></p>
</blockquote>
</aside>
<p>Thank you!  Your solution looks better than the monstrosity I just came up with, which I’ve put here as some sort of Friday horror show for you:</p>
<pre><code class="lang-auto">        x_width = input_image.size(1)
        y_width = input_image.size(0)
        input_array_flat = input_image.flatten(0)
        coord_sources_flat = coords_to_look_up.flatten(0)
        mult_y_by_x_size = torch.tensor([x_width, 1]).flatten(0).repeat(x_width * y_width)
        coords_y_scaled = coord_sources_flat * mult_y_by_x_size
        just_x_coords        = torch.as_strided(coords_y_scaled, (x_width * y_width,), (2,), 1)
        just_y_scaled_coords = torch.as_strided(coords_y_scaled, (x_width * y_width,), (2,))
        indices_flat = just_x_coords + just_y_scaled_coords
        result_i_want_flat = input_array_flat.index_select(0, indices_flat)
        
        result_i_want = result_i_want_flat.reshape((y_width, x_width))
</code></pre>
Cant have reproducible results with GPU

ozzyc: <p>Hi everyone,<br>
I am using pytorch(1.10.2)(cudnn=8.1.1_11.2) both with Jupyter and VSCode. I have reproducible results with CPU but not with GPU (use only 1 gpu). I use 3D Resnet for my prediction model. I remove the last two layers of ResNet34 (pooling and FC layers) and add my own classifier (including ConvTranspose3d and AvgPool3d (non-deterministic algorithms)).  I also do random cropping for data augmentation. I use the followings for the seeds:</p>
<hr>
<p>def setup_seed(seedno):<br>
torch.manual_seed(seedno)<br>
os.environ[‘PYTHONHASHSEED’] = str(seedno)<br>
torch.cuda.manual_seed(seedno)<br>
torch.cuda.manual_seed_all(seedno)<br>
np.random.seed(seedno)<br>
random.seed(seedno)<br>
torch.manual_seed(seedno)<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False</p>
<p>def _init_fn(worker_id):<br>
worker_seed = torch.initial_seed() % 2**32<br>
numpy.random.seed(worker_seed)<br>
random.seed(worker_seed)</p>
<p>setup_seed(int(42))<br>
g = torch.Generator()<br>
g.manual_seed(42)</p>
<p>train_loader = DataLoader(train_dataset, batch_size=4, shuffle=True, num_workers=40, prefetch_factor=256, drop_last=True, worker_init_fn=numpy.random.seed(int(42)))</p>
<p>val_loader = DataLoader(validation_dataset, batch_size=1, shuffle=False, num_workers=40, prefetch_factor=256, drop_last=True, worker_init_fn=numpy.random.seed(int(42)))</p>
<p>test_loader = DataLoader(test_dataset, batch_size=1, shuffle=False, num_workers=40, prefetch_factor=256,drop_last=True, worker_init_fn=numpy.random.seed(int(42)))</p>
<hr>
<p>When I use torch.use_deterministic_algorithms(True), I got the error of<br>
“RuntimeError: avg_pool3d_backward_cuda does not have a deterministic implementation…”.</p>
<p>The batches in train_loader are exactly the same for two runs but the test accuracies vary about  ~%5 when using GPU (the results are exactly the same with CPU). When I check <a href="https://pytorch.org/docs/stable/notes/randomness.html" class="inline-onebox" rel="noopener nofollow ugc">Reproducibility — PyTorch 2.0 documentation</a>, it says some algorithms (e.g. AvgPool3d) are non-deterministic and cannot provide reproducible results. I would appreciate any suggestions on this issue.</p>

ptrblck: <p>You could create a feature request on GitHub for the missing deterministic algorithms of the 3d pooling layer or move it to the CPU as a current workaround.</p>

ozzyc: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for your reply. I prefer to use GPU since it takes too long to train the model with CPU. Apparently, there is no way to get rid of this reproducibility issue while using some non-deterministic algorithms with GPU.</p>
Pre-calculated distance matrix in Dataset

ericyuan: <p>I’m working on a graph neural network involving 3d positional vectors R as nodes and the scaler distances D as edges, and I want to take derivatives of the network output with respect to the initial position R. Here is a minimum working example not involving any neural network.</p>
<pre><code class="lang-auto">import torch

R = torch.rand(10, 5, 3, requires_grad=True)  # Position vectors, (n_data, n_point, 3)
D = torch.norm(R.unsqueeze(2) - R.unsqueeze(1), dim=-1)  # Distances matrix, (n_data, n_point, n_point)
gradient = torch.autograd.grad(D, R, grad_outputs=torch.ones_like(D))[0]
print(gradient)  # Returns tensor with shape (10, 5, 3)
</code></pre>
<p>Since the distance matrix D does not change throughout the training, I want to pre-calculate the distance matrix when I create the PyTorch Dataset, rather than to re-calculate it every time I load a batch. Somehow the graph structure in PyTorch doesn’t allow me to do that, presumably because the batched R of the Dataset output is different from the original R used in the graph of D.</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import Dataset

class BatchDataset(Dataset):
    def __init__(self, R):
        self.R = torch.tensor(R, requires_grad=True)  # Position vectors, (n_data, n_point, 3)
        self.D = torch.norm(self.R.unsqueeze(2) - self.R.unsqueeze(1), dim=-1)  # Distances matrix, (n_data, n_point, n_point)
    def __getitem__(self, index):
        return self.R[index], self.D[index]
    def __len__(self):
        return self.R.size(0)

R = torch.rand(10, 5, 3, requires_grad=True)  # Position vectors, (n_data, n_point, 3)
batch_R, batch_D = BatchDataset(R)[:]
gradient = torch.autograd.grad(batch_D, batch_R, grad_outputs=torch.ones_like(batch_D))[0]
print(gradient)  # Returns RuntimeError: One of the differentiated Tensors appears to not have been used in the graph.
gradient = torch.autograd.grad(batch_D, batch_R, grad_outputs=torch.ones_like(batch_D), allow_unused=True)[0]
print(gradient)  # Returns None
</code></pre>
<p>Is there a way to pre-calculate the distance matrix in a Dataset while preserving the graph structure for gradient calculation? Or the distance matrix and the edge features have to be re-calculated if the gradients are needed? Thanks.</p>
Broadcasting Compare

MaKaNu: <p>I have following little example with the variable <code>bboxes</code> of shape NxMxWxW, where N is my batchsize, M is the max number of bounding boxes per image, W is the Width and Height of the Image. A BBox is represented as the class idx in each pixel the bbox is present. Additional I have the variable <code>candidate_classes</code> of shape NxO, where O is the number of candidates. If I run the following:</p>
<pre><code class="lang-python">bboxes_classes = bboxes.amax((2,3))
</code></pre>
<p>I will get for each bbox the class label in shape NxM. Where I now struggle is with my comparison. The goal is to get a Tensor with shape NxMxO with some vectorizing shenanigans, so that I have compared for each image and for each bbox its class against each  candidates class. If for loops are necessary I don’t need any help since this is already implemented.</p>
Disaggregated Out of Tree Backend

neeldug: <p>Hi I was looking to create a custom out-of-tree backend to support a disaggregated OS, I wish to essentially replicate the CUDA kernels but remotely using these disaggregated devices, but I’m having a little difficulty regarding understanding a few components of the backend.</p>
<p>My current understanding of the creation of a custom backend is that:</p>
<ul>
<li>A device guard must be created to handle multiple devices in a thread safe manner?</li>
<li>The allocator handles getting memory onto the device</li>
<li>Each kernel operates on the memory in the device</li>
</ul>
<p>My current thoughts on how I’d go about implementing this is by using the disaggregated OS’s RPCs for memory handling for the Allocator and then loading the CUDA kernels directly onto the device such that it should just work. In the topology that I’m working with, each device is associated with a single GPU, and each device has operations for loading CUDA kernels, creating CUDA streams, and allocating device or unified memory. I was hoping to understand what a good approach would be for extending PyTorch to support this OS as a backend?</p>
<p>My main concerns with my intended approach are that on the CUDA kernels that are used, several make direct use of the CUDADeviceGuard, I assume changing this to my device guard could be a large problem given that it uses no CUDA device guards directly, (since each device only has a single CUDA device on-board). I was hoping to clarify my assumptions regarding the various components and whether it is feasible to directly copy the CUDA kernels and load them directly onto my remote device?</p>
How to learn the implementation of basic operation

rianbrian: <p>Hi, I am new in Pytorch and GPU programming</p>
<p>I want to learn how some basic tensor operations in Pytorch (e.g. ‘.sum()’ ) is implemented in GPU, like how Pytorch utilizes  GPU parallelization for each operation.</p>
<p>Can somebody help me to show in which part of the Pytorch code I can starting to learn about this?</p>
<p>Thank you</p>

JuanFMontesinos: <p>Hmmm it’s hard to say but actually pytorch doesn’t implement anything (I’d say). Everything relies on CUDA and CuDNN. So learning pytorch is very far from understanding cuda programming.</p>
<p>To learn cuda programming you can have a look at nvidia courses.<br>
<a href="https://courses.nvidia.com/courses/course-v1:DLI+C-AC-02+V1/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://courses.nvidia.com/courses/course-v1:DLI+C-AC-02+V1/</a><br>
there is one with numba and python that is easy to follow.</p>

rianbrian: <p>I see, is there any ways to see how PyTorch tensor operation works in the lower level (e.g. how they use CuDNN)? My aim is to make sure and understand how the tensor basic operation works in the lower level</p>
Does binary cross entropy with logits = 0.5 equals random guess?

bornexmachina: <p>Hi, I have 256 samples labeled with 1 and 256 samples labeled with 0. My loss seems to converge to 0.51</p>
<p>Does it mean, the model only makes a random guess?</p>
<p>To be precise I have</p>
<p><code>domain_loss = F.binary_cross_entropy_with_logits(domain_predictions, domain_y)</code></p>
<p>and the printout converges to 0.51</p>

eqy: <p>Not exactly, as with two classes the loss should be about ~0.693 for “random” guessing where the model is outputting ~0.5 for each sample. In general we would expect the model at initialization to yield a loss of ln(num_classes), which is also why e.g., models trained on ImageNet typically start at a loss of about ln(1000) or 6.9.</p>
<p>For example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; output = (torch.ones(10000) * 0.5)
&gt;&gt;&gt; target = (torch.rand(10000) &gt; 0.5)*1.0
&gt;&gt;&gt; torch.nn.BCELoss()(output, target)
tensor(0.6931)
</code></pre>
<p>True “confident” random guessing would yield something a little different e.g.,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; output = (torch.rand(10000) &gt; 0.5)*1.0
&gt;&gt;&gt; target = (torch.rand(10000) &gt; 0.5)*1.0
&gt;&gt;&gt; torch.nn.BCELoss()(output, target)
tensor(50.4500)
</code></pre>
Embedding layer trained or not in Transformer?

IneedMrmeeseeks: <p>Hello, there!</p>
<p>I’m recently working on Transformer things(Self-Attention, Mutihead Attention).<br>
But I am genuinely curious<br>
whether the Embedding layer in the transformer is trained to have similarity suck like Skip-gram, CBOW<br>
or it’s just initialized random vector.</p>
<p>Does it use pre-trained vocab?</p>
<p>Thx for reading.<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji only-emoji" alt=":pray:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji only-emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
How to set the Conv2d to have a common filter over all input channels

Moslem_Yazdanpanah1: <p>How to set the Conv2d layer to convolve one common filter (one filter per kernel) over all input channels rather than creating one per each input channel?</p>

ptrblck: <p>By default each filter in a conv layer will use <em>all</em> input channels and create a single output channel.<br>
Could you describe your use case a bit more in case <code>nn.Conv2d(in_channels, 1, ...)</code> won’t work?</p>

Moslem_Yazdanpanah1: <p>Thanks. There is no special use case as it is a part of another bigger project. I need a conv layer with different kernels and one filter per kernel to be shared between all input channels. The “nn.Conv2d(in_channels, 1, …)” will do the job however, I need to define multiples of this in case of having output_channels &gt; 1.</p>
<p>Do I need to modify the C codes to implement this?</p>

ptrblck: <p>I’m not sure how to understand:</p>
<aside class="quote no-group" data-username="Moslem_Yazdanpanah1" data-post="4" data-topic="180252">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/moslem_yazdanpanah1/48/59347_2.png" class="avatar"> Moslem_Yazdanpanah1:</div>
<blockquote>
<p>and one filter per kernel to be shared between all input channels.</p>
</blockquote>
</aside>
<p>Filters and kernels are used as a synonym while it seems you put a different meaning to these terms. Could you explain what means what exactly?</p>

KFrank: <p>Hi Moslem!</p>
<aside class="quote no-group" data-username="Moslem_Yazdanpanah1" data-post="1" data-topic="180252" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/moslem_yazdanpanah1/48/59347_2.png" class="avatar"> Moslem_Yazdanpanah1:</div>
<blockquote>
<p>How to set the Conv2d layer to convolve one common filter (one filter per kernel) over all input channels</p>
</blockquote>
</aside>
<p>If I understand your use case correctly, you can reshape your input tensor<br>
to “move” the channels dimension into the batch dimension, pass it into a<br>
convolution with a single input channel, <code>in_channels = 1</code>, and then reshape<br>
the output to restore the initial batch dimension.  Something like this:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.0.0'
&gt;&gt;&gt; _ = torch.manual_seed (2023)
&gt;&gt;&gt; conv = torch.nn.Conv2d (1, 1, 3)
&gt;&gt;&gt; t = torch.randn (2, 3, 5, 5)
&gt;&gt;&gt; conv (t.reshape (6, 1, 5, 5)).reshape (2, 3, 3, 3).shape
torch.Size([2, 3, 3, 3])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Moslem_Yazdanpanah1: <p>Sorry  for the ambiguity in my description. <a class="mention" href="/u/kfrank">@KFrank</a> 's solution is roughly what I am looking for.</p>

Moslem_Yazdanpanah1: <p>Thank you Frank. You saved me a lot of time.</p>
Torch-Scripting the Swish function from the PyTorch Video implementation of X3D

JamesDickens: <p>I am trying to get a torchscript .pt file from a trained X3D network (a 3D CNN). One module<br>
is causing the problem, in particular an implementation of the Swish activation given by:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Swish(nn.Module):
    """
    Wrapper for the Swish activation function.
    """

    def forward(self, x):
        return SwishFunction.apply(x)


class SwishFunction(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        result = x * torch.sigmoid(x)
        ctx.save_for_backward(x)
        return result

    @staticmethod
    def backward(ctx, grad_output):
        x = ctx.saved_variables[0]
        sigmoid_x = torch.sigmoid(x)
        return grad_output * (sigmoid_x * (1 + x * (1 - sigmoid_x)))
</code></pre>
<p>When I try to script the X3D network, I get the error specific to this module, which has a custom back-pass implementation. The error reads:</p>
<pre><code class="lang-auto">Could not export Python function call 'SwishFunction'. Remove calls to Python functions before export. Did you forget to add @script or @script_method annotation? If this is a nn.ModuleList, add it to __constants__:
</code></pre>
<p>Any thoughts how to fix this?</p>
Is there a recommended way when using DDP to log to a different file per GPU?

divinho: <p>As I see it, this is not a problem if one is using one GPU per node as when submitting with <code>torchrun</code> one could specify what logging file to use for each process.</p>
<p>But what if one is using a node that use multiple GPUs? It seems like there is no way to redirect the outputs of different processes to different log files?</p>
torch.distributions.Bernoulli boolean support weird behaviour

zaRizk7: <p>Hello,</p>
<p>I’m trying to use the <code>torch.distributions.Bernoulli(prob).log_prob()</code>. I encounter an error where I have to use a boolean support for the input of <code>log_prob</code>. The error is</p>
<pre><code class="lang-auto">Expected value argument (Tensor of shape (32, 784)) to be within the support (Boolean()) of the distribution Bernoulli(probs: torch.Size([32, 784])), but found invalid values:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')
</code></pre>
<p>Oddly, when I wrapped the log_prob’s input i.e. <code>torch.bernoulli(targets)</code>, it somehow works which i don’t expect because the returned dtype for it is <code>torch.float32</code>. Are there any fixes for this? Thanks.</p>
<p>The code I’m using for it is wrapped in this function below,</p>
<pre><code class="lang-auto">def likelihood(outputs, targets):
    dist = torch.distributions.Bernoulli(probs=outputs)
    return dist.log_prob(targets).exp()
</code></pre>
<p>The outputs and targets are already within [0, 1] value range because the outputs are from a sigmoid activation while the targets are from a binary dataset.</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue as I’m currently unsure what exactly your inputs represent?</p>
Hessian of loss w.r.t. model parameters is all 0

Laurent1: <p>Hi,</p>
<p>I want to compute the second derivative (Hessian) of the loss w.r.t. the parameters of the model. I proceed batch-wise. Here is a code to illustrate my problem:</p>
<pre><code class="lang-auto">
import torch
from torch import nn
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
from torch.autograd.functional import hessian

# simple dataset
class data_set(Dataset):
    def __init__(self):
        self.input = torch.randn(10,2)
        self.output = torch.randn(10,1)
        
    def __len__(self):
        return len(self.input)

    def __getitem__(self, index):
        return self.input[index], self.output[index]

dataset = data_set()
dataloader = DataLoader(dataset, batch_size=2, shuffle=True)

# simple model
class Model(nn.Module):
    def __init__(self, input_size, layers_size, output_size):
        super(Model, self).__init__()
        self.mlp = nn.Sequential(
                         nn.Linear(input_size, layers_size),
                         nn.GELU(),
                         nn.Linear(layers_size, output_size) )
    
    def forward(self, x):
        return self.mlp(x)

model = Model(2,3,1)
flat_params = torch.nn.utils.parameters_to_vector(model.parameters())

loss_func = nn.MSELoss()

# compute Hessian
hess = 0.

for i, batch in enumerate(dataloader):
    inputs, targets = batch
    batch_size = len(inputs)
    
    def f(theta_):
        torch.nn.utils.vector_to_parameters(theta_, model.parameters())
        preds = torch.func.functional_call(model, model.state_dict(), inputs)
        loss = loss_func(preds, targets)
        return loss
    
    hess_batch = torch.autograd.functional.hessian(f, flat_params).detach()
    hess = hess + hess_batch * batch_size

print("Hessian =", hess)
</code></pre>
<p>Unfortunately, the Hessian is all 0.</p>
<p>Either I made a silly mistake or <code>theta_</code> is not “seen” in the computational graph.<br>
Big thanks for any help.</p>
Performance Impact of TORCH_CPP_LOG_LEVEL=INFO and TORCH_DISTRIBUTED_DEBUG=DETAIL

hui_chieh_yu: <h3>
<a name="more-info-1" class="anchor" href="#more-info-1"></a>More info</h3>
<p>I have a few questions regarding the performance impact of enabling the environment variables <code>TORCH_CPP_LOG_LEVEL=INFO</code> and <code>TORCH_DISTRIBUTED_DEBUG=DETAIL</code>. I would greatly appreciate any insights that anyone can provide on the following points:</p>
<ol>
<li>Code Implementation: Does anyone have information on where these variables should be set in the code? Specifically, I would like to know if enabling these variables has any impact on single-GPU performance.</li>
<li>Scaling to Multiple GPUs: As we scale our system to accommodate a large number of GPUs, potentially hundreds, I’m curious if enabling these variables leads to increased overhead or performance degradation?</li>
<li>Statistics and Iteration Values: The statistics reported when these variables are enabled are typically averaged. Is there a way to access the values for each iteration? We are particularly interested in analyzing potential slow nodes, and having access to individual GPU values could help uncover any hidden issues that might be masked by averaging.</li>
</ol>
<p>Thank you in advance to anyone who can provide insights or information on these topics!</p>

ptrblck: <ol>
<li>These are environment variables, which should be exported in your terminal. Setting them in your script could also work, but I won’t recommend it as env variables are often set too late in the script directly. Yes, printing to the terminal could slow down your application.</li>
<li>Same as 1 since you would be printing even more.</li>
<li>I don’t fully understand the question and which stats you are interested in as these env variables are used for debugging.</li>
</ol>

hui_chieh_yu: <p>Thank you for your reply. I apologize for any confusion caused by my previous question.</p>
<p>Regarding my first question, I am interested in understanding the specific locations within the PyTorch source code where these variables are set. By gaining this insight, I hope to obtain a deeper understanding of their impact on performance.</p>
<p>For the third question, when utilizing these environment variables, we can extract valuable information, such as the average forward compute time. This data is useful in analyzing and resolving performance issues. Our objective is to gather more detailed information to effectively address the performance problem at hand.</p>
<p>If you have any insights or suggestions regarding these questions, I eagerly await your response. Thank you in advance.</p>

ptrblck: <p>You can search the source code and will find that e.g. <code>TORCH_DISTRIBUTED_DEBUG</code> is read <a href="https://github.com/pytorch/pytorch/blob/ac1cf00085b30eadd164ccf02e5208a59ec1b38b/torch/csrc/distributed/c10d/debug.cpp#L22">here</a>.</p>
How to train T5?

randusr: <p>I followed <a href="https://pytorch.org/text/main/tutorials/t5_demo.html" rel="noopener nofollow ugc">this tutorial</a> on using <code>T5_BASE_GENERATION</code> for summarization. How can I train this model locally using my own dataset? I am new to nlp and pytorch, so any suggestions would be greatly appreciated!</p>