akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Managing AWS/GCP Vm's for Machine learning

maith93: <p>Hey all,</p>
<p>I’m thinking about building software to help manage AWS/GCP VM instances for ML. I’m thinking current pain points are around not having good cost visibility of VM’s, ability auto-start/stop VM’s and faster UI experience over using the CLI. Would this be valuable to anyone, or do most tend to use google colab/ their own machine over VM’s?</p>
Custome Dataset, Data loader

Saida2020: <p>Hello</p>
<p>I am new to pytorch .<br>
I have big dataset  consist of two txt files one for data and other for target data .<br>
In training file each line is list of length 340, In target each line is list of 136.</p>
<p>I would like to ask how i can define my dataset so I can use Dataloader to load my data to train pytorch model?</p>
<p>I apricate you answers</p>

AlperEkmekci: <p>Hello there,</p>
<p>You can follow up this <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">tutorial</a> for writing custom <code>Dataset</code> and <code>Dataloader</code> respectively.</p>
<p>Greetings.</p>
Libtorch no boolean not operator

Nathan_Wood: <p>Would be useful to be able to do <code>!tensor</code> for a boolean tensor. Also, there’s no unary not for integers, but this is less important (for my use).</p>

glaringlee: <p><a class="mention" href="/u/nathan_wood">@Nathan_Wood</a><br>
We don’t have ‘!’ or ‘not’ operator support on tensors in c++ apis. (you can do this in python version when tensor has only 1 item)<br>
You can file a feature issue here ‘<a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues</a>’  and put a label operator on the issue. We will triage it.</p>

mih.net: <p>if b - boolean var, than not_b = 1 - b or not_b = 1 ^ b; But ‘-’ operator is prohibited for boolean tensors, so:</p>
<p>(if B - boolean tensor) at::Tensor not_B = torch::ones_like(B) ^ B;</p>
<p>Hope this helps.</p>
Torch.exp(tensor) not working for cuda Long tensor

JDE65: <p>I want to compute exp() of each item of a cuda tensor<br>
<code>out_adj = torch.exp(out_adj)</code> where out_adj is a 1D tensor with 60 values.<br>
I get the error message<br>
<code>RuntimeError: "exp_cuda" not implemented for 'Long'</code></p>
<p>I tried to change the type of the tensor to torch.cuda.IntTensor and to torch.cuda.ShortTensor, but nothing works.<br>
I’d be happy to get help on this <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

albanD: <p>Hi,</p>
<p>The <code>exp</code> function is only implemented for continuous dtypes I’m afraid.<br>
You can convert your Tensor to float for example before evaluating it for it to work: <code>out_adj = torch.exp(out_adj.float())</code></p>

JDE65: <p>Thanks a lot.<br>
I managed to implement in multiplying out_adj * 1.0, but your proposal is by far cleaner and smarter.</p>
Overfitting on NLP ask

sripathisony_sony: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I am working on Nlp TASK<br>
Could you please let me know if my model is overfitting:</p>
<p>train accuracy:96</p>
<p>valid accuray:83</p>
<p>test accuracy:87</p>

sripathisony_sony: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks in advance</p>

Nihir: <p>You should probably provide more details than just your metrics… what is the NLP task you’re working on, what does the data look like, what kind of model are you using?</p>
<p>Anyway, based of those metrics, do you think your model is over fitting?</p>

sripathisony_sony: <p>I am working on text classification on bert model, I am confused is my model is overfitting based on the above score, Any suggestion, please</p>

Nihir: <p>Yes it probably is. Why? Because your train accuracy is significantly above your validation or test accuracy. Try investigating something known as early stopping to stop your model from overfitting</p>

sripathisony_sony: <p>Thanks, After doing early stopping, now<br>
train accuracy = 96<br>
val = 83<br>
test = 92<br>
now is it overfitting?</p>

Nihir: <p>Overfitting isn’t a difficult concept at all - I suggest you take a read through some resources (here’s one: <a href="https://elitedatascience.com/overfitting-in-machine-learning" class="inline-onebox" rel="noopener nofollow ugc">Overfitting in Machine Learning: What It Is and How to Prevent It</a>) to understand when your model is overfitting and strategies to prevent it from doing so</p>
Questions on distributed training when using gloo backend

lzypku: <p>I want to launch distributed training on two machines. I have two GPU cards on machine A, and one GPU card on machine B. And I’d like to use all these three cards for distributed training.</p>
<p>But when I try to launch the distributed training by specifying machine A with 2 cards and machine B with 1 card, the code cannot run.</p>
<p>PS: I use the “gloo” backend. I wonder if the gloo backend do not support this specifying(different machines with different number of cards), but I am not sure.</p>
Creating a tensor that counts difference between a target and a column number

JDE65: <p>I have a first tensor named “target” that is a 1D tensor that contain n integers from 0 to 3<br>
target = (2,0,3,2,1,1,2,3,0,…,2)</p>
<p>Within the framework of a custom loss function, I would like to create a 2D tensor of sise (n, 4) for which the value of each item would be equal to the absolute difference between the target and the column<br>
adj_target = ((2,1,0,1),(0,1,2,3),(3,2,1,0),…)<br>
Example :<br>
First row  of adj_target is a function of the first row of target = 2</p>
<ul>
<li>1st column = abs(0-2) = 2</li>
<li>2nd column = abs(1-2) = 1</li>
<li>3rd column = abs(2-2) = 0<br>
-4th column = abs(3-2) = 1<br>
Second row of adj_target is a function of the first row of target = 0</li>
<li>1st column = abs(0-0) = 0</li>
<li>2nd column = abs(1-0) = 1</li>
<li>3rd column = abs(2-0) = 2<br>
-4th column = abs(3-0) = 3<br>
Third row of adj_target is a function of the first row of target = 3</li>
<li>1st column = abs(0-3) = 3</li>
<li>2nd column = abs(1-3) = 2</li>
<li>3rd column = abs(2-3) = 1<br>
-4th column = abs(3-3) = 0<br>
etc…</li>
</ul>
<p>Is there an easy and efficient way to do it ?<br>
Thanks in advance</p>

Naruto-Sasuke: <pre><code class="lang-python">target = torch.tensor((2,0,3,2,1,1,2,3))
t_size = target.numel()
expand_tar = target.view(1, -1).expand(4,t_size )
adj_target = torch.abs(expand_tar - torch.arange(4).view(-1, 1).expand(4, t_size )).t()
</code></pre>
Keep the max value of the array and 0 the others

Dimitrisl: <p>Hello and good evening,</p>
<p>I’ve searched a lot but with no significant result so far so I’m posting in this forum <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a Tensor in the form of :<br>
(0 ,.,.) =<br>
1.0858 -3.2292 -0.6477</p>
<p>(1 ,.,.) =<br>
3.3863 -2.3600 -2.6776</p>
<p>(2 ,.,.) =<br>
-2.3978 -2.0387  2.0728</p>
<p>and i want to transform it in a way that all the max values of each row will get the numer 1 and the others the number 0.<br>
Any help would be very apprieciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

richard: <p>Try the following:</p>
<pre><code class="lang-auto">x == x.max(dim=0)[0]
</code></pre>

Dimitrisl: <p>well not exactly… you see the correct output would be something</p>
<p>(0 ,.,.) =<br>
1 0 0</p>
<p>(1 ,.,.) =<br>
1 0 0</p>
<p>(2 ,.,.) =<br>
0 0 1</p>
<p>with this i only get the max results and i can’t keep the shape of the array i need. ( I don’t want to be forced to do this with a for loop)</p>

richard: <p>Could you provide a solid example of a tensor? ie how many dimensions does this tensor have?</p>

Dimitrisl: <p>It has 3 dimensions it’s a 30x1x3 tensor</p>

richard: <p>Okay, so you have something like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c1fdfbc70aaff4d3b188d72e61cc58dbacb95d48.png" alt="image" data-base62-sha1="rG8j9B3oDwXa0CFPlotQtQ2Y8d2" width="325" height="337"><br>
What should the output be?</p>

Dimitrisl: <p>(0 ,.,.) =</p>
<p>0 0 1<br>
1 0 0<br>
0 0 1</p>
<p>(1 ,.,.) =</p>
<p>0 0 1<br>
0 1 0<br>
0 1 0</p>
<p>(2 ,.,.) =<br>
0 1 0<br>
1 0 0<br>
0 0 1</p>

richard: <p>Thank you. Try the following:</p>
<pre><code class="lang-auto">view = x.view(-1, 3)
(view == view.max(dim=1, keepdim=True)[0]).view_as(x)
</code></pre>

Dimitrisl: <p>Just saved me!!! thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

saluei: <p>if temp is your tensor this line do what you want<br>
mask=(temp == temp.max(dim=1, keepdim=True)[0]).to(dtype=torch.int32)<br>
and if you want also value of max this is<br>
temp=torch.mul(mask,temp)</p>

Javier: <p>Hi,<br>
I guess I’m failing to see how to extend this to the n-dimensional case. <a class="mention" href="/u/richard">@richard</a></p>
<p>Assume I have a tensor where the first two dims are batch and channel, and the last three correspond to xyz space:</p>
<p>A = torch.randn(b,c,32,32,32)</p>
<p>What I would like to do is to binarize along the x dimension (dim=2) for any batch or channel, i.e for every yz location I want to set the maximum value along x-axis to 1 and the rest to zero. Is there a way of doing this?</p>
<p>Thank you!</p>
Can't build libtorch with error message "C2210 'T' : pack expansions cannot be used as arguments to non-packed parameters in alias templates"

111158: <p>Hi~<br>
I followed the installing steps (<a href="https://pytorch.org/cppdocs/installing.html" rel="nofollow noopener">https://pytorch.org/cppdocs/installing.html</a>).<br>
When I build the the project (example-app) in visual studio 2015, there was a error message  “C2210 ‘T’ : pack expansions cannot be used as arguments to non-packed parameters in alias templates”.<br>
Can someone help me?</p>

tom: <p>I would recommend to try a newer Visual Studio. PyTorch does use pretty much all of C++11, so one would expect to need at least VS2017 or so.</p>
<p>Best regards</p>
<p>Thomas</p>

liuzt1024: <p>Have you solved the problem yet</p>

tom: <p>I think the answer here has been that VS2015 doesn’t cut the mustard and you need a newer VS to use with PyTorch. I’m afraid support isn’t going to expand to older version, if anything, you’ll need VS2019+ eventually. We keep <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#known-msvc-and-msvc-with-nvcc-bugs" rel="noopener nofollow ugc">a washlist of valid C++ idioms to avoid</a> as is, we would not want to expand it more than necessary.</p>
<p>Best regards</p>
<p>Thomas</p>

111158: <p>I had solved this problem by installing VS 2019.<br>
Here is my configuration:<br>
VS 2019<br>
Pytorch: torch 1.4.0 +cuda9.2<br>
Libtorch: nightly version 2019/12/02</p>

Dennis_Hugle: <p>Hey, I’m sorry to bump this old thread. However this “bug” still exists in VS2017. Updating to 2019 is not suitable for all of us, especially for people stuck in strict production environments. Is there any Idea how to go fix this?</p>
Impact of learning rate in Mixed Precision training

tam: <p>Is there any minimum learning rate, for training with amp (mixed precision training). Say if lr scheduler drops the lr to a high precision value (say 10^-20) then will the weight get updated in the that epoch?<br>
I understand that loss and some other operators are calculated in fp32 during mixed-precision training, so they may not have much impact, but how the operators that use fp16, will the weight update stop for them?</p>

seungjun: <p><code>toch.cuda.amp.autocast()</code> will use fp16 representation for <a href="https://pytorch.org/docs/stable/amp.html#ops-that-can-autocast-to-float16" rel="noopener nofollow ugc">allowed list of operations</a> regardless of the magnitude.<br>
If your loss / gradient is too small, they will fall to 0.</p>
<p>Gradient scaling (<code>torch.cuda.amp.GradScaler</code>) is there to prevent gradients from being 0 if possible.<br>
It</p>
<ol>
<li>scales your loss with a large value (e.g.1024) and then</li>
<li>compute the gradients</li>
<li>unscale the gradient</li>
</ol>
<p>If all the values required in the above process are in fp16 range, amp may work.<br>
If any of them is larger than the fp16 maximum or smaller than the minimum, amp may not work.<br>
You will have to find your optimal learning rate by running experiments and checking the gradient values.</p>
<p>You can refer to <a href="https://en.wikipedia.org/wiki/Half-precision_floating-point_format" rel="noopener nofollow ugc">wikipedia page</a> to check the fp16 precision limitation.<br>
I guess lr=10^-20 won’t work in fp16 for most of the cases.</p>
LightGBM port to LSTM network

hH1sG0n3: <p><strong>Background</strong></p>
<p>In binary classification problem of dataset consisting of sequences of events, performance varies across types of models:</p>
<ul>
<li>for sequence of events, LSTM gives an average precision score of <code>APS = 0.60</code>
</li>
<li>for features engineered with summary stats (first-last occurrence, overall frequency etc), LightGBM gives an <code>APS = 0.80</code>
</li>
</ul>
<p><strong>Dataset specs</strong></p>
<ul>
<li>The dataset is highly imbalance (<code>ratio=0.1</code>).</li>
<li>Number of unique tokens is <code>&lt;100</code>
</li>
</ul>
<p><strong>Questions</strong></p>
<ol>
<li>Would there be any obvious reason for the above that I am missing?</li>
<li>Importantly, what would be a way to combine the LSTM and lightGBM ports?</li>
</ol>
Monai Resized failing on some samples

nmvr: <p>Hi,</p>
<p>I’m having an issue with <a class="mention" href="/u/monai">@MONAI</a> <code>Resized</code> transformation where is does not rescale some samples to the given shape.</p>
<p>All volumes and corresponding masks are stored in a list of dictionaries (just like in the provided examples) and their original shapes are either (284,284,32), (320,320,32) or (640,640,32).</p>
<p>I’m using the Resized transform to make them (128,128,32), however, despite the majority being resized to that shape, some of them get resized to wrong values such as (128,127,32). This happens to random samples each time the transformations are applied.</p>
<p>My current hypothesis is that since the list of dicts contains multiple dicts with the same key in value for different ‘image’ keys it might be affecting the compose, but I’m not sure if that is the reason, as nothing points to that in the docs.</p>
<p>Below I will add both a snippet from the list of dicts and the transformations:</p>
<p><strong>[EDIT]</strong></p>
<p>I can confirm this issue is happening due to repetition of the image key,value pairs in the list of dicts.<br>
Using only 1 mask per volume removes this repetition and works as intended.<br>
Is this behaviour intended?</p>
<pre><code class="lang-auto">[Input]
    for i in range(5):
    print(train_files[i])
[Output]
    {'image': './data/0062_T2WAx.nii.gz', 'label': './data/0062_t2wax_mask.nii.gz'}
    {'image': './data/0062_T2WAx.nii.gz', 'label': './data/0062_t2wax_mask2.nii.gz'}
    {'image': './data/0154_T2WAx.nii.gz', 'label': './data/0154_t2wax_mask.nii.gz'}
    {'image': './data/0154_T2WAx.nii.gz', 'label': './data/0154_t2wax_mask2.nii.gz'}
    {'image': './data/0033_T2WAx.nii.gz', 'label': './data/0033_t2wax_mask.nii.gz'}


[Input]

train_transforms = Compose(
    [
        LoadImaged(keys=["image", "label"]),
        AddChanneld(keys=["image", "label"]),
        Resized(keys=["image", "label"], spatial_size=(128, 128, 32), mode=('trilinear', 'nearest')),
        CropForegroundd(keys=["image", "label"], source_key="image"),
        ToTensord(keys=["image", "label"]),
    ]
)

train_ds = Dataset(data=train_files, transform=train_transforms)
train_loader = DataLoader(train_ds, batch_size=1, shuffle=True, num_workers=4)

for check_data in train_loader:
    image, label = (check_data["image"][0][0], check_data["label"][0][0])
    print(f"image shape: {image.shape}, label shape: {label.shape}")

[Output]
# ' added on output for visibility
(...)
image shape: torch.Size([128, 128, 32]), label shape: torch.Size([128, 128, 32])
image shape: torch.Size([128, 128, 32]), label shape: torch.Size([128, 128, 32])
image shape: torch.Size([128, '127', 32]), label shape: torch.Size([128, '127', 32])
image shape: torch.Size([128, 128, 32]), label shape: torch.Size([128, 128, 32])
(...)</code></pre>
How to find the module dependency? 

Xuyuanjia2014: <p>If we run these codes below:</p>
<pre><code class="lang-auto">import torch
import torchvision
model = torchvision.models.resnet18()
inp   = torch.zeros([64, 3, 7, 7])
for temp in model.children():
    print(temp)
</code></pre>
<p>We can get several modules:</p>
<pre><code class="lang-auto">Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
ReLU(inplace=True)
MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
Sequential(
  (0): BasicBlock(
    (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (1): BasicBlock(
    (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
)
Sequential(
  (0): BasicBlock(
    (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (downsample): Sequential(
      (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
      (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (1): BasicBlock(
    (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
)
........
</code></pre>
<h2>My problems are:</h2>
<ol>
<li>We can only see constructed modules, but cannot see their input\output dependencies: In restnet18, the input of second Sequential module are both from the first Sequential and MaxPool2d. Is there any way we can figure out the depencies among different modules （maybe in Python client）?</li>
<li>Moudules are related to high-level operations, can we see related operations and their dependencies in Python client (the outputs of torch.jit._get_trace_graph are too low-level)?</li>
<li>How can wen find back propagation dependencies in Python client?</li>
</ol>
Defining operator schema to take variable number of inputs and return variable number of outputs

vdantu: <p>Is there a mechanism to define a custom JIT operator which returns variable number of outputs? (Tuple<br>
of any types)</p>
<p>I tried defining <code>ns::customOp(...) -&gt; (...)</code>, but this doesn’t seem to work.</p>
<p>When I save a model with a custom operator which returns 2 outputs <code>Tuple[Tensor, Tensor]</code> and load the saved model, I see error saying the number of outputs mismatch .</p>
<pre><code class="lang-auto">Return value was annotated as having type Tuple[Tensor, Tensor] but is actually of type Tuple[]:
</code></pre>
<p>My current operator registration is based off of some of the <code>prim</code> op registrations</p>
<pre><code class="lang-auto">RegisterOperators r({Operator(
         "ns::customOp(…) -&gt; (…)",
         [](const Node* node) -&gt; Operation {
           return [node](Stack&amp; stack) {
             myLogic(stack, node);
             return 0;
           };
         }, AliasAnalysisKind::FROM_SCHEMA)});
</code></pre>
<p>How do we define an operator which is dynamic and can take variable number of inputs and return variable number of outputs?</p>
<p>[Similar to <a href="https://discuss.pytorch.org/t/how-to-define-and-save-a-custom-operator-with-a-dynamic-schema/92188/5">This discussion</a>]</p>
ConvTranspose2D much slower than Conv2d

Alexander_Koch: <p>I am trying to figure out why my convolutional decoder is much slower than the encoder.<br>
Here are the architectures:</p>
<p>ConvEncoder(<br>
(model): Sequential(<br>
(0): Conv2d(3, 32, kernel_size=(4, 4), stride=(2, 2))<br>
(1): ReLU()<br>
(2): Conv2d(32, 64, kernel_size=(4, 4), stride=(2, 2))<br>
(3): ReLU()<br>
(4): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2))<br>
(5): ReLU()<br>
(6): Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2))<br>
(7): ReLU()<br>
)<br>
)<br>
ConvDecoder(<br>
(model): Sequential(<br>
(0): Linear(in_features=230, out_features=1024, bias=True)<br>
(1): Reshape()<br>
(2): ConvTranspose2d(1024, 128, kernel_size=(5, 5), stride=(2, 2))<br>
(3): ReLU()<br>
(4): ConvTranspose2d(128, 64, kernel_size=(5, 5), stride=(2, 2))<br>
(5): ReLU()<br>
(6): ConvTranspose2d(64, 32, kernel_size=(6, 6), stride=(2, 2))<br>
(7): ReLU()<br>
(8): ConvTranspose2d(32, 3, kernel_size=(6, 6), stride=(2, 2))<br>
)<br>
)</p>
<p>The batch shape is (50, 50, 3, 64, 64).<br>
The encoder takes about 16ms and the decoder 130ms on a GTX 1080TI.</p>
<p>I am using cuda.synchronize() for the timings.<br>
What is the reason for these very different times?</p>

JuanFMontesinos: <p>There are many reasons.</p>
<ol>
<li>The amount of filters of the decoder is bigger.</li>
<li>The kernel size of the decoder is bigger.</li>
<li>Probable the transposed convolution itself is slower. I guess this as you need to allocate more memory (you are upsampling) and to work with sparse kernels.</li>
</ol>
Tf.math.segment_min in PyTorch

GlebBrykin: <p>Is it possible to implement segment_min function from TensorFlow in PyTorch? Here is details of this method: <a href="https://www.tensorflow.org/api_docs/python/tf/math/segment_min?hl=ru" class="inline-onebox" rel="noopener nofollow ugc">tf.math.segment_min &nbsp;|&nbsp; TensorFlow Core v2.4.1</a></p>

ptrblck: <p>Based on the docs I think the operation would correspond to <code>scatter_min_</code> from <a href="https://github.com/rusty1s/pytorch_scatter">PyTorch Scatter</a>.<br>
There is also an open feature request <a href="https://github.com/pytorch/pytorch/issues/22378">here</a>.</p>
Convert Linear Layers (Fully Connected) with Conv2d Layers

marcogelli: <p>Hi everyone! In my CNN structure defined in the the <code>__init__</code> method I have different Linear layers which are fed with feature maps that are obtained from Conv2d layers. By leaving the network structure as-is, my network is dependent on the input image size. Is there a way to comvert the linear layers to Conv2d and obtain the same results?</p>
<p>This is the code:</p>
<pre><code class="lang-auto">...
self.smooth1 = nn.Conv2d(512, 256, kernel_size=3, stride=1, padding=1)
self.smooth2 = nn.Conv2d(768, 256, kernel_size=3, stride=1, padding=1)
self.smooth3 = nn.Conv2d(1024, 256, kernel_size=3, stride=1, padding=1)

# Fully Connected Layers
self.fc5 = nn.Linear(256*25*25, num_classes)
self.fc4 = nn.Linear(256*50*50, num_classes)
self.fc3 = nn.Linear(256*100*100, num_classes)
self.fc2 = nn.Linear(256*200*200, num_classes)
</code></pre>
<p>In the Linear layers <code>25</code>, <code>50</code>, <code>100</code> and <code>200</code> depend on the input image size. Is there a way to make the “independent”?</p>

ptrblck: <p>You could transform the linear layer to a conv layer with a spatial size of <code>1x1</code>, but the <code>in_features</code> of the linear layer would be translated to the <code>in_channels</code> of the conv layer, so you wouldn’t win anything.<br>
The usual approach to relax the size dependency is to add adaptive pooling layers after the feature extractor (conv-pool layers) in order to create a pre-defined activation output shape, which fits the <code>in_features</code> of the following linear layer.</p>
Parallel Layers in a single GPU

amoha107: <p>I have input tensor (x) of size (batch_size , 4 , 10). I would like to create 4 “small” separate fully connected (Linear) layers in parallel (no_inputs = 10 , no_outputs = 5).<br>
1st linear layer will be fed by x[: , 0:1 , :]<br>
2nd linear layer will be fed by x[: , 1:2 , :]<br>
3rd linear layer will be fed by x[: , 2:3 , :]<br>
4th linear layer will be fed by x[: , 3:4 , :]</p>
<p>I am using a single GPU for training. Is there a way to implement this in “Parallel”? So I won’t need to use a for loop as such:</p>
<p>output = torch.zeros(batch_size , 4 , 5)<br>
self.layers = [nn.Linear(10 , 5) , nn.Linear(10 , 5) , nn.Linear(10 , 5) , nn.Linear(10 , 5)]<br>
for i in range(x.shape[1]):<br>
output[: , i : i + 1 , :] = self.layers[i](x[: , i : i+1 , :])</p>
GTX 770 not working with PyTorch

Samuel_Bachorik: <p>Hello iam trying to train my model with my NVIDIA GTX 770. I am getting this error and i cant find way to make this work. It says my GPU is old. It is possible to make this GPU work with PyTorch ?</p>
<p>thank you</p>
<pre><code class="lang-auto">Found GPU0 GeForce GTX 770 which is of cuda capability 3.0.
    PyTorch no longer supports this GPU because it is too old.
    The minimum cuda capability that we support is 3.5.

GeForce GTX 770 with CUDA capability sm_30 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
</code></pre>

ptrblck: <p>You won’t be able to use PyTorch in its current version with this GPU architecture, as the min. compute capability is 3.5.<br>
I would recommend to use e.g. Colab with the free GPU runtimes.</p>
Effects of switching between amp and non-amp on training and inference

iordanis: <p>Is there any documentation or special considerations when switching between amp to non-amp and vice versa for both training and inference.</p>
<p>The documentation states the following, but it is still not clear the effects it would have on the model.</p>
<blockquote>
<p>" If a checkpoint was created from a run <em>without</em> Amp, and you want to resume training <em>with</em> Amp, load model and optimizer states from the checkpoint as usual. The checkpoint won’t contain a saved scaler state, so use a fresh instance of <code>GradScaler</code>.</p>
<p>If a checkpoint was created from a run <em>with</em> Amp and you want to resume training <em>without</em> Amp, load model and optimizer states from the checkpoint as usual, and ignore the saved scaler state."</p>
</blockquote>
How to calculate loss and gradient of index

DableUTeeF: <p>I have a misaligned point cloud and rotation matrix and I want to use backprop to tune the rotation matrix in 2d plane.</p>
<p>The problem is in the <code>forward</code> all I can get from the rotation matrix is index in the 2d plane which is duplicated like.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x
[500.99, 500.5, 411.2, 411.02, 411]
</code></pre>
<p>But the desired output for this case should be</p>
<pre><code class="lang-auto">[500, 411]
</code></pre>
<p>While the ground truth maybe</p>
<pre><code class="lang-auto">[505, 501, 460, 420]
</code></pre>
<p>I can’t compare the float <code>x</code> and ground truth using <code>mse</code> directly because the dimension is not equal.</p>
<p>Both the <code>x</code> and ground truth are index of a 512 dimension vector but the gradient disappear as soon as I use <code>x.long()</code> so I can’t use them as index either.</p>
<p>How to properly calculate loss of this problem?</p>
<p>Thanks</p>
How can I avoid this loss plot?

Pepi: <p>Hi all,</p>
<p>In some of my recent projects I have found that when training my model, the loss decreases very slowly, no matter the size of the dataset.</p>
<p>And by slow I mean falling by 1e-3 every 10 epochs slow.</p>
<p>I try many things like playing with the number of layers and the structure of the architecture and sometimes one of these experiments helps but it hasn’t resulted in how I believe the loss should be falling. When I let it train for long periods I get a plot like the one attached here, (x-epoch, y-mean loss for that epoch):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02ec1ea8e9eacb121006e3b81bf0a0b8fcb29b1c.png" alt="image" data-base62-sha1="pQQ1TQ6R27zRuGRRAEAd3SknFO" width="263" height="190"></p>
<p>I have tried playing with the features, including different scalers and feature engineering. In the image above, I am using a MinMaxScaler as when I tested the features in a linear regression, all the p-values were &lt; 0.05 with R_sq of 98%, so I don’t think it is the features here.</p>
<p>In the past I have been told to see if I can overfit a single batch. And when I train on a single batch it does train properly, but when I use the entire training dataset, this occurs.</p>
<p>I would really appreciate some help with this. I have added some relevant code below:</p>
<p><em>Model Architecture</em><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef768986951fa38f35bcfc5d43e8e58a7b6ef26f.png" data-download-href="https://discuss.pytorch.org/uploads/default/ef768986951fa38f35bcfc5d43e8e58a7b6ef26f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef768986951fa38f35bcfc5d43e8e58a7b6ef26f.png" alt="image" data-base62-sha1="yao4YqYR3oywYgvvNkHgkUbw1rN" width="366" height="375" data-dominant-color="F0F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">783×802 20.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am using Adam optimizer and MSE Loss function as I am trying to predict a single continuous value.</p>
<p>Thank you in advance!</p>
I want to give multiple input to the model where I want to give the input as Images and numerical data

pratik_mahankal: <p>Hello,<br>
I want to give the multiple input to the binary classifier where the one of the input are the images and the other one is the numerical data.<br>
Can anyone suggest me any article or give the code snippet for the pytorch by which I can give the multiple input to the models.</p>
<p>Regards</p>

VirginieBfd: <p>Hi, here an example: <a href="https://discuss.pytorch.org/t/multiple-input-model-architecture/19754/2" class="inline-onebox">Multiple input model architecture - #2 by ptrblck</a></p>
How to train a bias threshold for ReLU?

Kunpeng_GUO: <p>For the Relu layer, I would like to put a bias in it ( i.e. ReLU(x + b) ) and train it as a threshold. But it doesn’t seem to work when I train it. The bias doesn’t change when I update the params. Thanks in advance for your help.</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    
    def __init__(self):
        super(Net, self).__init__()
        # Bert
        self.bert = BertModel.from_pretrained('bert-base-uncased')
        
        # ReLU
        self.bias = nn.Parameter(torch.zeros(1), requires_grad=True)
        self.relu = nn.ReLU(inplace=True)
    
    def forward(self, ids_query, mask_query, ids_context, segment_ids_context, mask_context):
        # get embeddings
        query_emb = self.bert(ids_query, attention_mask=mask_query).last_hidden_state
        context_emb = self.bert(ids_context, token_type_ids=segment_ids_context, attention_mask=mask_context).last_hidden_state
        print(f"Shape_query: {query_emb.shape}, Shape_context: {context_emb.shape}")
        
        # multiply matrix
        out = torch.bmm(query_emb, torch.transpose(context_emb, 1, 2))
        print("Multiplication: ", out.shape)
        
        # max
        out, _ = torch.max(out, dim=2)
        print("Max-Pool: ", out.shape)
        
        # add bias =&gt; out_1 + b
        out = out + self.bias
        
        # relu =&gt; ReLU(out_1 + b)
        out = self.relu(out)
        print("ReLU: ", out.shape)
        
        # log
        out = torch.log(out+1)
        print("Log: ", out.shape)
        
        # summation
        out = torch.sum(out, dim=1)
        print("Summation: ", out.shape)
        print("========" * 12)
        
        return out
</code></pre>

ptrblck: <p>Your approach should work and <code>self.bias</code> is correctly defined as a parameter.<br>
Here is a minimal code snippet, which shows that <code>self.bias</code> gets a valid gradient:</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # ReLU
        self.bias = nn.Parameter(torch.zeros(1), requires_grad=True)
        self.relu = nn.ReLU(inplace=True)
    
    def forward(self, x):
        x = x + self.bias
        x = self.relu(x)        
        return x

model = Net()
x = torch.randn(1, 1)
out = model(x)
out.mean().backward()
print(model.bias.grad)
&gt; tensor([1.])
</code></pre>
<p>Depending on your use case and model, the gradient might be small and thus the bias wouldn’t be changed by a lot so you could check its <code>.grad</code> value.</p>

Kunpeng_GUO: <p>Thanks for your help.<br>
Yes, it actually changed, but not apparently to see. Checked with .grad works well.</p>
Measure time for backward pass in each module

Frank_Leeeee: <p>Hi,</p>
<p>I am measuring time in both forward pass and backward pass in each module. For forward pass, it is easy as we can measure the time taken in the forward method.</p>
<p>I am wondering what I should do if I want to do the same for the backward pass?</p>

googlebot: <p>Invoke backward() in autograd.profiler.profile context, instead of doing ad-hoc measurements. Then if you watch timeline from prof.export_chrome_trace(), you can maybe match operations with modules. I first thought that profiler.record_function() would allow tagging modules, but it seems to not work during backward.</p>
Out of place version of tensor.fill_

Kroshtan: <p>Relatively simple question, I believe: Due to the structure of my model, I cannot use inplace operations, but I would like to use the fill_ function from torch.Tensor. Is there an out of place variant or similar functionality available? For now I replaced some calls to fill_ with x.fill_(0) with x = torch.zeros(x.size), but some fill_ operations are nested, and so replacing them with an assignment is more cumbersome. Any more convenient alternatives?</p>

ptrblck: <p>You could use <code>torch.full_like</code> instead of the inplace <code>fill_</code> operation.</p>
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward

Hacking_Pirate: <p>Hi, I am trying to solve a computer vision problem but I am encountering this error:<br>
<code>RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward</code></p>
<p>I am using crossentropyloss. Here is my dataset class:</p>
<pre><code class="lang-auto">class CassavaDataset(Dataset):

    def __init__(self, df, root, transform=None):

        self.df= df

        self.root = root

        self.transform = transform

    def __len__(self):

        return self.df.shape[0]

    def __getitem__(self, idx):

        img_name = self.df.image_id[idx]

        target = self.df.label[idx]

        img_path = self.root + str(img_name)

        image = cv2.imread(img_path)

        image = image[:, :, ::-1]

        if self.transform:

            image = self.transform(image=image)["image"]

        

        return {

            "image": torch.tensor(image, dtype=torch.float32),
            "target": torch.tensor(target, dtype=torch.long)

        }
</code></pre>
<p>If someone can help me it would be great!<br>
Thanks.</p>

Hacking_Pirate: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Can you please help this is urgent.<br>
Thanks</p>

ptrblck: <p>As the error message explains, the target tensor for <code>nn.CrossEntropyLoss</code> should be a <code>LongTensor</code>, while apparently a <code>FloatTensor</code> is passed.<br>
The custom <code>Dataset</code> looks correct, so I assume the target is either generated in another code snippet or transformed after being returned by the <code>Dataset.__getitem__</code>.</p>
When to call backwards

mohammed_guermal: <p>Hi ,</p>
<p>‘’’ while steps &lt; max_steps:<br>
print (‘Step {}/{}’.format(steps, max_steps))<br>
print(’-’*10)</p>
<pre><code class="lang-auto">    for phase in ['train', 'val']:
        if phase == 'train':
            model.train(True)
        else:
            model.train(False)  # Set model to evaluate mode
        tot_loss = 0.0
        tot_loc_loss = 0.0
        tot_cls_loss = 0.0
        tot_acc = 0.0
        num_iter = 0

        for data in dataloaders[phase]:
            num_iter += 1

            # get the inputs
            inputs, labels = data
            #print(np.shape(inputs),'inputs')
            # wrap them in Variable
            inputs = Variable(inputs.cuda())
            # t = inputs.size(2)
            labels = Variable(labels.cuda())

            per_frame_logits = model(inputs)

            criterion = nn.CrossEntropyLoss().cuda()
            # cls_loss = F.binary_cross_entropy_with_logits(torch.max(per_frame_logits, dim=2)[0], labels)
            #print(np.shape(torch.max(labels,dim=1)[1].long()))
            cls_loss = criterion(per_frame_logits,torch.max(labels,dim=1)[1].long())
            tot_cls_loss += cls_loss.data
            optimizer.zero_grad()
            # loss = (0.5*loc_loss + 0.5*cls_loss)/num_steps_per_update
            loss = cls_loss
            tot_loss += loss.data
            loss.backward()
            optimizer.step()
            #print(np.shape(labels))
            #print(np.shape(per_frame_logits))
            acc = calculate_accuracy(per_frame_logits, torch.max(labels,dim=1)[1])# topk=(1,))
            tot_acc += acc
</code></pre>
<p>‘’’<br>
I have a question about the code before-mentioned, in phase of validation is it doing backward propagation because I m calling loss.backwards or is it okay because i do model.train(False).</p>
<p>I would be glad if someone can clear this for me.<br>
Thanksss</p>
How to ATSS setup in pytorch v1.7.1

sol: <pre><code class="lang-auto">Traceback (most recent call last):
  File "setup.py", line 76, in &lt;module&gt;
    include_package_data=True,
  File "/opt/conda/lib/python3.6/site-packages/setuptools/__init__.py", line 163, in setup
    return distutils.core.setup(**attrs)
  File "/opt/conda/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/conda/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/opt/conda/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.6/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/opt/conda/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/conda/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 87, in run
    _build_ext.run(self)
  File "/opt/conda/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)enter code here
  File "/opt/conda/lib/python3.6/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 649, in build_extensions
    build_ext.build_extensions(self)
  File "/opt/conda/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 194, in build_extensions
    self.build_extension(ext)
  File "/opt/conda/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
    _build_ext.build_extension(self, ext)
  File "/opt/conda/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension
    depends=ext.depends)
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 478, in unix_wrap_ninja_compile
    with_cuda=with_cuda)
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1233, in _write_ninja_file_and_compile_objects
    error_prefix='Error compiling objects for extension')
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1529, in _run_ninja_build
    raise RuntimeError(message)
RuntimeError: Error compiling objects for extension
</code></pre>
<p>source : <a href="https://github.com/sfzhang15/ATSS" class="inline-onebox" rel="noopener nofollow ugc">GitHub - sfzhang15/ATSS: Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection, CVPR, Oral, 2020</a><br>
I want to use it in pytorch 1.7 version.<br>
But I got the following error.<br>
My environment is nvidia docker 20.09.<br>
ubuntu :18.04<br>
python : 3.6<br>
pytorch : 1.7<br>
torchvision : 0.4<br>
cuda : 11.0</p>
<p>Please help me</p>
How to convert a ModuleList of Sequential to an entire Sequential

B.W_Zhang: <p>What I want to do is like this, for example:<br>
I have each layer = nn.Sequential(GRU(), LayerNorm()), and totally 4 layers.<br>
And finally a classifier which can also be a Sequential(Linear(), Softmax())</p>
<p>I know that I can put these totally 5 Sequentials into an nn.ModuleList(), and use a for loop in the forward() to make the input pass through all the layers. But I want to ask if there is some function that can directly convert a ModuleList that contains multiple Sequentials to an entire Sequential.</p>
<p>More generally it may look like this:<br>
func(Seq(A,B), Seq(C,D), Seq(A,E)) → Seq(A, B, C, D, A, E)</p>

Mendel123: <p>You can just use</p>
<pre><code class="lang-python">res = nn.Sequential(
seq_ab,
seq_cd,
seq_ae
)
</code></pre>
<p>. It works as what you want.</p>

B.W_Zhang: <p>Thank you for the reply. But when I use the way you suggest, it gives me<br>
Seqential(<br>
Seqential(A,B),<br>
Seqential(C,D),<br>
Seqential(A,E)<br>
)<br>
instead of Seqential(A,B,C,D,A,E) that I favor.</p>

Nihir: <p>You can use <code>*</code> to unpack a Sequential object. Regarding what you asked for, something along these lines should work:</p>
<pre><code class="lang-auto">all_layers = [*Seq(A,B), *Seq(C,D), *Seq(A,E)]
new_sequential = nn.Sequential(*all_layers)
</code></pre>

B.W_Zhang: <p>Thanks, worked for me.</p>
Inplace parameter updation without torch.no_grad()

mrityu: <p>Hello All,</p>
<p>I have just started learning this awesome tool called PyTorch but sadly I am stuck in an equivocal situation.</p>
<p>Below is a code snippet from one of the tutorials</p>
<pre><code class="lang-auto"> with torch.no_grad():
       weights -= weights.grad * lr
       bias -= bias.grad * lr
       weights.grad.zero_()
       bias.grad.zero_()
</code></pre>
<p>I am kind of confused that even if I will do parameter update without using torch.no_grad() ( <em>i.e.</em> only in-place ) and since the backward call has already been made in the code above this snippet( not included in the snippet above ) which basically means all the “grad” attributes are already computed and don’t require the “original” values again. Then, why is it illegal to do those operations without torch.no_grad()?</p>
<p>I know it will flag off the error in PyTorch but I just wanted to know where my line of thought is at fault?</p>

albanD: <p>Hi,</p>
<p>It is illegal because it would mean that the <code>weight</code> update would be done in a differentiable manner.<br>
So that means that in the next iteration, when computing the gradients, it will flow back through multiple iterations of your training loop which is most likely not what you want to do !</p>

mrityu: <p>Hey,</p>
<p>Will you please clear where am I wrong in the below thought-process?</p>
<p>Assume, I only have one <code>trainable weight</code> parameter to update. So, on calling <code>loss.backward()</code> the gradient of loss wrt that weight has been saved in the node corresponding to the tensor comprising of that weight.<br>
The way I perceive in-place operation is:- "The new value of <code>weight</code> oust the previous value in the ‘same’ node i.e. according to me( unsure if it’s correct ), there is no change in the computational graph other than the “value” update or no new node/edge has been added.</p>
<p>Going by my perception, since except values nothing else has changed, it shouldn’t be illegal.</p>
<p>Please correct the parts where I am wrong.</p>

albanD: <blockquote>
<p>there is no change in the computational graph other than the “value” update</p>
</blockquote>
<p>No there is actually a change. Because the autograd needs to take into account this change to compute the proper gradients.</p>
<p>Also this issue only happens when multiple iterations happen. For just one, it doesn’t change much indeed as you don’t reuse the graph.</p>

mrityu: <p>Thank you so much for bolstering my perception of  <code>PyTorch</code>.</p>
Training with DDP and SyncBatchNorm hangs at the same training step on the first epoch

ChickenTarm: <p>I am using pytorch-lightning as my training framework. And I am have tried training on 1, 2, 4 GPUs (all T4). My model, video action classification network, hangs at the same spot each time. It only hangs when I set the trainer flags</p>
<pre><code class="lang-auto">Trainer(
    gpus=(something greater than 1)
    sync_batchnorm=True,
    accelerator="ddp"
)
</code></pre>
<p>I noticed that when it hangs GPU utilization stays pinned at 100% with no power fluctuations.<br>
I am able to train my model with sync_batchnorm=False.<br>
Does anyone have experience or tips on what a solution might be or how to properly debug this?</p>
<p>I have also tested this on 2 V100s and it hangs not at the exact same spot but same issue.</p>
<p>Version/OS:<br>
Ubuntu 18.04LTS<br>
CUDA 11.1<br>
Driver Version 455.45.01<br>
Pytorch 1.7.1<br>
Pytorch-Lightning 1.1.5</p>

ChickenTarm: <p>[Solved] My problem was that I have random alternating training that go down different branches of my model. I needed to set the random seed that samples the probability of which alternating loss it will perform. This is probably because when pytorch does it reduce_all somewhere, it notices a difference in batch norm statistics since I believe it assumes some ordering on the statistics.</p>

rvarm1: <aside class="quote no-group" data-username="ChickenTarm" data-post="2" data-topic="109827">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/c4cdca/48.png" class="avatar"> ChickenTarm:</div>
<blockquote>
<p>I needed to set the random seed that samples the probability of which alternating loss it will perform</p>
</blockquote>
</aside>
<p>This sounds correct, there would probably be a hang if a different alternating loss is used on different ranks.</p>

ChickenTarm: <p>I think this should probably be noted in the docs since alternating training isn’t uncommon.</p>