akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Merge tensors with different dimmension

username: tomasz_z
cooked: <p>I have two tensors with different dimensions [423, 2] and [423, 10]. How can I merge these tensors (columns) to create one tensor with dimensions [423,10].</p>
<p>Many thanks in advance for your help.</p>
<p>print(categorical_data.shape)<br>
print(categorical_data[:5])</p>
<p>print(numerical_data.shape)<br>
print(numerical_data[:5])</p>
<p><span class="hashtag">#output:</span><br>
torch.Size([423, 2])<br>
tensor([[0, 4],<br>
[2, 0],<br>
[2, 1],<br>
[2, 2],<br>
[0, 4]])<br>
torch.Size([423, 8])<br>
tensor([[39.1440, 37.0711, 37.0711,  0.9470,  1.0000,  1.0000, 23.0000, 25.0000],<br>
[39.1440,  2.0729,  1.3524,  0.0530,  0.0000,  0.6524, 23.0000, 25.0000],<br>
[39.1440,  2.0729,  0.3276,  0.0530,  0.0000,  0.1581, 23.0000, 25.0000],<br>
[39.1440,  2.0729,  0.3929,  0.0530,  0.0000,  0.1895, 23.0000, 25.0000],<br>
[38.6237, 37.0152, 37.0152,  0.9584,  1.0000,  1.0000, 23.0000, 25.0000]],<br>
dtype=torch.float64)</p>

username: Greg-Tarr
cooked: <p>I believe you have a typo in your question, the two tensors are [423, 2] and [423, 8].</p>
<p>Here’s a link that may help:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/50424167/concatenating-two-tensors-with-different-dimensions-in-pytorch" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/3160065/adeelz92" target="_blank" rel="nofollow noopener">
    <img alt="adeelz92" src="https://www.gravatar.com/avatar/36af77dc3725c7d6beb4864e9460ccbb?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/50424167/concatenating-two-tensors-with-different-dimensions-in-pytorch" target="_blank" rel="nofollow noopener">Concatenating two tensors with different dimensions in Pytorch</a>
</h4>

<div class="tags">
  <strong>python, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/3160065/adeelz92" target="_blank" rel="nofollow noopener">
    adeelz92
  </a>
  on <a href="https://stackoverflow.com/questions/50424167/concatenating-two-tensors-with-different-dimensions-in-pytorch" target="_blank" rel="nofollow noopener">10:16AM - 19 May 18 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>You might have to change the ‘dim’ of the example.</p>
Why isn't pad_packed_sequence sufficient ? Why do we need to pad by ourselves when we supply the sequence lengths?

username: dans
cooked: <p>Why do we need to pad the input for variable length sequences for lstm when there is a pack_padded_sequence function that essentially tells the lstm to ignore the padded portion? Why isn’t the pack_padded_sequence function with the sequence lengths sufficient for training via mini batches? Why do we need to pre-pad the input ?</p>

username: Brando_Miranda
cooked: <p>Crossposted: <a href="https://www.quora.com/unanswered/Why-do-we-need-to-pad-sequences-by-ourselves-in-Pytorch-when-we-supply-the-sequence-lengths-Why-isn-t-pad_packed_sequence-sufficient" rel="nofollow noopener">https://www.quora.com/unanswered/Why-do-we-need-to-pad-sequences-by-ourselves-in-Pytorch-when-we-supply-the-sequence-lengths-Why-isn-t-pad_packed_sequence-sufficient</a></p>
<p>hopefully we’ll get an answer some day.</p>

username: cerisara
cooked: <p>It’s because every row in a tensor has to have the same dimension. There’s no other way, as pytorch does not support python lists for this method.</p>
Custom class names with ImageFolder or DatasetFolder not updating labels on custom dataset

username: redtailedhawk
cooked: <p>I created a custom Dataset, and in my <strong>init</strong> changed the classes to what I wanted them to be by calling a custom _find_classes method. (I wanted to use subfolders, and concatenate their names with the parents)This took my class count from something like 30 up to 964.</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, image_path, transform=None):
        super(MyDataset, self).__init__()
        self.data = datasets.ImageFolder(image_path, transform)
        self.data.classes, self.data.class_to_idx = self._find_classes(image_path)

    def _find_classes(self, dir):
        # Custom labels
</code></pre>
<p>When I call class_to_idx, I have 964 different classes as expected, but when I create a DataLoader, it is giving me the <em>old</em> count of labels. If I print labels, nothing is higher than 29, or the old mapping to the parent folders.</p>
<pre><code class="lang-auto">data = DataLoader(data, batch_size=32, shuffle=True, num_workers=0)
data_iter = iter(train_dataloader)
images, labels = data_iter.next()

print(labels)

out:
tensor([ 0,  2, 19, 19, 29, 20,  3, 20, 22, 27,  1, 18,  4, 20, 17,  1,  3, 23,
        20, 18,  6, 29,  6, 18,  9, 24,  8, 29, 13, 19, 21, 14])
</code></pre>
<p>And if I print data.targets, it shows me the old mapping, before I tried to customize my labels.</p>
<p>It seems like I changed the classes on my data, but not my targets. What is a better way for me to do this more efficiently? It’s clear that my _find_classes has nothing to do with the _find_classes being called by ImageFolder, and I’m changing classes after the dataset has been created. What can I do?</p>

username: ptrblck
cooked: <p>Answered <a href="https://discuss.pytorch.org/t/custom-label-for-torchvision-imagefolder-class/52300/5">here</a>.</p>
Torchvision._is_tracing()

username: AyanKumarBhunia
cooked: <p>What is the significance of  torchvision._is_tracing()?</p>

username: ptrblck
cooked: <p>This is an internal call to <code>torch._C._get_tracing_state()</code> to check, if you are currently trying to trace the model with the JIT.<br>
It’s used internally to e.g. use workarounds for methods, which are currently not supported for tracing.</p>
`weight_norm` does not perform normalization?

username: tengerye
cooked: <p>I would like to row normalize a fully connected weight using <a href="https://pytorch.org/docs/stable/nn.html?highlight=weight_norm#torch.nn.utils.weight_norm" rel="nofollow noopener"><code>weight_norm</code></a>. But the problem is it seems NOT to normalize at all.</p>
<pre><code class="lang-auto">l = nn.Linear(2, 4, bias=False)
# tensor([[ 0.1212, -0.1683],
#        [ 0.2399,  0.1841],
#        [ 0.2489,  0.6081],
#        [ 0.3547,  0.1641]], grad_fn=&lt;MulBackward0&gt;)

m = nn.utils.weight_norm(l)
x = torch.tensor([1., 1.])

l(x)
# tensor([-0.0471,  0.4241,  0.8571,  0.5188], grad_fn=&lt;SqueezeBackward3&gt;)

m(x)
# tensor([-0.0471,  0.4241,  0.8571,  0.5188], grad_fn=&lt;SqueezeBackward3&gt;)
</code></pre>
<p><code>m</code> here works exactly the same as <code>l</code>, including the <code>m.weight</code>. I expected the <code>m</code> should be normalized. What should I do then?</p>

username: ptrblck
cooked: <p>I might be mistaken, but I thought weight normalization would be a reparametrization of the <code>weight</code> parameter, which should yield faster convergence. The output should be the same as without weight norm.<br>
I just skimmed though the <a href="https://arxiv.org/abs/1602.07868">original paper</a> again, but please correct me, if I’m wrong.</p>
Problem about batch index

username: q13030880
cooked: <p>I have a tensor with size of (64, 20,128) (actually it’s decoder outputs).<br>
Now , I have index tensor and its size is (64) (it’s a sequence lengths).<br>
I want to select the last real decoder outputs, and i think the shape should be (64, 128).<br>
I have tried torch.gather and torch.index_select, but it didn’t work…<br>
Can anybody help me?</p>

username: ptrblck
cooked: <p>Would this code snippet work for you?</p>
<pre><code class="lang-python">x = torch.randn(64, 20, 128)
idx = torch.randint(1, 20, (64,))
result = x[torch.arange(x.size(0)), idx]
</code></pre>

username: q13030880
cooked: <p>Yes, it really works for me. Thank you very much.</p>
Train with pseudo label

username: oasjd7
cooked: <p>Hi, I want to train a classifier with pseudo label.<br>
In this process.</p>
<pre><code class="lang-auto">1. [Source data] -&gt; [train a classifier ]
2. [Target data] -&gt; [Generate a pseudo label with above the classifier]
3. [Target data] -&gt; [train the classifier with a pseudo label]
</code></pre>
<p>In this case, I’m not sure this codes are make sense.<br>
<code>target_pseudo_loss = criterion(target_pseudo_label, target_pseudo_label)</code><br>
this line makes me confusing because the predicted label always equals answer label…<br>
Does it make sense? if not how can I modify this code?</p>
<pre><code class="lang-auto"># 1. [Source data] -&gt; [train a classifier ]
...
source_out = classifier(source_img)
source_loss= criterion(source_out, source_label)
optimizer.zero_grad()
source_loss.backward()
optimizer.step()

# 2. [Target data] -&gt; [Generate a pseudo label with above the classifier]
target_pseudo_label = classifier(target_img.detach())

# 3. [Target data] -&gt; [train the classifier with a pseudo label]

target_pseudo_loss = criterion(target_pseudo_label, target_pseudo_label)
optimizer.zero_grad()
target_pseudo_loss.backward()
optimizer.step()

</code></pre>
What is the best practice of MLP in year 2019(or 2020)

username: jerron
cooked: <p>I want to use available several features to predict a variable. It is not related to vision or NLP. I believe there are good reasons that the variable to be predicted is a non linear function of these features.</p>
<p>currently what I did is like following:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(53, 200)
        self.fc2 = nn.Linear(200, 100)
        self.fc3 = nn.Linear(100, 36)
        self.fc4 = nn.Linear(36, 1)

    def forward(self, x):
        x = F.leaky_relu(self.fc1(x))
        x = F.leaky_relu(self.fc2(x))
        x = F.leaky_relu(self.fc3(x))
        x = self.fc4(x)
        return x
net = Net().to(device)
loss_function = nn.MSELoss()
optimizer = optim.Adam(net.parameters(), lr=0.001, weight_decay= 1e-6)#, momentum = 0.9, nesterov = True)


def train_normal(model, device, train_loader, optimizer, epoch):
    model.train ()
    for batch_idx, (data, target) in enumerate (train_loader):
        data = data.to (device)
        target = target.to (device)
        optimizer.zero_grad ()
        output = model (data)
        loss = loss_function (output, target)
        loss.backward ()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 100)
        optimizer.step ()
        if batch_idx % 100 == 0:
            print ('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format (
                epoch, batch_idx * len (data), len (train_loader.dataset),
                       100. * batch_idx / len (train_loader), loss.item ()))
</code></pre>
<p>It is a typical MLP. After trained 100 epochs the result shows the model did learn something but still not very accurate:</p>
<blockquote>
<p>Train Epoch: 99 [268800/276316 (97%)]	Loss: 0.217219<br>
Train Epoch: 99 [275200/276316 (100%)]   Loss: 0.234965<br>
predicted  actual  diff<br>
-1.18       -1.11   -0.08<br>
0.15        -0.15   0.31<br>
0.19        0.27    -0.08<br>
-0.49       -0.48   -0.01<br>
-0.05       0.08    -0.14<br>
0.44        0.50    -0.06<br>
-0.17       -0.05   -0.12<br>
1.81        1.92    -0.12<br>
1.55        0.76    0.79<br>
-0.05       -0.30   0.26</p>
</blockquote>
<p>How shall I improve the accuracy? To add more hidden layer or increase the neuron numbers? What is the best practice nowadays?</p>
Regarding tgt_mask: (T, T) set as optional in nn.TransformerDecoder

username: vainaijr
cooked: <p>shouldn’t tgt_mask be always there, because we do not want nn.TransformerDecoder to look at ahead words during training.<br>
for example if input to nn.TransformerDecoder is the sequence,</p>
<pre><code class="lang-auto">&lt;sos&gt; hello world.
</code></pre>
<p>in nn.TransformerDecoderLayer</p>
<pre><code class="lang-auto">tgt2 = self.self_attn(tgt, tgt, tgt, attn_mask=tgt_mask,
                              key_padding_mask=tgt_key_padding_mask)[0]
</code></pre>
<p>then if we do not provide tgt_mask, and pass (tgt, tgt, tgt) in self_attn, then the representation of ‘hello’ would consider ‘world’ also, when computing self attention.</p>
Convert trained caffe model to torch c++

username: amoakoRadjwan
cooked: <p>Hello everyone,<br>
I must confess I am a total newbie to pytorch framework and neural networks in general. I have been doing some reading and practice for about 2  months now. I am working on my thesis and I need to implement some classification at some point. I currently have a pretrained model in caffe and would like to convert it to c++ version of pytorch since I am a bit comfortable with pytorch now.</p>
<p>I would be glad if someone could direct me to a reading resource/library/tutorial that does this: caffe2–&gt;C++ torch. Most of what I have found is loading from caffe to pytorch in python not c++.</p>
<p>secondly, should I succeed in the convertion will the new model in torch still be a trained model or I woud I have to retain.<br>
I’ll be glad if someone could give me a prompt response as I am running out of time. Thanks in advance.</p>
How to average gradients when use DataParallel?

username: Catherine_Fang
cooked: <p>By default, DataParallel sums all gradients. How to average gradients when use DataParallel?</p>

username: Donk10
cooked: <p>The correct way to deal with it is to average your loss values - torch.mean(loss) - for each GPU you get a loss value and averaging the loss is the same as averaging the gradients from my point of view.<br>
Here a post where they also talk about that <a href="https://discuss.pytorch.org/t/is-average-the-correct-way-for-the-gradient-in-distributeddataparallel-with-multi-nodes/34260">https://discuss.pytorch.org/t/is-average-the-correct-way-for-the-gradient-in-distributeddataparallel-with-multi-nodes/34260</a></p>
Traning CNN and LSTM parameters

username: Mustapha
cooked: <p>Hello, I want to train a model to predict Spatio-temporal data. but before I train the model with LSTMs I want to use CNN to extract the patterns from my data, so to train my CNN parameter I’m dependent on the output of the LSTMs as the loss is calculated based on the output of the LSTM, I want to know is there a way to train both the parameters of the CNN and LSTM based on the output of the LSTM.<br>
my intuition tells me to do something like this.<br>
optimizer = optim.SGD([CNNmodel.parameters(),LSTMmodel.parameters()], lr=0.01, momentum=0.5)<br>
I didn’t try it yet but does something like this work<br>
You can understand better if you see the architecture of the model I’m trying to implement.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e8ab31e147b091ac823deadf92ae61667b758e22.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e8ab31e147b091ac823deadf92ae61667b758e22" title="torch%20forum"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8ab31e147b091ac823deadf92ae61667b758e22_2_690x248.jpeg" alt="torch%20forum" data-base62-sha1="xchyuOY6nncz5XevYB6Ee0h6cLM" width="690" height="248" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8ab31e147b091ac823deadf92ae61667b758e22_2_690x248.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e8ab31e147b091ac823deadf92ae61667b758e22.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e8ab31e147b091ac823deadf92ae61667b758e22.jpeg 2x" data-dominant-color="E2E4E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch%20forum</span><span class="informations">912×329 57.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Donk10
cooked: <p>Hi<br>
that should work as you suggested:</p>
<blockquote>
<p>params = list(LSTM.parameters()) + list(CNN.parameters())<br>
optimizer = optim.SGD(params, lr=learning_rate)</p>
</blockquote>
How to Caculate parameters using pytorch in Convolutional neural networks?

username: sriharsha0806
cooked: <p>Is there any pytorch module which automatically gives me result of the number of parameters in a Convolutional Neural Network layer (similar keras <a href="http://topology.py" rel="nofollow noopener">topology.py</a>)?</p>

username: jdhao
cooked: <p>I think it is easy to calculate the number of elements in PyTorch. Suppose you have a model called <code>net</code>. You can use the following snippet to calculate the number of parameter in your model:</p>
<pre><code class="lang-auto">count = 0
for p in net.parameters():
    count += p.data.nelement()
</code></pre>

username: Greg-Tarr
cooked: <p>That snippet can be reduced to:</p>
<blockquote>
<p><strong>sum</strong>(p.data.nelement() <strong>for</strong> p <strong>in</strong> net.parameters())</p>
</blockquote>
How to add bayesian conv layers( getting this error RuntimeError: Expected 3-dimensional input for 3-dimensional weight 1 28 3, but got 4-dimensional input of size [100, 1, 32, 32] instead)

username: Aakash_Bhatia
cooked: <p>def main():<br>
# import data<br>
kwargs = {‘num_workers’: 1, ‘pin_memory’: True} if FLAGS.cuda else {}</p>
<pre><code>train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('./data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),lambda x: 2 * (x - 0.5),
                   ])),
    batch_size=FLAGS.batchsize, shuffle=True, **kwargs)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('./data', train=False, transform=transforms.Compose([
        transforms.ToTensor(), lambda x: 2 * (x - 0.5),
    ])),
    batch_size=FLAGS.batchsize, shuffle=True, **kwargs)

# for later analysis we take some sample digits
mask = 255. * (np.ones((1, 28, 28)))
examples = train_loader.sampler.data_source.train_data[0:5].numpy()
images = np.vstack([mask, examples])

# build a simple MLP
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # activation
        self.relu = nn.ReLU()
        self.sigmoid = nn.Sigmoid()
        # layers
        self.conv1 = BayesianLayers.Conv1dGroupNJ(28, 1, 3, 1)
        #self.bn_conv1 = nn.BatchNorm2d(32)
        self.conv2 = BayesianLayers.Conv2dGroupNJ(28, 64, 3, 1)
        #self.bn_conv2 = nn.BatchNorm2d(64)
        
        '''self.conv3 = BayesianLayers.Conv3dGroupNJ(64,400,kernel_size = 5)
        
        self.flatten = FlattenLayer(1024)'''
        self.fc1 = BayesianLayers.LinearGroupNJ(9216, 128, clip_var=0.04, cuda=FLAGS.cuda)
        self.fc2 = BayesianLayers.LinearGroupNJ(128, 84, cuda=FLAGS.cuda)
        self.fc3 = BayesianLayers.LinearGroupNJ(84, 10, cuda=FLAGS.cuda)
        # layers including kl_divergence
        self.kl_list = [self.conv1,self.conv2,self.fc1, self.fc2, self.fc3]

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        '''x = self.conv3(x)
        x = F.sigmoid(x)'''
        x = torch.flatten(x, 1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        output = F.log_softmax(x, dim=1)
        return output</code></pre>

username: ptrblck
cooked: <p>I’m not sure, how <code>BayesianLayers</code> are defined, but it seems you are mixing 1d, 2d, and 3d versions of conv layers in your model.<br>
Based on your input shape, I guess you should stick to the 2d version.</p>
Race condition in Isend

username: seliad
cooked: <p>I get some bug during torch.distributed.isend and irecv: I think that there is a race condition there, but not sure how to debug this.<br>
I’m using MPI, so the buffers have to stay untouched until the send/recv is over.<br>
I see crazy “spikes” in error, which I get only with a certain degree of parallelism.</p>
<p>I wonder if pytorch/python garbage collector touches my buffers.<br>
I saved them in a list, just in case.<br>
Where can I check this in code? can I “guard” the buffers somehow?</p>
<p>I see pytorch tests barely check the Isend/Irecv,  and want to verify that the bug is not internal…</p>

username: mrshenli
cooked: <p>Hi <a class="mention" href="/u/seliad">@seliad</a></p>
<p>The code for MPI-based torch.distributed.isend is here: <a href="https://github.com/pytorch/pytorch/blob/cc16819028c325e2543d45752a875bd3c5e09b32/torch/lib/c10d/ProcessGroupMPI.cpp#L591" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/cc16819028c325e2543d45752a875bd3c5e09b32/torch/lib/c10d/ProcessGroupMPI.cpp#L591</a></p>

username: seliad
cooked: <p>I looked at the code and did not see something suspicious.<br>
However, the data corruption is still there.</p>
<p>I found that if I do <code>torch.distributed.synchronize(device)</code> explicitly before Isends the problem is mitigated and can be mistaken to “solved”, but I don’t like this solution at all.<br>
I don’t see any rational reason to do so, there is probably some bug there.</p>
<p>Reading the warnings <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="nofollow noopener">here</a> and <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing" rel="nofollow noopener">here</a> makes me believe that the MPI/distributed API probably does not do many stuff necessary for sharing tensors<br>
like handling references counts, using mutex to guard stuff and etc.<br>
I use CUDA-aware with openMPI, I thought it is supported.</p>
Runtime Error: variable modified by an inplace operation

username: azeeshan
cooked: <p>The forward function of my torch.nn module initially looked like this:</p>
<pre><code class="lang-auto">def forward(self, a, b, c):
   #blah blah blah
   inds = (c&lt;0)
   c_aug = c
   c_aug[inds] = a.shape[0]
   #more blah blah blah
</code></pre>
<p>This was giving me an error:<br>
<code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.LongTensor [82, 4]] is at version 2; expected version 1 instead.</code><br>
After some debugging using anomaly detection and looking at <a href="https://discuss.pytorch.org/t/setting-part-of-the-training-value-into-zero/52044">this</a> question, I changed the forward function as follows:</p>
<pre><code class="lang-auto">def forward(self, a, b, c):
   #blah blah blah
   inds = (c&lt;0)
   c_aug = torch.clone(c)
   c_aug[inds] = a.shape[0]
   #more blah blah blah
</code></pre>
<p>This does not give me that error. Is it the correct workaround for the error or there are better methods to handle it?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes this is the correct workaround. You just want to make sure that you don’t modify the <code>c</code> you get as input inplace. So you clone it before changing it inplace.</p>
<p>Note that we usually do <code>c_aug = c.clone()</code> instead of using <code>torch.clone</code>.</p>
Extending Select to support Int32

username: alannnna
cooked: <p>How can one go about extending Select to work on tensors with element type int32, specifically in the interpreter backend? Or, is it not recommended to extend a single op on a single backend?</p>

username: jfix
cooked: <p>Hi <a class="mention" href="/u/alannnna">@alannnna</a>, Totally reasonable to extend this. I don’t think we have a doc here on what to do, but off the top of my head it should be as simple as:</p>
<ul>
<li>Update <code>Interpreter::isOpSupported()</code> to include <code>ElemKind::Int32ITy</code>
</li>
<li>Update <code>BoundInterpreterFunction::fwdElementExpInst()</code> to support it</li>
<li>(probably not necessary) May need to update <code>SelectNode::verify()</code> if there is any verification that fails</li>
</ul>
How to create LSTMCell object in C++ API?

username: fulltopic
cooked: <p>It seemed that Only <em>torch::nn::LSTM</em> has been exported, <em>LSTMCell</em> and <em>CellParams</em> are just defined in cpp file that has not been exported for C++ API usage. Then how if I want to use <em>LSTMCell</em> directly?</p>
<p>Thank you very much!</p>
<p>Edit:  <a href="https://discuss.pytorch.org/t/lstmcell-equivalent-in-c-frontend/51221">The duplicated question</a>. The answer is NO equivalent API in C++ now.</p>
The precision problem during Back-Propagation

username: skhu1
cooked: <p>May I ask why the result of \partial L/\partial z_1 \partial L/\partial z_2 is not strictly equal?</p>
<p>import torch<br>
import torch.nn as nn<br>
input = torch.rand(3,3,256,256)<br>
label = torch.LongTensor([1,2,3])</p>
<p>relu = nn.ReLU(inplace=False)<br>
Conv = nn.Conv2d(3, 16, 3, padding=0, bias=False)<br>
bn = nn.BatchNorm2d(16, eps=1e-2)<br>
global_pooling = nn.AdaptiveAvgPool2d(1)<br>
classifier = nn.Linear(16, 5,bias=False)<br>
criterion = nn.CrossEntropyLoss()</p>
<p>z = torch.FloatTensor([1,1]).requires_grad_()<br>
out = relu(input)<br>
out = Conv(out)<br>
out = bn(out)<br>
out = global_pooling(out)<br>
logits = classifier(out.view(out.size(0), -1)*z[0])*z[1]</p>
<p>error_loss = criterion(logits, label)<br>
error_loss.backward()</p>
<p>torch.set_printoptions(precision=10)<br>
print(z.grad)<br>
torch.set_printoptions(precision=20)<br>
print(z.grad)</p>

username: skhu1
cooked: <p>If we have f(x)=Wx, then we can derive the formula \partial L / \partial x * x =  \partial L / \partial f(x) * f(x) =  \partial L / \partial z_1 =  \partial L / \partial z_2 based on the chain rule.</p>

username: skhu1
cooked: <p>I have the following result, i.e., tensor([0.00013380237214732915, 0.00013380234304349869]). And I can not understand this.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I did not check your code in detail, but given your last message, I think the answer is simply that single precision float numbers are only precise up to 6/7 digits. And there two numbers you show are precise up to that precision. So they are equal in terms of float numbers.<br>
You can use double precision numbers if you want more precise values.</p>

username: skhu1
cooked: <p>Thanks, I will check about this!</p>
How to pack / pad sequence of dictionary

username: Cakey
cooked: <p>Hi there!<br>
I am working with a custom dataset that has 1) multiple sequences, 2) each sequence is of different length, 3) at time step x for any sequence, we have a dictionary containing ‘field1’ ‘field2’ … Note that each data point contains multiple images and other different types of information, so we’d prefer to keep using dictionary.<br>
I wrote a custom dataset, returning seq_len x dictionaries.<br>
Now I need to write a collate_fn that combines all my data for the DataLoader.<br>
I know how to write it brutally : ) as follows:</p>
<pre><code class="lang-auto">def misc_seq_len_collate_fn(batch):
  # This is not the most efficient way but well dict
  # Input: batch containing BATCH_SIZE x seq_len x dictionary
  # Output: seq_len x dictionary, each key in dict: BATCH_SIZE x D
  #         add a "mask" to dictionary
  batch_size = len(batch)
  sample_lens = [len(x) for x in batch]
  max_len = max(sample_lens)

  # create mask
  lengths = torch.tensor(sample_lens)
  tmp = torch.arange(max_len).long().expand(batch_size, max_len)
  lengths = lengths.unsqueeze(1).expand_as(tmp)
  mask = (tmp &lt; lengths).t() # seq_len * batch_size

  # create filler
  dict_keys = batch[0][0].keys()
  dummy_tensor = {k : torch.as_tensor(batch[0][0][k]) for k in dict_keys}

  res = []
  for i in range(max_len):
    a_dict = {'mask' : mask[i]}
    for k in dict_keys:
      a_list = ([torch.as_tensor(b[i][k]) if i &lt; len(b)
            else torch.zeros_like(dummy_tensor[k]) for b in batch])
      a_dict[k] = torch.stack(a_list, dim=0)
    res.append(a_dict)
  return res
</code></pre>
<p>The collate function works but I am curious to see if anyone might point me to something maybe more efficient?</p>
Setup models on different gpus and use dataparallel

username: Hung_Nguyen
cooked: <p>Lets say I have 2 models A() and B() and 2 gpus. Outputs of A will be fed to B as inputs</p>
<p>Because 2 models are too big to fit on the same gpus so I have to manually instantiate A on gpu 0 and B on gpu 1. Hence, I have to manually change the device of A’s output to feed to B.</p>
<p>Sometimes, my batchsize is too large to run A() on gpu 0, but if I theoretically can utilize gpu1, I can still use that batch size without reducing it.</p>
<p>The question is can my models be placed on different gpus but still run in dataparallel mode?</p>
<p>Update:<br>
I saw a post mentioning about this:</p>
<pre><code class="lang-auto">model1 = nn.DataParallel(model1).cuda(device=0)
model1_feat = model1(input_image)

model2 = nn.DataParallel(model2).cuda(device=1)
model2_feat = model2(model1_feat, input_feat)
</code></pre>
<p>My question is does that mean model1 is replicated on both gpu?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="Hung_Nguyen" data-post="1" data-topic="37899">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hung_nguyen/48/8273_2.png" class="avatar"> Hung_Nguyen:</div>
<blockquote>
<p>My question is does that mean model1 is replicated on both gpu?</p>
</blockquote>
</aside>
<p>Yes, <code>DataParallel </code> will replicate the model, scatter inputs, and gather outputs in every iteration. So, for the above code snippet, everytime you run <code>model1_feat = model1(input_image)</code>, <code>model1</code> is replicated to all devices in the forward pass.</p>
I want to study Pytorch from scratch!

username: sunshower76
cooked: <p>Hello, everyone!</p>
<p>When we learn or research deep learning and the other things related to ML or DL we use PyTorch…</p>
<p>I’m also a user of Pytorch.</p>
<p>Using PyTorch, I suddenly think that I want to study PyTorch from scratch!</p>
<p>I’ve wondered how the tensor is defined and how the values are back-propagated.</p>
<p>So, I’ve decided that I study Pytorch which version of either c++ or python.</p>
<p>but, I don’t know where to start.</p>
<p>Could you give me some help to learn Pytorch?<br>
What is the first thing that I have to do?</p>
<p>It is my future goal to join in PyTorch open-source development project in Github.</p>

username: ptrblck
cooked: <p>I would suggest to pick an easy looking issue on GitHub, post that you are interested and new, and either someone will guide you through the PR or could suggest another one, if your pick is not beginner friendly.<br>
You will learn the internals of PyTorch while working on a real fix instead of staring at code, which sounds like more fun to me. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to combine data parallelism with model parallelism for multiple nodes?

username: Sword-Song
cooked: <p>For example, I have 4 nodes and every node has two gpus. I want to devide one model into four parts, every node run part of the model and  use data parallelism on its two gpus.</p>
<p>I use hook to get the gradients and use “dist.send” to send them to other node, it’s effective for<br>
model parallelism.</p>
<blockquote>
<h1>on node 1:</h1>
<p>dist.init_process_group(backend=“gloo”, init_method=‘tcp://172.22.4.11:28456’, rank=0,<br>
world_size=2)</p>
<h1>outputs is the result of node 1</h1>
<p>dist.send(tensor=outputs.to(‘cpu’), dst=1, tag=0)</p>
<h1>rec is the gradients send from node 2</h1>
<p>dist.recv(rec, src=1)<br>
outputs.backward(rec.cuda())</p>
<h1>on node 2:</h1>
<p>dist.init_process_group(backend=“gloo”, init_method=‘tcp://172.22.4.11:28456’, rank=1,<br>
world_size=2)</p>
<h1>rec is the result of node 1</h1>
<p>dist.recv(tensor=rec, src=0, tag=0)<br>
outputs2 = net2(rec)</p>
<h1>feta[0] is the gradients of node 2</h1>
<p>dist.send(tensor=feat[0].to(‘cpu’), dst=0)</p>
</blockquote>
<p>But when I try to combine data parallelism with model parallelism, it failed. I choose “torch.nn.parallel.DistributedDataParallel” to achieve data parallelism, but node2 can’t receive the  gradients from node1.</p>
<p>Question:<br>
So how to combine data parallelism with model parallelism for multiple nodes?</p>

username: mrshenli
cooked: <p>It might be easier to run model parallel on multiple GPUs in the same machine and distributed data parallel across machines. Checkout <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#combine-ddp-with-model-parallelism" rel="nofollow noopener">this section</a> for more details.</p>
<p>For your above use case, you will need to create multiple process groups. Given the above configuration, 4 nodes, and 2 GPUs per node, I assume you will have 8 processes, one process per GPU. Then you can create:</p>
<ol>
<li>Two process groups of world size 4, which will be responsible for send/recv outputs/gradients across machines.</li>
<li>One process group of world size 2 on EACH machine, which will do the distributed data parallel on the machine.</li>
</ol>
<p>The reason you need the above setting is because DistributedDataParallel would expect all processes in the same group are training the same model in a synchronized fashion. It won’t work if you only use 2 processes in the same group of size 8.</p>
<p>See the <a href="https://github.com/pytorch/pytorch/blob/46539eee0363e25ce5eb408c85cefd808cd6f878/torch/distributed/distributed_c10d.py#L1434" rel="nofollow noopener"><code>new_group</code></a> API.</p>
<p>BTW, the <a href="https://pytorch.org/docs/master/rpc.html" rel="nofollow noopener"><code>torch.distributed.rpc</code></a> API might make send/recv outputs/grads easier for you, and it also supports distributed autograd.</p>
Distributed training of multiple models on multiple nodes (CPU only)

username: gaussian
cooked: <p>Hi all,</p>
<p>I have been trying to figure out how to train a population of models on multiple nodes (which do not have GPUs, but that’s not the main point; I’m happy with training on CPUs). Ideally, I would like a single process per model running on a separate CPU. I can request hundreds or thousands of CPUs, and each model is <em>fully contained</em>, meaning that I don’t want to share any parameters from one model across nodes; rather, I want each model to train on its own CPU.</p>
<p>I have tried using a worker pool from <code>torch.multiprocessing</code> and passing models to the training function. I train each model for one epoch, then I perform some processing in the main process and then I map them again to the worker pool to train them for another epoch, and so on. That works fine if I run the models on a single machine, but it doesn’t scale up to a multi-node scenario because <code>torch.multiprocessing</code> is not aware of the additional nodes (I requested 256 CPUs on the cluster, which translates to 8 nodes with 16 CPUs each, but 7 of those remained idle).</p>
<p>As far as I can tell, all examples I found (for example, using <code>torch.distributed</code> <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="nofollow noopener">here</a>) assume that you have a single large model and you want to spread the training of one model across multiple workers. This is not my case - my models are small and I’d like to train them in parallel but independently of each other. They <em>are</em>, however, being trained on the same task using the same data, in case that’s relevant.</p>
<p>Any help would be appreciated! Apologies if I’m missing something obvious.</p>

username: mrshenli
cooked: <p>IIUC, DistributedDataParallel does not fit in this use case because you have a population of independent models to train on the same set of data instead one big model on different splits of input data. It looks like the experimental <a href="https://pytorch.org/docs/master/rpc.html" rel="nofollow noopener"><code>torch.distributed.rpc</code></a> might be helpful here, it would at least help you take care of the communication. But you would still need to write code to dispatch models to the workers in the pool.</p>
Torch.distributed does not support part of the model training?

username: Lausanne
cooked: <p>Hi, recently I tried to use torch.distributed package to train my model. In my case, I used one architecture like encoder-decoder. For accelerating my code, I pre-computed features of input data, but when I only trained decoder with calling model.module.decoder (input), it gave me some bug info like the following<br>
<strong>TypeError: _queue_reduction(): incompatible function arguments. The following argument types are supported:</strong><br>
<strong>1. (process_group: torch.distributed.ProcessGroup, grads_batch:List[List[at::Tensor]], devices: List[int]) -&gt; Tuple[torch.distributed.Work, at::Tensor]</strong><br>
I wonder if someone can give me some suggestions? Does torch.distributed package could not work normally if part of the model doesn’t participate in computing process?</p>

username: mrshenli
cooked: <p>In v1.1, we added a new <a href="https://github.com/pytorch/pytorch/blob/46539eee0363e25ce5eb408c85cefd808cd6f878/torch/nn/parallel/distributed.py#L196" rel="nofollow noopener"><code>find_unused_parameters</code></a> arg to DistributedDataParallel. If some of the model params are not involved in the forward pass, you can set  <code>find_unused_parameters</code> to <code>True</code>.</p>
Multiple replicas of the model on same GPU?

username: Joshua_Lian
cooked: <p>Hi, I am a newbee for pytorch distributed.<br>
My model is only a small component of a much more complicated problem.<br>
I noticed that if I train it using single-GPU, then it takes at most one quarter of the GPU memory and utility.<br>
So I wonder if it is possible to distribute four replicas of the model on the same GPU so that hopefully I can get 4x speedup.</p>
<p>I read the documents and there are many example of multi-gpu, but none of them is using fractional gpu like this. Anyone have ideas? Thanks.</p>

username: mrshenli
cooked: <p>It really depends. Even if 4 replicas of your model can fit into the memory of one GPU, they still need to compete for the same set of streaming multiprocessors and other shared resources on that GPU. You can try if using multiple streams would help, e.g.:</p>
<pre><code class="lang-python">s0 = torch.cuda.Stream()
s1 = torch.cuda.Stream()
with torch.cuda.stream(s0):
    output0 = model_replica0(input0)

with torch.cuda.stream(s1):
    output1 = model_replica1(input1)

s0.synchronize()
s1.synchronize()
</code></pre>

username: meilu_zhu
cooked: <p>Hi, Shen Li, Hi, the “DistributedDataParallel” automatically average the gradient when calling “loss.backward()”,<br>
But I didn’t find the corresponding script in pytorch source code, Do you know where it is ?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/meilu_zhu">@meilu_zhu</a></p>
<p>Sorry about the delay. The grad averaging algorithm is implemented in the <a href="https://github.com/pytorch/pytorch/blob/70013415c7acc109985a86f42d835fbbcfa45bf3/torch/csrc/distributed/c10d/reducer.cpp" rel="nofollow noopener">reducer</a>. Each DistributedDataParallel creates its reducer instance <a href="https://github.com/pytorch/pytorch/blob/70013415c7acc109985a86f42d835fbbcfa45bf3/torch/nn/parallel/distributed.py#L378" rel="nofollow noopener">in the constructor</a>. More specifically, allreduce is invoked <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/reducer.cpp#L415" rel="nofollow noopener">here</a>.</p>
Confused about computing multiple losses

username: Willzy-x
cooked: <p>I tried 2 kinds of computing multi-losses methods in the deeply-supervised nets.<br>
The first one needs to add <code>retain_graph=True</code> in <code>loss.backward()</code> while the second doesn’t and I don’t understand why.</p>
<pre><code class="lang-python">...
output = model(data)
loss = 0
for i, out in enumerate(output):
    ...
    loss+=(F.nll_loss(out, target))
loss.backward()
</code></pre>
<pre><code class="lang-python">...
output = model(data)
loss = 0
loss1 = []
for i, out in enumerate(output):
    ...
    loss1.append(F.nll_loss(out, target))

loss = reduce(lambda x, y: x+y, loss1)
loss.backward()
</code></pre>

username: ptrblck
cooked: <p>The first code snippet runs fine without <code>retain_graph=True</code> using your code as the base:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
data = torch.randn(10, 1)
target = torch.zeros(10).long()
output = model(data)

loss = 0.
for out, tar in zip(output, target):
    loss += F.nll_loss(out.unsqueeze(0), tar.unsqueeze(0))
loss.backward()
print(model.weight.grad)
</code></pre>

username: Willzy-x
cooked: <p>It seems that the model you defined is rather simple.<br>
My model looks like this, which return losses from different stages.<br>
Maybe that’s where the problem arises?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f3722aaedf882e62f17b316805f8170ddaada1e2.png" alt="Picture2" data-base62-sha1="yJCCPVutdjYHylw2e2tbfwcWp9w" width="675" height="356"></p>
Resize not support when using ONNX model converted from Pytorch

username: MandySuperTT
cooked: <p>I’m trying to convert Pytorch model to Onnx model, there is a block where I’ve used</p>
<blockquote>
<p>import torch.nn.functional as F<br>
F. interpolate(h, size=(288,528), mode=‘bilinear’, align_corners=False)</p>
</blockquote>
<p>to achieve <code>Upsampling </code></p>
<p>There is no Bug or any Error during converting process. However, when I ran the onnx model, I’ve got this error</p>
<blockquote>
<p>Onnx operator Resize is not supported now… By Passing</p>
</blockquote>
<p>Failed to run Onnx model.</p>
<blockquote>
<p>I’ve used Pytorch 1.3 and Opset version = 11</p>
</blockquote>
<p>Anyone knows how to solve this problem?</p>
Generate offline documentation

username: Alex5
cooked: <p>Hi,</p>
<p>I’m running make html in docs folder in order to generate offline documentation but I got an error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “source/scripts/build_activation_images.py”, line 68, in <br>
function = torch.nn.modules.activation.<strong>dict</strong><a>function_name</a><br>
KeyError: ‘GELU’<br>
Makefile:17: recipe for target ‘figures’ failed<br>
make: *** [figures] Error 1</p>
</blockquote>
<p>I cannot figure out how to fix it, can anyone help me ?<br>
Thanks !</p>

username: albanD
cooked: <p>Do you have properly recursively clones the repo and installed pytorch before?</p>

username: Alex5
cooked: <p>Yes everything is working fine expect the documentation</p>

username: albanD
cooked: <p>Can you give the full traceback please?</p>

username: wei_zhang
cooked: <p>Same Error, my torch version is 1.3.1</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "source/scripts/build_activation_images.py", line 68, in &lt;module&gt;
    function = torch.nn.modules.activation.__dict__[function_name]()
KeyError: 'GELU'
make: *** [figures] Error 1
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>I just tried again from master and it compiles properly for me <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p><code>GELU</code> has only been added in master (not in 1.3.1). Make sure that you are using the python where the brand new pytorch is installed in.<br>
Or build the 1.3.1 doc by first calling <code>git checkout v1.3.1</code>.</p>

username: Alex5
cooked: <p>Hi,</p>
<p>I am in the master branch. What do you mean by full traceback ?</p>

username: albanD
cooked: <p>If you are on the master branch, you need to have a pytorch from the master branch installed. You can check with <code>print(torch.__version__)</code> in a command line.<br>
If you have 1.3.1 installed, you will only be able to build the 1.3.1 doc.</p>
GPU model on shared memory?

username: wwiiiii
cooked: <p>Is there simple way to share model on GPU between processes?</p>
<p>I read <a href="https://pytorch.org/docs/stable/multiprocessing.html" rel="nofollow noopener">https://pytorch.org/docs/stable/multiprocessing.html</a> and <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html</a> but they only mention about CUDA shared memory in unit of tensor(not module) and CPU model.</p>
<p>Do we have simple method as <code>model.share_memory()</code> for GPU model?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Only the Tensors are actually stored on the GPU, the rest of the structure of your model is stored on CPU. So nothing will be copied on the GPU if you share the models.</p>
Handling edge cases/handling some inputs differently

username: azeeshan
cooked: <p>I am building a torch.nn.module for graph convolutional neural networks. For concreteness, let us imagine the input is a configuration of points with (x,y) coordinates and one has to predict the output (scalar) of the configuration. I want an elegant way to handle the different case (edge case) of a configuration with no (x,y) points i.e. empty configuration with the same torch.nn.module. The output in this case is a scalar and I want to be able to optimize this scalar corresponding to the empty configuration together with the neural network weights during the training stage of the neural network. The scalar may be considered as an additional weight in the model.<br>
One problem that may arise:<br>
-Each batch that is input to the neural network may have a combination of the empty and non-empty configurations</p>

username: albanD
cooked: <p>Hi,</p>
<p>To handle such things I usually do the following pseudo code:</p>
<pre><code class="lang-python">
cond = bath_data.is_empty? (gives a Tensor of size (n_batch,) containing 0/1)

out_true = your_non_empty_code(batch_data) # Compute everything as if non-empty

out_false = self.empty_value.expand_as(out_true) # Your param for the empty ones

res = out_true * (1 - cond) + out_false * cond
</code></pre>
Need some advice for face recognition project

username: hunar
cooked: <p>hi<br>
im new to with deep learning and computer vision<br>
i want to build my final year project with pytorch , for face recognition in real time !<br>
both detecting and recognition using neural network !<br>
i need some advice from advanced developers who done similar projects?!<br>
i have read alot , blog posts , youtube videos but i still confused , i have my own dataset<br>
and i have a basic understanding of neural network<br>
using a pre-trained model with transform learning is a good choice for those who use google colab service ?<br>
thanks for advice</p>

username: Eta_C
cooked: <ul>
<li>Face detection, get face bounding box and landmarks. <a href="https://paperswithcode.com/sota/face-detection-on-wider-face-hard" rel="nofollow noopener">Here’s the SOTA Papers and Codes</a>.</li>
<li>(Use landmarks) Aligned face is very beneficial for improve the performance of face recognition.</li>
<li><a href="http://megaface.cs.washington.edu/results/facescrub.html" rel="nofollow noopener">Here’s the Megaface Rank.</a></li>
<li>
<a href="https://github.com/deepinsight/insightface" rel="nofollow noopener">Insightface</a>, which include RetinaFace for face detection and ArcFace for face recognition, use MxNet instead of PyTorch, but you can find some third-party re-implementation in its README file.</li>
<li><strong>Dataset quality is more important than network structure.</strong></li>
<li>For faster inference: <a href="https://arxiv.org/abs/1905.02244" rel="nofollow noopener">MobileNet v3</a>, <a href="https://arxiv.org/abs/1905.11946" rel="nofollow noopener">EfficientNet</a> and <a href="https://arxiv.org/abs/1911.08609" rel="nofollow noopener">IdleBlock</a> are worth trying.</li>
</ul>
<p>Addition:</p>
<ul>
<li>The main idea is to enhance intraclass compactness and inter-class discrepancy.</li>
<li>More focused on intraclass compactness: <a href="https://arxiv.org/abs/1704.08063" rel="nofollow noopener">ShpereFace</a>, <a href="https://arxiv.org/abs/1801.09414" rel="nofollow noopener">CosFace</a> and <a href="https://arxiv.org/abs/1801.07698" rel="nofollow noopener">ArcFace</a> are a series.</li>
<li>More focused on inter-class discrepancy: <a href="http://ivg.au.tsinghua.edu.cn/people/Yueqi_Duan/CVPR19_UniformFace%20Learning%20Deep%20Equidistributed%20Representation%20for%20Face%20Recognition.pdf" rel="nofollow noopener">UniformFace</a> is one of the recent works.</li>
</ul>