A strange problem I meet when define a new loss function

KoalaSheep: <p>Hi, everybody, I met a strange problem when trying to define a weighted cross entropy myself.</p>
<p>My loss func is simple:</p>
<pre><code class="lang-auto">def weighted_BCE(sigmoid_x, targets):                  
#this is a weighted Binary Cross Entropy
    assert sigmoid_x.size() == targets.size()           
#make sure input and label have same shape
    count_p = (targets == 1.0).sum() + 1                 
    count_n = (targets == 0.0).sum() + 1                
#count 1(positive) and 0(negative), add 1 to avoid 0
    loss = -((targets * sigmoid_x.log()) * (1 / count_p)) - (((1 - targets) * (1 - sigmoid_x).log()) * (1 / count_n))
#divide the positive part and negative part with their count respectively
    return loss.mean()
</code></pre>
<p>then try it with a random input and all 0 label</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    a = np.random.uniform(low = 0.0, high = 1.0, size = (3, 3))
    a = torch.from_numpy(a).float().to('cuda:0')
    a.requires_grad = True
    b = np.zeros((3, 3))
    b = torch.from_numpy(b).float().to('cuda:0')
    b.requires_grad = False
    loss = weighted_BCE(a, b)
    loss2 = torch.mean(-((b * a.log()) * (1 / 1)) - (((1 - b) * (1 - a).log()) * (1 / 10)))
# calculate the loss in the same way without definition
    print(loss.item())
    print(loss2.item())
    print((loss - loss2).item())
</code></pre>
<pre><code class="lang-auto"># here is the result
0.0
0.15303052961826324
-0.15303052961826324 #this is the supposed to be zero
</code></pre>
<p>It may be a silly question, but I can’t figure out the reason why the loss func returns 0.0 in a definition and different result when calculate it in the main func.</p>
<p>My environment is</p>
<pre><code class="lang-auto">Os : Ubuntu 18.04.2 LTS
GPU: GTX 1080ti
pytorch: 1.2.0
</code></pre>

Nikronic: <p>Hi,</p>
<p>The logic you using is correct, actually the data types are causing this differences.<br>
For instance, if you use your own code in functions:</p>
<pre><code class="lang-auto">count_p = (b == 1.0).sum() + 1
count_n = (b == 0.0).sum() + 1                
loss2 = torch.mean(-((b * a.log()) * (1 / count_p)) - (((1 - b) * (1 - a).log()) * (1 / count_n)))
</code></pre>
<p><code>count_p</code> instead of <code>1</code> and <code>count_n</code> instead of <code>10</code>, you will get same result. Note if you add <code>1.</code> or <code>10.</code>, you will get different result.</p>
<p>So the most secure way to use constant numbers is to use <code>torch.tensor()</code> method.</p>
<pre><code class="lang-auto">loss2 = torch.mean(-((b * a.log()) * (1 / torch.tensor([1]).cuda())) - (((1 - b) * (1 - a).log()) * (1 / torch.tensor([10]).cuda())))
</code></pre>
<p><code>torch.tensor()</code> retains the datatype while moves tensor to the given device.</p>
<p>Bests<br>
Nik</p>

KoalaSheep: <p>Thank you and sorry for my late response. In fact what I want is a little different.<br>
Based on your advice, I change the</p>
<pre><code class="lang-auto">loss = -((targets * sigmoid_x.log()) * (1 / count_p)) - (((1 - targets) * (1 - sigmoid_x).log()) * (1 / count_n))
</code></pre>
<p>in my loss def to</p>
<pre><code class="lang-auto">loss = -((targets * sigmoid_x.log()) * (1 / count_p.float())) - (((1 - targets) * (1 - sigmoid_x).log()) * (1 / count_n.float()))
</code></pre>
<p>and it gives me the right result.</p>
<p>But I am still unsure about the int and float type here.<br>
As count_p is a int type and sigmoid_x.log() is a float type, float multiply a int type should result in a float type, isn’t it? Or is there anything different with tensor?</p>

Nikronic: <p>Actually, You are right about the casting to float. But something I am not sure about is the system dependent type sizes or even other parameters. I tried using <code>1.</code> to convert to python default float type and it did not work. I do not know much about C++ (backend of PyTorch) and Python variable system and type checking, so that’s why I suggest using <code>torch.tensor()</code> because it automatically handle type conversion of retaining current type of numpy ndarray object as argument.<br>
I did not check numpy float type number to check the answer because in the docs it says PyTorch tensors and numpy ndarrays can be converted to each other seamlessly. (Using sole constant number is python pure not numpy.)<br>
For instance, numpy has system dependent int and independent version too. There all can effect and an interface is needed to handle this situations which I think <code>torch.tensor</code> is doing it.</p>
<p>Something else I thing need to be checked is that GPU is considering different sizes rather than normal code or not.</p>
<p>My knowledge is very basic about these topics.<br>
I think you can create new question and demonstrating different answers regarding using python pure constant number or numpy array or even the tensor. PyTorch dev team is really good and almost all the time answer the question.</p>
<p>Bests</p>

KoalaSheep: <p>Thank you for your kindness and patience. I will take your advice.</p>
<p>edit:<br>
just found something alike on github.<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/5411" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/5411" target="_blank" rel="nofollow noopener">Integer division behavior is different from Python and NumPy</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-02-26" data-time="16:57:22" data-timezone="UTC">04:57PM - 26 Feb 18 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/mdraw" target="_blank" rel="nofollow noopener">
          <img alt="mdraw" src="https://avatars3.githubusercontent.com/u/6719909?v=4" class="onebox-avatar-inline" width="20" height="20">
          mdraw
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">PyTorch version: current master (10fd27)
How you installed PyTorch (conda, pip, source): from source
Python version: 3.6
When dividing two ints in pure Python,...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">medium priority (this tag is deprecated)</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: cpu</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: numpy</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">topic: numerical-reproducibility</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>