Reproducibility of weighted cross entropy loss

username: ryok
cooked: <p>Hi,</p>
<p>When I need to keep my calculations reproducible, shouldn’t I define a decimal point for the cross-entropy error weights? (Such as [1.0, 100.0, 50.0, 30.0, 20.0])</p>
<p>I am working on a semantic segmentation. The training set has 5 different classes. I applied torch.nn.CrossEntropyLoss and set weights option:</p>
<pre><code class="lang-python">def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
    catnum = \
        np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num], 
                      dtype=np.float32)
    cat_sum = catnum.sum()
    catnum = catnum / cat_sum
    catnum = 1.0 / catnum
    catnum = np.round(catnum, decimals=5) # !!!decimals=5!!!
    catnum = torch.FloatTensor(catnum)

    return catnum
</code></pre>
<p>But with this setting, the loss value is different every time the calculation is performed.<br>
On the other hand, the following settings always give the same result.</p>
<pre><code class="lang-python">def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
    catnum = \
        np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num], dtype=np.float32)
    cat_sum = catnum.sum()
    catnum = catnum / cat_sum
    catnum = 1.0 / catnum
    catnum = np.round(catnum, decimals=0) # !!!decimals=0!!!
    catnum = torch.FloatTensor(catnum)

    return catnum
</code></pre>
<p>Shouldn’t I define a decimal point for the cross-entropy error weights?</p>
<p>I already manually make the model deterministic by setting these properties:</p>
<pre><code class="lang-python">def init_seed(seed=1234):
    np.random.seed(seed)
    random.seed(seed)
    torch.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

def worker_init_fn(worker_id):
    new_seed = np.random.get_state()[1][0] + worker_id
    np.random.seed(new_seed)
    random.seed(new_seed)
    return
</code></pre>
<p>Pardon my broken English.<br>
Thanks,</p>
<p>Ryo.</p>

username: Kushaj
cooked: <p>For decimal=5 case, try printing value of catnum for two different runs and see if they are different. If they are same, then print the previous catnum value.</p>

username: ryok
cooked: <p>Thank you very much for your reply.</p>
<p>I tried printing value of “catnum” and here is my code.</p>
<pre><code class="lang-python">def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
    catnum = \
        np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num], 
                      dtype=np.float32)
    print(catnum) # -&gt; Standard output A
    cat_sum = catnum.sum()
    catnum = catnum / cat_sum
    catnum = 1.0 / catnum
    catnum = np.round(catnum, decimals=5) # !!!decimals=5!!!
    catnum = torch.FloatTensor(catnum)
    print(catnum) # -&gt; Standard output B
    return catnum
</code></pre>
<p>I run it two times and got same standard outputs.<br>
This is the standard outputs of this code.</p>
<pre><code class="lang-python"># Standard output A
[5.1066736e+07 1.0025117e+08 1.5670813e+07 8.5966000e+05 1.2038700e+05]

# Standard output B
tensor([   3.2892,    1.6755,   10.7186,  195.3898, 1395.2401])
</code></pre>
<p>If there is anything that you noticed about that please point them out for me.</p>
<p>Best regards,<br>
Ryo．</p>

username: Kushaj
cooked: <p>I don’t think the issue is due to this function. I got the same output when running this function. You can make the function shorter.</p>
<pre><code class="lang-python">def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
    catnum = np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num], dtype=np.float32)
    catnum = (catnum.sum()/catnum).round(5)
    catnum = torch.from_numpy(catnum)
    return catnum
</code></pre>