Using network to do classification on mixture of gaussian, weird behaviour

ElleryL: <p>Hey; I construct a very simple classification model to classify mixture of gaussian.</p>
<p>In this case, bivariate Gaussian. The data close to mode one has label 0 and data close to mode two has label 1.</p>
<p>Here is how I generate train samples</p>
<pre><code class="lang-auto">from torch.distributions.multivariate_normal import MultivariateNormal
m1 = MultivariateNormal(torch.zeros(2) + 300,torch.eye(2) * .01)
m2 = MultivariateNormal(torch.zeros(2) + 200.,torch.eye(2) * .01)

x1 = m1.sample((1000,)) # mode 1 
x2 = m2.sample((1000,)) # mode 2

c1 = torch.zeros(1000) # labels for mode 1
c2 = torch.ones(1000) # labels for mode 2


x = torch.cat([x1,x2],dim=0)
c = torch.cat([c1,c2],dim=0).view(-1,1)
</code></pre>
<p>The train sample look like this</p>
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAE8NJREFUeJzt3X+MXWWdx/H3d+lASUAqMCh0xp0iP8KPsIAF67LqLqjVrgH/cLHGRRRjs9goVURBEldNTKpsYDXuatjURLIEqFIBV35YEd3wB+2WWlpprVTB7bS41ioVw8/W7/5xn4FLudN7p3Ond+bp+5Xc3HOf85wz3yen/cy5z5x7T2QmkqR6/UWvC5AkTSyDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klS5ab0uAODII4/MoaGhXpchSVPKgw8++LvM7G/Xb1IE/dDQEKtWrep1GZI0pUTErzvp59SNJFXOoJekyhn0klS5STFH38rzzz/P8PAwzzzzTK9L2aPp06czMDBAX19fr0uRpJYmbdAPDw9z6KGHMjQ0RET0upyWMpPt27czPDzMrFmzel2OJLXUNugjYjrw38BBpf93MvOfI2IWcDNwOLAauCgzn4uIg4AbgNcB24H3ZOZjYy3smWeemdQhDxARHHHEEWzbtq3XpUiaIo676vvsbHG/p399z+m864yZE/IzO5mjfxY4NzP/CjgdeHtEzAG+BFyXmccDfwA+VPp/CPhDZh4HXFf67ZXJHPIjpkKNkiaH0UIeYNEta7jtp1sm5Oe2Dfps+FN52VceCZwLfKe0fwt4V1m+oLymrD8vTENJGjXkRyy6Zc2E/NyOrrqJiAMiYg3wW2A58EvgiczcWboMAyPvOWYCmwHK+h3AEd0sel+55JJLOOqoozj11FN7XYok7bWOgj4zd2Xm6cAAcDZwUqtu5bnV2fvLfo9FxIKIWBURqybrHPcHPvAB7r777l6XIUnjMqbr6DPzCeDHwBxgRkSM/DF3ANhaloeBQYCy/jDg9y32dX1mzs7M2f39bb+qoa3bfrqFcxb/iFlXfp9zFv+oK3Ndb3rTmzj88MPHvR9J6qW2QR8R/RExoywfDLwF2ADcB7y7dLsYuL0s31FeU9b/KDPbzEyNz20/3cJVy9ax5YmnSWDLE09z1bJ1E/aHDUmaSjo5oz8auC8i1gL/AyzPzP8CPg18IiI20ZiDX1L6LwGOKO2fAK7sftkvdc09G3n6+V0vaXv6+V1cc8/Gif7RktSxxxb//bjW762219Fn5lrgjBbtv6IxX797+zPAP3Slug5tfeLpMbVLUq9MVJjvSRXfdXPMjIPH1C5J+5Mqgv6KuSdycN8BL2k7uO8Arph74rj2+973vpc3vOENbNy4kYGBAZYsWdJ+I0maZCbtd92MxcjHhq+5ZyNbn3iaY2YczBVzTxz3x4lvuummbpQnST1VRdBDI+wn6nsiJGkqq2LqRpI0OoNekipn0EtS5Qx6SaqcQS9JlTPo27j77rs58cQTOe6441i8eHGvy5GkMTPo92DXrl0sXLiQu+66i/Xr13PTTTexfv36XpclSWNST9CvXQrXnQqfm9F4Xrt03LtcuXIlxx13HMceeywHHngg8+fP5/bbb2+/oSRNInUE/dql8L2PwY7NQDaev/excYf9li1bGBwcfOH1wMAAW7b41ceSppY6gv7eL8Dzu31T5fNPN9rHodXX6Hv7W0lTTR1Bv2N4bO0dGhgYYPPmzS+8Hh4e5phjjhnXPiVpX6sj6A8bGFt7h8466yweeeQRHn30UZ577jluvvlmzj///HHtU5L2tTqC/rzPQt9u3z3fd3CjfRymTZvG1772NebOnctJJ53EhRdeyCmnnDKufUrSvlbHt1eedmHj+d4vNKZrDhtohPxI+zjMmzePefPmjXs/ktQrdQQ9NEK9C8EuSbWpY+pGkjSqSR30rS5vnGymQo2S9m+TNuinT5/O9u3bJ3WQZibbt29n+vTpvS5FkkY1aefoBwYGGB4eZtu2bb0uZY+mT5/OwMD4LuOUpIk0aYO+r6+PWbNm9boMSZryJu3UjSSpOwx6SaqcQS9JlTPoJalyBr0kVc6gl6TKtQ36iBiMiPsiYkNEPBwRl5X20yPigYhYExGrIuLs0h4R8dWI2BQRayPizIkehCRpdJ1cR78TuDwzV0fEocCDEbEc+DLw+cy8KyLmldd/C7wDOL48Xg98vTxLknqg7Rl9Zj6emavL8pPABmAmkMArSrfDgK1l+QLghmx4AJgREUd3vXJJUkfG9MnYiBgCzgBWAIuAeyLiX2j8wvjr0m0msLlps+HS9vg4a5Uk7YWO/xgbEYcAtwKLMvOPwKXAxzNzEPg4sGSka4vNX/bNZBGxoMztr5rs32cjSVNZR0EfEX00Qv7GzFxWmi8GRpa/DZxdloeBwabNB3hxWucFmXl9Zs7OzNn9/f17U7skqQOdXHUTNM7WN2TmtU2rtgJvLsvnAo+U5TuA95erb+YAOzLTaRtJ6pFO5ujPAS4C1kXEmtL2GeDDwFciYhrwDLCgrLsTmAdsAp4CPtjViiVJY9I26DPzflrPuwO8rkX/BBaOsy5JUpf4yVhJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekirXNugjYjAi7ouIDRHxcERc1rTuoxGxsbR/uan9qojYVNbNnajiJUntTeugz07g8sxcHRGHAg9GxHLgVcAFwGmZ+WxEHAUQEScD84FTgGOAH0bECZm5a2KGIEnak7Zn9Jn5eGauLstPAhuAmcClwOLMfLas+23Z5ALg5sx8NjMfBTYBZ09E8ZKk9sY0Rx8RQ8AZwArgBOCNEbEiIn4SEWeVbjOBzU2bDZe23fe1ICJWRcSqbdu27U3tkqQOdBz0EXEIcCuwKDP/SGPa55XAHOAKYGlEBBAtNs+XNWRen5mzM3N2f3//XhUvSWqvo6CPiD4aIX9jZi4rzcPAsmxYCfwZOLK0DzZtPgBs7V7JkqSx6OSqmwCWABsy89qmVbcB55Y+JwAHAr8D7gDmR8RBETELOB5Y2e3CJUmd6eSqm3OAi4B1EbGmtH0G+CbwzYj4GfAccHFmJvBwRCwF1tO4YmehV9xIUu+0DfrMvJ/W8+4A/zjKNl8EvjiOuiRJXeInYyWpcga9JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klQ5g16SKmfQS1Ll2gZ9RAxGxH0RsSEiHo6Iy3Zb/8mIyIg4sryOiPhqRGyKiLURceZEFS9Jam9aB312Apdn5uqIOBR4MCKWZ+b6iBgE3gr8b1P/dwDHl8frga+XZ0lSD7Q9o8/MxzNzdVl+EtgAzCyrrwM+BWTTJhcAN2TDA8CMiDi6u2VLkjo1pjn6iBgCzgBWRMT5wJbMfGi3bjOBzU2vh3nxF4MkaR/rZOoGgIg4BLgVWERjOudq4G2turZoy5d1ilgALAB4zWte02kZkqQx6uiMPiL6aIT8jZm5DHgtMAt4KCIeAwaA1RHxahpn8INNmw8AW3ffZ2Zen5mzM3N2f3//+EYhSRpVJ1fdBLAE2JCZ1wJk5rrMPCozhzJziEa4n5mZvwHuAN5frr6ZA+zIzMcnbgiSpD3pZOrmHOAiYF1ErCltn8nMO0fpfycwD9gEPAV8cNxVSpL2Wtugz8z7aT3v3txnqGk5gYXjrkyS1BV+MlaSKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVrm3QR8RgRNwXERsi4uGIuKy0XxMRP4+ItRHx3YiY0bTNVRGxKSI2RsTciRyAJGnPOjmj3wlcnpknAXOAhRFxMrAcODUzTwN+AVwFUNbNB04B3g78e0QcMBHFS5Laaxv0mfl4Zq4uy08CG4CZmfmDzNxZuj0ADJTlC4CbM/PZzHwU2ASc3f3SJUmdGNMcfUQMAWcAK3ZbdQlwV1meCWxuWjdc2iRJPdBx0EfEIcCtwKLM/GNT+9U0pnduHGlqsXm22N+CiFgVEau2bds2tqolSR3rKOgjoo9GyN+Ymcua2i8G3gm8LzNHwnwYGGzafADYuvs+M/P6zJydmbP7+/v3tn5JUhudXHUTwBJgQ2Ze29T+duDTwPmZ+VTTJncA8yPioIiYBRwPrOxu2ZKkTk3roM85wEXAuohYU9o+A3wVOAhY3vhdwAOZ+U+Z+XBELAXW05jSWZiZu7pfuiSpE22DPjPvp/W8+5172OaLwBfHUZckqUv8ZKwkVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZWb1q5DRAwCNwCvBv4MXJ+ZX4mIw4FbgCHgMeDCzPxDRATwFWAe8BTwgcxc3fXK1y6FZR9uve5zO7r+4yRpqurkjH4ncHlmngTMARZGxMnAlcC9mXk8cG95DfAO4PjyWAB8vetV7ynkAT53WNd/pCRNVW2DPjMfHzkjz8wngQ3ATOAC4Ful27eAd5XlC4AbsuEBYEZEHN3Vqu/9Qld3J0k1G9McfUQMAWcAK4BXZebj0PhlABxVus0ENjdtNlzadt/XgohYFRGrtm3bNraqdwyPrb8k7cc6DvqIOAS4FViUmX/cU9cWbfmyhszrM3N2Zs7u7+/vtIyGwwbG1l+S9mMdBX1E9NEI+Rszc1lp/r+RKZny/NvSPgwMNm0+AGztTrnFeZ/t6u4kqWZtg75cRbME2JCZ1zatugO4uCxfDNze1P7+aJgD7BiZ4uma0y7s6u4kqWadnNGfA1wEnBsRa8pjHrAYeGtEPAK8tbwGuBP4FbAJ+A/gI90vmz1fQunllZL0grbX0Wfm/bSedwc4r0X/BBaOs67OGOiS1JafjJWkyhn0klQ5g16SKmfQS1LlDHpJqlw0LpLpcRER24Bfj3M3RwK/60I5U4Xjrd/+Nub9bbww/jH/ZWa2/WqBSRH03RARqzJzdq/r2Fccb/32tzHvb+OFfTdmp24kqXIGvSRVrqagv77XBexjjrd++9uY97fxwj4aczVz9JKk1mo6o5cktTAlgj4iBiPivojYEBEPR8Rlpf3wiFgeEY+U51eW9oiIr0bEpohYGxFn9nYEY7OH8V4TET8vY/puRMxo2uaqMt6NETG3d9XvndHG3LT+kxGREXFkeV3lMS7rPlqO48MR8eWm9iqPcUScHhEPlG/GXRURZ5f2qX6Mp0fEyoh4qIz386V9VkSsKLl1S0QcWNoPKq83lfVDXSsmMyf9AzgaOLMsHwr8AjgZ+DJwZWm/EvhSWZ4H3EXjWzfnACt6PYYujfdtwLTS/qWm8Z4MPAQcBMwCfgkc0OtxdGPM5fUgcA+Nz1ocWfkx/jvgh8BBZd1RtR9j4AfAO5qO648rOcYBHFKW+2jcgnUOsBSYX9q/AVxalj8CfKMszwdu6VYtU+KMPifjDcon0GjjzcwfZObO0u0BGnfvgsZ4b87MZzPzURr3Ajh7X9c9Hns4xgDXAZ/ipbekrPIYA5cCizPz2bJu5M5tNR/jBF5Ruh3Gi3ekm+rHODPzT+VlX3kkcC7wndK+e26N5Nl3gPPKjZ/GbUoEfbNu3qB8KthtvM0uoXG2AxWNF1465og4H9iSmQ/t1q2aMe92jE8A3ljeuv8kIs4q3aoZL7xszIuAayJiM/AvwFWl25Qfc0QcEBFraNxqdTmNd2JPNJ2wNY/phfGW9TuAI7pRx5QK+ujyDconu9HGGxFXAzuBG0eaWmw+5cYLLx0zjTFeDbS6SXAVY25xjKcBr6TxFv8KYGk5q6tivNByzJcCH8/MQeDjNG5dChWMOTN3ZebpNN59nw2c1KpbeZ6w8U6ZoI/JdoPyCTbKeImIi4F3Au/LMplHBeOFlmN+LY356Ici4jEa41odEa+mgjGPcoyHgWXlbf9K4M80vg9lyo8XRh3zxcDI8rd5cUqqijEDZOYTwI9p/AKfEREjd/drHtML4y3rDwN+342fPyWCvpzRTK4blE+g0cYbEW8HPg2cn5lPNW1yBzC//NV+FnA8sHJf1jxercacmesy86jMHMrMIRr/Ec7MzN9Q6TEGbqMxh0tEnAAcSONLr6o8xsVW4M1l+VzgkbI81Y9xf5Qr4yLiYOAtNP4ucR/w7tJt99waybN3Az9qOpkbn4n6i3M3H8Df0HgLsxZYUx7zaMxf3UvjH8a9wOH54l+7/43GfNg6YHavx9Cl8W6iMYc30vaNpm2uLuPdSLmCYSo9Rhvzbn0e48Wrbmo9xgcC/wn8DFgNnFv7MS7tD9K4qmgF8LpKjvFpwE/LeH8GfLa0H0vjl/QmGu9gRq6wml5ebyrrj+1WLX4yVpIqNyWmbiRJe8+gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcv8PBOEM4Sk0rtAAAAAASUVORK5CYII=" alt></p>
<p>Now I construct a simple classifier</p>
<pre><code class="lang-auto">class Classifier(nn.Module):
    def __init__(self,num_in_dim=2,num_hidden=100):
        super(Classifier, self).__init__()

        self.fc1 = nn.Sequential(
        nn.Linear(num_in_dim, num_hidden),
        nn.ReLU(inplace=True),
        nn.Linear(num_hidden, 1),
       nn.Sigmoid())


    def forward(self,x):
        return self.fc1(x)
</code></pre>
<p>Now I set up my training as</p>
<pre><code class="lang-auto">net = Classifier()
optimizer = optim.Adam(net.parameters(), lr=1e-3)
criterion = nn.BCELoss()
for i in range(100):
    optimizer.zero_grad()
    a = net(x)
    loss = criterion(a, c)
    loss.backward()
    optimizer.step()
    if i % 100 == 0:
        print(loss.item())
</code></pre>
<p>The weird behaviour I observe is that. Here are the train loss. I use the same set of batch samples train the same network with only initial parameters different. Each line is a one initialization of network. We see that sometimes the network doesn’t learn at all but some times it works really well. Though, there are many theory on optimization about local minimal etc. But I think example like this is too trivia and same thing happens even I use linear neural network.</p>
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEpNJREFUeJzt3X+QXWV9x/H3d3fDb2kCWTAmYIChKuoQnG0apbUUtUXUgh07lelY/qATO6MVW8cW7R/KTDujU5Xq1DITBUk7CFrEklL8kUEsY8eiGwwxISiiqJGUrIPIr0qyu9/+cc/N3iz37r3ZvTd377Pv18yde85znpPzPZzwOec+OfeeyEwkSYNvqN8FSJK6w0CXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFWLkSG5s5cqVuXbt2iO5SUkaeNu2bft5Zo6263dEA33t2rWMj48fyU1K0sCLiB930s8hF0kqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCnFE70Ofr6v/Yxf3P/JEv8uQpHk75wUn8oE3vbSn2/AKXZIKMRBX6L0+q0lSCbxCl6RCGOiSVAgDXZIKYaBLUiHaBnpEHBMR34qI+yJiV0RcXbXfEBE/iojt1Wtd78uVJLXSyV0uzwIXZuZTEbEM+EZEfKla9t7MvKV35UmSOtU20DMzgaeq2WXVK3tZlCTp8HU0hh4RwxGxHdgHbM3Me6pFfx8ROyLimog4umdVSpLa6ijQM3MqM9cBa4D1EfEy4H3Ai4HfAE4C/qbZuhGxMSLGI2J8YmKiS2VLkmY7rLtcMvNx4OvARZm5N2ueBT4DrG+xzqbMHMvMsdHRts84lSTNUyd3uYxGxPJq+ljgtcADEbGqagvgUmBnLwuVJM2tk7tcVgGbI2KY2gng85l5e0R8LSJGgQC2A3/ewzolSW10cpfLDuC8Ju0X9qQiSdK8+E1RSSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmF6OQh0cdExLci4r6I2BURV1ftZ0TEPRHxYER8LiKO6n25kqRWOrlCfxa4MDPPBdYBF0XEBuDDwDWZeTbwC+CK3pUpSWqnbaBnzVPV7LLqlcCFwC1V+2bg0p5UKEnqSEdj6BExHBHbgX3AVuAh4PHMnKy67AFWt1h3Y0SMR8T4xMREN2qWJDXRUaBn5lRmrgPWAOuBlzTr1mLdTZk5lpljo6Oj869UkjSnw7rLJTMfB74ObACWR8RItWgN8Eh3S5MkHY5O7nIZjYjl1fSxwGuB3cBdwFuqbpcDt/WqSElSeyPtu7AK2BwRw9ROAJ/PzNsj4n7g5oj4O+A7wHU9rFOS1EbbQM/MHcB5Tdp/SG08XZK0CPhNUUkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhejkIdGnRcRdEbE7InZFxJVV+wcj4mcRsb16Xdz7ciVJrXTykOhJ4D2ZeW9EPA/YFhFbq2XXZOZHeleeJKlTnTwkei+wt5p+MiJ2A6t7XZgk6fAc1hh6RKwFzgPuqZreGRE7IuL6iFjRYp2NETEeEeMTExMLKlaS1FrHgR4RJwBfAN6dmU8A1wJnAeuoXcF/tNl6mbkpM8cyc2x0dLQLJUuSmuko0CNiGbUwvzEzbwXIzEczcyozp4FPAet7V6YkqZ1O7nIJ4Dpgd2Z+rKF9VUO3NwM7u1+eJKlTndzlcj7wNuC7EbG9ans/cFlErAMSeBh4e08qlCR1pJO7XL4BRJNFd3S/HEnSfPlNUUkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhejkIdGnRcRdEbE7InZFxJVV+0kRsTUiHqzeV/S+XElSK51coU8C78nMlwAbgHdExDnAVcCdmXk2cGc1L0nqk7aBnpl7M/PeavpJYDewGrgE2Fx12wxc2qsiJUntHdYYekSsBc4D7gFOzcy9UAt94JQW62yMiPGIGJ+YmFhYtZKkljoO9Ig4AfgC8O7MfKLT9TJzU2aOZebY6OjofGqUJHWgo0CPiGXUwvzGzLy1an40IlZVy1cB+3pToiSpE53c5RLAdcDuzPxYw6ItwOXV9OXAbd0vT5LUqZEO+pwPvA34bkRsr9reD3wI+HxEXAH8BPij3pQoSepE20DPzG8A0WLxa7pbjiRpvvymqCQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQnTykOjrI2JfROxsaPtgRPwsIrZXr4t7W6YkqZ1OrtBvAC5q0n5NZq6rXnd0tyxJ0uFqG+iZeTfw2BGoRZK0AAsZQ39nROyohmRWtOoUERsjYjwixicmJhawOUnSXOYb6NcCZwHrgL3AR1t1zMxNmTmWmWOjo6Pz3JwkqZ15BXpmPpqZU5k5DXwKWN/dsiRJh2tegR4Rqxpm3wzsbNVXknRkjLTrEBE3ARcAKyNiD/AB4IKIWAck8DDw9h7WKEnqQNtAz8zLmjRf14NaJEkL0DbQF4Pb3/WHLHtoT7/LkKR5O3DWGt74iVt7ug2/+i9JhRiIK/Ren9UkqQReoUtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklSItoEeEddHxL6I2NnQdlJEbI2IB6v3Fb0tU5LUTidX6DcAF81quwq4MzPPBu6s5iVJfdQ20DPzbuCxWc2XAJur6c3ApV2uS5J0mOY7hn5qZu4FqN5PadUxIjZGxHhEjE9MTMxzc5Kkdnr+j6KZuSkzxzJzbHR0tNebk6Qla76B/mhErAKo3vd1ryRJ0nzMN9C3AJdX05cDt3WnHEnSfHVy2+JNwDeBF0XEnoi4AvgQ8LqIeBB4XTUvSeqjkXYdMvOyFote0+VaJEkL4DdFJakQBrokFcJAl6RCGOiSVIjBCPTv3Ai3/1W/q5CkRW0wAv3RXXDfzf2uQpIWtcEI9ONXwoGnYf8z/a5EkhatAQn06jdgnvl5f+uQpEVsQAJ9Ze39aX+tUZJaGZBAr67Qn/YKXZJaGZBA9wpdktoZkECvX6Eb6JLUymAE+lHHw7LjHHKRpDkMRqBDbdjFK3RJammAAn3UK3RJmsOABbpX6JLUyuAE+nErvUKXpDm0fWLRXCLiYeBJYAqYzMyxbhTVVH0MPRMierYZSRpUCwr0yu9mZu8vnY8fhekD8KtfwrHLe745SRo0gzPk4rdFJWlOCw30BL4aEdsiYmM3CmrJb4tK0pwWOuRyfmY+EhGnAFsj4oHMvLuxQxX0GwFOP/30+W/Jb4tK0pwWdIWemY9U7/uALwLrm/TZlJljmTk2Ojo6/435E7qSNKd5B3pEHB8Rz6tPA78H7OxWYc9x3Mm1d8fQJamphQy5nAp8MWq3EI4An83ML3elqmZGjoJjfs0hF0lqYd6Bnpk/BM7tYi3t+W1RSWppcG5bBH/PRZLmMGCB7i8uSlIrAxboDrlIUiuDF+jPPAbTU/2uRJIWncELdLIW6pKkQwxWoB+8F91hF0mabbAC3a//S1JLBrokFWJAA9170SVptsEK9GNXQAx5hS5JTQxWoA8N1Z4t6i8uStJzDFagg1//l6QWBjDQT3bIRZKaGIhAn56a5snHflWb8ev/ktTUQAT6f332e9z6D9t4+vFna4H+1D6Ymux3WZK0qAxEoL/sd9bwq2cm+c9/3sH+52+A/U/Blr+A6el+lyZJi8ZABPro6c/j9//spfz8p0+y9ZtnMf3q98F9n4UvvRcy+12eJC0KC3kE3RG19uUr+e0//nXuvvn7fHXkDaw8aSVx13/D9/+JOHE1REBARNSmiZnpejv1+QCGGqap3d9+cHl9uqHPwfWHmvQZavHnDtVutZzdFn34Dyipr844dyUnnnxsT7exoECPiIuAjwPDwKcz80NdqaqFl1+whqcef5Z7v/JjHsozgTPhe93cQlYvh3IkddfyU49bvIEeEcPAJ4HXAXuAb0fElsy8v1vFNfPKS89i/ZvOgGlIEp6aICf3Q06T01Ug5zRMJ2SSOV2NtWfVZ+rQPmTt99Wz3j5VzdfWr/32+nTVf6r2Xv8zD/adbnhNHdq3/mc3rpOz1m18n56slk3Xtltff3qqyXZnbePg+/SsbTese/DPb9zmFDBrXwZJ/VPS0DAwXHuPgBiufUKKelu931BDv6GGftWfMbt/NP4ZwzAUh84/ZzvV+9AwDI3UXlGfbngfXjbTPrxsVvsyGB45dNnwSNVeX7asYdlRM33Dj4CL0bKjh3u+jYVcoa8HflA9LJqIuBm4BOhpoAMMDw/VPhMArHh+rze39GS2OFFMNW9r1Xd2e+PJqunJqMM/5+AJcbp537lOck1Phs36T870mWpX4/TMOrP/W01PwvSBI3uSHKqH/VGHBv/wsjna6/ONJ4rZ88O16YMnmeGZ9Q45WY00zA/POqENNczPOnkeMj88c/KNxuUN8xFNTrwNr0OGWJeGhQT6auCnDfN7gN9cWDlaFKLhClTdUf+0N31g5n1qcibwpxrbD9Tapw60nj9ken/D/GRt/nCmD/zfoes33Wa91mr5QJn9b12zA78+3dg3mkzPfh9qWKdqgxb9gTd9HF74qp7u6UICvdlp7zm3nETERmAjwOmnn76AzUkDLKJ2tTs8MPchtHbwE9zkTPAfcrKa5JBPK1MHZj7J1Jc1/YQzxyetQ+anqxpmtR8cSp0ZYp3pW/9U2DDM2rgv5Myy+jrPma6/02SdevTNbsuZ7Rx1Qs8PzUL+du0BTmuYXwM8MrtTZm4CNgGMjY15j6E06Bo/wY0c3e9q1GAh96F/Gzg7Is6IiKOAtwJbulOWJOlwzfsKPTMnI+KdwFeo/RPl9Zm5q2uVSZIOy4IG9DLzDuCOLtUiSVqAgfjqvySpPQNdkgphoEtSIQx0SSqEgS5JhYg8gr8nHhETwI/nufpKYCk+HXop7vdS3GdYmvu9FPcZDn+/X5iZo+06HdFAX4iIGM/MsX7XcaQtxf1eivsMS3O/l+I+Q+/22yEXSSqEgS5JhRikQN/U7wL6ZCnu91LcZ1ia+70U9xl6tN8DM4YuSZrbIF2hS5LmMBCBHhEXRcT3IuIHEXFVv+vphYg4LSLuiojdEbErIq6s2k+KiK0R8WD1vqLftXZbRAxHxHci4vZq/oyIuKfa589VP89clIhYHhG3RMQD1TF/ZenHOiL+svq7vTMiboqIY0o81hFxfUTsi4idDW1Nj23UfKLKth0R8YqFbHvRB3rDw6hfD5wDXBYR5/S3qp6YBN6TmS8BNgDvqPbzKuDOzDwbuLOaL82VwO6G+Q8D11T7/Avgir5U1VsfB76cmS8GzqW2/8Ue64hYDbwLGMvMl1H7ye23UuaxvgG4aFZbq2P7euDs6rURuHYhG170gU7Dw6gzcz9Qfxh1UTJzb2beW00/Se1/8NXU9nVz1W0zcGl/KuyNiFgDvAH4dDUfwIXALVWXEvf5RODVwHUAmbk/Mx+n8GNN7ee6j42IEeA4YC8FHuvMvBt4bFZzq2N7CfAvWfM/wPKIWDXfbQ9CoDd7GPXqPtVyRETEWuA84B7g1MzcC7XQB07pX2U98Y/AXwPT1fzJwOOZOVnNl3i8zwQmgM9UQ02fjojjKfhYZ+bPgI8AP6EW5L8EtlH+sa5rdWy7mm+DEOgdPYy6FBFxAvAF4N2Z+US/6+mliHgjsC8ztzU2N+la2vEeAV4BXJuZ5wFPU9DwSjPVmPElwBnAC4DjqQ03zFbasW6nq3/fByHQO3oYdQkiYhm1ML8xM2+tmh+tfwSr3vf1q74eOB/4g4h4mNpQ2oXUrtiXVx/LoczjvQfYk5n3VPO3UAv4ko/1a4EfZeZEZh4AbgVeRfnHuq7Vse1qvg1CoC+Jh1FXY8fXAbsz82MNi7YAl1fTlwO3HenaeiUz35eZazJzLbXj+rXM/BPgLuAtVbei9hkgM/8X+GlEvKhqeg1wPwUfa2pDLRsi4rjq73p9n4s+1g1aHdstwJ9Wd7tsAH5ZH5qZl8xc9C/gYuD7wEPA3/a7nh7t429R+6i1A9hevS6mNqZ8J/Bg9X5Sv2vt0f5fANxeTZ8JfAv4AfBvwNH9rq8H+7sOGK+O978DK0o/1sDVwAPATuBfgaNLPNbATdT+neAAtSvwK1odW2pDLp+ssu271O4Cmve2/aaoJBViEIZcJEkdMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSrE/wP/vFE8OwfZdQAAAABJRU5ErkJggg==" alt></p>

ptrblck: <p>I tried a few approaches using different weight init methods etc., but the main issue seems to be the loss in numerical precision using <code>sigmoid</code> + <code>nn.BCELoss</code>.<br>
If you remove the sigmoid in your model and use <code>nn.BCEWithLogitsLoss</code> as your criterion (which uses the LogSumExp trick for stability), your model seems to converge in all runs.</p>

ElleryL: <p>Thanks for the reply; so I guess that this is also true that using <code>nn.CrossEntropyLoss</code> is more numerically stable than using  <code>nn.LogSoftmax</code> +  <code>nn.NLLLoss</code> ?</p>

ptrblck: <p>No, internally <code>F.nll_loss(F.log_softmax)</code> will be used as seen in <a href="https://github.com/pytorch/pytorch/blob/5254b12002caf44a4df0ec56f6aea83877dd82f7/torch/nn/functional.py#L1993" rel="nofollow noopener">this line of code</a>.<br>
However, <code>F.log_softmax</code> is more numerically stable than <code>F.log(F.softmax)</code>.</p>

ElleryL: <p>I see. Thanks. So in the neural network with binary classification, <code>nn.BCEWithLogitsLoss</code> as my loss function and in my neural network has no activation in the last layers. In the test time, I just manually apply sigmoid function to my output from neural network</p>

ptrblck: <p>Yes, you could do that to e.g. apply a probability threshold to get the predicted class.</p>