How to get predict interval using Monte Carlo Dropout

username: Takumi3
cooked: <p>Hi. I am trying to get standard deviation of predict value to describe predict interval using Monte Carlo Dropout. I can run my code without any problems. However I got standard deviation which all values are zero. I think there  are problems in dataset or algorithm. So give me some advices to fix my code.<br>
My code is here.</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    def __init__(self, x1, x2, p, m, mode=0):
        super().__init__()
        input_dim = p*m 
        hidden_dim = 128
        self.dropout = nn.Dropout(p=0.1)
        if mode == 1:
            self.embed = Encoder1(x1, x2, p)
        elif mode == 2:
            self.embed = Encoder2(x1, x2, p)
        else:
            self.embed = Encoder(x1, x2, p)

        self.share_block = nn.Sequential(
            # nn.BatchNorm1d(input_dim),
            nn.utils.weight_norm(nn.Linear(input_dim, hidden_dim)),
            # nn.Dropout(p=0.1),
            nn.SELU(),
            # nn.BatchNorm1d(hidden_dim),
            nn.utils.weight_norm(nn.Linear(hidden_dim, hidden_dim)),
            # nn.Dropout(p=0.1),
            nn.SELU(),
            # nn.BatchNorm1d(hidden_dim),
            nn.utils.weight_norm(nn.Linear(hidden_dim, hidden_dim)),
            # nn.Dropout(p=0.1),
            nn.SELU(),
            # nn.BatchNorm1d(hidden_dim),
            nn.utils.weight_norm(nn.Linear(hidden_dim, hidden_dim)),
            # nn.Dropout(p=0.1),
            nn.SELU()
        )
        self.head = nn.Sequential(
            # nn.BatchNorm1d(hidden_dim),
            nn.utils.weight_norm(nn.Linear(hidden_dim, 1))
        )

    def forward(self, x1, x2):
        x1 = x1.to(device)
        x2 = x2.to(device)
        x = self.embed(x1, x2)
        x = self.share_block(x)
        _out = self.head(x)
        _out = self.dropout(_out)
       

        return _out

    def predict(self, x1, x2, alpha, samples=100):
        x1 = x1.to(device)
        x2 = x2.to(device)
        # z_c = norm.ppf(alpha)
        samples_ = []
        for i in range(samples):
            pred = self.forward(x1, x2).detach()
            samples_ += pred
            
        # print(len(samples_))
        pred_sample = torch.cat(samples_, dim=-1)
        pred_sample = torch.reshape(pred_sample, (100, -1))
        # print(pred_sample.shape)
        pred_mean = torch.mean(pred_sample, dim=0)
        print(pred_mean)
        pred_std = np.sqrt(torch.var(pred_sample, dim=0))

        return pred_mean, pred_std

def train_and_test_gpu(model, num_epochs, dataset):
    # model.cuda()
    print(model)
    train_dataset = utils.data.Subset(dataset, train_idx)
    train_loader = utils.data.DataLoader(
        train_dataset, batch_size=64, worker_init_fn=seed_worker, generator=g, shuffle=True)

    test_dataset = utils.data.Subset(dataset, test_idx)
    test_loader = utils.data.DataLoader(
        test_dataset, batch_size=64, shuffle=False)

    optimizer = optim.Adam(model.parameters(), lr=1e-6)
    criterion = nn.MSELoss()

    # このschedulerはかなり特殊でscheduler.step()を呼び出す位置も違うので注意
    scheduler = optim.lr_scheduler.OneCycleLR(
        optimizer,
        max_lr=1e-3,
        steps_per_epoch=len(train_loader),
        epochs=num_epochs,
    )

    model.train()
    for epoch in tqdm(range(num_epochs)):
        for x1, x2, y in train_loader:
            # print(x1.shape, x2.shape)

            x1 = x1.to(device)
            x2 = x2.to(device)
            y = torch.Tensor(y).to(device)

            # print(x1.is_cuda)
            # print(x2.is_cuda)
            # print(y.is_cuda)
            optimizer.zero_grad()
            y_pred = model(x1, x2)

            # y_pred = torch.take_along_dim(y_pred, task)
            loss = criterion(y_pred, y.unsqueeze(1))
            loss.backward()
            optimizer.step()
            scheduler.step()

    model.eval()
    with torch.no_grad():
        y_true_list = []
        y_pred_list = []
        pred_std_list = []
        for x1, x2, y_true in test_loader:
            x1 = torch.Tensor(x1).to(device)
            x2 = torch.Tensor(x2).to(device)

            y_pred, pred_std = model.predict(x1, x2, alpha=0.95)
            # print(pred_std)
            # y_pred = torch.take_along_dim(y_pred, task)
            y_pred_list.extend(y_pred.detach().flatten().cpu().tolist())
            y_true_list.extend(y_true.detach().cpu().tolist())
            pred_std_list.extend(pred_std.detach().cpu().tolist())
        # print(y_pred_list)
    print(np.array(y_true_list).shape, np.array(y_pred_list).shape)
    print(np.array(pred_std_list).shape)
    plot_prediction(y_true_list, y_pred_list, color="red")
    return y_true_list, y_pred_list, pred_std_list
</code></pre>
<p>and output standard deviation is here.</p>
<pre><code class="lang-auto">[0.0,
 0.0,
 0.0,
 0.0,
 0.0,
・・・
 0.0]
</code></pre>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/takumi3">@Takumi3</a> The monte carlo sounds interesting. Can you share a link of the paper/blog etc</p>
<p>In your code, I can see an issue here</p>
<p>In the predict function</p>
<pre><code class="lang-auto">samples_ = []
for i in range(samples):
    pred = self.forward(x1, x2).detach()
    samples_ += pred
</code></pre>
<p>Although you are iterating over a range of 100,  you are performing the forward on x1,x2 without updating the weights. This will give you the same results for all the 100 iterations. You need to perform some weight updation like <code>.step()</code> etc</p>

username: Takumi3
cooked: <p>Thanks for your help. I can correctly run my code.</p>
<p>and I share blog and paper about Monte Calro dropout and predict, confidence interval.<br>
blogs:<br>
“What My Deep Model Doesn’t Know…”<br>
<a href="http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html" class="inline-onebox" rel="noopener nofollow ugc">What my deep model doesn't know... | Yarin Gal - Blog | Cambridge Machine Learning Group</a></p>
<p>" Prediction Intervals for Deep Learning Neural Networks"</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://machinelearningmastery.com/prediction-intervals-for-deep-learning-neural-networks/">
  <header class="source">
      <img src="https://machinelearningmastery.com/wp-content/uploads/2019/09/icon-16x16.png" class="site-icon" width="16" height="16">

      <a href="https://machinelearningmastery.com/prediction-intervals-for-deep-learning-neural-networks/" target="_blank" rel="noopener nofollow ugc" title="06:00PM - 21 February 2021">Machine Learning Mastery – 21 Feb 21</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/458;"><img src="https://machinelearningmastery.com/wp-content/uploads/2021/05/Prediction-Intervals-for-Deep-Learning-Neural-Networks.jpg" class="thumbnail" width="690" height="458"></div>

<h3><a href="https://machinelearningmastery.com/prediction-intervals-for-deep-learning-neural-networks/" target="_blank" rel="noopener nofollow ugc">Prediction Intervals for Deep Learning Neural Networks</a></h3>

  <p>Prediction intervals provide a measure of uncertainty for predictions on regression problems. For example, a 95% prediction interval indicates that […]</p>

  <p>
    <span class="label1">Est. reading time: 16 minutes</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>" Uncertainty in Neural Networks"</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://jonnylaw.rocks/posts/2021-08-16-mc-dropout-uncertainty/#ref-gal2016dropout">
  <header class="source">

      <a href="https://jonnylaw.rocks/posts/2021-08-16-mc-dropout-uncertainty/#ref-gal2016dropout" target="_blank" rel="noopener nofollow ugc">Bayesian Statistics and Functional Programming</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/424;"><img src="https://jonnylaw.rocks/posts/2021-08-16-mc-dropout-uncertainty/mc-dropout-uncertainty_files/figure-html5/unnamed-chunk-1-1.png" class="thumbnail" width="690" height="424"></div>

<h3><a href="https://jonnylaw.rocks/posts/2021-08-16-mc-dropout-uncertainty/#ref-gal2016dropout" target="_blank" rel="noopener nofollow ugc">Bayesian Statistics and Functional Programming: Uncertainty in Neural Networks</a></h3>

  <p>Using MC Dropout to get probability intervals for neural network predictions.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>paper:<br>
" High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach"</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1802.07167">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.2.8/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/1802.07167" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/1802.07167" target="_blank" rel="noopener nofollow ugc">High-Quality Prediction Intervals for Deep Learning: A Distribution-Free,...</a></h3>

  <p>This paper considers the generation of prediction intervals (PIs) by neural
networks for quantifying uncertainty in regression tasks. It is axiomatic that
high-quality PIs should be as narrow as possible, whilst capturing a specified
portion of data....</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: anantguptadbl
cooked: <p><a class="mention" href="/u/takumi3">@Takumi3</a> thanks for sharing your work. It helps in keeping the community vibrant</p>