Ensembling Predictions for combined Uncertainty-based Loss and Dice

username: mattj
cooked: <p>I’m trying to train multiple models in parallel, combine the predictions to produce an uncertainty estimate, then measure the quality of this estimate with Expected Calibration Error.</p>
<p>For each model, I want to then use the loss as the dice of its prediction + the ECE of the ensembled models’ predictions.</p>
<p>I’m not sure how the backprop process will work with this. I understand that you can simply combine losses then call <code>.backward()</code>, however if I combine multiple predictions, then run ECE on these predictions, is using the same calculate ECE fine to use in the sum with the specific model’s prediction?</p>
<p>To put it in psuedocode:</p>
<pre><code class="lang-python">for epoch in epochs:
  preds = predict_all_models(models)
  stacked_pred = stack_preds(preds)
  uncertainty = calc_entropy_uncertainty(stacked_pred)
  ece = calc_ece(stacked_pred, uncertainty, truth)
  for model in models:
    dice = dice(preds[model])
    combined_loss = dice + ece
    combined_loss.backward()
    optimisers[model].step()
</code></pre>
<p>Further, I’m not sure how <code>autograd</code> works - is there anything special I need to do to make <code>backward()</code> work, or can I just calculate the ECE using PyTorch functions and it’ll magically work?</p>

username: soulitzer
cooked: <p>As long as ECE is implemented with differentiable torch ops then yes the gradient should propagate properly through it. As long as you don’t do weird things like create new tensors, convert to numpy and back, etc, it should be fine.</p>

username: mattj
cooked: <p>The updated gradients will propagate properly through all models, despite the predictions being stacked to produce the ECE?</p>

username: soulitzer
cooked: <p>If the predictions are continuous &amp; you used torch.stack (rather than constructing a new tensor yourself), then yes.</p>

username: mattj
cooked: <p>I’m now getting a <code>RuntimeError: Trying to backward through the graph a second time</code>.</p>
<p>Am I right in thinking that because <code>stacked_pred</code> and <code>uncertainty</code> come from the same <code>preds</code> variable, when I call backward on the ECE, it causes the gradients to be propagated from both <code>stacked_pred</code> and <code>uncertainty</code> to the original predictions.</p>
<p>Therefore, since they’re both combined to produce ECE, am I okay to call <code>.detach()</code> on one of the values, solving the problem? Or would this cause the gradients from ECE to not properly backprop?</p>
<p>EDIT: Maybe not… maybe instead because I’m combining the dice and ECE which both come from the same prediction?</p>
<p>It’s probably worth mentioning that the <code>dice</code> in my original psuedocode is the dice for an individual model’s prediction.</p>
<p>EDIT 2: Thinking about it a bit more, I should probably do</p>
<pre><code class="lang-python">ece.backward(retain_graph=True)
for model in models:
    dice = dice(preds[model])
    dice.backward()
    optimisers[model].step()
</code></pre>
<p>otherwise I’ll be calling <code>.backward()</code> on the same loss function multiple times.</p>