Repository: CitrineInformatics/lolo (language: python)

Issue:
Bugfix/rescale ratios
When training a bagged model, if `uncertaintyCalibration` is true then a rescaling ratio is calculated. This ratio is used to rescale the bootstrap standard deviation into a reasonable estimate of the observational standard deviation. It is calculated by computing the out-of-bag error for each prediction, dividing by the bootstrap standard deviation, ordering, and taking the 68-percentile value. Assuming the out-of-bag errors are a reasonable estimate of the error on unknown points, then this means that, post-rescaling, approximately 68% of observations will lie within 1 error bar.

If `uncertaintyCalibration` is false then the ratio is 1.0, and the observational standard deviation is equal to the tree standard deviation, which is known to overestimate the uncertainty.

These rescale ratios were not being passed through when training a multitask random forest. This PR fixes that bug.


Discussion / hints:
I think we have an inconsistency in our defaults.

When getting uncertainty, the [default](https://github.com/CitrineInformatics/lolo/blob/main/src/main/scala/io/citrine/lolo/PredictionResult.scala#L24) is to get the observational uncertainty, which for a bagged model means calculating the bootstrap standard deviation and re-scaling.

But the default for [RandomForest](https://github.com/CitrineInformatics/lolo/blob/main/src/main/scala/io/citrine/lolo/learners/RandomForest.scala#L41) is to set `uncertaintyCalibration = false`, which means that the rescaling ratio isn't calculated and users are getting the uncorrected bootstrap standard deviation. This is also the default for [Bagger](https://github.com/CitrineInformatics/lolo/blob/main/src/main/scala/io/citrine/lolo/bags/Bagger.scala#L31) and [MultiTaskBagger](https://github.com/CitrineInformatics/lolo/blob/main/src/main/scala/io/citrine/lolo/bags/MultiTaskBagger.scala#L26).

I think that these defaults should be trained so that `uncertaintyCalibration = true`.

Fix patch:
diff --git a/src/main/scala/io/citrine/lolo/bags/MultiTaskBagger.scala b/src/main/scala/io/citrine/lolo/bags/MultiTaskBagger.scala
index 79a55dd4..7e34a4ae 100644
--- a/src/main/scala/io/citrine/lolo/bags/MultiTaskBagger.scala
+++ b/src/main/scala/io/citrine/lolo/bags/MultiTaskBagger.scala
@@ -113,7 +113,7 @@ class MultiTaskBaggedTrainingResult(
                                      trainingWeights: Seq[Double]
                                    ) extends MultiTaskTrainingResult {
 
-  lazy val model = new MultiTaskBaggedModel(models, Nib, useJackknife, biasModels, trainingLabels, trainingWeights)
+  lazy val model = new MultiTaskBaggedModel(models, Nib, useJackknife, biasModels, rescaleRatios, trainingLabels, trainingWeights)
 
   // Each entry is a tuple, (feature vector, seq of predicted labels, seq of actual labels).
   // The labels are of type Option[Any] because a given training datum might not have a value for every single label.
@@ -196,6 +196,7 @@ class MultiTaskBaggedModel(
                             Nib: Vector[Vector[Int]],
                             useJackknife: Boolean,
                             biasModels: Seq[Option[Model[PredictionResult[Double]]]],
+                            rescaleRatios: Seq[Double],
                             trainingLabels: Seq[Seq[Any]],
                             trainingWeights: Seq[Double]
                           ) extends MultiTaskModel {
@@ -203,9 +204,9 @@ class MultiTaskBaggedModel(
   lazy val groupedModels: Seq[BaggedModel[Any]] = Seq.tabulate(numLabels) { i =>
     val thisLabelsModels = models.map(_.getModels(i))
     if (getRealLabels(i)) {
-      new BaggedModel[Double](thisLabelsModels.asInstanceOf[ParSeq[Model[PredictionResult[Double]]]], Nib, useJackknife, biasModels(i))
+      new BaggedModel[Double](thisLabelsModels.asInstanceOf[ParSeq[Model[PredictionResult[Double]]]], Nib, useJackknife, biasModels(i), rescaleRatios(i))
     } else {
-      new BaggedModel(thisLabelsModels, Nib, useJackknife, biasModels(i))
+      new BaggedModel(thisLabelsModels, Nib, useJackknife, biasModels(i), rescaleRatios(i))
     }
   }
 


Test patch:
diff --git a/src/test/scala/io/citrine/lolo/bags/MultiTaskBaggerTest.scala b/src/test/scala/io/citrine/lolo/bags/MultiTaskBaggerTest.scala
index 139db001..a27e39a9 100644
--- a/src/test/scala/io/citrine/lolo/bags/MultiTaskBaggerTest.scala
+++ b/src/test/scala/io/citrine/lolo/bags/MultiTaskBaggerTest.scala
@@ -33,7 +33,7 @@ class MultiTaskBaggerTest {
     val inputs = trainingData.map(_._1)
     val labels = trainingData.map(_._2)
     val DTLearner = MultiTaskTreeLearner()
-    val baggedLearner = MultiTaskBagger(DTLearner, numBags = trainingData.size, randBasis = TestUtils.getBreezeRandBasis(10478L))
+    val baggedLearner = MultiTaskBagger(DTLearner, numBags = trainingData.size, randBasis = TestUtils.getBreezeRandBasis(10478L), uncertaintyCalibration = true)
     val RFMeta = baggedLearner.train(inputs, Seq(labels))
     val RF = RFMeta.getModels().head
 
@@ -43,6 +43,7 @@ class MultiTaskBaggerTest {
     val sigma: Seq[Double] = results.getUncertainty().get.asInstanceOf[Seq[Double]]
     assert(sigma.forall(_ >= 0.0))
 
+    assert(results.asInstanceOf[MultiPredictionBaggedResult].rescale != 1.0, "uncertainty calibration ratio was not included in prediction result")
     assert(results.getGradient().isEmpty, "Returned a gradient when there shouldn't be one")
     assert(RFMeta.getLoss().get < 1.0, "Loss of bagger is larger than expected")
   }
@@ -183,7 +184,8 @@ class MultiTaskBaggerTest {
       learner,
       numBags = numBags,
       biasLearner = Some(RegressionTreeLearner(maxDepth = 2)),
-      randBasis = TestUtils.getBreezeRandBasis(78495L)
+      randBasis = TestUtils.getBreezeRandBasis(78495L),
+      uncertaintyCalibration = true
     )
     val RF = baggedLearner.train(inputs, Seq(realLabel, catLabel))
 
@@ -191,6 +193,9 @@ class MultiTaskBaggerTest {
     val predictionResult = RF.getModel().transform(testInputs)
     assert(predictionResult.predictions.length == numBags)
 
+    // because the uncertainty is recalibrated, the prediction result should have a rescale value that is not equal to 1.0
+    assert(predictionResult.baggedPredictions.head.asInstanceOf[MultiPredictionBaggedResult].rescale != 1.0)
+
     // The prediction made by the full model and the prediction made by just the categorical model should agree
     // and both be equal to the training label.
     val expected = predictionResult.getExpected()