Repository: lightly-ai/lightly

Issue:
AL detection scorer fails for object detection
## Error
![image](https://user-images.githubusercontent.com/1522766/116398967-ad9f7980-a828-11eb-8583-135e29b8e7b4.png)
## Reproduction:
`np.partition([[1.]],-2, axis=1)` yields `ValueError: kth(=-1) out of bounds (1)`
## Possible cause of error:
https://github.com/lightly-ai/lightly/blob/b064ab7f077867ab1fe4ca1a3977090701856c26/lightly/active_learning/utils/object_detection_output.py#L1[…]27
tries to do a one-hot-encoding without knowing the number of classes. Thus if max(labels) is only 1, only one class is encoded and the probability vector only has a length of 1. Thus partitioning it into 2 fails.

## Possible Solutions:
### 1. Prevent the error cause
Create the class_probabilities vector correctly with a given number of classes.
### 2. Handle the error (MUST DO, We all agree on it)
In the ScorerClassification handle the case where there is only a single class
### 3. General solution
The class_probabilities do not make sense at all if the ObjectDetectionOutput is created from scores. Thus we could also not create the `class_probabilities` in this case and not compute any classification scores if the `class_probabilities` are unset.

Solution 1 and 3 are exclusive, solution 2 can be combined with the other solutions.

## How to work on this issue
- First write a unittest reproducing the error by using ClassificationScorer.calculate_scores()
- Next fix this unittest with solution 2.
- Do not work on Solution 3 yet (another issue)


Discussion:
Let's do solution 2 for sure. There's no reason not to safeguard this.

As for the other two I think both solutions are fine. What are your favorites @MalteEbner and @IgorSusmelj ?
I'm also in favor of resolving this in the `ScorerClassification` (option 2)
I am for solutions 2+3. It simply does not make sense to calculate scores which will always be 0 (for all three uncertainty scorers).
Ok for me, @IgorSusmelj ?

Problem is then that we have scores which work for some cases and not for others.
Let's, for now, make sure we can compute scores for all cases. Thus option 2. Other object detection frameworks give class_probabilities. I can have a look at whether I can update the tutorial without a lot of modifications to get them as well.
> Problem is then that we have scores which work for some cases and not for others.

We have that problem anyway, as scores that are always 0 work in the sense of no errors but not in the sense of sampling better. And I prefer getting a clear error to doing a sampling and only finding out much later that it did not work that well because I did not provide class probabilities. Better fail with an error than silently.

Let's do 2) now and 3) next sprint, as @IgorSusmelj suggested. This whole thing got really complicated with the introduction of the classification scores for object detection. Let's simplify quickly.

Resolving patch:
diff --git a/lightly/active_learning/scorers/classification.py b/lightly/active_learning/scorers/classification.py
--- a/lightly/active_learning/scorers/classification.py
+++ b/lightly/active_learning/scorers/classification.py
@@ -1,3 +1,4 @@
+import warnings
 from typing import *
 
 import numpy as np
@@ -93,9 +94,25 @@ class ScorerClassification(Scorer):
 
     """
     def __init__(self, model_output: Union[np.ndarray, List[List[float]]]):
-        if isinstance(model_output, List):
+        if not isinstance(model_output, np.ndarray):
             model_output = np.array(model_output)
-        super(ScorerClassification, self).__init__(model_output)
+
+        validated_model_output = self.ensure_valid_model_output(model_output)
+
+        super(ScorerClassification, self).__init__(validated_model_output)
+
+    def ensure_valid_model_output(self, model_output: np.ndarray) -> np.ndarray:
+        if len(model_output) == 0:
+            return model_output
+        if len(model_output.shape) != 2:
+            raise ValueError("ScorerClassification model_output must be a 2-dimensional array")
+        if model_output.shape[1] == 0:
+            raise ValueError("ScorerClassification model_output must not have an empty dimension 1")
+        if model_output.shape[1] == 1:
+            # assuming a binary classification problem with
+            # the model_output denoting the probability of the first class
+            model_output = np.concatenate([model_output, 1-model_output], axis=1)
+        return model_output
 
     @classmethod
     def score_names(cls) -> List[str]:


Test patch:
diff --git a/tests/active_learning/test_ScorerClassification.py b/tests/active_learning/test_ScorerClassification.py
--- a/tests/active_learning/test_ScorerClassification.py
+++ b/tests/active_learning/test_ScorerClassification.py
@@ -25,22 +25,70 @@ def test_score_calculation_random(self):
 
     def test_score_calculation_specific(self):
         model_output = [
-                    [0.7, 0.2, 0.1],
-                    [0.4, 0.5, 0.1]
-                ]
+            [0.7, 0.2, 0.1],
+            [0.4, 0.5, 0.1]
+        ]
         model_output = np.array(model_output)
         scorer = ScorerClassification(model_output)
         scores = scorer.calculate_scores()
 
-        self.assertListEqual(list(scores["uncertainty_least_confidence"]), [(1 - 0.7) / (1 - 1./3.), (1 - 0.5) / (1 - 1./3.)])
+        self.assertListEqual(list(scores["uncertainty_least_confidence"]),
+                             [(1 - 0.7) / (1 - 1. / 3.), (1 - 0.5) / (1 - 1. / 3.)])
         self.assertListEqual(list(scores["uncertainty_margin"]), [1 - (0.7 - 0.2), 1 - (0.5 - 0.4)])
-        for val1, val2 in zip(scores["uncertainty_entropy"], _entropy(model_output)/np.log2(3)):
-            self.assertAlmostEqual(val1, val2,places=8)
+        for val1, val2 in zip(scores["uncertainty_entropy"], _entropy(model_output) / np.log2(3)):
+            self.assertAlmostEqual(val1, val2, places=8)
 
+    def test_score_calculation_binary(self):
+        model_output = [
+            [0.7],
+            [0.4]
+        ]
+        model_output = np.array(model_output)
+        scorer = ScorerClassification(model_output)
+        scores = scorer.calculate_scores()
+
+        self.assertListEqual(list(scores["uncertainty_least_confidence"]),
+                             [(1 - 0.7) / (1 - 1. / 2.), (1 - 0.6) / (1 - 1. / 2.)])
+        self.assertListEqual(list(scores["uncertainty_margin"]), [1 - (0.7 - 0.3), 1 - (0.6 - 0.4)])
+        model_output = np.concatenate([model_output, 1-model_output], axis=1)
+        for val1, val2 in zip(scores["uncertainty_entropy"], _entropy(model_output) / np.log2(2)):
+            self.assertAlmostEqual(val1, val2, places=8)
 
     def test_scorer_classification_empty_model_output(self):
         scorer = ScorerClassification(model_output=[])
         scores = scorer.calculate_scores()
         self.assertEqual(set(scores.keys()), set(ScorerClassification.score_names()))
 
+    def test_scorer_classification_variable_model_output_dimension(self):
+
+        for num_samples in range(5):
+            for num_classes in range(5):
+
+                with self.subTest(msg=f"model_output.shape = ({num_samples},{num_classes})"):
+                    if num_samples > 0:
+                        preds = [1. / num_samples] * num_classes
+                    else:
+                        preds = []
+                    model_output = [preds] * num_samples
+
+                    if num_classes == 0 and num_samples > 0:
+                        with self.assertRaises(ValueError):
+                            scorer = ScorerClassification(model_output=model_output)
+                    else:
+                        scorer = ScorerClassification(model_output=model_output)
+                        scores = scorer.calculate_scores()
+                        self.assertEqual(set(scores.keys()), set(ScorerClassification.score_names()))
+                        for score_values in scores.values():
+                            self.assertEqual(len(score_values), len(model_output))
+
+    def test_scorer_classification_variable_model_output_tensor_order(self):
 
+        for tensor_order in range(1, 5):
+            model_output = np.ndarray((3,) * tensor_order)
+            with self.subTest(msg=f"model_output.shape = {model_output.shape}"):
+                if tensor_order == 2 or tensor_order == 0:
+                    scorer = ScorerClassification(model_output=model_output)
+                    scores = scorer.calculate_scores()
+                else:
+                    with self.assertRaises(ValueError):
+                        scorer = ScorerClassification(model_output=model_output)