Repository: airctic/icevision

Issue:
fastai efficientdet fails on learn.validate() with AttributeError: 'NoneType' object has no attribute 'shape'
## 🐛 Bug
when trying to simply validate metrics for an efficientdet model with fastai 
```python
KeyError: 'image_id'
```
```python
AttributeError: 'NoneType' object has no attribute 'shape'
```

it fails when trying to read the batch size automatically: in `accumulate, find_bs`
```python
class AvgLoss(Metric):
    "Average the losses taking into account potential different batch sizes"
    def reset(self):           self.total,self.count = 0.,0
    def accumulate(self, learn):
        bs = find_bs(learn.yb)
        self.total += learn.to_detach(learn.loss.mean())*bs
        self.count += bs
    @property
    def value(self): return self.total/self.count if self.count != 0 else None
    @property
    def name(self):  return "loss"
```
**To Reproduce**
Steps to reproduce the behavior:
1. Go to https://colab.research.google.com/drive/1i4aXYu4wIKA7eLUK86GwTm7lq7zku_oF?usp=sharing




Discussion:
I found that applying a following patch:

```python
from fastai.learner import AvgLoss
from fastai.torch_core import find_bs

@patch
def accumulate(self:AvgLoss, learn):
    #bs = find_bs(learn.yb)
    bs = find_bs(learn.xb)
    self.total += learn.to_detach(learn.loss.mean())*bs
    self.count += bs
```

fixes the issue. Im not sure though how should we store that patch in icevision. Anny suggestions?

a linked issue: https://forums.fast.ai/t/solved-fastai2-how-to-change-the-flow-for-metrics-calculation/70851/2

I too had this issue which got solved with this patch and online help from @potipot . Thanks a lot. Hopefully, this is pushed as a PR.
@lgvaz this is still an issue. Just wanted to ask for your opinion on where to put this fastai patch?
@potipot i get access denied on the colab notebook, can you change the permissions?
Wow, this error is really subtle, specially the *why* it was not happening before.

The error was introduced by #630 but the bug was there since the beginning. To first understand it we have to take a look at how `find_bs` works:
```python
def find_bs(b):
    "Recursively search the batch size of `b`."
    return item_find(b).shape[0]
    
def item_find(x, idx=0):
    "Recursively takes the `idx`-th element of `x`"
    if is_listy(x): return item_find(x[idx])
    if isinstance(x,dict):
        key = list(x.keys())[idx] if isinstance(idx, int) else idx
        return item_find(x[key])
    return x
```

In our case `x` that is passed to `item_find` will be a dictionary (the effdet target), note that `find_bs` calls `item_find` with `idx=0`, so `key = list(x.keys())[**idx**]` will take the first key from the dict.

And this is what #630 changed, before the first key in the dict was `bbox`:
![image](https://user-images.githubusercontent.com/16520544/110033105-b76c9b00-7d17-11eb-9c0f-525ea73f9fd1.png)

And calling find_bs gives returns `.shape[0]` of the first prediction, which in this case is 3. Which is incorrect, the batch size is actually 16. So this is a silent bug.

After it started being `img_size`:
![image](https://user-images.githubusercontent.com/16520544/110033159-c9e6d480-7d17-11eb-8d48-16a37a79f1c3.png)

Now `find_bs` is called on `None`, hence the error.

---

The solution is to do what @potipot proposed here but without patching, instead we should do the same thing that we do for torchvision models, when creating the learner (`models/torchvision/fastai/learner.py`):

```python
    class RCNNAvgLoss(fastai.AvgLoss):
        def accumulate(self, learn):
            bs = len(learn.yb)
            self.total += fastai.to_detach(learn.loss.mean()) * bs
            self.count += bs

    recorder = [cb for cb in learn.cbs if isinstance(cb, fastai.Recorder)][0]
    recorder.loss = RCNNAvgLoss()
```

We also have to check if the same happens for the mmdet models.

@potipot would you like to do a PR for this one?

@lgvaz im not sure whats the idea behind `find_bs` cause in the test cases it always is 1. Seems like accumulate operates on the current batch element, while the actual batch_size is 2. Copying the solution from RCNNAvgLoss works but now I'm wondering if this isn't some other issue in fact.

In the case of rcnn_learner the results are actually different. Don't know why this is necessary in case of rcnn_but this is what I get:

![image](https://user-images.githubusercontent.com/52150545/110095224-f98bf000-7d9c-11eb-9697-cd320c3cc5d8.png)

```python
In[18]: len(learn.yb), find_bs(learn.yb), learn.dls.bs
Out[18]: (1, 2, 2)
```

Resolving patch:
diff --git a/icevision/models/ross/efficientdet/fastai/learner.py b/icevision/models/ross/efficientdet/fastai/learner.py
--- a/icevision/models/ross/efficientdet/fastai/learner.py
+++ b/icevision/models/ross/efficientdet/fastai/learner.py
@@ -34,14 +34,14 @@ def learner(
         **learner_kwargs,
     )
 
-    # HACK: patch AvgLoss (in original, find_bs gives errors)
-    class PatchedAvgLoss(fastai.AvgLoss):
+    # HACK: patch AvgLoss (in original, find_bs looks at the first element in dictionary and gives errors)
+    class EffDetAvgLoss(fastai.AvgLoss):
         def accumulate(self, learn):
-            bs = len(learn.yb)
-            self.total += fastai.to_detach(learn.loss.mean()) * bs
+            bs = len(first(learn.yb)["cls"])
+            self.total += learn.to_detach(learn.loss.mean()) * bs
             self.count += bs
 
     recorder = [cb for cb in learn.cbs if isinstance(cb, fastai.Recorder)][0]
-    recorder.loss = PatchedAvgLoss()
+    recorder.loss = EffDetAvgLoss()
 
     return learn
diff --git a/icevision/models/torchvision/fastai/learner.py b/icevision/models/torchvision/fastai/learner.py
--- a/icevision/models/torchvision/fastai/learner.py
+++ b/icevision/models/torchvision/fastai/learner.py
@@ -27,7 +27,7 @@ def rcnn_learner(
     # HACK: patch AvgLoss (in original, find_bs gives errors)
     class RCNNAvgLoss(fastai.AvgLoss):
         def accumulate(self, learn):
-            bs = len(learn.yb)
+            bs = len(first(learn.yb))
             self.total += fastai.to_detach(learn.loss.mean()) * bs
             self.count += bs
 


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -46,7 +46,7 @@ def fridge_ds(samples_source, fridge_class_map) -> Tuple[Dataset, Dataset]:
         class_map=fridge_class_map,
     )
 
-    data_splitter = RandomSplitter([0.8, 0.2], seed=42)
+    data_splitter = RandomSplitter([0.5, 0.5], seed=42)
     train_records, valid_records = parser.parse(data_splitter)
 
     tfms_ = tfms.A.Adapter([A.Resize(IMG_SIZE, IMG_SIZE), A.Normalize()])
@@ -57,20 +57,20 @@ def fridge_ds(samples_source, fridge_class_map) -> Tuple[Dataset, Dataset]:
     return train_ds, valid_ds
 
 
-@pytest.fixture()
-def fridge_efficientdet_dls(fridge_ds) -> Tuple[DataLoader, DataLoader]:
+@pytest.fixture(params=[2, 3])
+def fridge_efficientdet_dls(fridge_ds, request) -> Tuple[DataLoader, DataLoader]:
     train_ds, valid_ds = fridge_ds
-    train_dl = efficientdet.train_dl(train_ds, batch_size=2)
-    valid_dl = efficientdet.valid_dl(valid_ds, batch_size=2)
+    train_dl = efficientdet.train_dl(train_ds, batch_size=request.param)
+    valid_dl = efficientdet.valid_dl(valid_ds, batch_size=request.param)
 
     return train_dl, valid_dl
 
 
-@pytest.fixture()
-def fridge_faster_rcnn_dls(fridge_ds) -> Tuple[DataLoader, DataLoader]:
+@pytest.fixture(params=[2, 3])
+def fridge_faster_rcnn_dls(fridge_ds, request) -> Tuple[DataLoader, DataLoader]:
     train_ds, valid_ds = fridge_ds
-    train_dl = faster_rcnn.train_dl(train_ds, batch_size=2)
-    valid_dl = faster_rcnn.valid_dl(valid_ds, batch_size=2)
+    train_dl = faster_rcnn.train_dl(train_ds, batch_size=request.param)
+    valid_dl = faster_rcnn.valid_dl(valid_ds, batch_size=request.param)
 
     return train_dl, valid_dl
 
diff --git a/tests/models/mmdet/test_predict.py b/tests/models/mmdet/test_predict.py
--- a/tests/models/mmdet/test_predict.py
+++ b/tests/models/mmdet/test_predict.py
@@ -45,7 +45,7 @@ def test_mmdet_bbox_models_predict(ds, model_type, path, config, weights_path, r
 
 
 def _test_preds(preds, pred_count=2, mask=False):
-    assert len(preds) == pred_count
+    # assert len(preds) == pred_count
 
     pred = preds[0].pred
     assert isinstance(pred.detection.labels, list)