Fusing EfficientNet & YoloV5 — Advanced Object Detection 2 stage pipeline tutorial | by Mostafa Ibrahim | Towards Data Science

In this article, I am going to be explaining a concept which I call the “2 class filter”. This is an ensembling technique for object detection and classification models that was heavily used during a Kaggle competition that I have been doing for the last few weeks. This technique has been used by almost everyone doing the competition and it seems to boost the performance by around 5–25% which is quite useful.
We start off by training a YoloV5 model on our dataset whilst using weighted boxes fusion (WBF) for post/pre-processing, if you want to find out more, I suggest checking these 2 articles:
towardsdatascience.com
towardsdatascience.com
I don’t want to dive into the details of training YoloV5 with WBF again. But, essentially all you need to do is to eliminate duplicate boxes using WBF, then preprocess the data to run YoloV5 on it. YoloV5 needs a certain hierarchy for the dataset to be present to start training and evaluation.
The next thing to do is to train a classification network on the dataset. But, the interesting bit is that although the object detection models were trained on 14 different classes (13 different types of diseases and 1 no-disease class), we are going to train the classification network on only 2 classes (disease and no disease). You can think of this as a modeling trick that simplifies our data science problem because classifying 2 classes for a network is much easier than classifying 14 and when we fuse those 2 networks we don’t really need the specifics of every disease, we will just need one of those 2 classes. Of course, this might be a bit different for your problem, so you will probably need to experiment with different settings, but hopefully, you can get an idea or 2 from this article.
One of the best State-of-the-art current classification networks is EfficientNet. For this dataset, we are going to be using a B6 EfficientNet, trained with Keras (TensorFlow), along with these augmentations:
(    rescale=1.0 / 255,    rotation_range=40,    width_shift_range=0.2,    height_shift_range=0.2,    shear_range=0.2,    zoom_range=0.2,    horizontal_flip=True,    fill_mode="nearest",)
If you want to see the full tutorial, I suggest checking out this:
towardsdatascience.com
This is where the 2 class filter comes in to boost performance and what the article is really about. I didn’t want to talk too much about training YoloV5 and EfficientNet since there are a lot of resources for those.
The main idea that I wanted to highlight is that although the classification predictions of Yolo are quite good if you can mix them with the classification of another more powerful network, you can get a quite decent boost in performance. Let’s see how this can be implemented.
The idea being used here is to set a high and a low threshold. And then we are going to be checking each classification prediction. If the probability is less than the low threshold, we set whatever the prediction is to “No Disease”. Recall that our original problem was classifying one of the 14 diseases or the “No Disease” class. This low threshold can be any value between 0 and 1, but will probably fall be something between 0 and 0.1. Moreover, if the classification prediction is between the low and high threshold, we add a “No Disease” prediction with EfficientNet’s confidence for that prediction (Not Yolo’s) because there is a higher chance of having any disease in this scenario. Finally, if the classification prediction is higher than the high threshold, we do nothing because this means that the network is highly confident.
This can be implemented as shown here:
low_thr  = 0.08high_thr = 0.95def filter_2cls(row, low_thr=low_thr, high_thr=high_thr):    prob = row['target']    if prob<low_thr:        ## Less chance of having any disease        row['PredictionString'] = '14 1 0 0 1 1'    elif low_thr<=prob<high_thr:        ## More chance of having any disease        row['PredictionString']+=f' 14 {prob} 0 0 1 1'    elif high_thr<=prob:        ## Good chance of having any disease so believe in object detection model        row['PredictionString'] = row['PredictionString']    else:        raise ValueError('Prediction must be from [0-1]')    return row
Source: Kaggle
Final thoughts
I have experimented with this 2 class filter on various different scenarios and models during the competition and it always seems to boost the performance by up to 25% which is amazing. I think if you want to apply it to your custom scenario, you will need to think about what cases the classification network prediction can help your object detection model. It’s not exactly swapping the prediction confidences, it's more about “fusing” them in a smart way.