[Kaggle - Petfinder2] 25th place silver medal solution

Luckily I was able to finish a kaggle data science competition with **25th place out of >3,500 teams** and received **silver medal**! 
Here I would like to share my solution and findings hoping that it will be somewhat valuable to DS community :)

## Overview of the competition
In this competition, animals' pictures are given and you predict attractiveness of them.
The competition is held by pet adoption organization and solutions are supposed to be contributing to help users to take cutter photos of animals to accelerate pet adoption.

Input: Images of photos, metadata of photos
Output: attractiveness score (based on page view)
Details: [kaggle](https://www.kaggle.com/c/petfinder-pawpularity-score/overview)


## Overall model architecture
![petfinder-model drawio](https://user-images.githubusercontent.com/33713884/149642878-35bd9ecd-ba1c-495f-adfe-c45355afe702.png)

## CV strategy
- 10 stratified fold on binned Pawpularity score 
- 100% trusted cv as it has been reported that cv and LB do not really correlate in this competition

## Model

| Model | cv | pub LB | priv LB | note |
| --- | --- | --- | --- | --- |
| **LightGBM stacking** | **17.1585** | **17.8592** | **16.9833** | **Stacking of the below 5 models** |
| LightGBM tabular model | 20.0685 | - | - |  Provided metadata <br> Inferred cat or dog label with Yolov5 <br> Inferred breed label with swin tiny |
| Swin large 1 | 17.4748 | - | - | fastai, mixup |
| Swin large 2 | 17.4862 | - | - | fastai, mixup  |
| Efficientnet b0 | 19.1828 | - | - | fastai, mixup  |
| SVR | 17.5398 | - | - | Swin large 3 (headless) |


### Other models tried but not used in the final submission
- Efficientnet bX  (X={1,2...,7})
    - Takes longer time to train and predict than b0 but did not really outperform b0
- ConvNext
    - Single model performance was almost same as Swin large but did not contribute in ensemble
- Swin tiny, resnet
    - Did not get good single model performance

## Feature engineering

### Feature engineerings that worked
- MixUp
    - Makes model very robust. Worsen training loss but improves CV.
    - [arxiv pdf](https://arxiv.org/pdf/2111.09833.pdf)
- Head less Swin large
    - Swin large was trained with BCEWithLogitsLoss as classification problem but extracted feature was trained RMSE with SVR (SVM on GPU) as regression problem. This gave diversity in models.
    - [kaggle discussion](https://www.kaggle.com/c/petfinder-pawpularity-score/discussion/276724)
- HueSaturationValue
- Cat/dog label
    - Pet photos are either cats or dogs in this competition. With the assumption that cat(/dog) adopter will only look for cats(/dogs), I believed that cat adopters and dog adopters are different customer segments and optimal photo of cats and dogs are different.
    - [yolov5 github](https://github.com/ultralytics/yolov5)
- Breed
    - Some people takes breeds really seriously, and it actually contributed to CV.
    - [kaggle discussion](https://www.kaggle.com/alexvishnevskiy/breed-identification?kernelSessionId=79111656)


### Feature engineerings that did not work
- Holizontal/Vertical Flip
- object center clip
    - Yolo could detect object (cats/dogs) boundary very well but it seems that surrounding information is also important.
    - [yolov5 github](https://github.com/ultralytics/yolov5)

## Lessons learned 
- CV strategy is a curtail key for good modeling
    - In this competition public test data seemed very skewed and score did not correlate with whole test data.
    - In the real setting, this can happen as well when cv strategy is simple train-validation-test random split.
    - It is very important to utilize as many labeled data as possible for both train and validation/test.
    - Therefore, kfold (possibly stratified-kfold) cross validation is a key to make your model more precise and also robust.
- Explainability and model complexity are often trade-off but there are some mitigations 
    - NN based models perform very well in image data (and other data) but due to its complexity it is hard to diagnose the model in the way human can understand.
    - However, there are some tools to understand where in image NN models are seeing.
    - In CNN, Grad-CAM, and in transformer, attention-map.
    - It was very useful in this competition to understand which model is seeing which part of image in order to decide how to conduct feature engineering.