Guide on Submitting Offline Kernels to a Kaggle Competition | by イルカ Borovec | Towards Data Science

In the previous post, we have talked about the need for hyperparameter searchers, and we have walked throw Grid.ai Runs which simplifies fine-tuning in the cloud. We can even monitor our training online and eventually terminate some by our choice with all default configurations.
towardsdatascience.com
Now that we have our best model, it’s time to submit our results. This challenge and several others have restricted access to the test data, which means that the real test dataset is provided only for isolated prediction run without any external connection.
In this challenge, the user is asked to prepare a notebook that reads a folder with test images and saves its prediction to a specific submission.csv file. You can debug the flow as a few test images are provided, and you can work with the kernel in an interactive session. While the final evaluation is performed, the dummy test folder is replaced by the true one.
Said so, the user had typically two options — prepare a kernel which:a) does the full training followed by prediction on test images;b) uses saved checkpoint and performs the prediction only.
Making multiple iterations with a) scenario is difficult, especially since the Kaggle kernel runtime is limited. The b) option allows us to outsource the heavy computing to any other platform and come back with the trained model or even an ensemble of models.
For test data leakage, these evaluation kernels are required to run offline, which means that we have to have all the needed packages and checkpoints included in the Kaggle workspace.
For this purpose, we create private storage (Kaggle calls it dataset even you can have there almost anything) where we upload the required packages (all needed python packages not included in the default Kaggle kernel) and our trained model — checkpoint.
For installing all packages, you can call:
! pip install /kaggle/input/my-storage-name/*.whl
The Kaggle kernels are rich in standard machine learning libraries, and most are included in the base configuration. But still, it can very simply happen that your task requires some other libraries, just different versions. In such a case, it is very annoying to try to install one of the top dependencies (package that you use directly), see it failing for missing lower dependency, which you need to download, update private datastore, and run the installation again. This can be overcome exquisite way assuming we have our package which can be installed and we have listed all our dependencies in standard requirements.txt.
create own package locally python setup.py bdist_wheeldownload dependencies locally pip download -r requirements.txt -d package_freeze[optional] you may remove some standard heavy packages which you know that are included in Kaggle, for example, torch or tensorflowrename all source packages as Kaggle perform recursive extraction of all archives, which yield in breaking packagesimport glob, os, shutilpkgs = glob.glob("package_freeze/*.tar.gz")[shutil.move(p, p.replace(".tar.gz", ".xyz")) for p in pkgs]upload this folder to the dataset other with own package kaggle_plantpatho created earlierin the Kaggle kernel, rename packages back and install our package by pointing the dependencies to our folder on Kaggle
create own package locally python setup.py bdist_wheel
download dependencies locally pip download -r requirements.txt -d package_freeze
[optional] you may remove some standard heavy packages which you know that are included in Kaggle, for example, torch or tensorflow
rename all source packages as Kaggle perform recursive extraction of all archives, which yield in breaking packagesimport glob, os, shutilpkgs = glob.glob("package_freeze/*.tar.gz")[shutil.move(p, p.replace(".tar.gz", ".xyz")) for p in pkgs]
upload this folder to the dataset other with own package kaggle_plantpatho created earlier
in the Kaggle kernel, rename packages back and install our package by pointing the dependencies to our folder on Kaggle
! cp -r /kaggle/input/plant-pathology/package_freeze package_freezeimport glob, os, shutil# finad all packagespkgs = glob.glob("package_freeze/*.xyz")# rename them back to correct name format[shutil.move(p, p.replace(".xyz", ".tar.gz")) for p in pkgs]! pip install -q kaggle_plantpatho --no-index \    --find-link /kaggle/working/package_freeze
towardsdatascience.com
The prediction kernel is straightforward, as you can check it over here. The two primary steps (except installing all needed packages from the storage) are the following: 1) loading saved checkpoint, 2) perform predictions on test images and 3) saved them to a standard CSV file.
The checkpoint loading is performed by the native PyTorch method and parsed by our Lightning model with load_state_dict. For loading, we recommend sticking with the CPU to avoid any eventual side effects.
ckpt = torch.load(    '/kaggle/input/plant-pathology-model-zoo/epoch=4-step=119.ckpt',    map_location=torch.device('cpu'))# this can be any model as we caal load from checkpoint jsut aftermodel = MultiPlantPathology("resnet18")model.load_state_dict(ckpt['state_dict'])
To mention, our PlantPathologyDM has also implemented exploring the test folder to collect all images in it. In contrast to the parsing training dataset, we return the image data paired with the image name instead of the label (which is unknown anyway). The prediction iterates over all images from test_dataloader. and saves test labels. In the end, all paired image names and labels are exported.
That is it, now make this kernel and submission kernel and see how you score in the leaderboard! Good luck!
The last post covered how to prepare the submission kernel, suitable for running in a completely isolated environment. Follow me to learn me!
devblog.pytorchlightning.ai
Stay tuned and follow me to learn more!
devblog.pytorchlightning.ai
Jirka Borovec has been working in Machine learning and Data science for several years in a few different IT companies. In particular, he enjoys exploring interesting world problems and solving them with state-of-the-art techniques. In addition, he developed several open-source python packages and actively participated in other well-known projects. He works in Grid.ai as Research Engineer and serves as a lead contributor of PyTorchLightning.ai.