Package: koilerplate
koilerplate - boilerplate code for kaggleOne of my great frustrations when building notebooks forkaggleis the amount of boilerplate code I always add to the top of each notebook.The whole idea behind this package is that a simple import can pull in a lot of this
boilerplate and allow me to focus on the competition in hand.foldersThefoldersmodule defines some simple constants that can be used as the root names
for the three primary folder paths used in Kaggle.These are all defined relative to the working folder (which for a standard kaggle
notebook is the/kaggle/workingfolder).This module is also OS agnostic. This means that it's possible to download and use
the notebook outside kaggle. If this is the case then it will define the folders
relative to your local project working folder. For example:c:\
  ├── dev                     # Somewhere you put all your project files
  .   ├── kaggle-projects     # You put all your kaggle projects here
  .   .   ├── input           # Put downloaded datasets here "INPUT_ROOT"
  .   .   ├── my-competition  # Your competition notebook in here "WORKING_ROOT"
          └── temp            # This is where temp files/folders will go "TEMP_ROOT"UsagefromkoilerplateimportINPUT_ROOT,WORKING_ROOT,TEMP_ROOTprint(f"INPUT_ROOT={INPUT_ROOT}")print(f"WORKING_ROOT={WORKING_ROOT}")print(f"TEMP_ROOT={TEMP_ROOT}")INPUT_ROOT=/kaggle/inputWORKING_ROOT=/kaggle/workingTEMP_ROOT=/kaggle/tempExample kaggle notebookkoilerplate folders and zipout (public)zipoutThezipoutfolder is designed to overcome a limitation in the Kaggle working
folder. There is an unspecified limit to thenumberof files that can be placed
in the working folder, and sub-folders of the working folder.This means that if you are manipulating a datasetit may fail silentlyif you
place too many files in the folder, even if it is within the disk space quota defined
for the notebook.To overcome this, the recommendation through the Kaggle forum is to first write the
files to a folder in /kaggle/temp and then zip up this folder and copy it to the
working folder. (The temp folder does not have this same restriction.)This module contains a single python classZipOutthat does the heavy
lifting to accomplish this.UsagefromkoilerplateimportZipOut# Create the temporary foldermy_dataset=ZipOut("my-dataset")# Print out the path that was createdprint(my_dataset.folder_path)/kaggle/temp/my-dataset# Make a trivial file and place it into the foldertest_file_path=os.path.join(my_dataset.folder_path,"test.txt")withopen(test_file_path,"w")asf:for_inrange(1,1000):f.write("Hello world!\n")# Convert the temporary folder into zip filezip_file_path=my_dataset.make_zip_file()print(zip_file_path)/kaggle/working/my-dataset.zipExample kaggle notebookkoilerplate folders and zipout (public)pushoverPushover is a subscription based notification service:pushoverThe servicepricingis trivially inexpensive (at
the time of writing a one off payment of USD 5.00 per platform; I have subscribed
only for my phone).On an internet enabled workbook you can use this to notify you of steps that are
occurring during a notebook run on the Kaggle servers (or elsewhere).To use the service you need to set up an account onpushover.netand obtain from
your account settings an API token and your user id.UsageCreate yourself aprivatedataset calledpushover-credentialswhich should
consist of a single file calledpushover_credentials.json(note underscore in
filename) which has the following structure:{"token":"<<<<<--- Your token hex code string --->>>>>""user":"<<<<<--- Your user hex code string --->>>>>"}Now, add this dataset to your competition notebook. You're now all set up to use
pushover in your notebook.# Optionally define this before importing koilerplateNOTEBOOK_NAME="My competition notebook"fromkoilerplateimportpushover# Send a notificationpushover("Hello Kaggle World!")Example kaggle notebookSorry, none yet.offlineOften, in code competitions on Kaggle, the notebook submission has to be run without
internet access. This is usually to stop notebooks handing off tasks to an external
processor farm. Usually, these competitions permit the use of external datasets and
libraries.However, it is a non-trivial matter using a library that's not included in the standard
notebook image. You need to build yourself a custom dataset containing all the
information that's needed forpipto load them into your notebook whilst it's
running without an internet connection.Theofflinemodule is designed to take the legwork out of preparing the dataset
that you will need to load the modules off-line. It contains a single functionoffline_pip_prepare()which will copy the modules needed into the WORKING folder
so that they can be uploaded to kaggle as a dataset.UsageIn this example we're going to prepare the libraries to be loaded as a specific verison
of PyTorch plus the very good cell segmentation librarycellpose. If we look at the install instructions for
these libraries then we would see the recommended installion scripts for both are:pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 \
    --find-links https://download.pytorch.org/whl/torch_stable.html

pip install cellpose==0.6.1In order to prepare these for offline loading we have to note the module names and
versions and any locations wherepipshould find links from. These should be each
placed into a list and these passed as parameters to theoffline_pip_prepare()function.In a kaggle internet enabled notebook enter the following code cell:fromkoilerplateimportoffline_pip_preparePACKAGES=["torch==1.7.1+cu110","torchvision==0.8.2+cu110","torchaudio==0.7.2","cellpose==0.6.1",]LINKS=["https://download.pytorch.org/whl/torch_stable.html"]offline_pip_prepare(packages=PACKAGES,links=LINKS)Note that this will take some time to run, as the function downloads the necessary
installation information and, if necessary, builds the wheel files thatpipwill
use.After the cell completes the WORKING folder will contain the following:/
└── kaggle
      ├── working                   # Working folder of your kaggle notebook
      .   ├── requirements.txt      # requirements file that you will use with pip
      .   └── wheels.zip            # the offline wheels files that pip will useOnce this is completed you need to save the notebook as a new version; Kaggle will
then execute your notebook in another kernel. Wait for this to finish then navigate
to the output section of the notebook and create a new dataset, call it, for example,competition-packages.Note that the above example produces a dataset of approximate
size 1.2GB.Now, start a new notebook that will become your competition entry notebook and add
to it your newly created dataset. You will now see that this appears in your input
folder for the notebook(note that kaggle will drop the hyphen in the dataset name):/
└── kaggle
      ├── input                     # Kaggle loads all datasets here
      │   ├── competitionpackages   # This is your newly created dataset
      │   .   ├── requirements.txt  # requirements file that you will use with pip
      │   .   └── wheels
      │           ├── ... .whl      # the offline wheels files that pip will use
      │           └── ... .whl      
      └── working                   # Working folder of your kaggle notebookYou now need to add the following code cell somewhere near the start of your competition
notebook:!pip install \
   --requirement /kaggle/input/competitionpackages/requirements.txt \
   --no-index \
   --find-links file:///kaggle/input/competitionpackages/wheelsNote that it's very important to include the--no-indexflag or this will not
work when the internet access for the notebook is disabled.Don't forget that you also need toimportthe libaries, as you normally would,
before you use them:importtorchimportcellposeExample kaggle notebooks/datasetsNotebook describing process:koilerplate offline (public)Example dataset using process:competition-packagesExample competition notebook:koilerplate offline pip install exampleSuggestionsPlease add any suggestions to theissue trackerand I'll try and get back to you.LicenseThis is licensed under the MIT license. See LICENSE.txt for details.