How to organize code in Python if you are a scientist | by Victor Serban | Towards Data Science

If you work in computational research or data science and you don’t come from a Computer Science background, you probably generated amazing pieces of scientific knowledge, but the code you wrote to do it prevents you from indulging in academic glory. You might need a week yourself to make your scripts do the same things again. Even if you are an experienced coder, you might find it hard to do both the job of a developer and that of a scientist.
I have faced this challenge myself as a research student. Although I took classes in Software Engineering, I found it hard to divert from the research questions I was handed to think about how I can create useful tools out of my code. After some trial and error, I designed a recipe to help me make the transition seamlessly and now I use it in my exploratory work as a Data Scientist.
In this article, I share the way I organize my coding workflow, give you some tips and tricks and show you the tool stack I use. The goal is to make the transition from experimentation to tool development easier.
For research to be reproducible, you need to log all the stuff you are doing. This might be a burden if you just want to experiment as fast as possible and do ad-hoc analysis.
The tool I use for experimentation is Jupyter Notebook. The interactive programming interface allows you to check what your code does instantaneously, so you can build algorithms step by step. Moreover, you can use Markdown cells to write down your thinking and conclusions, along with the code. This makes it really easy to log and report everything you do.
You can find plenty of Jupyter tutorials on the web. I recommend getting familiar with the Keyboard Shortcuts (they are described under the Help menu of the editor), as they will accelerate your work significantly.
In digital experiments, looping through experimental variables is really simple. After all, the power of computers comes from their capacity to do things over and over again.
I structure the formal experiments as in the code below.
I keep all the important variables for the current experiment in lists at the beginning of the file for transparency.
A thing to consider here is what to do with the output of each iteration: should you process it into a visualization, compute a metric or store it? The application and desired form of reporting dictates this choice.
A Notebook file should be created for each significant piece of experimentation, and it is up to you to decide what that means. I save an experiment with a relevant title and a versioning number for each time I repeat it using different parameters or a new way of doing things.
This is different from version control in that I keep all the records of my work in place, not only the latest version of it. Moreover, I put all the experiments in the same folder regardless of the project they are assigned to.
Although this creates a huge pile of files, you can easily keep the relevant ones on top through sorting by the date modified. Centralization makes it easy to find specific files by simple search, rather than walking through a hierarchy of folders you don’t remember the logic of.
The greatest advantage of keeping things together is that you can implement cross referencing within and between notebooks. This is simply done with html links in Markdown.
To create a reference to a section of your Notebook, add the following code in a Markdown cell before the referenced part:
<a id='label_of_your_choice'></a><!--referenced section in file experiment00.ipynb-->
Now, in the place where you want your link to the reference to appear write one of the following:
[description](#label_of_your_choice) <!--to make link in the same notebook-->[description](experiment00.ipynb#label_of_your_choice) <!--to make link in another notebook-->
For example, [previous experiment](experiment00.ipynb#label_of_your_choice) produces the link:
Now you can jump straight to the referenced section in the old experiment with one click.
Reports should be nothing else but summaries of the best versions of your experiments and accompanying comments. Fast lookup through your records is key to reportability of your work and it is a matter of clicking your way through notebooks if the cross references are set in place.
Sometimes you need to save some data results, either for reporting, sharing or use in other experiments. The file format is up to you and the situation. For this cases I save the data in a data folder within the experiments folder and name them with something that makes clear the experiment they originated from and the version number of that experiment.
To make things tractable I keep a log in a spreadsheet where I list the data inputs and outputs of all my experiments.
You are probably familiar with writing functions within a script to make the code neat, enhance understanding and most importantly to DRY (Don’t Repeat Yourself). As the experiments evolve, so do your functions, encapsulating more and more logic and other functions within them.
If you followed my approach this far, you started separating your experiments into multiple notebook files, one for each experiment and version of it. When the baggage (of functions) you carry on from one experiment to another grows considerably, it is time for you to start thinking about the reusability of your code. Wouldn’t it be nice if you could just import the functions that passed the test of time smoothly into your code as you do with any other Python library?
You should simply start by saving functions within .py files together with your experiments. You can edit those files in your favourite IDE (I use VS Code). Then you can just use import statements on the file names to get all that functionality back into your experiments:
# if you save the experiment_with() function in the# experiment_functions.py file in the same folder as # your experiment, you can import it into your codefrom experiment_functions import experiment_with
As your functions become more complex and flexible, they will need a lot of inputs which become hard to manage. Luckily Python facilitates setting up default values of function arguments. Having defaults speeds up the experimentation process considerably, hiding away unnecessary complexity. It also helps someone new to your functions get started with them right away.
Sometimes you will come across situations where the default needs to be defined as a function of other parameters. In this case, the strategy I use is to set the default to None and immediately check in the body of the function whether the argument to be set was defined at call. If it was not, then I assign a value to it accordingly. See the example below.
When functions become so large that they encapsulate entire algorithms with lots of parameters to tune, the simple interface of a function might not be enough. In this situation, I prefer the flexibility of the object oriented approach.
With algorithms wrapped as objects you can change parameters on the go by defining them as attributes of the class. You can set up very specific configurations and only make tiny adjustments to the algorithm as needed.
Classes can also be used to build data structures specific to your application and encapsulate the functionality associated with them. There is a whole science of design patterns using OOP.
Now that your reusable functions are in modules separate from the experiments code, keeping things tractable and reproducible might become challenging. You will probably want to modify the reusable bits of your code to make them better in time.
This is the moment where reproducibility and reusability come into conflict. If you want to reproduce your two year old experiment using your super evolved algorithm you might end up with totally different results. Therefore, you want to keep the exact version of the modules you used in experiments together with the notebook files in which you used them (unless you want to cover up some stupid thing you’ve done in the past). If you save a numbered file for each modification you make and keep that along with the experiments for the record, things get really entangled.
Enter Version Control. Software Engineers make things reproducible by versioning and using environments to keep things compatible with one another. Exactly the same idea can be applied to our experiments. You can start version controlling your functions and classes as you improve them, and at the same time maintain a reference to the corresponding versions in the experiment files.
This is how I do it. I sort out those .py files into project folders named <project>_func, within which I set up repositories and start version controlling with git. If you didn’t use git before, go on and find a git tutorial, then come back.
Whenever I create a new experiment file, I make sure I’ve committed the code version I am using and write the following lines in a cell at the top of the experiment file:
%%bash cd <project>_func # move inside the directory where you have the                           # repository you needgit checkout <git commit id> # restore the version of the code                             # you will use
Then in the experiment, I import my functions as needed. For example, let’s say I moved to France recently and now I write all my reports in French. Importing and running the latest version of this function
from magic_func.magic import do_some_magicdo_some_magic()
prints:
Salut Le Monde!
The problem is that I have an old experiment script that generated a report in English, and I would like to rerun it. Luckily, I referenced the commit id at the top of the experiment notebook as shown earlier:
%%bashcd magic_funcgit checkout cff052f7c36bb09ccc101d9ce3652dce87f2acbc
The output of the same code from above within this context is:
Hello World!
Voilà! The magic of Version Control and well written records.
Eventually, you will want to pack the .py files together in proper Python packages. Then you will be able to share them with other people and install them in the environments of the Ipython kernels you use in experiment notebooks.
You can follow the official Python Packaging guide to learn how to package code and this tutorial to upload it on PyPi. To use a specific version of your package in Jupyter Notebook, I recommend you to install it in a conda environment. Then you can run the Jupyter kernel in that environment if you follow the installation instructions of the nb_conda_kernels extension.
As you add functionality to your package, you can adopt the test-driven development philosophy. It states that you should write unit tests before you write the bit of code which they test. In this way you can make the task you want to achieve from your package concrete.
You probably play around with your modules in the experiment files already. You can design tests in your Notebooks and just paste their code into the formal structure of unit tests. For my packages, I use the pytest framework.
When you reach this point, you are no longer just an experimentalist. You are also a developer and you need to share, maintain and improve the tools you created. Have fun coding!