Django for Data Scientists — An Introduction to Django and Django’s ORM | by Dominik Polzer | Towards Data Science

The extensive feature set of Django as a high-level Web framework can scare off Data Analysts and Data Scientists who never really had contact with Web development. Unfortunately, this could keep them from experiencing Django’s impressive and easy to use Data Management abilities.
While the most Django tutorials cover the entire range of functions in several hours of video footage, I would like to concentrate and highlight Django’s great database interaction features without overloading the reader with the almost limitless feature set. The article should explain it as simple as possible, so you could use Django for your Data Management tasks without understanding Django in every detail. But why use Django at all?
Use the language you are comfortable with: Django ORM allows you to interact with your database without using any SQL commandsAll in one place: Simply use the amazing functionalities and modules for Data Processing, Data Management and Data Visualization provided by Python while interacting with your databaseDjango comes with a powerful built-in database management GUI — Django AdminIn case you want to visualize insights of the data in your database and share them in an App — Django gives you everything you need for this
Use the language you are comfortable with: Django ORM allows you to interact with your database without using any SQL commands
All in one place: Simply use the amazing functionalities and modules for Data Processing, Data Management and Data Visualization provided by Python while interacting with your database
Django comes with a powerful built-in database management GUI — Django Admin
In case you want to visualize insights of the data in your database and share them in an App — Django gives you everything you need for this
A short introduction — What is Django and Django ORM?
Prerequisites to interact with your database — Install Django, start a project and connect it to your Database
Use Django ORM in standalone python scripts
A short introduction: Django as a Web Framework
Django Admin —A built-in Web Interface for Data Management
Summary
Django is a Python Web framework which helps developers to bring applications from concept to completion as fast as possible.1 A High-level framework like Django offers a comprehensive set of features for web development like an HTTP application server, a storage mechanism such as a database, a template engine, a request dispatcher and an authentication module.2
As I mentioned before, I try to focus on the part of Django that allows you to interact with your relational databases, the Object-Relation Mapper (ORM).3 The ORM brings you all functions to create and manipulate data and tables of your database without using any SQL commands.4
What I tried to explain with the following image, each Django App includes a module named mocels.py, which defines the structure of the database tables you want to create. To translate the Python Objects into database tables, the ORM comes into place. The ORM is responsible for the communication with your database, which includes the translation of your defined models into the right database structure and the execution of any database operations.
In case you are new to Django, you can install it with pip and the following command:
python -m pip install Django
When you start a new project with Django-admin, a number of files and code is already automatically generated for you, giving you the basic structure, example templates and a default configuration. On the command line, navigate to the folder you want to use as the location for your project folder and run the following command, to start the project data_management:
django-admin startproject data_management
Django-admin startproject creates a set of files that primarily determines the basic configuration of the Django project. Since we will mainly be using Django’s ORM in the following, we can ignore most of these settings for now.
The project folder you just created represents the environment you are moving in. In order for your project to be able to really do something, you still need the actual web application. This is achieved with the help of manage.py, which gives you the possibility to interact with your project. To create your first app, just make sure you are in you project folder and execute the following command:
python manage.py startapp data_processig
The last two steps will create the following project structure.
To tell your project which apps to use, you still need to register your app in settings.py. Add the created App data_processing_app to the list INSTALLED_APPS :
#settings.pyINSTALLED_APPS = [    'django.contrib.admin’,    'django.contrib.auth’,    'django.contrib.contenttypes’,    'django.contrib.sessions’,    'django.contrib.messages’,    'django.contrib.staticfiles’,    'data_processing_app']
Before we can go into more detail about the Django ORM, we first need a link to the database of our choice, in my case MySQL. At first, we need to make sure that the required MySQL DB API Driver is installed. Django’s recommended choise is mysqlclient.5
pip install mysqlclient
In the following you link Django to an existing database, so you must first create a new database/schema. The name of my created schema is personal_dashboard .
To connect Django to your MySQL and the right database, you have to define the connection parameters in settings.py :
Thats it, Django should be connected to your database! A easy way to test it to just use python manage.py runserver . This comand will run a lightweight development Web server on the local machine. During this process Django checks the settings and will tell you if the connection to the database cannot be established.
Go to models.py and create a new model. The model will later represents the table in your database. To make it a bit more interesting, let me show it with an example.
I am tracking my working hours with a tool called Toggl Track. To use these recordings for my own visualizations and calculations, Toggl provides an API that allows me to query this data. The data returned by the API has approximately the following structure:
In the following, I will try to reproduce the structure in my own database. The table workspaces contains only the workspace-id wid and the name of the workspace. To map the table via Django go to models.py and create a new class.
Start with creating a model that holds the information about your workspaces. You will find the name of your Python class/model later in your database table. The full name of the created table is composed of <appname>_<modelname>, for the case shown here, the table takes the following name: data_processing_app_toggl_workspaces
To add a field/column to the table, add a field with the appropriate field type. You can find a summary of all field types in the Django docs. In this case we only need an integer field that holds the ID of the workspaces and a char field that holds the name of the workspace. To roll out the changes to your database, you need to run the following two commands.
python manage.py makemigrations,which is creating new migrations based on the changes you have made to your models
python manage.py migrate, applies the migrations
After you have successfully executed both commands, the table data_processing_app_toggl_workspaces should appear in your database.
Next, I need a model that I will later populate with the clients model. Each client is assigned to a workspace. Since each workspace usually has multiple clients, this is a many-to-one relationship, which is established in Django using the ForeignKey field.
Similarly, the creation and linking of the other tables is performed. The following code shows the mapping of all required tables to Django models.
If you want to use the Django functionality without a web server, you have to set the settings.py as environment variable and make it accessible. This is done by adding the following lines to the beginning of your script:
import osimport djangoos.environ.setdefault('DJANGO_SETTINGS_MODULE', 'data_management.settings')django.setup()
Now to create a new entry of the just generated table data_processing_app_toggl_workspaces, first create a new object to your toggl_workspaces class:
new_workspace = models.toggl_workspaces(wid="12512465",                                  name="test_workspace")
With .save() you save the new entry in your database.
new_workspaces.save()
The following script stores the contents of the sample data frames in the corresponding tables.
In order to be able to display, evaluate and visualize your data on a website, the next paragraph shows you how the communication between HTML pages and your database takes place.
As mentioned before, I don’t want to go too deep into Django as a web framework here. Nevertheless, I think its usefull to get a common understanding of the Django moduls and what they do. The following figure describes how the individual components of the project structure are interrelated. The displayed request-response cycle shows only a simplified form of the processes and do not contain the used Middlewares, but it puts the just created files into a rough context.
The incoming Requests are mainly handled by Middlewares, which have been omitted from the figure for simplicity. Django allready includes some built-in Middlewares like the AuthenticationMiddleware, CsrfViewMiddleware or the SessionMiddleware. Each of these are responsible for one specific task.6
You can find a list of used Middlewares in settings.py :
After the WSGI (Web Server Gateway Interface) and the first Middlewares7, like the Security and AuthentificationMiddleware, the ‘Request will reach the URL Router, which uses the in urls.py defined variable urlpatterns and is looking for a urlpattern which matches the url in the Request.8
As mentioned before, you can start a web server on your local machine with the command python manage.py runserver . Unless otherwise specified, the local web server runs on port 8000 and the address 127.0.0.1. So if you call up the address http://127.0.0.1:8000/example in your browser, Django searches the urlpatterns list for the entry ‘example. The URL router in this case points to the function example in views.py in your first created app data_processing_app .
The function render is used to combine a given context dictonary and a HTML tempate and returns a HttpResonse.9 In the example, I pass the variable test_string to my template. Alternatively I have the possibility to pass data from my database to my HTML template. I will explain how this works in the following.
To access the data from your database and use it within your HTML template is now pretty simple.
To be able to retrieve the data from your database in the following, we cronstruct a queryset via a manager on your model class.10 Manger describes the interface that allows you to perform database query operations. For each model there is at least one manager called objects.11
To return all object in the database you are using the all() method. So if you apply the all() method to the objects manager of your model time_entries , you will get back a QuerySet with all entries in your database.
import data_processing_app.models as modelsQuerySet = models.time_entries.objects.all()
To build and test your querries, you can use the manage.py shell. After starting a new shell with the command python manage.py shell you need to import your models (import data_processing_app.models as models ). Now you are able to create a new QuerySet.
To be able to display this query on your HTML page, you have to include it in the context of your request. For this it is enough to include your QuerySet in the dictonary context in the corresponding function in your views. Within your HTML you can then simply output the passed values with {{ <variable_name> }} .
The following figure shows this for the example we used.
Django comes with a lot of methods to manipulate and create QuerySets. You will find a comprehensive documentation in the Django docs. So much in advance, there is hardly anything that is not possible. The querys can be extended and refined as required.
Here is just an example that I use to filter the entries in time_entries by the client Dans Computer Serviceand then sum up the duration of all entries to the client by calendar week. This way I calculate the number of hours I have worked each week for a particular client.
The first step is to filter the table time_entries by the field name in the table toggl_clients . Since the tables connected via the ForeignKeys project and client you can do that by:
queryset = models.time_entries.objects.filter(project__client__name="Dans Computer Service")
Now, calculate the sum of hours per week: To generate per-object summaries you can use the annotate() clause. I am using .annotate() in the following to calculate the sum of the duration for each calendar week.
queryset_sum_week =   QuerySet.annotate(week=ExtractWeek('start')).values('week') .annotate(week_total=Sum('duration') / 3600).order_by('week')
This results in the following QuerySet, with the entries for calendar weeks and the sum of the durations of all entries in that week.
In order not to go beyond the scope of this article, I don’t want to go into this in detail, but you will find all methods in the Django docs to be able to build your queries as you wish.
The next paragraph shows you what is probably Django’s most important data management tool, an integrated user interface for maintaining data.
Django Admin is a powerful built-in feature. It provides an interface with all the models metadata and gives you the ability to easily manage your database entries. The Groups and Users tables are created by automatically. In Users you can directly create new users and give them the appropriate access permissions. Access to Django Admin is granted to so-called superusers and staff users.
If you are launching the site for the first time and have not yet created a user, you can create a superuser using the command python manage.py createsuperuser and then log in to Django admin.
If you have registered your models in admin.py, you can manage all entries of your tables in the admin area and also create new ones. Django Admin is really well thought out and is available without any development effort.
I tried to explain all the required Django functionalities you need to create tables in your database, populate them, generate queries and maintain your data via Django Admin.
For performance reasons, there are certainly reasons not to use Python and Django for operations with large amounts of data. But, Django gives you all the functionality to interact with your database in a easy, understandable and pythonic way. And especially the last point opens you a whole new world of Data Analytics, Data Science and Data Visualization. Python offers you a huge treasure chest of data analysis and visualization tools and that gives you the chance to possibly uncover an insight or two in your data.
Hope the article gives you a quick and easy introduction to Django and shows that Django is a powerful tool not only for web development.
Thanks for reading!
[1] Meet Django, https://www.djangoproject.com/
[2] Web Frameworks for Python, https://wiki.python.org/moin/WebFrameworks
[3] Django ORM, https://www.fullstackpython.com/django-orm.html
[4] Katie McLaughlin, An introduction to the Django ORM, https://opensource.com/article/17/11/django-orm
[5] Databases, https://docs.djangoproject.com/en/3.1/ref/databases/#mysql-notes
[6] Middleware, https://docs.djangoproject.com/en/3.1/topics/http/middleware/
[7] URL dispatcher, https://docs.djangoproject.com/en/3.1/topics/http/urls/
[8] Django Request-Response Cycle — An easy to follow guide, https://data-flair.training/blogs/django-request-response-cycle/
[9] Django shortcut functions, https://docs.djangoproject.com/en/3.1/topics/http/shortcuts/
[10] Making queries, https://docs.djangoproject.com/en/3.1/topics/db/queries/
[11] Manager names, https://docs.djangoproject.com/en/3.1/topics/db/managers/#django.db.models.Manager