Automatic Update of Django Models from a Google Spreadsheet | by Angelica Lo Duca | Towards Data Science

Often Data Model definition and Data Model Implementation are performed by different people and a little change in Data Model definition should be transformed into an implemented model as soon as possible.
The speed with which changes in the model can be translated into implementation depends on how definition and implementation are related to each other.
In this tutorial I propose a mechanism to directly translate changes in the model into changes in the implementation. I exploit the following tools:
Google Spreadsheet for the Data Model definition
Python to translate changes in the spreadsheet into changes for Django
Django to render the model in a Web interface.
Three steps are needed to automatically update a Data Model in Django through a Google Spreadsheet:
Data Model Definition in Google Spreadsheet
Model Translation from a Google Spreadsheet to a Django Model
Model Update in Django
Firstly, I define a basic Data Model. For example, I can consider three tables:
Person(id,name,surname,birthDate,sex)
Organization(id,name,address,website,locality)
Staff(person_id,organization_id)
Then, I create a new Google Spreadsheet and I devote one sheet for each table, as shown in this example. For each table, I define the following columns:
attributes — the list of attributes for a table
description — what the attribute represents
data type — the attribute data type
max length — the max length of char type variable. It is valid only for char attributes
key — specifies if an attribute is a key (primary or foreign)
foreign key table — if the attribute is a foreign key, specify the associated table
The following figure shows the built sheet for the Person table:
The Data Model can be dynamically updated by the Model Designer, without taking care of any implementation. They can even add other tables to the spreadsheet or modify existing ones, by adding or removing attributes.
It is sufficient that the Model Designer respects the structure of the table.
The next step involves Model Translation from the Google Spreadsheet to Django. This can be achieved by dynamically building the Django Model with the content of the Google Spreadsheet. In practice, I can write a Python code, which generates the Django model.py script on the fly, depending on the Google Spreadsheet.
Model Translation can be achieved in three steps:
Read the Google Spreadsheet
Create a Mapping between tables and Django model classes
Create the content of the model.py script
In my previous article, I described how to load the content of a Google Document in Python. Regarding Google Spreadsheets, the procedure for authentication is quite similar, with the difference that Google Spreadsheets APIs should be enabled in the Google Console, instead of Google Documents API.
For completeness, I report here the Python code used for authentication:
from __future__ import print_functionimport os.pathfrom googleapiclient.discovery import buildfrom google_auth_oauthlib.flow import InstalledAppFlowfrom google.auth.transport.requests import Requestfrom google.oauth2.credentials import Credentials# If modifying these scopes, delete the file token.json.SCOPES = ['https://www.googleapis.com/auth/spreadsheets.readonly']# The ID and range of a sample spreadsheet.SPREADSHEET_ID = 'YOUR DOCUMENT ID'creds = None# The file token.json stores the user's access and refresh tokens, and is created automatically when the authorization flow completes #for the first time.if os.path.exists('token.json'):    creds = Credentials.from_authorized_user_file('token.json', SCOPES)# If there are no (valid) credentials available, let the user log in.if not creds or not creds.valid:    if creds and creds.expired and creds.refresh_token:        creds.refresh(Request())    else:        flow = InstalledAppFlow.from_client_secrets_file(            'credentials.json', SCOPES)        creds = flow.run_local_server(port=0)    # Save the credentials for the next run    with open('token.json', 'w') as token:        token.write(creds.to_json())
After authentication, I can load the content of the Google Spreadsheet:
service = build('sheets', 'v4', credentials=creds)sheet = service.spreadsheets()sheet_metadata = service.spreadsheets().get(spreadsheetId=SPREADSHEET_ID).execute()
and transform it into a dictionary of Pandas Dataframe. In practice, each item of the dictionary is a sheet of the Google Spreadsheet and the associated key is the sheet name:
df_dict = {}properties = sheet_metadata.get('sheets')for  item in properties:    table = item.get('properties').get('title')    df_dict[table] = pd.DataFrame()        result = sheet.values().get(spreadsheetId=SPREADSHEET_ID,                                range=table + '!A1:F').execute()    header = result.get('values', [])[0]        values = result.get('values', [])[1:]     if not values:        print('No data found.')    else:        all_data = []        for col_id, col_name in enumerate(header):            column_data = []            for row in values:                if col_id < len(row):                    column_data.append(row[col_id])                else:                    column_data.append('')            ds = pd.Series(data=column_data, name=col_name)            all_data.append(ds)        df_dict[table] = pd.concat(all_data, axis=1)
This step can be easily achieved by building a table, which maps columns of each dataframe to the properties of a Django model class:
tables = {}for table,df in df_dict.items():    tables[table] = {}    for i in range(0, df.shape[0]):        attr = {}        attr['default'] = 'None'        if df['MAX LENGTH'][i] != '':            attr['max_length'] = df['MAX LENGTH'][i]        if df['KEY'][i] == 'primary key':            attr['primary_key'] = 'True'        tables[table][df['ATTRIBUTES'][i]] = [df['DATA TYPE'][i], attr]
which produces the following output:
{'PERSON': {  'id'    : ['Integer', {'default': 'None', 'primary_key': 'True'}],  'Name'  : ['Char', {'default': 'None', 'max_length': '32'}],  'Surname': ['Char', {'default': 'None', 'max_length': '32'}],  'BirthDate': ['Date', {'default': 'None'}],  'Sex'.  : ['Char', {'default': 'None', 'max_length': '1'}]}, 'ORGANIZATION': {  'id'  : ['Integer', {'default'  : 'None', 'primary_key': 'True'}],  'Name'    : ['Char', {'default': 'None', 'max_length': '64'}],  'Address' : ['Char', {'default': 'None', 'max_length': '128'}],  'WebSite' : ['Char', {'default': 'None', 'max_length': '128'}],  'Locality': ['Char', {'default': 'None', 'max_length': '64'}]}, 'STAFF'.   : {  'person_id': ['Integer', {'default': 'None'}],  'organization_id': ['Integer', {'default': 'None'}]}}
In practice, for each table, the list of attributes is built and for each attribute, table columns are translated to Django model properties.
The final step of Model Translation involves the dynamically generation of the model.py script.
I define a function which parses each item of the previous table and builds the code for each attribute within a Django class:
def get_type(attr_type):    if isinstance(attr_type, list):        attr = attr_type[0] + 'Field('        for k,v in attr_type[1].items():            attr = attr + k + '=' + v + ','        attr = attr[:-1]        return attr + (')\n')    else:        return attr_type + 'Field()\n'
Then I build a string, which will contain all the content of the model.py file. I initialize it with the static code:
script = 'from django.db import models\n'
Then, I build a Django Model class for each item in the table:
for model,attributes in tables.items():    script = script + "class " + model + "(models.Model):\n"    for attr_name,attr_type in attributes.items():        script = script + '\t' + attr_name + ' = models.' + get_type(attr_type)
Finally I save the result into the file named model.py :
root = 'mywebsite/myapp/'file_name = root + 'models.py'with open(file_name, "w") as py_file:    py_file.write(script)
Now I’m ready to apply changes to Django. I simply run the following commands:
python3 manage.py makemigrations
which should give the following output:
and finally:
python3 manage.py migrate
This command should add the three tables to the database:
In this tutorial, I have illustrated how to automatically update a Django model from a Google Spreadsheet. This process allows to synchronize Data Model and Data Implementation, although the two processes are managed by different people.
If you have come this far to read, for me it is already a lot for today. Thanks! You can read more about me in this article.
You could subscribe for a few dollars per month and unlock unlimited articles — click here.
I can configure the admin console in Django in order to insert records manually. This can be achieved by registering each created Model class to the admin console. I can do it automatically by creating a admin.py script:
script = """from django.contrib import admin\nfrom .models import *"""for model in tables.keys():    script = script + "admin.site.register(" + model + ")\n"
Then, I can save the file:
file_name = root + 'admin.py'with open(file_name, "w",encoding='utf-8') as py_file:    py_file.write(script)
Then, I can apply migrations again and create a a new superuser:
python3 manage.py makemigrationspython3 manage.py migratepython3 manage.py createsuperuser
Finally, I can run the app:
python3 manage.py runserver
and connect to the admin console:
http://127.0.0.1:8000/admin
where I can see my tables and add new records:
All the code of this tutorial can be downloaded from my Github Repository.