Evolutionary Database Design
By: Alexandra Chakeres
Tags: Database, Ruby on Rails, Postgresql, Postgres

As a Back End Engineering student at the Turing School of Software & Design, I started learning about web apps and databases about two months ago. We’re mostly using Ruby on Rails and PostgreSQL. There has been a suggestion passed down from class to class that when things are not working with your database, you should just blow it up with rails db:{drop,create,migrate,seed} , also known as “DCMS”. Our instructors mentioned that this is a bad habit to get into, because on the job you’ll never be able to rely on dropping the database to solve your issues. But we’ve also never really been told the right way of doing things. As a career changer trying to prove myself as a new software engineer, I have no desire to lose my future clients’ data (or wear the “broke the build” hat of shame).

I had some major database changes to make on my current project “Oddio”, and was counseled not to worry about dropping the production database (because we’re still early in production and still have no real-life users). But I wanted to figure out the right way of doing things now so I don’t form a bad habit that persists after my schooling is done.

Research lead me to the concept of Evolutionary Database Design. Instead of laying out the full schema anticipated when a project begins, Evolutionary Database Design allows for agile (iterative/adaptable) development of your database. Things can start simple, and as your app needs more functionality or needs refactoring, this approach allows your database to smoothly morph over time.

The Rails framework (and several other tools) include database “migration” functionality by default. It allows you to add tables & columns, drop tables & columns, add/remove foreign keys & indexed columns, etc. These migrations are like version control for your database — a history of the incremental changes that got your schema to its current state both on your local machine and in production. Just like version control for your code, you generally shouldn’t mess with history — avoid editing migration files that have already been run outside your local machine. Also avoid messing with your schema file — instead use migrations to make and keep track of the changes.

Let’s walk through how I avoided “DCMS” and more elegantly migrated existing data from one table to another in my project.

The Problem

Our Oddio app initially registered & authenticated users through the bcrypt gem and its has_secure_password method in the User model. The users table had columns for username and password_digest , among other things.

the starting point

But we wanted to add OAuth through Google (& possibly Twitter in the future). Users authenticated through Google would not have a username or a password with us. The bcrypt gem we were using didn’t allow a User to be created without a password. Which meant if we wanted both types of authentication (and didn’t want to randomly assign some bogus password for users who utilize OAuth), we’d have to get the password_digest (and username ) field out of the users table. Here is the desired endpoint:

the desired endpoint

But our app had already been deployed! There may be users whose accounts would be deleted if we were to just add a new table and delete the old username and password columns from users ! I had to think more carefully about how to move the data.

Step 1: Get the App Working with the New Tables

At this point we start a new branch in git. Without deleting any columns from the users table, we’re going to add the two new tables app_auths and google_auths (as well as a display_name column to the users table). Get the app working with the new configuration. By the end of this refactor, nothing in our app should be using the username or password_digest columns in the users table.

schema at the mid-way point, with the unused columns shown in purple

Here are the migrations I ran:

class CreateAppAuths < ActiveRecord::Migration[5.2] def change create_table :app_auths do |t| t.references :user, foreign_key: true t.string :username t.string :password_digest t.timestamps

end end end

class AddDisplayNameToUsers < ActiveRecord::Migration[5.2] def change add_column :users, :display_name, :string end end class CreateGoogleAuths < ActiveRecord::Migration[5.2] def change create_table :google_auths do |t| t.references :user, foreign_key: true t.string :uid t.string :token t.timestamps end end end

This is what my models look like now (in part):

class User < ApplicationRecord has_one :app_auth has_one :google_auth validates :email, uniqueness: { case_sensitive: false }, presence: true validates :display_name, uniqueness: { case_sensitive: false }, presence: true validates_presence_of :first_name validates_presence_of :role end

class AppAuth < ApplicationRecord belongs_to :user, optional: true validates :username, uniqueness: { case_sensitive: false }, presence: true before_save { self.username = username && username.downcase } has_secure_password end

class GoogleAuth < ApplicationRecord belongs_to :user, optional: true validates :uid, uniqueness: true, presence: true validates_presence_of :token end

Step 2: Write a Script to Move the Production Data

In the branch created in Step 1 above, we should also include a script to extract the username and password_digest data from one table to another. We’ll set the users’ new display_name to be their username .

desc 'Move existing login info from users table to app_auths table' task migrate_existing_user_logins: :environment do User.all.each do |user| begin AppAuth.create!( user: user, username: user.username, password_digest: user.password_digest, created_at: user.created_at, updated_at: user.updated_at ) user.update!(display_name: user.username) user.update!(username: nil, password_digest: nil) puts "User #{user.id} successfully migrated".green rescue Exception => e puts "User #{user.id} failed to migrate! Here is the stack trace:".red e.backtrace.each { |line| puts line } end end end

At this point I probably should have abstracted this into a service so I could more thoroughly test it, but I ran several scenarios in my development environment and it worked as expected. Because our app doesn’t have real-live users to worry about at the moment, and the fact that the script should only be run once, I didn’t want to over-analyze it in this exercise.

Step 3: Deploy & Move the Data

It’s go time!

Submit a pull request Merge the branch into master Deploy it Run the database migrations on the production database Run the script created in Step 2 to move the existing users’ data from users to app_auths .

Step 4: Clean Up

Yay, our app is up and running with both bcrypt logins to our site, as well as Google OAuth! But we still have those two unused columns in our users table.

Start a second branch. Add a database migration to remove the two unused columns:

class RemoveColumnsFromUsers < ActiveRecord::Migration[5.2] def change remove_column :users, :username, :string remove_column :users, :password_digest, :string end end

In the home stretch now:

Run the database migration in the test/development environments Delete the rake task from Step 2 — it’s already served its purpose and won’t be relevant anymore Make sure everything still works (test suite is happy, nothing weird in localhost) Submit another PR Merge the branch into master Deploy it Run the migration on the production database Enjoy your new and improved app!

Resources