lschallenges/data-engineering — pull request #318 by jeffo-pdx (2014-10-22 17:19:47+00:00)
Title: Jeff Olfert

Setup:

bundle install

rake db:create db:migrate

rails s

open browser at localhost:3000

Test:

rake test

Design Notes:

This design is simply a file uploader - with a main upload model that tracks each file upload and
the individual revenue per file. I followed the basic rails MVC pattern, looking for simplicity,
as ease of implementation over performance. Along with a very simple summation method to calculate
the total revenue across all file uploads. Error handling is light as rails logging will show any
errors - and I chose not to add to much without knowing who will use the system. When the number
of uploads reaches a large count, the design should change - for example adding paging, archive, etc.

I used Devise for Authentication with a default registration and login. For simplicity I attached
each upload to the user who performed the upload. Other designs, could allow total revenue to cross
users.

As this is just an exercise, I left some items open - such as performance tuning. For example if
the system is loading millions of records - then careful insertion, selection, and fk / indexing
will help with performance. I also did not try to solve issues such as how to remove files, or records.

I added very basic indexing on foreign keys. See schema for detail.

The main model that does most of the work in this very simple design is the Upload - it is also tied
to using Carrierwave as a mounted file uploader. Upload performs the simple revenue calculations
for each file, as well as the total revenue across all files. Note I tied the file upload to
a user, so each file is also tied to the user along with revenue and total revenue.

Since almost all of the important behavior resides in Upload, I added have the most tests in the upload
model. As a long time TDD expert I realize there a many ways to test drive. Here I stuck to the basic
rails testing patterns and let the so called rails model "unit" tests read and write the file system
and database. Since there are only a few tests in the exercise waiting a few seconds to load rails is
not an issue. I leveraged active record validations to ensure that the data is correctly created
and saved in the db - there are model tests using active support to make sure this works.

Again as this is a simple exercise I added very little error handling - in a prod system I would want
to know error detail, how many records had errors, etc. Here if the file import errors out you would
need to view logs which is ok for an exercise.

I also left the database configuration with sql lite because it is so easy to change to a non in memory
db - and the design intent is clearly shown as is.