Build The World’s Simplest ETL (Extract, Transform, Load) Pipeline in Ruby With Kiba
By: Chris I.
Tags: Ruby, Programming, Data Science, Business Intelligence, Analytics

The Code

We’re going to bundle this up in a tiny ruby project.

Create our directory

$ mkdir kiba-etl && cd kiba-etl/

Add the source CSV

Create a CSV file with touch phone.csv and paste in the following.

id,number

1,123.456.7891

2,222

3,303-030-3030

4,444-444-4444

5,900-000-00001

6,#1000000000

7,#9898989898

8,800-000-00000

9,999.999.9999

10,1.1.1.1.1.1.1.1.1.1

11,(112)233-4455

12,(121)212-0000

In a real situation, you might use a service like Twilio to detect if they’re real phone numbers.

Create Gemfile

Run touch Gemfile and add the following to the file.

source ' https://rubygems.org' gem 'kiba'

gem 'rake', '~> 11.2', '>= 11.2.2'

Install these packages with bundle install .

Create a class for Extract

Create a file to store our ETL classes. We’ll call this common.rb .

Write our first class in common.rb . We could name this class anything but I’ve chosen SourceCSV .

require 'csv'

def initialize(filename:)



end class SourceCSVdef initialize(filename:) @filename = filenameend

# time to transform in the next step

def each

csv = CSV.open( # Iterates over each line in the CSV so we can pass them one at a# time to transform in the next stepdef eachcsv = CSV.open( @filename , headers:true) csv.each do |row|

yield(row.to_hash)

end csv.close

end

end

Create a Rakefile

We’ll use a rake task to kickstart the Kiba job.

Create a file called Rakefile (no extension), and add the following. We’ll add more logic to this in a moment.

require 'kiba'

require_relative 'common' task :kiba_run do

puts "The job is running..."



Kiba.run(

Kiba.parse do

##############################################################

source SourceCSV, filename: 'numbers.csv'

##############################################################

end

)

puts "...The job is finished"

end

Note that our rake task is named, kiba_run . Let’s run our task from the console.

$ rake kiba_run

The output should be:

The job is running...

...The job is finished

If we see that, everything works so far.

Create a class for Load

Now we’ll create the class that saves our data after it’s modified. Obviously we haven’t added any modification logic yet.

In common.rb , add.



def initialize(filename:)





end class DestinationCSVdef initialize(filename:) @csv = CSV.open(filename, 'w') @headers_written = falseend

if !@headers_written

# Add headers on the first row

# after which, don't add headers





end



end def write(row)if !@headers_written# Add headers on the first row# after which, don't add headers @headers_written = true @csv << row.keysend @csv << row.valuesend



end

end def close @csv .closeendend

This stores our loaded data into a new CSV. Alternatively you could dump this data to a db like postgres.

Add the load step to the rake file.

...

##############################################################

source SourceCSV, filename: 'numbers.csv' destination DestinationCSV, filename: 'numbers-cleaned.csv'

##############################################################

...

Give it another run with rake kiba_run .

A new file called numbers-cleaned.csv should now exist which looks exactly like numbers.csv .

Congratulations! You’ve completed the “E” and “L” steps in ETL.

Create classes for Transform

Now the fun part. We’ll create 2 classes related to transform which each do something different.

The 1st removes punctuation from our numbers, and counts the number of characters. Add this class to common.rb .



def initialize(field:)



end class TransformCleandef initialize(field:) @field = fieldend

number = row[ def process(row)number = row[ @field # remove non-numeric characters

row[:number_cleaned] = number.tr('^0-9', '') # count number of characters

row[:digit_count] = row[:number_cleaned].length row

end

end

And update Rakefile .

...

#################################################################

source SourceCSV, filename: 'numbers.csv' transform TransformClean, field: 'number' destination DestinationCSV, filename: 'numbers-cleaned.csv'

#################################################################

The 2nd removes any numbers that are not 10 digits long. Add the second transform class to common.rb .



def initialize(field:)



end class TransformDropFakedef initialize(field:) @field = fieldend

number = row[

row[:digit_count] == 10 ? row :nil

end

end def process(row)number = row[ @field row[:digit_count] == 10 ? row :nilendend

And update our rake task.

...

##################################################################

source SourceCSV, filename: 'numbers.csv' transform TransformClean, field: 'number'

transform TransformDropFake, field: 'number' destination DestinationCSV, filename: 'numbers-cleaned.csv'

##################################################################

...

Now run our rake task again and look at the outputted file. It should look like.

id,number,number_cleaned,digit_count

1,123.456.7891,1234567891,10

3,303-030-3030,3030303030,10

4,444-444-4444,4444444444,10

6,#1000000000,1000000000,10

7,#9898989898,9898989898,10

9,999.999.9999,9999999999,10

10,1.1.1.1.1.1.1.1.1.1,1111111111,10

11,(112)233-4455,1122334455,10

12,(121)212-0000,1212120000,10

Awesome!

If we pretty printed the outputted CSV, it would look like this. Try: cat numbers-cleaned.csv | column -t -s, | less -S .

id number number_cleaned digit_count

1 123.456.7891 1234567891 10

3 303-030-3030 3030303030 10

4 444-444-4444 4444444444 10

6 #1000000000 1000000000 10

7 #9898989898 9898989898 10

9 999.999.9999 9999999999 10

10 1.1.1.1.1.1.1.1.1.1 1111111111 10

11 (112)233-4455 1122334455 10

12 (121)212-0000 1212120000 10

We’ve implemented ETL on a CSV!