Rails: mirroring uploads to other s3-compatible services
By: Med Bouzid
Tags: Ruby on Rails, Mirror, S3

Recently I wanted my uploaded files (to s3) to be mirrored in other s3 compatible services,

Note: if you want to copy a whole bucket (with tons of files maybe) to another bucket within another cloud storage provider, you may want to check https://rclone.org/ so just keep in mind that what I’m discussing here is different.

The idea is… when a file gets uploaded (I do that using direct upload) the file should be mirrored to another s3 service like digitalocean spaces, wasabi or backblase b2, or dreamobjects, basically any other s3 compatible service (there are a lot), like that I have at least another copy/backup of the file, it should be also possible to mirror the file in more than one S3-like service.

If you are using ActiveStorage, you have already this functionality out of the box, but if you are not (which is my case), then you might be interested how mirroring files can be done!

I was curious how ActiveStorage does it, so I jumped right into ActiveStorage repository and started analyzing the code… don’t get me wrong, I love Rails, but I was surprised how much complexity is involved in their code, I never try to look saavy, I love simple ways, simplicity works, but let’s see how Rails do it…

The first file I came across was https://github.com/rails/rails/blob/a6f42938390138ae88d4d32a3a16ddcb1e61d3f2/activestorage/lib/active_storage/service/s3_service.rb

When you read it, you will find a method “upload”

def upload(key, io, checksum: nil, filename: nil, content_type: nil, disposition: nil, **)

instrument :upload, key: key, checksum: checksum do

content_disposition = content_disposition_with(filename: filename, type: disposition) if disposition && filename if io.size < multipart_upload_threshold

upload_with_single_part key, io, checksum: checksum, content_type: content_type, content_disposition: content_disposition

else

upload_with_multipart key, io, content_type: content_type, content_disposition: content_disposition

end

end

end

This method checks if the file is not greater than a certain size threshold (like 5.megabytes or something) then upload it at once, otherwise it will use a multipart upload function

uploading small files at once is done via s3 sdk gem “object.put” like so:

def upload_with_single_part(key, io, checksum: nil, content_type: nil, content_disposition: nil)

object_for(key).put(body: io, content_md5: checksum, content_type: content_type, content_disposition: content_disposition, **upload_options)

rescue Aws::S3::Errors::BadDigest

raise ActiveStorage::IntegrityError

end

In the other hand, uploading large files is done with multipart upload, using aws s3 gem method called “upload_stream”, it’s as follows:

def upload_with_multipart(key, io, content_type: nil, content_disposition: nil)

part_size = [ io.size.fdiv(MAXIMUM_UPLOAD_PARTS_COUNT).ceil, MINIMUM_UPLOAD_PART_SIZE ].max object_for(key).upload_stream(content_type: content_type, content_disposition: content_disposition, part_size: part_size, **upload_options) do |out|

IO.copy_stream(io, out)

end

end

I’m not going to go much deeper in the explanation, but before uploading your file to mirrors, ActiveStorage downloads the file from the primary storage that was uploaded to, then re-upload it to all configured mirrors, one of the method you may notice is “download” which looks like this:

def download(key, &block)

if block_given?

instrument :streaming_download, key: key do

stream(key, &block)

end

else

instrument :download, key: key do

object_for(key).get.body.string.force_encoding(Encoding::BINARY)

rescue Aws::S3::Errors::NoSuchKey

raise ActiveStorage::FileNotFoundError

end

end

end

It checks for a block and either stream download the file (in case it’s a large file or it download it at once via aws s3 sdk “get” method, let’s for example see how large files gets downloaded (notice the line “stream(key, &block)”) large file are downloaded using the method stream:

def stream(key)

object = object_for(key) chunk_size = 5.megabytes

offset = 0 raise ActiveStorage::FileNotFoundError unless object.exists? while offset < object.content_length

yield object.get(range: "bytes=#{offset}-#{offset + chunk_size - 1}").body.string.force_encoding(Encoding::BINARY)

offset += chunk_size

end

end

As you see, a while loop will call #get method of aws s3 sdk (https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#get-instance_method) with a range option to download only a small chunck of the big object, until the object is fully downloaded into your desk.

object.get(range: "bytes=#{offset}-#{offset + chunk_size - 1}").body.string.force_encoding(Encoding::BINARY)

I don’t want to overwhelm you with the full code used, I just exposed some code from one file, but if you try to connect the dots, you will find that the code is bigger than what you think and it’s spread through different files, here is more 2 files to check if you are curious:

https://github.com/rails/rails/blob/a6f42938390138ae88d4d32a3a16ddcb1e61d3f2/activestorage/lib/active_storage/downloader.rb

https://github.com/rails/rails/blob/5cfd58bbfb8425ab1931c618d98b649bab059ce6/activestorage/lib/active_storage/service/mirror_service.rb

Honestly after reading how this is done with ActiveStorage, I was like:

Noway I can write similar code, it’s just complex and to be honest I don’t know how to write complex stuff, writing simple things make me sane.

So how you can avoid insane code and write SUPER SIMPLE code, that just works?

Maybe… you should just avoid things that distract you when you are writing code (like your phone, twitter, or any other noise out there) and learn how to concentrate…

The Aws s3 sdk already has 2 (out of the box) methods/actions to download and upload files, why reinventing the wheel?

The download method is called: download_file (so simple isn’t it?) and it will do everything for you, if the file is big, it will download it part by part, if it’s a small file, it will download it at once.

2. The upload method is called similarly: upload_file (make sense) and like the other method it will upload the file in multipart if the file is big, otherwise it will upload it at once!

By using just 2 simple methods download_file and upload_file you will be able to write in few lines what Rails ActiveStorage does in multi-complex lines, how comes?

Someone will come and tell me that they know what they are doing, maybe yes… maybe not (collaboration sometimes is a mess, and being smart more than normal is the way to show the community how clever you are)! but at the end, my way also works, and it just works even though it’s not that fancy o smart :) I even wrote some logic around it and put it out there as a Gem ( a one short class gem that does it all ) called s3mirror check it out: https://rubygems.org/gems/s3mirror

In the next write up, I will maybe explain how I write it :) Let me know what do you think.