How to get robbed by insecure practices
By: Richardson Dackam
Tags: Web Development, Hacking, Rails, Security, AWS

I was recently victim of an insecure malpractice in Rails involving Carrierwave, Fog, Amazon S3 and a Hacker. I wanted to share my story with other developers so that it doesn’t happen to someone else.

The Story

It’s Thursday, 3AM in Toronto. I go on my startup website before deploying and I realize that all the pictures are down. Alert: Something is wrong with my image hosting server. I then try to login in my AWS console and Amazon is telling me that my account has been shut down. ??? Still not sure what’s going on, I click everywhere and realize that the Bill & Payment page is still working ( of course..)… There, something terrifying is waiting for me…..

It’s been only 17 days in April, how the heck did my bill mount to that amount?

What Happened

I was recently overwhelmed with all my contract jobs so I hired a remote developer to help me with my startup. After asking him to sign a Non Disclosure Agreement (NDA), I added him to my private Github repository. He then forked my repository and publicly exposed a copy of it on his own repository. (What was he thinking?) It still absurd to me that he didn’t think of keeping it private even after signing an NDA… This negligence leaded to an exploit. Someone out there found the key and used it to create 68 EC2 instances in my AWS it was at least 12 per server region. My bill suddenly went from $15.00/mo to +$10,000 in a matter of a few days. It took me sometimes to realized that because my AWS billing alerts weren’t setup. It would have probably taken a few more days before I found out about if I didn’t decide to wake up that night and check one of my site.

As soon as I noticed it, I immediately removed my compromised key then called Amazon and terminated all the instances.

In order to allow me to shutdown the instance Amazon had to give me back access to my account since it was shut down. During that process my bill moved to:

What’s Then?

Well, “Amazon feel very sorry for the concern the charges are causing me due to unauthorized access on my account”. They are currently processing a concession request for all the unauthorized charges. There are not guaranteeing that they will wave the fees on my amount but I am really hopeful for it.

ABC. Always Be Cautious

The developer wasn’t the only person at fault, I should have removed the hard coded value before handing him access to my repository. Actually, they shouldn’t have been there at the first place.

The only reason I put them there was because I was migrating my application from one server to another (Heroku to DigitalOcean) so I temporarily hardcoded the value so that I could refer to them if needed in development (especially when I quickly switch from one laptop to another). I would recommend to always play it safe with this even when you are the only developer. I learned a lot of thing from this negligeance. Unfortunately I have the curse to learn my lessons the HARD WAY. I have made many mistake , ran into many bugs and obstacle but I’ve always learned from each experience.

A man must be big enough to admit his mistakes, smart enough to profit from them, and strong enough to correct them. (John C. Maxwell )

Always avoid hard coded security keys

I had temporally hardcoded the security keys because I was the only one working on the project and when I gave access to the developer; I forgot to remove them and use AWSIdentity and Access Management instead. We tend to be negligent with security keys even with services we pay for. That’s a habit I noticed on a lot of developer.