Elastic infrastructure using AWS, Terraform and NodeJs
By: Guilherme Viebig
Tags: AWS, Nodejs, Ibm Watson, Terraform, Cloud Computing

This article demonstrates how development paradigms are being leveraged to levels once unreached by our vain existence.

Any mortal can do it

That’s right. It is not necessary to be a scientist supported by a research center, university or large corporation. Any developer can have access to hundreds of computing instances spending less than $ 1.00 per hour.

$ 1.00 /hour !? Hundreds of machines!? Are you sure ?

Absolute! And I will demonstrate some ways to achieve this by setting the example with a REAL application, not a skeleton or code aesthetic exercise.

The app: Scrapper + Captcha breaker

I need to access a URL, read an image or audio from a captcha, decode that captcha, fill in that captcha, and access the page that has the information valuable to the business needs.

Step 1: Making an inventory

Inventory? If you’re a developer and never stopped to inventory everything involving a project, you’re probably making less than you could. As in an Adventure or RPG, you have to take into account everything you have in your bag or chest, the external environment, your financial resources and study the possible combinations of elements to balance your character, in this case your application. Let's fullfil this quest!

The business

I need to check information from 150,000 companies in 5 days. I can charge $ 0.03 per company. I disintegrate this in my mind as follows:

30,000 per day, 1,250 per hour, 20 per minute … That’s if the app could work 24 hours a day.

per day, per hour, per minute … That’s if the app could work a day. $ 4,500 bill … Is it worth it? Let’s figure it out.

NodeJS —my fellow libraries

Request (https://github.com/request/request), requests, cookieJar, etc.

Cheerio (https://github.com/cheeriojs/cheerio) to extract

Lodash (https://github.com/lodash/lodash)

Bluebird (https://github.com/petkaantonov/bluebird)

Winston (https://github.com/winstonjs/winston)

AWS SDK (https://github.com/aws/aws-sdk-js)

Amazon AWS

EC2 provides machines … prices starting at $ 0.003 per hour.

S3 allows me to save files without worrying about barriers and physical provisions, free tier, very cheap.

Lambda is awesome. Performs code on demand, $ 0.50 per 1,000,000 runs.

SQS is a messaging queue service, perfect for my demand, free tier

2Captcha

Manual recognition service for captchas. People typing captchas, API, $ 1.00 every 1000 captchas. (plus the ~20% typos)

IBM Watson

With the text-to-speech service I could either stream the audio or send a file to break the audio captcha. Free tier.

Nuance Loquendo, Verbio, etc.

Another option for Automatic Speech Recognition (ASR), ~ $ 1,000 per channel. Expensive, 30% — 70% effective.

Terraform

Very cool. You can automate the management of your infrastructure in the cloud. With a keystroke you'll have multiple computational instances from an existing image at your service. Do it again and you update or destroy all those instances under your demand. It saves me a lot of time and manual labor that would make my great idea impossible.

The target

It’s the site I’m going to scrap, let’s see the perks of this guy.

It has high availability: this is great! I can fire simultaneous requests.

this is great! I can fire simultaneous requests. It has an integrated firewall and rate limiter: It will block my IP’s as soon as they receive many/unusual requests.

It will block my IP’s as soon as they receive many/unusual requests. Extremely strong image captcha: you cannot count on these Python and Matlab libs with examples of image processing.

you cannot count on these Python and Matlab libs with examples of image processing. Captcha audio ok: the audio is not so difficult to be understood, but recently they charged it with some noise.

Step 2: Making the best decisions

After inventorying almost everything that has come to mind it is time to choose the right ingredients to my recipe.

Chosen architecture

Computing

I could use Lambda, even for free, but it would not be able to manage IP rotation and blocks.

Let't go with EC2 and Terraform. I can prepare an image with my deployed application, and run multiple instances on demand.

20 scraps per minute = 20 instances t2.nano each with its own IP. That will cost $ 7.2 for these 5 days. Thats way good!

Database / Queue

Amazon SQS is perfect and cheap. After the first million requests (which are free), I paid $ 0.00000050 in subsequent ones. Let’s assume 150,000 * $ 0.00000050 = $ 0.15, since the storage in S3, add $ 1.5.

Breaking the captcha

I quickly gave up the possibility of using audio in this case. IBM Watson misses a lot in Portuguese, and solutions like Loquendo are very expensive.

Let’s pay xinglings to manually enter 150,000 captchas 24 hours a day. 150,000 * 0.001 = $ 150.

The time

The highest cost of the entire operation. My time or the time of another developer will cost, after all, there is no free lunch and we do not study and we perfect ourselves to live on crumbs.We want a sustainable and long life. 20h * $ 100.00 = $ 2,000.00

The Profit: $ 2,341.15

Cool, you can not get rich but it helps to finance part of a smartphone, notebook, travel or guitar. In my present case, a retaining wall or ladder.

The Profit of Learning

XP gained during this process could be worth much more $$$ and opportunities in the future. I would do it all again if I had time left over, even without financial gain.

Keep in mind that Amazon AWS costs are quite affordable.

Risk

Any business involves risk. Client is postpaid, I may not be able to perform the job. Amazon may fall, and 2captcha may close at any time. When entering a business it is necessary to evaluate all the risks, even that of not receiving anything or being processed for misuse of computing resources:>

Step 3: Hands on

NodeJs application overview

Concurrency

When climbing the first instance of the scrapper, I was able to determine the maximum speed that each worker could operate without raising suspicions that he was not a real human: 1 (one) query every 30 seconds.

Terraform

With just a few apply and destroy commands, I was able to upload multiple instances. Every time an IP was blocked the instance was terminated automatically by the application, so I could even automate the process of deploying an instance by detecting that another had dropped.

Surprisingly, it was not necessary to deal with significant falls, because by respecting the limit of 30 seconds per company per worker could operate 24 hours without raising suspicions.

The advantage of EC2 is that IP is dynamic for each instance, and does not follow an easily detectable lease range.

Aftermath

I was able to accomplish this work in accordance with my expectation of profit and break using combined technologies in an unprecedented way in my experience.

And you, would you do it differently? Share with me!