EC2 Pricing Data

#Background
We often want to check for EC2 instance pricing data. Since the volume of instance types are vast and there are also many additional dimensions like *Region*, *Operation*, *UsageType*, etc to define the pricing factors, it's been a time-consuming and terrible experience to find the exact pricing we want to know quickly.

#Where to find pricing data
There are many places to find pricing data:

- [AWS EC2 Pricing Page](https://aws.amazon.com/ec2/pricing/on-demand/) this is the official page on showing prices, but it's kinda slow and hard to find the information you need unless you clicked a fleet of buttons and select boxes. And there is still no way you can search.
- [AWS Cost Calculator](https://calculator.s3.amazonaws.com/index.html) grab instance type and along with other costs to define the budget.
- [ec2instances.info](https://www.ec2instances.info/) A third-party site that you can query and search for pricing data on various dimensions. 
- [AWS Price List API](https://aws.amazon.com/blogs/aws/new-aws-price-list-api/) This is what I want to share in this post.

#AWS Price List API

This API contains all services pricing data which Amazon offer. 

To get the offer code and URLs for all supported services by accessing the Offer Index:

```
https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json
```

It returns like:

```javascript

{
  "formatVersion" : "v1.0",
  "disclaimer" : "This pricing list is for informational purposes only. All prices are subject to the additional terms included in the pricing pages on http://aws.amazon.com. All Free Tier prices are also subject to the terms included at https://aws.amazon.com/free/",
  "publicationDate" : "2015-11-19T02:10:02Z",
 "offers" : {
    "AmazonS3" : {
      "offerCode" : "AmazonS3",
      "currentVersionUrl" : "/offers/v1.0/aws/AmazonS3/current/index.json"
    },
    "AmazonRedshift" : {
      "offerCode" : "AmazonRedshift",
      "currentVersionUrl" : "/offers/v1.0/aws/AmazonRedshift/current/index.json"
    },
    "AmazonEC2" : {
      "offerCode" : "AmazonEC2",
      "currentVersionUrl" : "/offers/v1.0/aws/AmazonEC2/current/index.json"
    },
    "AmazonCloudWatch" : {
      "offerCode" : "AmazonCloudWatch",
      "currentVersionUrl" : "/offers/v1.0/aws/AmazonCloudWatch/current/index.json"
    }
  }
}
```

To take EC2 price index as an example, it contains a huge list of around 400,000 items and a single API call won't return that all information within a body. Luckily, AWS provides all of the data in a CSV format (file size around 304MB), it can be download from this link (don't open it if you are running on a low spec laptop):

https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/index.csv

A screenshot of the CSV (opened on my Mac):

<img width="2062" alt="Screenshot_2018-08-17 10.12.09_glSMiB.png" src="https://qiita-image-store.s3.amazonaws.com/0/73251/cfd856c5-6597-b40f-60f6-3eb27c573d18.png">

As you can see, the **count of items is more than 400,000**! And why are there so many pricing rates data?

While I filtered the CSV by columns like 

- `InstanceType`, 
- `Operation`, 
- `Location`, 
- `TermType`, 
- `UsageType`,
- `Tenancy`, 
- `...`

then I got few lines left which contain the pricing data in a column called `PricePerUnit`, and the number on this cell is presenting the pricing rate for that specific instance in a specific region with specific dimensions. For example:

> After filtering `Region` and `InstanceType`, an m3.medium instance in Tokyo region contains ***204 pricing rate items*** . And the 204 pricing rate items of m3.medium instance are with different rate because there are dimensions like *Operation System* (Windows, Linux/Unix, SUSE, Enterprise Linux, etc are with different prices), *Reservation Terms* (either 3-year or 1-year with upfront option like No-upfront, Partial-upfront and Full-upfront, under Convertible or Standard types), *Pre-installed Softwares*, etc to factor the pricing.

#Dump CSV into dynamoDb
There are many big data solutions to visualize a huge CSV. Stuff like BigQuery, QuickSight, etc. But I decided to dump that into DynamoDb for the sake of fast querying. So after filtered some unused values, I got 309,365 items of EC2 offering data available globally:


<img width="975" alt="Screenshot_2018-08-17 09.51.59_cFq5b1.png" src="https://qiita-image-store.s3.amazonaws.com/0/73251/989fe78a-656a-ef16-b34c-26b0d690ed80.png">

Click on an Item, the pricing data of this specific `m5.12xlarge` rate info looks like this:

<img width="879" alt="Screenshot_2018-08-17 10.11.06_RVlKUt.png" src="https://qiita-image-store.s3.amazonaws.com/0/73251/42a1432f-a073-2e38-4713-b7a791704a6a.png">

Now, I can easily query the pricing info against the instance I wish to search:

<img width="1569" alt="Screenshot_2018-08-17 10.35.30_cOefol.png" src="https://qiita-image-store.s3.amazonaws.com/0/73251/5e2f6b83-b272-8062-bc4c-fa7c9142fdf4.png">

#Interesting Dimensions
The pricing data are so vast because of dimensions. And when you are about to calculate the [true unblended cost](https://mobingi.com/jp/products/wave/faq) by yourself you gotta understand all of these dimensions. I picked some of the most important ones:

- `TermType`: _OnDemand, or Reserved_
- `PriceDescription`: _The description of the pricing item_
- `ProductFamily`: _Compute Instance_, _Data Transfer_ or _EBS Storage_, etc
- `serviceCode`: _AmazonEC2_
- `Location`: _Region, eg: Asia Pacific (Tokyo)_
- `InstanceType`: _The type of instance, eg: c1.large_
- `UsageType`: _This is the dimension used by AWS officially to calculate your monthly invoices. By matching this dimension with the column found in your monthly [CUR](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-reports-costusage.html) csv, aws applies the unit price and generate the total usage cost._
- `Tenancy`: _Shared, Dedicated, or Host_
- `Operation`:  _This defines what OS is the instance running, eg:_
    - `RunInstances` _Amazon EC2 running Linux/UNIX_
    - `RunInstances:000g`  _Amazon EC2 running SUSE Linux_
    - `RunInstances:0010` _Amazon EC2 running Red Hat Enterprise Linux_
    - `RunInstances:0002` _Amazon EC2 running Windows_
    - `RunInstances:0006` _Amazon EC2 running Windows with SQL Server Standard_
    - `RunInstances:0102` _Amazon EC2 running Windows with SQL Server Enterprise_
    - `RunInstances:0202` _Amazon EC2 running Windows with SQL Server Web_
    - `RunInstances:0800` _Amazon EC2 running Windows (Bring your own license)_
    - `RunInstances:SV006` _Amazon EC2 running Linux/UNIX Spot Instance-hour in US East (Virginia) in VPC Zone #6 ..._
    - ...

#About CUR & Self-calculation
CUR (Cost and Usage Report) is a CSV file which aws provides to our S3 bucket (via payer account) on an hourly/daily basis. This CSV file contains all of the services usage data in detail. 

We can hence calculate the cost base on that CUR csv file and get detailed insight of what our AWS usage is allocated to. More detail on understanding CUR and consolidated billing on [this AWS documentation page](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/con-bill-blended-rates.html).

Self-calculation can be tough and human intensive. Especially when the RI blending happens across multiple aws accounts we will have to re-calculate the correct true usage. 

Luckily there are SaaS vendors who are providing such insight services:

- [Cloudability](https://www.cloudability.com/)
- [Mobingi Wave](https://mobingi.com/jp/products/wave)
- [Mobingi Ripple for CSP/MSP](https://mobingi.com/jp/products/ripple)
- [CloudCheckr](https://cloudcheckr.com/)

I'm working at Mobingi team, and we've been helping many aws customers to calculate their billings every month in a total of multi-million dollars. The [getting started](https://docs.mobingi.com/v/ripple/mobingi-ripple/mobingi-ripple-toha) guide on Mobingi Wave is very simple.

I'll share more on the CUR csv format in another article.