Performance and Cost Evaluation ofPublic Cloud Cold Storage Services for
Astronomy Data Archive and Analysis
November 2021
Hiroshi Yoshida, Kento Aida
Center for Cloud Research and Development,
National Institute of Informatics© National Institute of Informatics 2021

Cloud Cold Storage Services
 Major public IaaS providers provide cold storage services.
AWS GlacierAWS S3 Infrequent AccessAWS S3
GCP Coldline Storage
GCP Regional Storage

0.005
0.006

Azure block BLOB Cool
Azure block BLOB Hot
Oracle Archive Storage
Oracle Cloud Storage

0.019

0.025
0.023

0.015

0.020.0026

0.0255

Price unit: USD/(GB*month) [as of September 2020, Japan region

 Data store charge █ is relatively inexpensive
compared to standard object storage services █ (2/3 - 1/10).
 Drawbacks• Time consuming restoration process (hours)• Extra charge for data retrieval• Minimal retention period (30 – 90 days)• Limited performance, or extra charge for additional performance• Reduced availability
2© National Institute of Informatics 2021

Experiment in Cloud Cold Storage Services Is it possible to adopt cloud cold storage to store
a large amount of scientific research data for a long time?
 Expectation• Data store in clouds: Reduction of TCO and storage management labor• Data analyses in clouds: Flexibility to adapt to resource requirement
 Performance and cost of public clouds in scientific applications
have not been well studied.
→Difficult to determine whether cold storage services meet the performancerequirements of research applications→Difficult to assess the feasibility of storing and accessing research data incold storage services in terms of cost
 Evaluation using astronomical research data and applications
 Store the observation and analysis data of the ALMA telescope project Port the data archive system “NGAS” to AWS
 Run common analysis applications “CASA” on a variety of instances
to analyze the retrieved observation data inside cloud3© National Institute of Informatics 2021

Data and Application Used in the Experiment Archive data
Datacontents

ALMA radio telescope Observation/analysis data
provided by National Astronomical Observatory of Japan (NAOJ)

Quantity

58.5TiB, 1,380,000 files

Size

Average 44MiB (falls between smaller than 1MiB and larger than 100GiB)

Application Archive management: NGAS (Next Generation Archive System) Analyzed data
Category

Number ofdatasets

Estimated analysis
time

Size (GiB)

Number of included
files

Small

3１hour

0.4～ 0.6

99～ 267

Medium35 hours

2.2～ 3.9

240～4,000Large

31 day9.0～26.1

2,421～3,879

XLarge1≫ 1 day

87.3456

 Application: CASA (Common Astronomy Software Applications)
4© National Institute of Informatics 2021

Experiment Environment
AWS (Tokyo)
Cold storage

InternetStandard
object storage
s3fs
cloudfuse
HDD

Glacier
S3-IA

S3
CLI

S3

GCP (Tokyo)
ColdlinegsutilStandard

HDD
HDD

swift

oci

Standard server
Xeon E5-2650 2.0GHz 32 cores
Memory 64GiB 10G ether
External disks configured as RAID10s3fs

S3 CLI
Analysis instances

:

…
CASA
CASA
CASA

Lifecycle management

gsutils3fs

Analysis instances

:

…
CASA
CASA
CASA

Oracle (Tokyo)
ociArchive

NII
On-premise server in NII

OracleRDS

az
CLI

Standard:
cloudfuseAnalysis instances…
CASCASCAS
A A A

Azure (Japan East)az CLIBLOB coolHigh performance server
Xeon E5-2660 2.6GHz 40 cores RAM
128GiB 10G ether
External RAID connected via iSCSIBLOB hot

5Analysis instances:…
CASCASCAS
A A A© National Institute of Informatics 2021

Target of experiments and System Configuration
 First step (FY2017 – FY2018)
 Evaluate performance, cost, and manageability
by porting archive management system NGAS to AWSand storing archive data in S3-IA and Glacier Second step (FY2019-)
 Analyze observation data on public cloud instances
to evaluate performance and cost
 Investigate optimal selection and usage of instances

Analysis instance
CASA
Block storage(SSD or HDD)NGAS instance
NGAS
s3fs

Metadata DBRDS

Mounted via s3fs

Archive data(S3-IA/Glacier)Store each file as an object6© National Institute of Informatics 2021

First Step Experiment First step (FY2017 – FY2018)
 Evaluate performance, cost, and manageability
by porting archive management system NGAS to AWSand storing archive data in S3-IA and Glacier Second step (FY2019-)
 Analyze observation data on cloud instances on the instances of public
cloud services to evaluate performance and cost
 Investigate optimal selection and usage of instances

Analysis instance
CASA
Block storage(SSD or HDD)NGAS instance
NGASs3fs

Metadata DBRDS

Mounted via s3fs

Archive data(S3-IA/Glacier)Store each file as an object7© National Institute of Informatics 2021

Performance of Uploading Archive Data via Internet900.0• Uploaded with CLIs offered by cloudproviders (multiple object upload).
• Because S3-IA, GCP coldline, and Azure
BLOB cool showed similar performances
to standard object storage services, somemeasurements were substituted bystandard object storage.

800.0
700.0
600.0
500.0
400.01380Kobj 1380Kobj
58.5TB 58.5TB
281.1h117.4h300.0

191Kobj
9.0TB
11.7h

259Kobj
11.6TB
19.0h

68Kobj
2.58TB
4.21h

244.8

200.0
100.0

886.7

Throughput (MiB/s)

145.3

1380Kobj 445Kobj
58.5TB 17.5TB
5.75h
103.2h

736.5

736.7

68Kobj
2.58TB
1.0h

68Kobj
2.58TB
1.0h

1380Kobj 68Kobj
58.5TB 2.58TB
153.6h6.43h68Kobj
2.58TB
6.57h

68Kobj
2.58TB
6.28h

68Kobj
2.58TB
6.37h

116.8

114.4

119.7

118.1

GCP
GCP
GCP
GCP
GCP
Azure
Azure
Azure
Azure
Oracle参考Coldline Regional Regional Regional Regional BLOB
BLOB
BLOB
BLOB ArchiveAWS
2017
2018
2018
2019
2020 Cool 2017 Hot 2018 Hot 2019 Hot 2020 2019
S3-S3Old server Old server
copy 2020 Old server Old server

OracleArchive2020

68Kobj
2.58TB
3.13h

1380Kobj
58.5TB
103.2h

960Kobj
41.2TB
197.9h

240.4
177.3

178.4

165.2
111.0

97.8
60.6

60.7

0.0
AWS S3 AWS S3 AWS S3 AWS S3 AWS S3IA 2017 IA 2018 IA 2018 20192020
Old server Old server

 The performance requirements can be generally fulfilled.
 Significant Performance improvement between 2017 and 2018
 No performance increase after 20188

© National Institute of Informatics 2021

Data Download Performance of NGAS
Throughput (MiB/s)
150

Archive data is stored in S3 IA100500512

1024

2048

4096

819216384

Downloaded amount of data (MiB)

Cloud (network performance= "high")
On-premise 1G

Cloud (25Gps Network >"high")
On-premise 10G

 Performance of cloud NGAS is lower than that of on-premise NGAS Possible causes• Performance difference between S3 and on-premise storage• Overhead of s3fs
• Too small RDS instance Throughputs can be increased by increasing network performance→ Practical performance can be achieved with appropriate sizing.
9© National Institute of Informatics 2021

Download Performance and Cost of NGAS on AWSStore charge ($)

Access charge ($)

Elapsed time (min)S3 IA
Glacierstandardrestore
Glacierbulk restore
0Download time200Restore time

S3 IA: actual measurement
Glacier performances are estimatedfrom manual restore operations

012Restore charge
Read charge
Internet transfer
SINET transfer

00.1

0.2

0.3

Download data length =16,253MiB

 A retrieval from cold storage requires read cost and retrieve cost.
 Restore time and cost are required in Glacier, instead of low store cost.
 Large egress transfer costs are required.
10© National Institute of Informatics 2021

Cost Estimation Model of NGAS on AWS
 Parameters
 Charge for NGAS and RDS instances
based on instance types and local disk capacity
 Capacity of archive data store Amount of archive data retrieval/read• Read request charge can be converted
to the proportional charge to the retrieval amount approximately .
 Amount of egress data transfer
Analysis instance NGAS instance
Data transfer

CASA

via SINET
or via Internet

Local disk

RDSNGAS

Archivedataretrieve
Local disk /read

S3-IA or GlacierArchive data store
11© National Institute of Informatics 2021

Yearly Cost Estimation of NGAS on AWS
 Store 900TiB archive data in S3- IA / Glacier and retrieve 550TiB/year$6,195$28,053$210,125$23,654

S3 IA$268,516$9,580$28,053$0$55,296$50,000$23,654$100,000■ Instance (NGAS,DB)■ Archive data store■ Archive data retrieve/read■ Data transfer (via SINET direct connection)

Glacier$116,584$150,000$200,000$250,000$300,000

 Store cost of Glacier is less than one third of that of S3 IA Tiered storage including Glacier, S3 IA, and on-premise archive can be asolution to balance cost saving with the disadvantages of retrieval time.
 Relatively high egress transfer cost
→ Motivation for the second step experiment (analyses in cloud)12© National Institute of Informatics 2021

Hybrid Configuration of Archive System
 Tiered storage including Glacier, S3 IA, and on-premise storage
OracleDBNGAS

Archive
data

on-premise

Data transferMax.
750TiB Estimation of cloud charge
in a hybrid configuration

NGAS

cloud

Migrated by lifecycle
management

Amount ofdata (TiB)$250,000

2,500
Glacier store cost

 Amount of archive data is 900TiB$200,000in year 0; increases 250TiB/year Datasets are stored in on-premisestorage for 3 years, S3 IA for 2 years, $150,000
and then migrated to Glacier.
 Amount of downloads is 550TiB/year; $100,000
20% from S3 IA and 10% from Glacier(i.e. only 10% of downloads
require additional 200 minuterestore time).

Archive data store
Glacier
S3-IA

S3 IA store cost

2,000

transfer cost
Instances (NGAS, DB)

1,500

Accumulated dataamount (TiB)
1,000$50,000

500$00year year year year year year year year year year year year year
-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 613© National Institute of Informatics 2021

Second Step Experiment First step (FY2017 – FY2018)
 Evaluate performance, cost, and manageability
by porting archive management system ”NGAS” to AWS
and storing archive data in S3-IA and Glacier Second step (FY2019-)
 Analyze observation data on public cloud instances
to evaluate performance and cost
 Investigate optimal selection and usage of instances

Analysis instance
CASA
Block storage(SSD or HDD)NGAS instance
NGAS
s3fs

Metadata DBRDS

Mounted via s3fs

Archive data(S3-IA/Glacier)Store each file as an object

14© National Institute of Informatics 2021

Elapsed Time and Cost of Data Analyses in AWS
 Enough performance for practical use
 Performance differences are not caused by the differences of memory
capacity but by the differences of instance generation.
 The analyses of chosen datasets don’t require large memory capacity.
 61GB and 244GB memory instances belong to the older generations.
1.50

Dataset #1 (small: analysis time¥300 8.00≅ 1 hour)Dataset #4 (medium: ≅ 5 hours)¥2,000 25.00

6.00
1.00

¥1,500Dataset #9 (large: ≅ 24hours)¥10,000

20.00¥8,00015.00¥6,00010.00¥4,0005.00¥2,000

¥200

4.00

0.50

¥1,000

¥100
2.00

0.00

¥0On-premise m5.2xlarge r4.2xlarge x1e.2xlarge
32GB
61GB
244GB
Analysis elapsed time (h)Cost (JPY)

¥500

0.00

¥0On-premise m5.2xlarge r4.2xlarge x1e.2xlarge
32GB
61GB
244GB
Analysis elapsed time (h)15Cost (JPY)

0.00

¥0
On-premise m5.2xlarge r4.2xlarge x1e.2xlarge
32GB
61GB
244GB
Analysis elapsed time (h)Cost (JPY)© National Institute of Informatics 2021

Elapsed Time and Cost of Data Analyses in AWS
 Instances with large capacity memories are expensive.
 A new generation instance is usually less expensive
than old generation one with the same specification.
→ Adopt new generation instances whenever possible
Select instances with appropriate memory capacity

1.50

Dataset #1 (small: analysis time¥300 8.00≅ 1 hour)Dataset #4 (medium: ≅ 5 hours)¥2,000 25.00

6.00
1.00

¥1,500Dataset #9 (large: ≅ 24hours)¥10,000

20.00¥8,00015.00¥6,00010.00¥4,0005.00¥2,000¥200

4.000.50¥1,000

¥100
2.00

0.00

¥0On-premise m5.2xlarge r4.2xlarge x1e.2xlarge
32GB
61GB
244GB
Analysis elapsed time (h)Cost (JPY)

¥500

0.00

¥0On-premise m5.2xlarge r4.2xlarge x1e.2xlarge
32GB
61GB
244GB
Analysis elapsed time (h)16Cost (JPY)

0.00

¥0
On-premise m5.2xlarge r4.2xlarge x1e.2xlarge
32GB
61GB
244GB
Analysis elapsed time (h)Cost (JPY)© National Institute of Informatics 2021

Cost Optimization Points (1): SSD vs HDD
 SSD and HDD can be chosen as block storage media.
 Analysis data, working data, final results are stored in block storage.
 Up to 20% reduction of elapsed time by using SSD
 The increase of instance charge caused by the increase of elapsed time
is higher than the cost difference between SSD and HDD.
→In CASA cases, SSD is always better.
m5.2xlarge/SSD vs HDD

25.000¥1,00020.000¥800

15.000¥600

10.000

¥400

5.000

¥200

0.000

¥0Dataset #1(small)On-premise

Dataset #1
SSD

Dataset #1
HDD

Dataset #4 Dataset #4 Dataset #4 Dataset #9(medium)SSDHDD(large)
On-premise
On-premise
Elapsed time (h)Instance cost (JPY)EBS cost (JPY)

17Dataset #9
SSD

Dataset #9HDD$1=¥110© National Institute of Informatics 2021

Cost Optimization Points (2): Instance Swap
Imaging100

 Observation
80
usr
 Calibration is performed mostly on 1 core.
60
 Imaging is performed on more cores.
Calibration40
 Cloud block storage volumes can20
be dynamically attached to/detached from0
an instance using CLI and/or API
 Instance swap
1) Perform calibration on a 1-core instance2) Attach the volume including working data to a multi-core instanceand perform imaging
 The workflow is automated by using literate computing technology
developed by NII to reduce operation labor.

EBSCalibration
Swap

作業dataImaging

18© National Institute of Informatics 2021

Advantage of Instance Swap Example: dataset#4
46% cost reduction as the elapsed time is retainedor 49% higher performance with additional 12% cost
Elapsed time (dataset #4)

Total cost (dataset #4)

10.00¥400.09.00¥300.08.00$1=¥110¥200.07.00¥100.06.00¥352.4¥192.1¥172.6¥0.0

5.00

Single 1 core instance

2.32

9.14

Single 4 core instance2 instance execution

4.00

dataset #4-1/m5.large (1 core)
6.21

3.00

100

2.00

80

3.80

1.00

60

0.00

40
Single 1 core instanceSingle 4 core instance1 core instance(r1.large 16GB or m5.large 8GB)

#4-2/m5.2xlarge (4 core)

2 instance execution

20

4 core instance(m5.2xlarge 32GB)

0usr19© National Institute ofPerformance and Cost from Archive to Analysis
 Retrieving and analyzing datasets archived in S3 IA and Glacier Total elapsed time
■Total cost
Total elapsed time:
S3-IA
1,200
1,000

5.9

1,200

Total elapsed time:
Glacier
6.0

1,000

221

800

800

600

600
4.3

400

216
400

00.4
54372

Dataset #1 Dataset #4 Dataset #9(small)(medium)(large)IA retrieve time (m)Analysis time (m)

200
0198

¥11

¥900

¥800

¥700

¥700

¥600

¥600
¥500
¥875

¥400932¥5¥300¥10054
Dataset #1 Dataset #4 Dataset #9(small)(medium)(large)Glacier retrieve time (m)Glacier restore time (m)

¥0

¥18

¥400

¥875

¥300

¥200372

¥26

¥900

¥800

¥500

0.3200¥1,000¥1,0003.8932

Total cost: Glacier

Total cost: S3-IA

¥200

¥352
¥1
¥49

¥100

Dataset Dataset Dataset#1#4#9 (large)(small) (medium)

¥0

¥352
¥2
¥49
Dataset #1 Dataset #4 Dataset #9(small)(medium)(large)Glacier restore+retrieve cost (m)S3-IA retrieve cost (JPY)
Instance cost (JPY)

Instance cost (JPY)$1=¥110

 Glacier is advantageous to store cost in spite of long elapsed time and extra charge
 200 minute restore time before analysis■Restore operation charge
20© National Institute of Informatics 2021Hybrid Configuration Including Analysis System Optimization points of archive location and analysis location
 Access frequency of datasets based on the age of data Performance (1) data retrieval (2) data transfer (3) analysis
 Cost ▪Cloud charge (1) data store (2) data retrieve/read(3)egress data transfer (4) analysis instances▪CAPEX+OPEX of the on-premise system
 Resource deployment flexibility in cloud Usability and automatability of workflow
OracleNGASNGASAnalysis

DBAnalysis NGAS
Data transfer instances instanceCASACASACASANGAS

Archive Archive data store
S3-IA Glacierdataretrieve/readCASA

Archive data
CAPEX+OPEX

cloudon-premise
21

Migrated bylifecyclemanagement© National Institute of Informatics 2021

Summary Practical results on storing and analyzing scientific research data in cloudare acquired.
 Practically acceptable performance of data access can be achieved
with appropriate sizing of resources and tuning of the system.
 Inexpensive cold storage services (such as Glacier) are significant
options in terms of store cost, although they require hourly restore time
before accessing data.
• Can be mitigated with an appropriate tiered storage architecture The cost estimation model enables to estimate total cost in cloud.
• Data store and egress transfer costs are major parts of the total cost of
the archive system.
• Data retrieval costs of cold storage services have little effect.
 The model is also capable to estimate cost on a hybrid system organizedby clouds and on-premise systems.
22© National Institute of Informatics 2021

Next Steps Establish methodology to estimate required resources such as
number of cores, CPU usage patterns, memory capacity, and block storage
capacity based on the dataset characteristics to choose optimal instance Optimization of the hybrid system configuration and archive/analysis
locations Investigate optimizations of applications and usages of cloud services
 Improve mapping between files and objects to accelerate restore
operatione.g., 1 file to 1 object → multiple files to 1 object
 Adopt cloud-native object storage API to improve performance Instance swap
 Share the practical information and the best practices of cloud usage withresearchers of other scientific field23© National Institute of Informatics 2021

© National Institute of Informatics 2021