Performance and Cost Evaluation ofPublic Cloud Cold Storage Services for
Astronomy Data Archive and Analysis
Hiroshi Yoshida, Kento Aida
Center for Cloud Research and Development,
National Institute of Informatics

Cloud Cold Storage Services
AWS GlacierAWS S3 Infrequent AccessAWS S30.005

GCP Coldline Storage
GCP Regional Storage

0.01

Azure block BLOB Cool
Azure block BLOB Hot
Oracle Archive Storage
Oracle Cloud Storage

0.019

0.023
0.015

0.00260.025

0.02
0.0255

Price unit: USD/(GB*month) [as of July 1, 2018, Japan region except Oracle] Storage charge per GB*month is relatively inexpensive
compared to standard object storage services.
 2/3 - 1/10 less expensive
 Drawbacks in exchange for inexpensive storage charge Time consuming restoration process (hours) Extra charge for data retrieval
 Minimal retention period (30 – 90 days) Limited performance, or extra charge for additional performance Reduced availability

Experiment in Cloud Cold Storage Services
 Issue“Is it possible to adopt cloud cold storage to store a large amount ofscientific research data for a long time?” Reduction of storage management labor and TCO is expected.
 However, as very few precedents exist, feasibility in terms ofperformance, manageability, and cost remains unknown.
 Aim of experiments
To acquire practical information to determine the suitability of storing
research data in cloud cold storage, and to design an overall data
storage architecture.
-> Experiments in cold storage services of commercial public clouds Basic benchmark tests including storing up to 1PB data Collaborative case study analyses using actual research data andapplications• High-energy physics (High Energy Accelerator Research Organization)• Astronomy (National Astronomical Observatory of Japan)
3

Experiments Using Astronomy Data
 First step (FY2017 - FY2018)
 Store observation and analysis data of ALMA radio telescopein cloud cold storage services
 Evaluate performance, cost, and manageability
by porting archive management system ”NGAS” to AWS
and storing archive data in S3-IA and Glacier
S3-IA: S3 Infrequent Access Second step (started in FY2019)
 Analyze observation data on cloud instances Expected advantages- Reduction of outbound data transfer cost- Flexible extension of compute resourcesin case of on-premise resource shortage
 Evaluate performance and cost by running analysis software
package “CASA” on the instances of public cloud services
 Investigate optimal selection and usage of instances
based on the measurement results
4Astronomy Data and Application
 For archive
Data contents ALMA radio telescope Observation/analysis data provided by NAOJ *1
Quantity

58.5TiB, 1,380,000 files

Size

Average 44MiB (falls between smaller than 1MiB and larger than 100GiB)

Application

Archive management: NGAS (Next Generation Archive System)*1: National Astronomical Observatory of Japan For analysisDataset #Small (analysis time≃1 hour)
Medium (analysis time≃5 hours)
Large (analysis time≃1 day)
Extra large (analysistime≫1 day)

Datasetid

Number of

Data size

12
34
5
6
7
8
9objects
147
267
99
2,076
4,000
240
3,384
3,879
2,421(GiB)
0.5
0.4
0.6
3.9
3.5
2.2
26.1
16.7
9.0

10

456

87.3

 Application: CASA (Common Astronomy Software Applications)
5Experiment Environment
Cold storage

InternetStandard
object storage

HDD

s3fs
cloudfuse

HDD

S3CLI

gsut
ilHDD

swift

NII
On-premise server in NII
Standard server
Xeon E5-2650 2.0GHz 32 cores
Memory 64GiB 10G ether
External disks configured as RAID10High performance server
Xeon E5-2660 2.6GHz 40 cores RAM
128GiB 10G ether
External RAID connected via iSCSI

ociaz
CLI

AWS (Tokyo）
Glacier
S3-IA

GCP (Tokyo)

S3

Coldline
Standard

OracleRDS

s3fs

S3 CLI:

Analysis instances
…
CASA
CASA
CASA

Lifecycle management

gsutils3fs

Analysis instances
…

:
CASA
CASA
CASA

Oracle (Tokyo)

ArchiveStandard

Azure (Japan East)

BLOB cool
BLOB hotoci:
cloudfuseaz CLI

Analysis instances…
CASACASA
CASA

Analysis instances

:…
CASA
CASA
CASA

6

1st Step Experiment: NGAS on AWS
 Store observation and analysis data of ALMA radio telescopein cloud cold storage services
 Evaluate performance, cost, and manageability
by porting archive management system ”NGAS” to AWS
and storing archive data in S3-IA and Glacier
NGAS instanceClientm4.2xlarge(Virtual 8core memory 32GB)

Metadata DB

Science data
download tool

NGAS
s3fs

RDS (Oracle)
Mounted via s3fs

Archive data
S3-IAGlacier

7

Storing eachfile as a object

Performance of Uploading Archive Data

Throughput (MiB/s)

886.7

900.0
800.0

736.5

700.0
600.0
500.0
400.0
300.0

1380Kobj
58.5TB
281.1h

200.0
100.0

1380Kobj
58.5TB
117.4h

145.3

191Kobj
9.0TB
11.7h

244.8

259Kobj
11.6TB
19.0h

1380Kobj
58.5TB
103.2h

177.3

165.0

60.6

445Kobj
17.5TB
5.75h

6.8万obj
2.58TB
1.0h• Uploaded with CLIs offered by cloudproviders (multiple object upload).
• Because S3-IA, GCP coldline, andAzure BLOB cool show similar
performances to standard object
storage services, some measurements
were substituted by standard storage.
960Kobj
41.2TB
197.9h

60.7

1380Kobj
58.5TB
153.6h68Kobj
2.58TB
6.43h

111.0

116.8

68Kobj
2.58TB
6.28h

119.7

0.0
AWS S3 AWS S3 AWS S3 AWS S3GCP
GCP
GCP
Azure
Azure
Azure
OracleIA 2017 IA 2018 IA 2018
Regional Regional Regional
BLOB BLOB Hot BLOB Hot Archive
2019
High perf. High perf.
2018
2018
2019
Cool 2017
2018
2019
2019
server
Highperf.
Highperf.
Highperf.
Highperf.
ServerServer
ServerServerServer Significant Performance improvement between 2017 and 2018
 More performance improvement in the higher performance server cases
 Performance seems to be degraded between 2018 and 2019.
 Internet connection of providers’ data centers might be more congested
because other measurements of internal transfer (copy objects betweenS3 buckets) in AWS show the same performance (≒ 240MB/s).
8

Cost Estimation of NGAS on AWS
 Store 900TiB archive data in S3- IA  Store 900TiB archive data in Glacierand retrieve 550TiB per year
and retrieve 550TiB per year
Archive data in Glacier $117,128

Archive data in S3-IA $268.516$23,654$6,195$23,654$28,053$28,053$9,580$210,125$55,296

NGAS instance & RDS

s3 IA data storages3 IA data retieval

Outbound data transfer

NGAS instance & RDS
Glacier data storage
Glacier data retrieval with standard restore
Outbound data transfer

 Yearly operation cost: 249,033USD  Yearly operation cost: 95,435USD
 Storage cost is 84% of the total cost.  Restore time (standard: 3.3hours) is required for each
retrieval.
9

2nd Step Experiment: Analyses on AWS
 Evaluate data analysis performance and cost by
running analysis software package “CASA”
on the VM instances of public cloud services.
 Analysis datasets are actually stored in S3-IA and Glacierand retrieval operations from NGAS were simulated.
Analysis instance
CASAEBS（gp2:SSD
st1: HDD)NGAS instance
NGASs3fs

Retrieve data to beanalyzed

Mounted via s3fs

Archive data

S3-IA
Glacier
Storing each file as a object
10

Metadata DBRDS(Oracle)Performance and Cost of Data Analyses in AWS Elapsed times and costs of analyses
using AWS instances with various memory capacities (32 – 244GB)
 Achieved performance is suitable for practical use.
 Performance differences are mainly caused by the differences ofinstance generation rather than memory capacity, because thechosen datasets don’t require large memory capacity for analyses.
1.50

Dataset #1 (small: analysis time¥300 8.00≅ 1 hour)Dataset #4 (medium: ≅ 5 hours)¥2,000 25.00

6.00
1.00

¥1,500Dataset #9 (large: ≅ 24hours)¥10,000

20.00¥8,00015.00¥6,00010.00¥4,0005.00¥2,000

¥200

4.00

0.50

¥1,000

¥100
2.00

0.00

¥0On-premise m5.2xlarge r4.2xlarge x1e.2xlarge
32GB
61GB
244GBAnalysis elapsed time (h)Cost (JPY)

¥500

0.00

¥0
On-premise m5.2xlarge r4.2xlarge x1e.2xlarge
32GB
61GB
244GB
Analysis elapsed time (h)11Cost (JPY)

0.00

¥0
On-premise m5.2xlarge r4.2xlarge x1e.2xlarge
32GB
61GB
244GB
Analysis elapsed time (h)Cost (JPY)Hybrid Configuration
 To optimize performance and cost, it is worth considering a hybrid
cloud configuration including on-premise environment and cloudenvironment with tiered storage.
 Data storing locations and analysis locations should be determinedbased on data retrieval/analysis performance, data transfer
time/cost, cost of cloud/on-premise, and expandability/flexibility.
MetadataDBMS

NGAS

OracleNGAS

Archive data Analysis

CASA

NGAS instance
NGAS(SINET)

Metadata DBAnalysis
instances
CASA
CASA

Metadata DBEBS

RDSArchive data

Migrated bylifecycle
management

S3 Infrequent
Access (IA)Glaciercloud

on-premise12Next Steps Establish methodology to estimate required number of cores andmemory capacity based on the dataset characteristics to choose
optimal instance Investigate optimizations of applications and usages of services
considering the characteristics of cloud cold storage Optimize mapping between files and objects
to improve handling of multiple objects
and to reduce time and cost of restore processinge.g., 1 file to 1 object -> multiple files accessed at a time to 1 object

 Adopt cloud-native object storage API
to improve performance, stability, and predictability of charge
 Instance Swapping Share the practical information on performance and cost
and the best practices of cloud usage
with researchers of other scientific field
13