TITLE: Check the integrity of a dataset stored on Amazon S3
AUTHORS: Sonia  García-Ruiz

[DESCRIPTION]

Background

Amazon Simple Storage Service (Amazon S3) has become a widely used and reliable platform for storing large biomedical datasets. However, unintended changes to the original data can occur during the data writing and transmission, ultimately altering the original contents of the transferred object and generating unexpected results when later accessed. Despite the interest in verifying end-to-end data integrity, there are no existing open-source and easy-to-use tools to accomplish this mission; 

Results

To bridge this gap, here we present aws-s3-integrity-check, a user-friendly, lightweight and reliable bash tool to verify the integrity of a dataset stored within an Amazon S3 bucket. By using this tool, we completed the integrity verification of >392K genomic and transcriptomic records ranging between 5 Bytes and 10 Gigabytes (GB) in size and occupying a total of 4.5 Terabytes (TB) of Amazon S3 cloud storage space in less than 13.5 hours. The aws-s3-integrity-check tool also provides file-by-file on-screen and log-file-based information about the status of each individual integrity check; 

Conclusions

To the best of our knowledge, the aws-s3-integrity-check bash tool is the only open-source tool that allows verifying the integrity of a dataset uploaded to the Amazon S3 Storage system in a quick, reliable and efficient manner. The aws-s3-integrity-check tool can be used to test any file type and file size and it is freely available for use and download at https://github.com/SoniaRuiz/aws-s3-integrity-check and https://hub.docker.com/r/soniaruiz/aws-s3-integrity-check.



[BEFORE_START]
1. This protocol will guide you through the installation of the following software dependencies:

jq (version jq-1.5-1-a5b5cbe, https://stedolan.github.io/jq/) . jq is a command-line JSON processor.
xxd (version 1.10 27oct98 by Juergen Weigert, https://manpages.ubuntu.com/manpages/bionic/en/man1/xxd.1.html). xxd is a command-line binary/hexadecimal conversion tool.
AWS Command Line Interface (CLI), (version 2, https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) 
s3md5 (https://github.com/antespi/s3md5)

2. This protocol will also ask you to authenticate on AWS using either the 'aws configure' or 'aws configure sso' command in your Linux command-line terminal.

IMPORTANT: for the correct operation of this protocol, JSON must be chosen as the preferred output format during the 'aws configure' and aws configure sso' commands execution.

For example, in case you choose to authenticate on AWS using the "aws configure" command:
 
3. This protocol requires a dataset stored on an Amazon S3 bucket and an additional copy of that dataset stored locally. This requirement is due to the fact that the 'aws-s3-integrity-check' tool checks the integrity of a dataset on Amazon S3 by comparing the contents of those files with the contents of a local version of those files, so it can verify that the integrity of the files has not been compromised during the data transmission. Please, skip this requirement in case your files are already stored on an Amazon S3 bucket and there is an existing copy of them stored locally.

To upload your local files to your S3 bucket, you can use the AWS CLI 'sync' command as follows:

 

To download a dataset from Amazon S3, you can use the AWS CLI 'sync' command as follows:

 


[STEPS]
SECTION: Clone GitHub repositories and tool configuration
5. Clone the s3md5 repo:

 

SECTION: Clone GitHub repositories and tool configuration
6. Grant execution access to the s3md5 script file:

 

SECTION: Clone GitHub repositories and tool configuration
7. Clone the aws-s3-integrity-check repo:

 

SECTION: Clone GitHub repositories and tool configuration
8. Move the 's3md5' folder within the 'aws-s3-integrity-check' folder:

 

SECTION: Clone GitHub repositories and tool configuration
9. The 'aws-s3-integrity-check' folder should now look similar to the following structure:

 

SECTION: Bash Tool Execution
10. Run the 'aws-check-integrity.sh' bash script following the instructions below:

 
Usage :

[-l|--local <path_local_folder>]. This argument is required. Path to a local folder containing the original version of the files uploaded to Amazon S3. Example: -l /data/nucCyt/raw_data/
[-b|--bucket <bucket_name>]. This argument is required. The name of the Amazon S3 bucket containing the files uploaded from the local folder [-l|--local <path_local_folder>]. Example: -b nuccyt
[-p|--profile <aws_profile>]. This argument is optional. AWS profile in case the user has authenticated on AWS using the command *aws configure sso*. Example: -p my_aws_profile
[-h|--help]. This argument is optional. Shows further help options. 

Example #1 (if the user has authenticated on Amazon s3 using SSO):
 

Example #2 (if the user has authenticated on Amazon s3 using SSO):
 

Example #3 (if the user has authenticated on Amazon s3 using KEY + SECRET):
 

Example #4:
 

SECTION: General info
11. The 'aws-s3-integrity-check' tool will:

Process and evaluate the arguments [-l|--local], [-b|--bucket] and/or [-p|--profile] received.
If the arguments received are correct, this script will test whether the user had read access over the files contained within the Amazon S3 bucket indicated using the parameter: [-b|--bucket <bucket_name>]. 
If the user has correctly authenticated on AWS and has read access over the files stored on the indicated Amazon S3 bucket, this script will interact with Amazon S3 and retrieve the ETag number assigned to the totality of the files contained within the bucket. This query will return a JSON metadata object.
Next, in case the local path indicated through the parameter [-l|--local <path_local_folder>] exists, is a directory, and the user has read access to it, this tool will loop through its files. Per each file, it will check whether its name exists among the entries retrieved on the JSON metadata object, indicated within the “Key” field. If that is the case, the local file exists on the indicated remote Amazon S3 bucket.
If the file exists on the S3 bucket, this tool will evaluate the size of the local file prior to calculating its checksum value. If the data content of the file is smaller than 8 Megabytes (MB), it calculates its Content-MD5 value by using the function md5sum. If the file is larger than 8 MB, the aws-s3-integrity-check tool will automatically call the function s3md5 (version 1.2, https://github.com/antespi/s3md5) using the command “s3md5 8 local_file_path”. 
The aws-s3-integrity-check' tool will then filter the metadata JSON object by the fields “ETag” and “Key” using the function select (jq JSON processor, version jq-1.5-1-a5b5cbe, https://stedolan.github.io/jq/) and will compare the local and remote checksum values for the local file. In case both numbers are identical, the integrity of the local file will be proven. 
This process will automatically be repeated for each of the files contained within the local folder [-l|--local <local_folder_path>].

Finally, the aws-s3-integrity-check tool will inform the user about the outcome of each aforementioned step by using on-screen messages and log files in a .txt format. The name of the log files will follow the following pattern: bucketname_S3_integrity_log.timestamp.txt. The log file will be stored within a folder named 'logs'. In case this folder doesn't exist, it will be automatically created by the script in the same path in which the aws-s3-integrity-check.sh has been executed.
SECTION: AWS Authentication
4. Authenticate on Amazon Web Services (AWS) using AWS CLI. Depending on the authentication mode chosen, the AWS command to use may differ.

If you log in on AWS using an IAM role (KEY + SECRET), more info:

 

If you log in on AWS using the AWS Single Sign-On (SSO) service, more info:

 

**IMPORTANT: for the correct function of this tool, you should choose json as the preferred output format during the AWS authentication process.
SECTION: Install software dependencies
1. Check if the command-line JSON processor jq is installed on your Ubuntu computer:

 

In case it is not installed, please run the following command to install it 
 

SECTION: Install software dependencies
2. Check if the command-line binary/hexadecimal conversion tool xxd is installed on your Ubuntu computer:

 


In case it is not installed, please run the following command to install it 
 

SECTION: Install software dependencies
3. Check if the AWS Command Line Interface (CLI) is installed on your Ubuntu computer:

 

In case it is not installed, please run the following command to install it 
 

AWS CLI is an open-source tool to interact with AWS services using commands in a command-line shell. AWS CLI also provides direct access to all Amazon S3 API operations.
SECTION: Docker
12. To run the Dockerised version of this tool, please follow the substeps indicated below.

SECTION: Docker
12.3. The Docker image will require the user to indicate the following volumes as parameters:

[-v <local_folder>:<local_folder>]. Required argument. Please, replace the strings [<local_folder>:<local_folder>] with the absolute path to a local folder containing the files to be tested. This argument is used to mount the local folder as a local volume to the Docker image to allow the Docker image to have read access to the files to test. Important: reference the local folder only using the absolute path to the folder. Example: -v /data/nucCyt:/data/nucCyt
[-v "$PWD/logs/:/usr/src/logs"]. Required argument. This argument should not be changed and sent to the Docker image as it is shown. It represents the path to the local log folder. This argument is used to mount the local logs folder as a local volume to the Docker image. It allows the Docker image to record the outputs produced during the tool execution within the local “logs/” folder.
[-v "$HOME/.aws:/root/.aws:ro"]. Required argument. This argument should not be changed and, therefore, it should be sent to the Docker image as it is shown. It represents the path to the folder containing the information about the user authentication on Amazon S3. This parameter is used to mount the local AWS credential directory as a read-only volume to the Docker image. It allows the Docker image to have access to the authentication information of the user on AWS.

**IMPORTANT: volumes (-v) need to be specified before the name of the Docker image.
SECTION: Docker
12.5. Usage. Considering the local folder"/data/nucCyt/"containing a dataset uploaded to the Amazon S3 bucket"nuccyt":

Example #1 (if the user has authenticated on Amazon s3 using SSO):
docker run -v /data/nucCyt:/data/nucCyt -v "$PWD/logs:/usr/src/logs" -v ${HOME}/.aws/credentials:/root/.aws/credentials:ro soniaruiz/aws-s3-integrity-check:latest -l /data/nucCyt/ -b nuccyt

Example #2 (if the user has authenticated on Amazon s3 using an IAM role (KEY + SECRET) ):
 

SECTION: Docker
12.2. Download the Docker image:

 

SECTION: Docker
12.4. The Docker image will require the user to indicate the following arguments:

[-l|--local <local_folder>].Required argument. Path to a local folder containing the original version of the files uploaded to Amazon S3. Example: -l /data/nucCyt/raw_data/
[-b|--bucket <bucket_name>]. Required argument. Name of the Amazon S3 bucket containing the files uploaded from the local folder '-l <local_folder>'. Example: -b nuccyt
[-p|--profile <aws_profile>]. Optional argument. AWS profile in case the user has logged in using the commandaws configure sso. Example: -p my_aws_profile
[-h|--help]. Optional argument. Shows further help options.
SECTION: Docker
12.1. Check if the Docker engine is installed on your Ubuntu machine:

 

In case it is not installed, please install it on your Ubuntu machine. This step requires multiple commands, so please check the latest guidance here.