Package: s3torchconnector
Amazon S3 Connector for PyTorchThe Amazon S3 Connector for PyTorch delivers high throughput for PyTorch training jobs that access or store data in
Amazon S3. Using the S3 Connector for PyTorch
automatically optimizes performance when downloading training data from and writing checkpoints to Amazon S3,
eliminating the need to write your own code to list S3 buckets and manage concurrent requests.Amazon S3 Connector for PyTorch provides implementations of PyTorch'sdataset primitivesthat you can use to load
training data from Amazon S3.
It supports bothmap-style datasetsfor random data
access patterns anditerable-style datasetsfor
streaming sequential data access patterns.
The S3 Connector for PyTorch also includes a checkpointing interface to save and load checkpoints directly to
Amazon S3, without first saving to local storage.Getting StartedPrerequisitesPython 3.8 or greater is installed (Note: Using 3.12+ is not recommended as PyTorch does not support).PyTorch >= 2.0 (TODO: Check with PyTorch 1.x)Installationpipinstalls3torchconnectorAmazon S3 Connector for PyTorch supports only Linux via Pip for now. For other platforms, seeDEVELOPMENTfor build instructions.ConfigurationTo uses3torchconnector, AWS credentials must be provided through one of the following methods:If you are using this library on an EC2 instance, specify an IAM role and then give the EC2 instance access to
that role.Install and configureawscliand runaws configure.Set credentials in the AWS credentials profile file on the local system, located at:~/.aws/credentialson Unix or macOS.Set theAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYenvironment variables.ExamplesAPI docsare showing API of the public components.
End to end example of how to uses3torchconnectorcan be found under theexamplesdirectory.Sample ExamplesThe simplest way to use the S3 Connector for PyTorch is to construct a dataset, either a map-style or iterable-style
dataset, by specifying an S3 URI (a bucket and optional prefix) and the region the bucket is located in:froms3torchconnectorimportS3MapDataset,S3IterableDataset# You need to update <BUCKET> and <PREFIX>DATASET_URI="s3://<BUCKET>/<PREFIX>"REGION="us-east-1"map_dataset=S3MapDataset.from_prefix(DATASET_URI,region=REGION)iterable_dataset=S3IterableDataset.from_prefix(DATASET_URI,region=REGION)# Randomly access to an item in map_dataset.object=map_dataset[0]# Learn about bucket, key, and content of the objectbucket=object.bucketkey=object.keycontent=object.read()len(content)# Datasets are also iterators.forobjectiniterable_dataset:print(object.key)In addition to data loading primitives, the S3 Connector for PyTorch also provides an interface for saving and loading
model checkpoints directly to and from an S3 bucket.froms3torchconnectorimportS3Checkpoint

importtorchvision
importtorchCHECKPOINT_URI="s3://<BUCKET>/<KEY>/"REGION="us-east-1"checkpoint=S3Checkpoint(region=REGION)model=torchvision.models.resnet18()# Save checkpoint to S3withcheckpoint.writer(CHECKPOINT_URI+"epoch0.ckpt")aswriter:torch.save(model.state_dict(),writer)# Load checkpoint from S3withcheckpoint.reader(CHECKPOINT_URI+"epoch0.ckpt")asreader:state_dict=torch.load(reader)model.load_state_dict(state_dict)Using datasets or checkpoints withAmazon S3 Express One Zonedirectory buckets requires only to update the URI, followingbase-name--azid--x-s3bucket name format.
For example, assuming the following directory bucket namemy-test-bucket--usw2-az1--x-s3with the Availability Zone ID
usw2-az1, then the URI used will look like:s3://my-test-bucket--usw2-az1--x-s3/<PREFIX>(please note that the
prefix for Amazon S3 Express One Zone should end with '/'), paired with region us-west-2.ContributingWe welcome contributions to Amazon S3 Connector for PyTorch. Please
seeCONTRIBUTINGFor more information on how to report bugs or submit pull requests.DevelopmentSeeDEVELOPMENTfor information
about code style, development process, and guidelines.Compatibility with other storage servicesS3 Connector for PyTorch delivers high throughput for PyTorch training jobs that access or store data in Amazon S3.
While it may be functional against other storage services that use S3-like APIs, they may inadvertently break when we
make changes to better support Amazon S3. We welcome contributions of minor compatibility fixes or performance
improvements for these services if the changes can be tested against Amazon S3.Security issue notificationsIf you discover a potential security issue in this project we ask that you notify AWS Security via ourvulnerability reporting page.Code of conductThis project has adopted theAmazon Open Source Code of Conduct.
SeeCODE_OF_CONDUCT.mdfor
more details.LicenseAmazon S3 Connector for PyTorch has a BSD 3-Clause License, as found in theLICENSEfile.