Question: How can I reduce unexpected high costs for Amazon S3?

The charges for Amazon Simple Storage Service (Amazon S3) usage on my AWS bill are higher than expected, and I want to reduce them.

Short descriptionAmazon S3 charges are based on storage, requests and data retrievals, data transfer (including S3 Transfer Acceleration usage), and data management.To identify where you can reduce your Amazon S3 charges, review your current billing and usage for Amazon S3. Then, review how you incur costs.The following tips can help you reduce your Amazon S3 billing costs:Clean up incomplete multipart uploads.Delete previous versions of objects that you don't need.Review your storage-class transition costs.Review your data retrieval costs.Track the requests to your bucket.Check for changes in the size of your bucket.Review the cost of individual buckets.Understand how your usage relates to your charges.You can also use the following Amazon S3 features to help you investigate billing concerns:Amazon S3 Storage Lens to obtain a single view of usage and activity across your Amazon S3 storage.Amazon S3 storage class analysis to observe data access patterns.

ResolutionClean up incomplete multipart uploadsAfter you initiate a multipart upload, Amazon S3 retains all the uploaded parts of the file until you complete or shut down the multipart upload. If you don't complete the multipart upload, then Amazon S3 continues to store the uploaded parts. As a result, you're charged for the storage of uploaded parts.If you use the AWS Command Line Interface (AWS CLI) to shut down the multipart upload, then the operation deletes the incomplete uploaded parts. However, if you use other tools that use the Amazon S3 multipart upload API, then incomplete multipart uploads might leave the uploaded parts.Here are some additional best practices for when you clean up incomplete multipart uploads:Create a lifecycle policy to clean up incomplete multipart uploads after a certain number of days.Use Amazon S3 API calls to list your multipart uploads. Then, shut down multipart uploads that failed or that you want to cancel.Use S3 Storage Lens to determine the number and size of incomplete multipart uploads for buckets in your account.Amazon S3 Storage Lens to identify which S3 buckets contain multipart uploads.Delete previous versions of objects that you don't needImportant: Amazon S3 can't restore any data that you delete from your buckets.If you activated versioning on your bucket, then each object can have multiple versions. Each object version contributes to your storage costs.To review your object versions, use the Amazon S3 console or run the ListObjectVersions API call. Then, delete any object versions that you don't need.Note: In a version-activated bucket, you must specify the object version ID to permanently delete an object version. If you run a simple delete request on an object in a version-activated bucket, then Amazon S3 adds a delete marker to the object. The delete marker becomes the current version of the object, and the actual object becomes the previous version. However, this doesn't erase the object and its versions.You can also use the following methods:A lifecycle rule to manage the current and non-current versions of your objects. However, because you can't reverse permanent deletions in Amazon S3, review the parameters for your lifecycle rule.S3 Storage Lens to determine the number of current versions, previous versions, and delete markers for specific buckets or all buckets in your account.Review your storage-class transition costsYou might incur costs when you transition objects from one storage class to another.For example, when objects transition from S3 Standard to the S3 Standard-IA storage class, you're charged $0.01 for every 1,000 lifecycle transition requests. When objects transition from any storage class to the S3 Glacier Flexible Retrieval storage class, you're charged $0.03 for every 1,000 lifecycle transition requests. For more information and pricing details, see Amazon S3 pricing.If you set up a lifecycle configuration rule that transitions an entire bucket's objects to another storage class, then expect an increase in Amazon S3 charges. To determine if an increase in your charges is because of a lifecycle rule, review your AWS CloudTrail event history. You can also review your Amazon S3 server access logs for PutBucketLifecycleConfiguration calls to identify when the rule was applied.You can also use S3 Storage Class Analysis to determine how much data is retrieved from the different storage classes.Note: You must configure S3 storage class analysis on a per bucket basis.Review your data retrieval costsObjects that are stored in the following storage classes have costs that are associated with data retrievals:S3 Standard-IAS3 One Zone-IAS3 Glacier Flexible RetrievalS3 Glacier Deep Archive storage classesFor example, you retrieve 15 million objects from the S3 Glacier Flexible Retrieval storage class. The total cost of retrieval is based on the bulk retrieval option that you choose.The following example costs are for a standard retrieval:The retrieval request is 15,000,000 / 1000 * $0.05 = $750.The retrieval is 100,000 * $0.01 = $1,000.The total cost is $1750.The following example costs are for an expedited retrieval:The retrieval request is 15,000,000 / 1000 * $10 = $150,000.The retrieval is 100,000 * $0.03 = $3,000.The total cost is $153,000.For objects that you store in S3 Glacier Flexible Retrieval, there's no charge for bulk retrievals.Tip: To help you monitor the different storage requests and data usage, you can also use the S3 Storage Lens dashboard.Track the requests to your bucketYou can monitor requests to your bucket in the following ways:Server access logging. For more information on how to review access logs, see Amazon S3 server access log format.Object-level logging through AWS CloudTrail.Amazon CloudWatch metrics, such as AllRequests and BytesDownloaded.To reduce you costs from requests, you can also use bucket policies or AWS Identity and Access Management (IAM) policies. These policies prevent unauthorized access or limit public access to your buckets.To reduce your costs from data transfer OUT charges, you can use an Amazon Elastic Compute Cloud (Amazon EC2) instance to access your bucket. The Amazon EC2 instance must be in an Amazon Virtual Private Cloud (Amazon VPC) within the same AWS Region as your bucket.Check for changes in the size of your bucketTo isolate storage costs, compare the current size of your bucket with the size of your bucket from a previous month.1.    Open the Amazon S3 console.2.    Choose Buckets, and then select your bucket.3.    Choose the Management tab.4.    Choose Metrics.You can also use the AWS CLI to review your bucket's storage size, or you can monitor your storage metrics with Amazon CloudWatch. To monitor bucket sizing changes on a daily, weekly, and monthly basis, use S3 Storage Lens.Review the cost of individual bucketsTo identify which bucket incurs higher costs, activate the cost allocation tags on your buckets. Then, review your buckets with the AWS Cost Explorer. For more information, see How do I find the cost of my Amazon S3 buckets?Understand how your usage relates to your chargesYour usage report records storage usage in byte hours, and your billing report records storage usage in GB months. To understand how your storage usage relates to your storage charges, convert byte hours into GB months. For an example of this conversion, see Understanding your AWS billing and usage reports for Amazon S3.You can download your Amazon S3 usage report from the AWS Billing and Cost Management dashboard. Or, to obtain detailed storage and billing metrics, create an S3 Storage Lens dashboard for all the buckets in your account.