VishaalY/solutions-architect-hf-dataset huggingface dataset:
Question: How do I use Auto Scaling to improve the fault tolerance of an application behind my load balancer?

I have an application that's running on Amazon Elastic Compute Cloud (Amazon EC2) instances behind a load balancer. I want to use Auto Scaling to improve the fault tolerance of my application.

ResolutionElastic Load Balancing (ELB) supports the following types of load balancers: Application Load Balancer, Network Load Balancer, and Classic Load Balancer. If you have a preexisting load balancer, then you can create an Auto Scaling group to automatically terminate unhealthy instances and launch new, healthy ones. This improves the fault tolerance of your application. You can also configure Auto Scaling to dynamically scale up your application in response to demand.You can create an Auto Scaling group that launches copies of instances that you've already configured. You can also create a launch template that uses an Amazon Machine Image (AMI) instead. After you create the Auto Scaling group, attach your load balancer to it.Note: You must create your Auto Scaling group in the same AWS Region and Availability Zone as your load balancer.

Related informationTutorial: Set up a scaled and load-balanced application
Question: How do I create Amazon Data Lifecycle Manager policies from selected EBS volumes in the EC2 console?

I want to create Amazon Data Lifecycle Manager policies from my Amazon Elastic Block Store (Amazon EBS) volumes in the Amazon Elastic Compute Cloud (Amazon EC2) console. How do I do this?ResolutionYou can select one or more Amazon EBS volumes in the Amazon EC2 console to create an Amazon Data Lifecycle Manager policy that's used to manage snapshots of the EBS volumes. An Amazon Data Lifecycle Manager policy creates and deletes snapshots according to the policy schedule. This saves you on costs and effort because you don't have to match the snapshots manually.To create an Amazon Data Lifecycle Manager policy:Open the Amazon EC2 console.Make sure that all the EBS volumes that you want Amazon Data Lifecycle Manager to manage have at least one tag with a key-value pair.Select the desired EBS volumes, choose Actions, and then choose Create snapshot lifecycle policy.All the tags for the EBS volumes selected in the previous step are automatically prefilled under Target resource tags.Define the rest of the Amazon Data Lifecycle Manager policy, including the creation frequency and retention period.For step-by-step instructions, see Automating Amazon EBS snapshot and AMI management using Amazon Data Lifecycle Manager.After creating the policy, Amazon Data Lifecycle Manager automatically creates and deletes snapshots based on the policy schedule.Troubleshooting tipsTo create an EBS snapshot lifecycle policy, your EBS volumes must have at least one tag with both a key and a value. If you select one or more volumes that don't have any tags, and then select Create a snapshot lifecycle policy in the Amazon EC2 console, you receive an error.When you select an EBS volume with multiple tags, and at least one tag has both a key and a value, Create snapshot lifecycle policy is prefilled with tags with both keys and values. Any tags with a key but no value aren't prefilled.All the tags of your selected EBS volumes are prefilled with both keys and values. When other volumes aren't selected, but they have at least one of the prefilled tags, the Amazon Data Lifecycle Manager policy creates and manages the snapshots for those volumes.The total number of unique tags in your selected EBS volumes can't be greater than 45.When you remove tags from EBS volumes after a policy is created, the Amazon Data Lifecycle Manager can no longer create snapshots of those volumes. Refer to the Amazon Data Lifecycle Manager documentation for more information.
Question: How can I use Apache Iceberg with a cross-account AWS Glue Data Catalog in Spark?

I want to use Spark with Amazon EMR or AWS Glue to interact with Apache Iceberg from a cross-account AWS Glue Data Catalog.

ResolutionSet the following parameters to use Spark to interact with Apache Iceberg tables from the AWS Glue Data Catalog:--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \  --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog \  --conf spark.sql.catalog.dev.glue.id=<CROSS_ACCOUNT_ID> \  --conf spark.sql.catalog.glue_catalog.warehouse=s3://<your-warehouse-dir>/ \  --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \  --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIOYou can set these parameters in a number of ways, depending on whether you use an AWS Glue job or an Amazon EMR cluster.For AWS Glue jobs, use job parameters. For example:Key:  --conf  Value: spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.dev=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.dev.glue.id=<CROSS_ACCOUNT_ID> --conf spark.sql.catalog.dev.warehouse=s3://<WAREHOUSE_DIR>/ --conf spark.sql.catalog.dev.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.dev.io-impl=org.apache.iceberg.aws.s3.S3FileIOFor an Amazon EMR cluster that runs version 6.5 or later, set the parameters when you submit the job. Or, use the Spark default configuration (/etc/spark/conf/spark-defaults.conf). For more information, see Use an Iceberg cluster with Spark.Note: For cross-account scenarios, you must always use the glue.id property to specify the corresponding AWS Glue Data Catalog ID (AWS account ID).If you're using Amazon EMR version 6.5 or later, then use the following spark-defaults configuration:]    }          "configurations": []          },              "spark.sql.catalog.dev.io-impl": "org.apache.iceberg.aws.s3.S3FileIO"              "spark.sql.catalog.dev.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog",              "spark.sql.catalog.dev.warehouse": "s3://<WAREHOUSE_DIR>/",              "spark.sql.catalog.dev.glue.id": "<CROSS_ACCOUNT_ID>",              "spark.sql.catalog.dev": "org.apache.iceberg.spark.SparkCatalog",              "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",              "spark.jars": "/usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar",          "properties": {          "classification": "spark-defaults",      {      },          "configurations": []          },              "iceberg.enabled": "true"          "properties": {          "classification": "iceberg-defaults",      {  [Note: The Amazon EMR or AWS Glue job must have sufficient AWS Identity and Access Management (IAM) permissions to access the cross-account AWS Glue Data Catalog. For more information, see Making a cross-account API call.

Related informationRead an Iceberg table from Amazon S3 using Spark
Question: How can I grant access to Cost Explorer for an IAM user with a member account of an AWS Organizations?

I want to allow an AWS Identity and Access Management (IAM) user belonging to an AWS Organizations member account access to AWS Cost Explorer.

ResolutionFollow these instructions to allow IAM users for an Organizations member account to access Cost Explorer.1.    Using the Organizations management account, turn on Cost Explorer access for all your organization accounts by following the instructions for Controlling member accounts' access using Cost Explorer preferences.Note: You can't grant access to individual member accounts. All member accounts in your organization are granted access. Member account owners can't view Cost Explorer information for other accounts in your organization.2.    Follow the instructions for Activating access to the AWS Billing console.Note: Only the root user of an account can access the Billing and Cost Management console until this step is completed.3.    Attach an IAM policy to the IAM user to allow access to Cost Explorer information. For example policies, see AWS Billing policy examples.

Related informationBest practices for the management accountBest practices for member accountsManaging the AWS accounts in your organizationAWS Organizations terminology and concepts
Question: How do I create an IAM policy to control access to Amazon EC2 resources using tags?

I want to create an AWS Identity and Access Management (IAM) policy that controls access to Amazon Elastic Compute Cloud (Amazon EC2) instances using tags.

Short descriptionControl access to smaller deployments of Amazon EC2 instances as follows:1.    Add a specific tag to the instances you want to grant the users or groups access to.2.    Create an IAM policy that grants access to any instances with the specific tag.3.    Attach the IAM policy to the users or groups that you want to access the instances.

ResolutionAdd a tag to your group of EC2 instancesOpen the Amazon EC2 console, and then add tags to the group of EC2 instances that you want the users or groups to be able to access. If you don't already have a tag, then create a new tag.Note: Be sure to read and understand the tag restrictions before tagging your resources. Amazon EC2 tags are case-sensitive.Create an IAM policy that grants access to instances with the specific tagCreate an IAM policy that does the following:Allows control over the instances with the tag.Contains a conditional statement that allows access to Amazon EC2 resources if the value of the condition key ec2:ResourceTag/UserName matches the policy variable aws:username. The policy variable ${aws:username} is replaced with the friendly name of the current IAM user when the policy is evaluated by IAM.Allows access to the ec2:Describe* actions for Amazon EC2 resources.Explicitly denies access to the ec2:CreateTags and ec2:DeleteTags actions to prevent users from creating or deleting tags.Note: This prevents the user from taking control of an EC2 instance by adding the specific tag to it.The finished policy looks similar to the following:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": "ec2:*",      "Resource": "*",      "Condition": {        "StringEquals": {          "ec2:ResourceTag/UserName": "${aws:username}"        }      }    },    {      "Effect": "Allow",      "Action": "ec2:Describe*",      "Resource": "*"    },    {      "Effect": "Deny",      "Action": [        "ec2:CreateTags",        "ec2:DeleteTags"      ],      "Resource": "*"    }  ]}Note: This policy applies to Amazon EC2 instances that use the ec2:ResourceTag condition key. To restrict launching new Amazon EC2 instances using tags, see How can I use IAM policy tags to restrict how an EC2 instance or EBS volume can be created?Attach the IAM policy to the users or groups you want to access the instancesFinally, attach the IAM policy that you created to the users or groups you want to access the instances. You can attach the IAM policy using the AWS Management Console, AWS CLI, or AWS API.

Related informationGranting required permissions for Amazon EC2 resourcesIAM policies for Amazon EC2
Question: How do I prevent ThrottlingException or Rate exceeded errors when using AWS Glue?

When I make an AWS Glue API call, I receive a ThrottlingException error message indicating that my rate is exceeded.

Short descriptionWhen you make an AWS Glue API request, you might receive an error similar to one of the following:"Error Code: ThrottlingException; Request ID: 07512ede-123f-49c8-8933-1298742297b9; Proxy: null)" } }{ "taskSubmitFailedEventDetails": { "resourceType": "glue", "resource": "startJobRun.sync", "error": "Glue.AWSGlueException", "cause": "Rate exceeded (Service: AWSGlue; Status Code: 400;""TaskSubmitFailed": {"resourceType": "glue", "resource": "startJobRun.sync", "error": "Glue.AWSGlueException", "cause": "Rate exceeded (Service: AWSGlue; Status Code: 400; Error Code: ThrottlingException; Request ID: 70ea5ab7-2bb8-4a47-b957-799ef46c129e; Proxy: null)" }""botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the GetCrawler operation (reached max retries: 4): Rate exceeded cause": "Rate exceeded (Service: AWSGlue; Status Code: 400; Error Code: ThrottlingException; Request ID: 70ea5ab7-2bb8-4a47-b957-799ef46c129e; Proxy: null)"AWS Glue API requests are throttled for AWS accounts on a per-Region basis to help service performance. For more information, see AWS Glue endpoints and quotas.

ResolutionFollow these best practices to mitigate a ThrottlingException or rate exceeded error:Reduce the frequency of the API calls.Stagger the intervals between your API calls so that they don't all run at the same time.Use APIs that return more than one value. For example, GetPartitions supports a call of 1000 values with one API call. Similarly, GetTables supports a call of 100 parameters with one API call.Implement error retries and exponential backoff when you make API calls.Using the AWS CloudTrail console, check which and how many API calls are sent during a given period of time.If you've tried these troubleshooting steps but still receive rate exceeded errors, then request a service quota increase.Note: Before submitting a request to increase your quota, identify the API call that caused the error and the call rate.

Related informationCommon errors
Question: How can I create notifications for when an Amazon EMR cluster or step changes state?

I want to get a notification when my Amazon EMR cluster or step changes state. How can I accomplish this with Amazon EventBridge?Short descriptionWhen you create an EventBridge rule, you choose a target, such as an Amazon Simple Notification Service (Amazon SNS) topic. The target triggers when a specified event occurs, such as when a cluster state changes.

ResolutionTo create an EventBridge rule that sends an SNS message when an Amazon EMR cluster or step changes state, do the following:Create an SNS topic. This topic is the target for the EventBridge rule.Follow the steps for Creating Amazon EventBridge rules that react to events.For Define pattern, choose Event pattern.Choose Pre-defined pattern by service.For Service provider, choose AWS.For Service Name, choose EMR.For Event type, choose State Change.For Target, select SNS topic.For Topic, select the topic that you created in step 1.Note: You can choose more than one target per rule. For example, you can perform an AWS Lambda function, send an SNS notification, and send a message to an Amazon Simple Queue Service (Amazon SQS) queue at the same time, in the same rule.If the EventBridge rule doesn't behave as expected, see Troubleshooting Amazon EventBridge.

Related informationAmazon EMR events
Question: How can I use the AWS CLI to register a Lambda function as a target behind my Application Load Balancer?

I want to register a Lambda function as a target behind my Application Load Balancer using the AWS Command Line Interface (AWS CLI). How can I do this?ResolutionNote: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.1.    Create a target group with the target type set to Lambda. Be sure to replace [your target group name] with the name of your target.aws elbv2 create-target-group \    --name [your target group's name] \    --target-type lambda2.    Use the add-permission command to allow Elastic Load Balancing to invoke your Lambda function. Be sure to replace [your Lambda function's name] and [your target group's Amazon Resource Name (ARN)] with your respective names.aws lambda add-permission \    --function-name [your Lambda function's name] \    --statement-id load-balancer \    --principal elasticloadbalancing.amazonaws.com \    --action lambda:InvokeFunction \    --source-arn [your target group's Amazon Resource Name (ARN)]3.    Use the register-targets command to register Lambda as the target. Be sure to replace [your target group's ARN] and **[your Lambda function's ARN]**with your ARNs.aws elbv2 register-targets \     --target-group-arn [your target group's ARN] \     --targets Id=[your Lambda function's ARN]
Question: Why does my Amazon EMR cluster fail to launch and return the error "bootstrap action returned a non-zero return code"?

I created a custom bootstrap action for my Amazon EMR cluster. The cluster fails to launch, and the bootstrap action returns an error similar to the following: "bootstrap action 1 returned a non-zero return code".

Short descriptionBootstrap actions run after Amazon EMR provisions the Amazon Elastic Compute Cloud (Amazon EC2) instances in the cluster. When a bootstrap action fails, Amazon EMR terminates the instance. For more information, see Understanding the cluster lifecycle.If a bootstrap action fails because of an error in the bootstrap script, then the cluster can't launch. For more information, see Bootstrap action basics.

ResolutionTo determine why a bootstrap action failed, review the stderr logs for your bootstrap action. These logs are located in the Amazon Simple Storage Service (Amazon S3) path or LogUri that you configured earlier when you created the Amazon EMR cluster.Example: s3://example-log-location/example-cluster-id/node/example-instance-id/bootstrap-actions/If you can't determine why the script failed after reviewing the stderr logs, modify your script to provide additional debug information. For example, set the -ex parameters in the bash script. This allows you to view the bash script flow in the bootstrap action log files.#!/bin/bashset -x -eNote: If the failed bootstrap action isn't a bootstrap action that you created—for example, if you created six bootstrap actions and you see the error message is "bootstrap action 7 failed with non-zero exit code")—that error message indicates that Amazon EMR couldn't install applications or start services.To troubleshoot this issue, check the Amazon EMR provisioning logs. Amazon EMR uses Puppet to install and configure applications on a cluster. The logs might provide information on errors caused during the provisioning of the cluster.You can access these logs on the cluster or S3 if you configured the logs to be pushed to S3. The logs are stored in /var/log/provision-node/apps-phase/0/example-UUID}/puppet.log on the disk and s3://example-log-location/example-cluster-id/node/example-instance-id/provision-node/apps-phase/0/{UUID}/puppet.log.gz.

Related informationUse custom bootstrap actions
Question: How can I configure my Amazon VPC to privately connect to my S3 bucket without using authentication?

I want to create a private connection from my Amazon Virtual Private Cloud (Amazon VPC) to an Amazon Simple Storage Service (Amazon S3) bucket. However, I don't want to use authentication, such as AWS Identity and Access Management (IAM) credentials. How do I create this type of private connection?Short descriptionYou can access an S3 bucket privately without authentication when you access the bucket from an Amazon Virtual Private Cloud (Amazon VPC). However, make sure that the VPC endpoint used points to Amazon S3.Follow these steps to set up VPC endpoint access to the S3 bucket:1.    Create a VPC endpoint for Amazon S3.2.    Add a bucket policy that allows access from the VPC endpoint.

ResolutionBefore you begin, you must create a VPC that you'll access the bucket from.Create a VPC endpoint for Amazon S31.    Open the Amazon VPC console.2.    Using the Region selector in the navigation bar, set the AWS Region to the same Region as your S3 bucket.3.    From the navigation pane, choose Endpoints.4.    Choose Create Endpoint.5.    For Service category, verify that "AWS services" is selected.6.    For Service Name, select the "s3" service name and "Gateway" type. For example, the service name in the US East (N. Virginia) Region is com.amazonaws.us-east-1.s3.7.    For VPC, select your VPC.8.    For Configure route tables, select the route tables based on the associated subnets that you want to be able to access the endpoint from.9.    For Policy, verify that Full Access is selected.10.    Choose Create endpoint.11.    Note the VPC Endpoint ID. You'll need this endpoint ID for a later step.Add a bucket policy that allows access from the VPC endpointUpdate your bucket policy with a condition that allows users to access the S3 bucket when the request is from the VPC endpoint that you created.To allow those users to download objects ( s3:GetObject), use a bucket policy like this one:{   "Version": "2012-10-17",   "Id": "Policy1415115909152",   "Statement": [     {       "Sid": "Access-to-specific-VPCE-only",       "Principal": "*",       "Action": "s3:GetObject",       "Effect": "Allow",       "Resource": ["arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"],       "Condition": {         "StringEquals": {           "aws:sourceVpce": "vpce-1a2b3c4d"         }       }     }   ]}For the value of aws:sourceVpce, make sure to enter the VPC endpoint ID of the endpoint that you previously created.Important: This policy allows access from the VPC endpoint, but it doesn't deny all access from outside the endpoint. If a user from the same account is authenticated, this policy still allows the user to access the bucket from outside the VPC endpoint. For a more restrictive bucket policy, use a policy that explicitly denies access to any requests from outside the endpoint.

Related informationGateway endpoints for Amazon S3
Question: "How can I determine why I was charged for EventBridge usage, and then how can I reduce future charges?"

I'm seeing high Amazon EventBridge charges in my AWS bill. How can I see why I was charged for EventBridge usage, and then how can I reduce future charges?Short descriptionReview your AWS Cost and Usage Reports to understand your EventBridge charges. Look for charges for the following service:CloudWatch Events: AWSEvents PutEvents region-Event(-64K-Chunks)|RequestNote: Items in bold are similar to what you might see in your reports. In your reports, region represents the  abbreviation for your AWS Regions.

ResolutionCharges are based on the number of events published to EventBridge event buses. Events originated from other AWS services are free. Custom events, events originating from third parties (SaaS), and events from another event bus incur a fee. Events sent from one AWS account to another are billed to the sender account as custom events.To reduce costs:Minimize PutEvents calls.Reduce the number of events that are sent to an event bus. These are also charged as custom events.Keep in mind that events ingested by Scheme discovery and Event replay incur charges.There are similar rules for API destination targets that forward events to web-based applications. In this scenario, charges are based on the number of target invocations.

Related informationAmazon EventBridge pricing
Question: How can I resolve the IAM Identity Center error message "This permission set is currently provisioned in xx AWS accounts"?

I tried to delete the AWS IAM Identity Center (successor to AWS Single Sign-On) permission set and received an error similar to the following:"This permission set is currently provisioned in xx AWS accounts".

ResolutionTo delete the permission set, you must first remove it from the AWS account that it's provisioned for.Open the IAM Identity Center console, and then in the navigation pane choose AWS accounts.In AWS Accounts, choose the AWS organization tab.Choose the AWS account that is associated with the permission set that you want to delete.Expand Permission sets, and then choose the permission set that you want to remove.In the Remove permission set dialog box, choose Remove access.If you have multiple AWS accounts associated with the permission set, repeat steps 3-5.Choose AWS Accounts in the navigation pane.Choose the Permissions sets tab, choose the permission set, and then choose Delete.The permission set is now deleted. For more information, see Delete permission sets.

Related informationHow do I use IAM Identity Center permission sets?How do I get started with using IAM Identity Center and access the AWS access portal?