VishaalY/solutions-architect-hf-dataset huggingface dataset:
Question: How do I monitor AWS VPN tunnels using Amazon CloudWatch alarms?

I want to configure Amazon CloudWatch alarms to monitor my AWS VPN tunnels, and then receive notifications when my AWS VPN connection is  down.

Short descriptionMonitor the status of an AWS VPN through its metric data, which is automatically sent to CloudWatch. The VPN tunnel state is reported as a Boolean value in the CloudWatch metric TunnelState. A value of 0 indicates that the tunnel is down and 1 indicates that the tunnel is up. Set up a CloudWatch alarm based on this CloudWatch metric to notify you when one or both VPN tunnels are down.

ResolutionSet up your AWS VPN and confirm that both tunnels are available. Then, configure a CloudWatch alarm to monitor the connection.Set up a topic for your alarm in Amazon Simple Notification Service (Amazon SNS). If you don't have an existing topic, then create an Amazon SNS topic. Then, subscribe to the topic.Create a CloudWatch alarm to monitor the VPN tunnel state.(Optional) Test your alarm by making the tunnels of your AWS VPN unavailable to invoke a notification from your SNS topic.

Related informationMonitoring VPN tunnels using Amazon CloudWatchHow do I check the current status of my VPN tunnel?
Question: How can I use AWS Glue workflows to automatically start a job when a crawler run completes?

I want to use AWS Glue workflows to automatically start a job when a crawler run completes.

Short descriptionTo start a job when a crawler run completes, create an AWS Glue workflow and two triggers: one for the crawler and one for the job. This method requires you to start the crawler from the Workflows page on the AWS Glue console.Note: You can also use an AWS Lambda function and an Amazon EventBridge rule to automate job runs. When you choose this option, the Lambda function is always on. The function monitors the crawler regardless of where or when you start the function. For more information, see How can I use a Lambda function to automatically start an AWS Glue job when a crawler run completes?ResolutionBefore completing the following steps, be sure that you have:An AWS Glue extract, transform, and load (ETL) jobAn AWS Glue crawlerAn AWS Identity and Access Management (IAM) role for AWS Glue that has the AWSGlueServiceRole policy attached to itCreate the workflowOpen the AWS Glue console.In the navigation pane, choose Workflows, and then choose Add workflow.Enter a name for the workflow, and then choose Add workflow. The new workflow appears in the list on the Workflows page.Create the trigger for the crawlerOn the Workflows page, select your new workflow, and then choose the Graph tab.Choose Add trigger, and then choose the Add new tab. For Trigger type, choose On demand.Choose Add. The trigger appears on the graph.On the graph, choose Add node.On the Crawlers tab, select your crawler, and then choose Add.Create the trigger for the AWS Glue jobOn the Action menu above the graph, choose Add trigger.Choose the Add new tab, and then select the following options: For Trigger type, choose Event. For Trigger logic, choose Start after ALL watched event.Choose Add. The trigger appears on the graph.On the graph, to the left of the job trigger that you just created, choose Add node.On the Crawlers tab, select your crawler, and then choose Add. The trigger appears on the graph.On the graph, to the right of the job trigger that you just created, choose Add node.On the Jobs tab, select the job that you want to start when the crawler run completes, and then choose Add.Test the workflowOn the Actions menu, next to the Add workflow button, choose Run. The Last run status column changes to Running.Check the Graph tab to see the status of the workflow. Or, open your corresponding crawler or job to confirm that it's running.

Related informationCreating and building out a workflow manually in AWS Glue
Question: How can I prevent an increasing backlog of messages in my Amazon SQS queue?

The backlog of messages in my Amazon Simple Queue Service (Amazon SQS) queue is increasing. Why does this happen, and how can I prevent the backlog from increasing?Short descriptionStandard and FIFO SQS queuesThe backlog of messages increases under the following circumstances:Producers send messages at a faster rate than the messages are consumed.Consumers dont delete messages within the visibility timeout period. When the SQS queue is polled, messages reappear in the queue.FIFO SQS queuesFor FIFO (First-In-First-Out) SQS queues, an increase in the backlog of messages can also result from either of the following:The FIFO queue 20,000 message buffer limit.A message that belongs to a message group is stuck with a consumer, preventing other messages from the same message group from processing.

ResolutionFollow these best practices to prevent the backlog of messages from increasing.Standard and FIFO SQS queuesSet the SQS queue optimum visibility timeout to allow the consumer to delete messages after processing them within the visibility timeout period. If you dont know how long it takes to process a message, then create a heartbeat for your consumer process. Specify the initial visibility timeout (for example, 2 minutes). Then, if the consumer needs more time to process the message, continue to increase the visibility timeout using the ChangeMessageVisibility API call.Increase the batch size when you make ReceiveMessage API calls. Set the MaxNumberOfMessages parameter value to more than 1 and up to a maximum of 10.Monitor the SQS queue metric Approximate Number of Messages Visible. This metric allows you to see if producers start to produce messages at a higher rate than consumers can consume the messages. To scale horizontally, increase the number of consumers or clients that consume the SQS queue, or increase the number of threads that poll the queue.FIFO SQS queues20,000 message bufferFIFO queues allow a maximum of 20,000 inflight messages. Inflight messages include those received from a queue by a consumer, but not yet deleted from the queue. If you reach the 20,000 quota, Amazon SQS doesnt return error messages. A FIFO queue looks through the first 20,000 messages to determine available message groups. If you have a backlog of messages in a single message group, you cant consume messages from other message groups until you successfully consume messages from the backlog.Scaling over message groupsMessages that belong to the same message group are processed one by one, in the order relative to the message group. When receiving messages with multiple message group IDs, Amazon SQS tries first to return as many messages with the same message group ID as possible. This allows other consumers to process messages with a different message group ID. When messages that belong to a specific message group ID are invisible, no other consumer can process messages with the same message group ID. However, consumers can process messages from other message groups. Try increasing the number of message groups where the order isnt important.
Question: How do I back up my Amazon EBS volumes?

I want to back up my Amazon Elastic Block Store (Amazon EBS) volumes, but I don't know how to do it.

ResolutionTo back up your Amazon EBS volumes, use Amazon EBS snapshots.Use the Amazon Elastic Compute Cloud (Amazon EC2) console, or the AWS Command Line Interface (AWS CLI) to create a snapshot. Make sure that you create the snapshot in the same AWS Region as the volume. You can also create multi-volume snapshots that are point-in-time snapshots of volumes that are attached to an Amazon EC2 instance. To automate snapshot creation and retention, you can use Amazon Data Lifecycle Manager.Note: Creating snapshots incurs charges. You're billed for the amount of data that you store. For more information, see How is my Amazon EBS snapshot bill calculated?
Question: How do I resolve the error message "OOM command not allowed" for an ElastiCache for Redis cluster node?

I received the "OOM command not allowed when used memory greater than 'maxmemory'" error when querying my Amazon ElastiCache for Redis cluster node? How do I resolve this?Short descriptionAn OOM error occurs when an ElastiCache for Redis cluster can't free any additional memory.ElastiCache for Redis implements the maxmemory-policy that's set for the cache node’s parameter group when out of memory. The default value (volatile-lru) frees up memory by evicting keys with a set expiration time (TTL value). When a cache node doesn't have any keys with a TTL value, it returns an error instead.To resolve this error and to prevent clients from receiving OOM command not allowed error messages, do some combination of the following:Set a TTL value for keys on your node.Update the parameter group to use a different maxmemory-policy parameter.Delete some existing keys manually to free up memory.Choose a larger node type.Note: The exact combination of the resolutions you use depends on your particular use case.

ResolutionSet a TTL value for keys on your nodeYou can set a TTL value for keys on your node. The default volatile-lru memory management policy then evicts expired keys to free up memory.If you set a TTL value, you can also use the volatile-ttl setting. This setting makes space by evicting only those keys with an expire time set, starting with the keys with the lowest TTL.Choose a different maxmemory-policy settingSet the maxmemory-policy for a cache node parameter group to one of the following values:allkeys-lru: Make space by evicting less recently used keys first.allkeys-random: Make space by evicting random keys.volatile-random: Make space by evicting random keys with an expire set.Additionally, Redis 4.0 offers least frequently used (LFU) eviction modes:allkeys-lfu: Make space by evicting the key that was accessed the fewest number of times.volatile-lfu: Make space by evicting keys with an expire set, starting with the one that was accessed the fewest number of times.Delete existing keys to free up memoryYou can delete existing keys to free up memory using the DEL command. For more information, see Del key [key...] on the redis.io website.Important: Deleting existing keys temporarily resolves OOM errors. Be sure to also reconsider your memory management strategy.Choose a larger node typeEach node type has a different amount of available memory. You can't modify the amount of available memory on a node. However, you can use a larger node type with more storage.Important: Using a larger node type temporarily resolves OOM errors. Be sure to also reconsider your memory management strategy.

Related informationRedis 4.0.10 parameter changesRedis node-type specific parametersKey eviction - Overview of Redis key eviction policies (LRU, LFU, etc.)
Question: How do I create a backup VPN for my AWS Site-to-Site VPN connection using the same transit gateway?

I want to create a backup AWS Virtual Private Network (AWS VPN) for my AWS Site-to-Site connection using the same transit gateway.

Short descriptionWith dynamic routing, you can use the Border Gateway Protocol (BGP) parameters such as local preference, AS_Path, and multi-exit discriminator (MED) values with your VPN connection. A backup AWS Site-to-Site VPN connection is preferred when primary AWS VPN tunnels are down.Note: The AWS Site-to-Site VPN connection must be dynamic, not static. This is because you can't use the same LAN routes in the routing table for more than one transit gateway attachment. Static routes are then blocked in the transit gateway routing table.

ResolutionFollow these instructions to create a backup AWS VPN using transit gateway:Note: Equal Cost Multipath (ECMP) doesn't need to be enabled or disabled on the transit gateway. The BGP attribute values that's configured for the prefixes determine the tunnel for the ingress and egress direction.1.    Follow the instructions to create a transit gateway attachment to a VPN.2.    For Customer Gateway, choose Existing, and then select the primary AWS VPN gateway ID.3.    For Routing options, choose Dynamic.4.    Follow the instructions to view your VPN attachments and note the new VPN attachment.5.    Follow the instructions to associate a transit gateway route table with the new VPN attachment.To be sure that the preferred tunnel sends traffic from AWS to an on-premises network, see How do I configure my Site-to-Site VPN connection to prefer tunnel A over tunnel B?Related informationExample customer gateway device configurations for dynamic routing (BGP)
Question: How do I troubleshoot Amazon ECS tasks stopping or failing to start while my container exits?

My Amazon Elastic Container Service (Amazon ECS) container exits unexpectedly, and tasks stop or fail to start. How can resolve the issue?Short descriptionYour containers can exit due to application issues, resource constraints, or other issues.For task failure due to image issues, see How do I resolve the "Image does not exist" error when my tasks fail to start in my Amazon ECS cluster?For AWS Fargate tasks that fail due to network configuration or resource constraint issues, see Stopped tasks error codes.

ResolutionTo identify why your tasks stopped, follow these troubleshooting steps:Check for diagnostic information in the service event log.Check stopped tasks for errors. Note: You can see stopped tasks in the returned results for at least one hour.If you already have a log driver configured, then check your application logs for application issues. Otherwise, use the log configuration options in your task definition to send logs to a custom log driver for the container. For example, you can send the logs to Amazon CloudWatch or use a supported log driver. Note the following information on logs, depending on your task's launch type:For ECS tasks other than Fargate: If you're using the default json-file logging driver with the Amazon Elastic Compute Cloud (Amazon EC2) launch type, run the docker logs yourContainerID command. This command checks the Docker logs of the container on your ECS container instance.For Fargate tasks: Captured logs show the command output that you see in an interactive terminal if you run the container locally, in the STDOUT and STDERR I/O streams. The awslogs log driver passes these logs from Docker to Amazon CloudWatch Logs.To address memory constraint issues, follow the instructions at How can I allocate memory to tasks in Amazon ECS?If the awslogs log driver is configured in your task definition, then check Viewing awslogs container logs in CloudWatch Logs.
Question: How do I troubleshoot 504 errors returned while using a Classic Load Balancer?

I see HTTP 504 errors in Classic Load Balancer access logs, Amazon CloudWatch metrics, or when connecting to my service through a Classic Load Balancer.

ResolutionAn HTTP 504 error is an HTTP status code that indicates a gateway or proxy has timed out. When troubleshooting, investigate the following:Check your load balancer’s idle timeout, and then modify if necessaryThe most common reason for an HTTP 504 error is that a corresponding instance didn't respond to the request within the configured idle timeout. By default, the idle timeout for Classic Load Balancer is 60 seconds.If CloudWatch metrics are turned on, check CloudWatch metrics for your load balancer. When latency data points are equal to your configured load balancer timeout value, and HTTPCode_ELB_5XX has data points, at least one request has timed out.To resolve this, you can do one of two things:Modify the idle timeout for your load balancer so that the HTTP request is completed within the idle timeout period.Tune your application to respond more quickly.Make sure that your backend instances keep connections openWhen the backend instance closes a TCP connection to the load balancer before it reaches its idle timeout value, an HTTP 504 error might display.To resolve this, activate keep-alive settings on your backend instances, and then set the keep-alive timeout to a value greater than the load balancer’s idle timeout.(Apache only) Turn off TCP_DEFER_ACCEPTWhen TCP_DEFER_ACCEPT is activated for Apache backend instances, the load balancer initiates a connection by sending a SYN to the backend instance. The backend instance then responds with a SYN-ACK, and the load balancer sends an empty ACK to the backend instance.Because the last ACK is empty, the backend doesn't accept the ACK, and instead resends a SYN-ACK to the load balancer. This can result in a subsequent SYN retry timeout. When FIN or RST aren’t sent before the backend instance closes the connection, the load balancer considers the connection established, but it's not. Then, when the load balancer sends requests through this TCP connection, the backend responds with an RST, generating a 504 error.To resolve this, set both AcceptFilter http and AcceptFilter https to none.(Apache only) Turn off the event MPM, and optimally configure the prefork and worker MPMsIt’s a best practice to turn off event MPM on backend instances that are registered to a load balancer. Because the Apache backend dynamically closes connections, these closed connections might result in HTTP 504 errors.For optimal performance when using the prefork and worker MPMs, and presuming the load balancer is configured with a 60-second idle timeout, use these values:mod_prefork MPMmod_worker MPMTimeout6565KeepAliveTimeout6565KeepAliveOnOnMaxKeepAliveRequests100000AcceptFilter httpnonenoneAcceptFilter httpsnonenone

Related informationMonitor your Classic Load BalancerTroubleshoot a Classic Load Balancer: HTTP errorsWhat are the optimal settings for using Apache or NGINX as a backend server for ELB?
Question: How do I initiate rolling updates when there are no changes to the launch configuration in CloudFormation?

I want to initiate rolling updates in my Auto Scaling group on every AWS CloudFormation stack update, without modifying the launch configuration.

Short descriptionYou can initiate rolling updates for an Auto Scaling group only if you meet specific conditions of the UpdatePolicy attribute.To initiate rolling updates, you can create a toggle parameter in the launch configuration of your CloudFormation template. However, if you change the value of the toggle parameter during a stack update, then the UserData property is modified. Any modification to UserData requires replacement. CloudFormation detects the modification to UserData, and then replaces the LaunchConfiguration resource. This replacement initiates the Auto Scaling rolling update, as defined by the UpdatePolicy attribute.

ResolutionThe following steps assume that your AutoScalingRollingUpdate policy is configured for your Auto Scaling group, and your Auto Scaling group is configured to reference LaunchConfiguration.Important: Make sure that you don't disrupt other elements in the UserData property when you add the toggle parameter to your template. Also, don't add the toggle parameter before cfn-signal.1.    In your CloudFormation template, define Toggle as the parameter.JSON:"Parameters": {        "Toggle": {            "Type": "String",            "AllowedValues": ["true","false"],            "Default": "true"        }    }YAML:Parameters:  Toggle:    Type: String    AllowedValues:      - 'true'      - 'false'    Default: 'true'2.    In the launch configuration of your template, reference the toggle parameter in the UserData property, and then launch your stack. See the following JSON and YAML examples.JSON:"LaunchConfig" : {  "Type" : "AWS::AutoScaling::LaunchConfiguration",  "Properties" : {     "ImageId" : { "Ref" : "ImageID" },     "UserData" : { "Fn::Base64" : { "Ref" : "Toggle" }         ...         ...     },          "InstanceType" : { "Ref" : "Type" }  }}YAML:LaunchConfig:  Type: 'AWS::AutoScaling::LaunchConfiguration'  Properties:    ImageId: !Ref ImageID    UserData:      'Fn::Base64': !Ref Toggle      ...      ...    InstanceType: !Ref TypeImportant: To initiate rolling updates when you update your stack, change the value of the parameter from true to false or false to true, depending on its initial setting.Note: You can use the toggle parameter solution on properties where an update requires replacement, such as LaunchConfigurationName, and for resources such as, AWS::EC2::LaunchTemplate.
Question: Can I safely delete the empty files with the _$folder$ suffix that appear in my Amazon S3 bucket when I use Amazon EMR with Amazon S3?

When I use Amazon EMR to transform or move data into or out of Amazon Simple Storage Service (Amazon S3), several empty files with the "_$folder$" suffix appear in my S3 buckets. What are these files, and is it safe to delete them?ResolutionThe "_$folder$" files are placeholders. Apache Hadoop creates these files when you use the -mkdir command to create a folder in an S3 bucket. Hadoop doesn't create the folder until you PUT the first object. If you delete the "_$folder$" files before you PUT at least one object, Hadoop can't create the folder. This results in a "No such file or directory" error.In general, it's a best practice not to delete the "_$folder$" files. Doing so might cause performance issues for the Amazon EMR job. The exception is if you manually delete the folder from Amazon S3 and then try to recreate the folder in an Amazon EMR job or with Hadoop commands. If you don't delete the "_$folder$" files before you try to recreate the folder, you get the "File exists" error.

Related informationUpload data to Amazon S3Configure an output location
Question: Why did my Auto Scaling group scale down?

My Auto Scaling group scaled down without my intervention. Why did this happen?Short descriptionScale-downs are user initiated, or triggered by configured scale-down policies and scheduled scaling. When a scale-down occurs, an instance is terminated according to the configured termination policy.

ResolutionView your Auto Scaling group scaling history in the Amazon EC2 console, using the AWS Command Line Interface (AWS CLI), or using the AWS API.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.Policy-based scale downIf your Auto Scaling group scales down because of a scale-down policy, a message similar to the following appears in the scaling history:At 2016-05-08T13:55:14Z a monitor alarm My-Scale-Down-Alarm in state ALARMtriggered policy Decrease Group Size changing the desired capacity from 4 to 3.You can adjust your Auto Scaling group scaling policy using the Amazon EC2 console or with the AWS CLI put-scaling-policy command. To configure when a scale-down occurs, adjust the associated Amazon CloudWatch alarm. Or, create a new alarm and then associate the new alarm with the Auto Scaling group scaling policy.User-initiated scale downA scale-down triggered by a user displays an event similar to the following in the scaling history:At 2016-05-13T15:03:47Z a user request update of AutoScalingGroup constraintsto min: 12, max: 20, desired: 13 changing the desired capacity from 14 to 13.You can determine the user that made the API call by viewing your AWS CloudTrail logs.Note: CloudTrail must be configured and enabled before you can begin recording API calls.Scheduled scalingA scale-down initiated by a scheduled scale down action displays an event similar to the following in the scaling history:At 2016-02-12T16:01:25Z a scheduled action update of AutoScalingGroup constraintsto min: 1, max: 5, desired: 2 changing the desired capacity from 3 to 2. At2016-02-12T16:01:25Z the scheduled action ScaleDown executed. Setting max sizefrom 1 to 5. Setting desired capacity from 3 to 2.To view scheduled scaling using the AWS CLI, run the following command. Replace MY-ASG-NAME with the name of your Auto Scaling group.aws autoscaling describe-scheduled-actions --auto-scaling-group-name MY-ASG-NAMEYou can also view and manage scheduled scaling using the Amazon EC2 console. For more information, see Create and manage scheduled actions (console).For more information about schedule-based Amazon EC2 Auto Scaling, see Scheduled scaling.

Related informationTutorial: Set up a scaled and load-balanced applicationMonitoring CloudWatch metrics for your Auto Scaling groups and instancesGetting Amazon SNS notifications when your Auto Scaling group scalesLogging Amazon EC2 Auto Scaling API calls with AWS CloudTrailAWS CLI autoscaling commands