VishaalY/solutions-architect-hf-dataset huggingface dataset:
Question: How do I allow my IP address while blocking other IP addresses using AWS WAF?

I've set up AWS WAF and I need to allow my IP address while blocking other IP addresses using AWS WAF. How can I do this?ResolutionAWS WAF can inspect the source IP address of a web request against a set of IP addresses and address ranges. You can create a rule that blocks requests from all IPs except the specific IPs in an IP set.First, create an IP setOpen the AWS WAF console.In the navigation pane, choose IP sets, and then choose Create IP set.Enter an IP set name and Description - optional for the IP set. For example: MyTrustedIPs.Note: You can't change the IP set name after you create the IP set.For Region, choose the AWS Region where you want to store the IP set. To use an IP set in web ACLs that protect Amazon CloudFront distributions, you must use Global (CloudFront).For IP version, choose the version that you want to use.For IP addresses, enter one IP address or an IP address range per line that you want to allow in CIDR notation.Note: AWS WAF supports all IPv4 and IPv6 CIDR ranges except for /0.Examples:To specify the IPv4 address 10.20.0.5, enter 10.20.0.5/32.To specify the IPv6 address 0:0:0:0:0:ffff:c000:22c, enter 0:0:0:0:0:ffff:c000:22c/128.To specify the range of IPv4 addresses from 10.20.0.0 to 10.20.0.255, enter 10.20.0.0/24.To specify the range of IPv6 addresses from 2620:0:2d0:200:0:0:0:0 to 2620:0:2d0:200:ffff:ffff:ffff:ffff, enter 2620:0:2d0:200::/64.Review the settings for the IP set. If the IP set matches your specifications, choose Create IP set.Then, create an IP match ruleIn the navigation pane, under AWS WAF, choose Web ACLs.For Region, select the AWS Region where you created your web ACL.Note: Select Global if your web ACL is set up for Amazon CloudFront.Select your web ACL.Choose Rules, and then choose Add Rules, Add my own rules and rule groups.For Name, enter a name to identify this rule. For example: Block-Other-IPs.For Type, choose Regular rule.For If a request, choose doesn't match the statement (NOT).On Statement, for Inspect, choose Originates from IP address in.For IP Set, choose the IP Set you created earlier. For example: MyTrustedIPs.For IP address to use as the originating address, choose Source IP address.For Action, choose Block.Choose Add rule.Choose Save.The IP match rule blocks any IP not added to the IP set. For IPs added to an IP set, the request is evaluated by other rules below the rule. If there isn't a match, the web ACL default action is applied. For more information, see Processing order of rules and rule groups in a web ACL.

Related informationHow do I use AWS WAF to block HTTP requests that don't contain a User-Agent header?
Question: How do I configure Amazon EMR to run a PySpark job using Python 3.4 or 3.6?

Python 3.4 or 3.6 is installed on my Amazon EMR cluster instances, but Spark is running Python 2.7. I want to I upgrade Spark to Python 3.4 or 3.6.

Short descriptionIn most Amazon EMR release versions, cluster instances and system applications use different Python versions by default:Amazon EMR release versions 4.6.0-5.19.0: Python 3.4 is installed on the cluster instances. Python 2.7 is the system default.Amazon EMR release versions 5.20.0 and later: Python 3.6 is installed on the cluster instances. For 5.20.0-5.29.0, Python 2.7 is the system default. For Amazon EMR version 5.30.0 and later, Python 3 is the system default.To upgrade the Python version that PySpark uses, point the PYSPARK_PYTHON environment variable for the spark-env classification to the directory where Python 3.4 or 3.6 is installed.

ResolutionOn a running clusterAmazon EMR release version 5.21.0 and laterSubmit a reconfiguration request with a configuration object similar to the following:[  {     "Classification": "spark-env",     "Configurations": [       {         "Classification": "export",         "Properties": {            "PYSPARK_PYTHON": "/usr/bin/python3"          }       }    ]  }]Amazon EMR release version 4.6.0-5.20.x1.    Connect to the master node using SSH.2.    Run the following command to change the default Python environment:sudo sed -i -e '$a\export PYSPARK_PYTHON=/usr/bin/python3' /etc/spark/conf/spark-env.sh3.    Run the pyspark command to confirm that PySpark is using the correct Python version:[hadoop@ip-X-X-X-X conf]$ pysparkThe output shows that PySpark is now using the same Python version that is installed on the cluster instances. Example:Python 3.4.8 (default, Apr 25 2018, 23:50:36)Welcome to      ____              __     / __/__  ___ _____/ /__    _\ \/ _ \/ _ `/ __/  '_/   /__ / .__/\_,_/_/ /_/\_\   version 2.3.1      /_/Using Python version 3.4.8 (default, Apr 25 2018 23:50:36)SparkSession available as 'spark'.Spark uses the new configuration for the next PySpark job.On a new clusterAdd a configuration object similar to the following when you launch a cluster using Amazon EMR release version 4.6.0 or later:[  {     "Classification": "spark-env",     "Configurations": [       {         "Classification": "export",         "Properties": {            "PYSPARK_PYTHON": "/usr/bin/python3"          }       }    ]  }]Related informationConfigure SparkApache SparkPySpark documentation
Question: How can I share an encrypted Amazon RDS DB snapshot with another account?

I have an encrypted snapshot of an Amazon Relational Database Service (Amazon RDS) DB instance. It uses the default AWS Key Management Service (AWS KMS) key. I want to share an encrypted snapshot of a DB instance with another AWS account.

Short descriptionYou can't use the default AWS KMS encryption key to share a snapshot that's encrypted. For more information about the limitations of sharing DB snapshots, see Sharing encrypted snapshots.To share an encrypted Amazon RDS DB snapshot, complete the following steps:Add the target account to a custom (non-default) KMS key.Use the customer managed key to copy the snapshot, and then share the snapshot with the target account.Copy the shared DB snapshot from the target account.Note: You can also follow the steps in the AWSSupport-ShareRDSSnapshot AWS Systems Manager Automation document to share your snapshot. Provide a snapshot to copy and share with the target account. You can also provide the DB instance or DB cluster ID to share with snapshots. Provide an existing KMS Key, or keep it blank to create a new key. For more information, see Add a key policy statement in the local account and Run an automation.

ResolutionAllow access to the target account on the AWS KMS key of the source accountLog in to the source account, and then open the AWS KMS console in the same AWS Region as the DB snapshot.Choose Customer managed keys from the navigation pane.Choose the name of your customer managed key. If you don't have a key, then choose Create key. For more information, see Creating keys.From the Key administrators section, Add the AWS Identity and Access Management (IAM) users and roles who can administer the AWS KMS key.From the Key users section, Add the IAM users and roles who can use the AWS KMS key (KMS key) to encrypt and decrypt data.In the Other AWS accounts section, choose Add another AWS account, and then enter the AWS account number of the target account. For more information, see Allowing users in other accounts to use a KMS key.Copy and share the snapshotOpen the Amazon RDS console, and then choose Snapshots from the navigation pane.Choose the name of the snapshot that you created, choose Actions, and then choose Copy Snapshot.Choose the same AWS Region that your KMS key is in, and then enter a New DB Snapshot Identifier.In the Encryption section, choose the KMS key that you created.Choose Copy Snapshot.Share the copied snapshot with the target account.Copy the shared DB snapshotLog in to the target account, and then open the Amazon RDS console.Choose Snapshots from the navigation pane.From the Snapshots pane, choose the Shared with Me tab.Select the DB snapshot that you shared.Choose Actions. Then, choose Copy Snapshot to copy the snapshot into the same AWS Region and with a KMS key from the target account.After you copy the DB snapshot, you can use the copy to launch the instance.

Related informationHow can I change the encryption key used by my Amazon RDS DB instances and DB snapshots?Encrypting Amazon RDS resourcesCopying a DB snapshot
Question: How do I view my Reserved Instance utilization and coverage?

I want to see my Reserved Instance (RI) utilization and coverage.

Short descriptionYou can use AWS Cost Explorer to generate the RI utilization and RI coverage reports. With these reports, you can dive deeper into RIs for the following services:Amazon Elastic Compute Cloud (Amazon EC2)Amazon ElastiCacheAmazon MemoryDB for RedisAmazon OpenSearch ServiceAmazon RedshiftAmazon Relational Database Service (Amazon RDS)

ResolutionRI utilization and coverageYou can use your RI utilization reports and RI coverage reports to understand where your highest RI costs are. Also, how you can potentially lower RI costs.To view your RI utilization and RI coverage reports, do the following:Open the AWS Cost Management console.In the left navigation pane, choose Reports.To view the RI utilization report, choose RI Utilization.To view the RI coverage report, choose RI Coverage.In the right navigation pane for Report parameters, select the service for which you require the utilization data.For information on how to use the RI utilization reports, see How do I use the Reserved Instance utilization report in Cost Explorer?For information on how to use the RI coverage reports, see How do I use the Reserved Instance coverage report in Cost Explorer?RI recommendationsTo view RI purchase recommendations that could help reduce your costs, in the navigation pane choose Reservations, and then choose Recommendations. For more information, see Accessing Reserved Instance recommendations.

Related informationReserved Instance reportsUnderstanding your reservations with Cost ExplorerReserved Instance reportingHow can I use Cost Explorer to analyze my spending and usage?
Question: How do I resolve an HTTP 503 Service Unavailable error in Amazon OpenSearch Service?

When I query my Amazon OpenSearch Service domain, I get an HTTP 503 Service Unavailable error. How do I resolve this error?Short descriptionA load balancer sits in front of each OpenSearch Service domain. The load balancer distributes incoming traffic to the data nodes. An HTTP 503 error indicates that one or more data nodes in the cluster is overloaded. When a node is overloaded by expensive queries or incoming traffic, it doesn't have enough capacity to handle any other incoming requests.Note: You can use the RequestCount metric in Amazon CloudWatch to track HTTP response codes.

ResolutionUse one of the following methods to resolve HTTP 503 errors:Provision more compute resourcesScale up your domain by switching to larger instances, or scale out by adding more nodes to the cluster. For more information, see Creating and managing OpenSearch Service domains.Confirm that you are using an instance type that is appropriate for your use case. For more information, see Choosing instance types and testing.Reduce the resource utilization for your queriesConfirm that you are following best practices for shard and cluster architecture. A poorly designed cluster can't use all available resources. Some nodes might be overloaded while other nodes sit idle. OpenSearch Service can't fetch documents from overloaded nodes. For more information about shard and cluster best practices, see Get started with OpenSearch Service: How many shards do I need?Reduce the number of concurrent requests to the domain.Reduce the scope of your query. For example, if you run a query for a specific time frame, reduce the date range. You can also filter the results by configuring the index pattern in OpenSearch Dashboards.Avoid running select * queries on large indices. Instead, use filters to query a part of the index and search as few fields as possible.Re-index and reduce the number of shards. The more shards you have in your cluster, the more likely it will result in a courier fetch error. Because each shard has its own resource allocation and overheads, a large number of shards can strain your cluster. To lower your shard count, see Why is my Amazon OpenSearch Service domain stuck in the "Processing" state?Related informationHow can I prevent HTTP 504 gateway timeout errors in Amazon OpenSearch Service?Best practices for Amazon OpenSearch ServiceTroubleshooting Amazon OpenSearch Service
Question: How do I run security assessments or penetration tests on AWS?

I want to run a security test or other simulated event on my AWS architecture.

ResolutionTo carry out penetration tests against or from resources on your AWS account, follow the policies and guidelines at Penetration Testing. You don't need approval from AWS to run penetration tests against or from resources on your AWS account. For a list of prohibited activities, see Customer service policy for penetration testing.If you plan to run a security test other than a penetration test, see the guidelines at Other simulated events.Note: You're not permitted to conduct any security assessments of AWS infrastructure that isn't on your AWS account. You also aren't permitted to conduct security assessments of AWS services themselves. If you discover a security issue within any AWS service in the course of your security assessment, contact AWS Security immediately.To request permission for network stress-testingBefore stress-testing your network, review the Amazon EC2 Testing Policy. If your planned tests exceed the limits outlined in the policy, then submit a request using the Simulated Event form.Important: Submit the simulated event request at least 14 business days before your planned test. Provide a full description of your plan, including expected risks and outcomes.To request permission for other simulated eventsFor any other simulated events, submit a request using the Simulated Event form. Provide a full description of your planned event, including details, risks, and desired outcomes.Other simulated event types can include:Red, blue, or purple teamCapture the flagDisaster recoverySimulated phishingMalware testing
Question: How do I select the best Amazon API Gateway Cache capacity to avoid hitting a rate limit?

My API Gateway is rate limiting and I want to prevent throttling. How can I select the best API Gateway Cache capacity for my workload?Short descriptionAmazon API Gateway throttles requests to your API to prevent it from being overwhelmed by too many requests. Turn on API caching to reduce the number of calls made to your endpoint.There are multiple API Gateway Cache sizes available. To select the appropriate cache size, run a load test on your API and then review the Amazon CloudWatch metrics.

ResolutionTurn on API Gateway cachingTurn on Amazon API Gateway caching for your API stage. The cache capacity depends on the size of your responses and workload.Note: Cache capacity affects the CPU, memory, and network bandwidth of the cache instance. As a result, cache capacity can affect the performance of your cache.After creating your cache, run a load test to determine if the cache size is high enough to prevent throttling.Run a load testRun a load test on your API. You can use AWS Distributed Load Testing to simulate the load test.Run the load test for at least 10 minutes and mirror your production traffic. When the load test is running, monitor related CloudWatch metrics using the steps in the following section.Monitor API metrics in CloudWatchOpen the CloudWatch console.In the navigation pane, select Metrics.Choose the ApiGateway metric.Monitor Latency, 4XXError, 5XXError, CacheHitCount, and CacheMissCount metrics for the API that you are load testing against.If you see an increase in Latency, 4XXError, 5XXError or CacheMissCount with a CacheHitCount decrease, then resize your API Gateway cache to a larger capacity.If you see an increase in CacheHitCount and no corresponding increase in CacheMissCount, then resize your API Gateway cache to a smaller capacity.After any changes to your cache's capacity, run the load test again until there are no sudden increases or decreases.

Related informationAmazon API Gateway pricingTurn on API caching to enhance responsivenessAmazon API Gateway dimensions and metrics
Question: Do I need to have a configuration set to get Amazon SES bounce details?

I want to get more information about my Amazon Simple Email Service (Amazon SES) messages that result in bounces. Do I need to use a configuration set to get bounce details?ResolutionNo, configuration sets (for Amazon SES event publishing) aren't the only way to get details about bounces on Amazon SES. You can set up bounce notifications in one of these ways:You can send bounce notifications to an Amazon Simple Notification Service (Amazon SNS) topic. Because Amazon SNS notifications use the JSON format, you can process the notifications programmatically.You can use email feedback forwarding, which is enabled by default. If you previously disabled email feedback forwarding, you can re-enable the feature using the Amazon SES console.
Question: Why can't I create a WorkSpaces BYOL image?

I tried to create an Amazon WorkSpaces bring your own license (BYOL) image, but it failed. What can I do to fix this issue?ResolutionFollow these steps to resolve a WorkSpaces BYOL image validation failure:1.    Connect to the virtual machine (VM) in the on-premises environment. Download and run the BYOL Checker PowerShell script. The BYOL Checker script verifies that the image you want to import meets all the requirements for a successful import. The script also provides a detailed report of any issues that must be corrected before image creation.Important: All BYOL Checker script tests must pass for WorkSpaces image validation to succeed.The BYOL Checker script generates two log files: BYOLPrevalidationlogYYYY-MM-DD_HHmmss.txt and ImageInfo.text. These files are located in the directory that contains the BYOL Checker script files. Review the log files for errors.If issues persist after running the BYOL Checker script and reviewing the log files, then perform the following steps.2.    If your imported BYOL image has a firewall turned on, verify in the VM that the firewall isn't blocking any necessary ports. Certain ports are required for downloading scripts used for image creation.3.    Verify that the applications installed on your WorkSpace are compatible with Sysprep. Run the following PowerShell command to list all the AppX packages installed:Get-AppxPackage -AllUsers | Format-List -Property PackageFullName,PackageUserInformationIf image creation fails at the Sysprep stage, see the article about Sysprep failures on the Microsoft website.4.    Make sure that the disk isn't encrypted. Image creation fails if the disk used for importing a VM is encrypted.

Related informationBring Your Own Windows desktop licensesHow do I troubleshoot WorkSpaces image creation issues?
Question: How can I identify the usage associated with an API key for API Gateway?

How can I get the usage associated with an API key for Amazon API Gateway?ResolutionFollow these instructions to get the API key usage using either the AWS Management Console or the AWS Command Line Interface (AWS CLI).Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.Use the AWS Management ConsoleOpen the API Gateway console.In the navigation pane, choose APIs.Choose your API, and then choose Usage Plans.Choose your Usage Plan.Choose Actions, and then choose Export Usage Data.Choose the export From and To date range.For Export as, choose JSON or CSV, and then choose Export.For more information, see Create, configure, and test usage plans with the API Gateway console.Use the AWS CLIYou can use the AWS CLI command get-usage to get the usage data of a usage plan in a date range similar to the following:aws apigateway get-usage --usage-plan-id <usage-plan-id> --start-date "20xx-xx-xx" --end-date "20xx-xx-xx" --key-id <api-key-id>Note: The usage date range can't exceed 90 days.For more information, see Create, configure, and test usage plans using the API Gateway CLI and REST API.

Related informationBest practices for API keys and usage plans
Question: How do I resolve "no space left on device" stage failures in Spark on Amazon EMR?

I submitted an Apache Spark application to an Amazon EMR cluster. The application fails with a "no space left on device" stage failure similar to the following:Job aborted due to stage failure: Task 31 in stage 8.0 failed 4 times, most recent failure: Lost task 31.3 in stage 8.0 (TID 2036, ip-xxx-xxx-xx-xxx.compute.internal, executor 139): org.apache.spark.memory.SparkOutOfMemoryError: error while calling spill() on org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@1a698b89 : No space left on device

Short descriptionSpark uses local disks on the core and task nodes to store intermediate data. If the disks run out of space, then the job fails with a "no space left on device" error. Use one of the following methods to resolve this error:Add more Amazon Elastic Block Store (Amazon EBS) capacity.Add more Spark partitions.Use a bootstrap action to dynamically scale up storage on the core and task nodes. For more information and an example bootstrap action script, see Dynamically scale up storage on Amazon EMR clusters.

ResolutionAdd more EBS capacityFor new clusters: use larger EBS volumesLaunch an Amazon EMR cluster and choose an Amazon Elastic Compute Cloud (Amazon EC2) instance type with larger EBS volumes. For more information about the amount of storage and number of volumes allocated for each instance type, see Default Amazon EBS storage for instances.For running clusters: add more EBS volumes1.    If larger EBS volumes don't resolve the problem, attach more EBS volumes to the core and task nodes.2.    Format and mount the attached volumes. Be sure to use the correct disk number (for example, /mnt1 or /mnt2 instead of /data).3.    Connect to the node using SSH.4.    Create a /mnt2/yarn directory, and then  set ownership of the directory to the YARN user:sudo mkdir /mnt2/yarnchown yarn:yarn /mnt2/yarn5.    Add the /mnt2/yarn directory inside the yarn.nodemanager.local-dirs property of /etc/hadoop/conf/yarn-site.xml. Example:<property>    <name>yarn.nodemanager.local-dirs</name>    <value>/mnt/yarn,/mnt1/yarn,/mnt2/yarn</value></property>6.    Restart the NodeManager service:sudo stop hadoop-yarn-nodemanagersudo start hadoop-yarn-nodemanagerAdd more Spark partitionsDepending on how many core and task nodes are in the cluster, consider increasing the number of Spark partitions. Use the following Scala code to add more Spark partitions:val numPartitions = 500val newDF = df.repartition(numPartitions)

Related informationHow can I troubleshoot stage failures in Spark jobs on Amazon EMR?