VishaalY/solutions-architect-hf-dataset huggingface dataset:
Question: How do I troubleshoot the failure of my Amazon EMR Spark job using Amazon Athena?

My Spark job on Amazon EMR has failed. I want to troubleshoot the failure by querying the Spark logs using Amazon Athena.

ResolutionWhen Amazon EMR applications run on Amazon EMR, they produce log files. You can create a basic table for the EMR log files and then use Athena to query these EMR logs. You can identify events and trends for applications and clusters by querying the EMR logs.Run a command similar to the following to create a basic table myemrlogs based on the EMR log files saved to your Amazon S3 log location:CREATE EXTERNAL TABLE `myemrlogs`(  `data` string COMMENT 'from deserializer')ROW FORMAT DELIMITED  FIELDS TERMINATED BY '|'LINES TERMINATED BY '\n'STORED AS INPUTFORMAT   'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'LOCATION  's3://aws-logs-111122223333-us-west-2/elasticmapreduce/j-1ABCDEEXAMPLE/containers/application_1111222233334_5555/'Replace the following in the above query:myemrlogs with the name of the table111122223333 with the AWS account numberj-1ABCDEEXAMPLE with the clusterIDus-west-2 with your preferred Regionapplication_1111222233334_5555 with the application IDNote: The S3 bucket mentioned in the example is the default bucket used by Amazon EMR. To verify your log bucket path, open the Amazon EMR console, choose your cluster, and then check the Log URI field in the Summary tab.Then, run a command similar to the following to check for occurrences of FAIL, ERROR, WARN, EXCEPTION, FATAL, or CAUSE in myemrlogs:SELECT *,"$PATH" FROM myemrlogs WHERE regexp_like(data, 'FAIL|ERROR|WARN|EXCEPTION|FATAL|CAUSE') limit 100;Note: Replace myemrlogs with the name of the table that you’ve created from the EMR log files.The EMR logs can be queried in different ways to find out at which step the Spark application failed. Here are a few ways to query the logs to troubleshoot if the application failed at the job, stage, task, or executor level.Run a command similar to the following to get the exit code of the application:SELECT *,"$PATH" FROM myemrlogs WHERE regexp_like(data, 'exitCode');Run a command similar to the following to check which host the Spark executor is running on:SELECT *,"$PATH" FROM myemrlogs WHERE regexp_like(data, 'executor ID');Run a command similar to the following to track the mapping of tasks to stages:SELECT *,"$PATH" FROM myemrlogs WHERE regexp_like(data, 'TID');Run a command similar to the following to check the heap memory details of containers:SELECT *,"$PATH" FROM myemrlogs WHERE regexp_like(data, 'space');Run a command similar to the following to track the progress of each job/stage on the Directed Acyclic Graph (DAG) scheduler:SELECT *,"$PATH" FROM myemrlogs WHERE regexp_like(data, 'DAGScheduler');You can also create a partitioned table based on Amazon EMR logs and then use Athena to query these logs. For more information, see Creating and querying a partitioned table based on Amazon EMR logs.

Related informationHow can I troubleshoot stage failures in Spark jobs on Amazon EMR?
Question: How can I automate the RefreshCache operation on Storage Gateway?

I have a Network File System (NFS) or Server Message Block (SMB) file share on my AWS Storage Gateway file gateway. The file gateway is associated with an Amazon Simple Storage Service (Amazon S3) bucket. However, when I upload files directly to the bucket, I can't see the files on my file share. How can I automate the refresh of my file share?ResolutionStorage Gateway updates the file share cache automatically when you write files to the cache locally using the file share. However, Storage Gateway doesn't automatically update the cache when you upload a file directly to Amazon S3. When you do this, you must perform a RefreshCache operation to see the changes on the file share.Storage Gateway allows you to automate the RefreshCache operation based on a Time To Live (TTL) value. TTL is the length of time since the last refresh. When a user accesses the file directory after the TTL value, the file gateway refreshes the directory's contents from the S3 bucket. Valid TTL values for automating the RefreshCache operation range from 300 seconds to 2,592,000 seconds (5 minutes to 30 days).Note: After your specified TTL value expires, the file gateway invokes the ListObjects Amazon S3 API call at the folder level. Setting the TTL to a low value can result in more Amazon S3 API requests.To automate the RefreshCache operation using the Storage Gateway console, follow these steps:Open the Storage Gateway console.Choose File shares.Choose the file share that you want to automate the RefreshCache operation for.Choose Actions, and then choose Edit File Share Settings.For Automated cache refresh from S3 after, select Set refresh interval.Set the time in days, hours, and minutes to refresh the file share's cache using TTL.Choose Save.To automate the RefreshCache operation using the Storage Gateway API, use the UpdateNFSFileShare API call or the UpdateSMBFileShare API call. For both API calls, you need to update the CacheAttributes parameter to automate the RefreshCache operation.To run RefreshCache at one point in time instead of automating the operation, see Why aren't changes on my Amazon S3 bucket showing on the Storage Gateway file share?Note the following considerations about the RefreshCache operation:The time that the refresh process takes to complete depends on the following:Number of objects cached on the gateway.Number of objects added to or removed from the bucket since the last refresh.Refreshing the cache initiates the refresh operation. However, file refresh is a separate process. You can set up refresh cache notifications to get updates on the file refresh process. The refresh-complete notification indicates that the file refresh process is complete.
Question: How do I troubleshoot ReceiptHandleIsInvalid and InvalidParameterValue errors when calling the AWS SQS DeleteMessage API?

I receive a ReceiptHandleIsInvalid or InvalidParameterValue error when I try to delete messages from my Amazon Simple Queue Service (Amazon SQS) queue. How do I successfully delete messages?Short descriptionEvery time you receive a message from an Amazon SQS queue, you receive a receipt handle for the message. The handle is associated with the action of receiving the message rather than the message itself. To delete the message, you must provide the receipt handle.The error message varies depending on whether you use FIFO queues or standard SQS queues, and whether the receipt handle is incorrect or expired.FIFO SQS queuesIf the receipt handle is incorrect, then you see an error similar to the following:An error occurred (ReceiptHandleIsInvalid) when calling the DeleteMessage operation: The input receipt handle 'YOUR RECEIPT HANDLE' is not a valid receipt handle.For FIFO queues, a receipt handle expires after a visibility timeout period. When a receipt handle expires, the message returns to the queue. When you receive the message again, you get a new receipt handle. If you try to delete the message outside of the visibility timeout period, then you see an error similar to the following:An error occurred (InvalidParameterValue) when calling the DeleteMessage operation: Value 'YOUR RECEIPT HANDLE' for parameter ReceiptHandle is invalid. Reason: The receipt handle has expired.Standard SQS queuesIf the receipt handle is incorrect, then you see an error similar to the following:An error occurred (ReceiptHandleIsInvalid) when calling the DeleteMessage operation: The input receipt handle 'YOUR RECEIPT HANDLE' is not a valid receipt handle.For standard queues, you might sometimes succeed in using an older receipt handle. However, it’s a best practice to use the latest receipt handle for DeleteMessage API calls. Using an older receipt handle can result in an error similar to the following:An error occurred (InvalidParameterValue) when calling the DeleteMessage operation: Value 'YOUR RECEIPT HANDLE' for parameter ReceiptHandle is invalid. Reason: The receipt handle has expired.

ResolutionTo prevent invalid or expired receipt handle errors, you must delete the message within the SQS queue visibility timeout period using the correct receipt handle.If you receive a message more than once, review the most recent ReceiveMessage API call response and identify the ReceiptHandle value. Then, use the identified ReceiptHandle value when you call the DeleteMessage API.

Related informationReceiveMessageDeleteMessage
Question: What outbound caller ID is used when calls in my Amazon Connect contact center are transferred to an external number?

When agents in my Amazon Connect contact center transfer calls to an external number, what outbound caller ID appears on the destination number's side?ResolutionNote: There are three ways agents in your Amazon Connect contact center can transfer calls to an external number. The outbound caller ID that appears on the external destination number's side depends on the transfer method used.If the call is transferred with an external quick connectThe outbound caller ID depends on if the agent is on an active call or not.If the agent is on an active callThe original queue that the call is serviced from provides the outbound caller ID.For more information, see Set up outbound caller ID.If the agent isn't on an active callThe outbound queue of the agent's routing profile provides the outbound caller ID.Note: You can override the outbound caller IDs in your agents' routing profiles by using the Call phone number block in a custom outbound whisper flow.For more information, see Create quick connects.If the call is transferred with the Quick Connect number padThe original queue that the call is serviced from provides the outbound caller ID.For more information, see Transfer calls to a quick connect or external number.If the call is transferred with the Transfer to phone number contact block in a contact flowThe outbound caller ID depends on if a caller ID is configured in your Transfer to phone number contact block or not.If a caller ID is configuredThe configured caller ID appears as the outbound caller ID on the external destination number's side.If a caller ID isn't configuredThe caller's caller ID appears as the outbound caller ID on the external destination number's side.For more information, see the Properties section of Contact block: Transfer to phone number.Important: If you're using Amazon Connect outside of the United States, it's a best practice to set the Caller ID number as an Amazon Connect number. Local regulations may cause telephony providers to block or redirect non-Amazon Connect phone numbers.

Related informationWhy your caller ID might not appear correctly to customers
Question: How can I use AWS Health Aware to send events to channels across my organization in Organizations?

I want to use AWS Health to send events to channels across my organization in AWS Organizations.

Short descriptionYou can use AWS Health Aware (AHA) to send events to channels such as Slack, Microsoft Teams, and Amazon Chime across your organization. With AHA, you can get aggregated public or account-specific events for accounts in your AWS Organizations.For more information on the benefits of using AHA, see AWS Health Aware – customize AWS Health alerts for organizational and personal AWS accounts.

ResolutionTo deploy AHA, see Deployment Options on the GitHub website.Note: Because AHA uses the Health API, you must have a Business or Enterprise Support plan.

Related informationAre you a first-time AWS Health user?
Question: How can I access and download the results of an Amazon Athena query?

I want to find the results of my Amazon Athena queries.

ResolutionUse one of the following options to access the results of an Athena query:Download the query results files using the Athena console.Download the query results files from the Amazon Simple Storage Service (Amazon S3) that you specified for the query location. For more information, see Specifying a query result location.

Related informationViewing recent queries
Question: How do I configure my CloudFront distribution to use an SSL/TLS certificate?

I want to configure my Amazon CloudFront distribution to use an SSL/TLS certificate.

ResolutionCloudFront assigns a default domain name to your distribution, for example, d111111abcdef8.cloudfront.net. If you use this domain name, then you can use the CloudFront default SSL/TLS certificate already selected for your distribution. If you use a different domain name for your distribution, then it's a best practice to do one of the following to avoid domain-name-related certificate warnings:Request a public certificate from AWS Certificate Manager.Import certificates into AWS Certificate Manager.If you use an Amazon issued certificate:You must request the certificate in the US East (N. Virginia) Region.You must have permission to use and request the ACM certificate.If you use an imported certificate with CloudFront:Your key length must be 1024 or 2048 bits and cannot exceed 2048 bits.You must import the certificate in the US East (N. Virginia) Region.You must have permission to use and import the SSL/TLS certificate.Note: If you are missing permissions, the CloudFront console displays Missing permission acm:ListCertificates in the Custom SSL Certificate settings. If you don't have a certificate in the US East (N. Virginia) Region, or if your key size exceeds 2048 bits, the setting for Custom SSL Certificate is grayed out.For more information, see requirements for using SSL/TLS certificates with CloudFront.Then, configure your CloudFront distribution to use the new certificate and require HTTPS between viewers and CloudFront. For more information, see viewing and updating CloudFront distributions.After you save the changes to your CloudFront distribution configuration, CloudFront propagates the changes to all edge locations. The status of your CloudFront distribution in the CloudFront console changes from InProgress to Deployed when propagation is complete.If you require HTTPS for communication between CloudFront and your custom origin, then you can also use SSL/TLS certificates on a custom originTo update settings for your CloudFront distribution, see updating your CloudFront distribution.

Related informationAWS Certificate Manager benefitsIssuing and managing certificatesHow can I troubleshoot issues with using a custom SSL certificate for my CloudFront distribution?Clients are receiving certificate error messages when trying to access my website using HTTPS connections. How do I resolve this?
Question: How do I set up dynamic port mapping for Amazon ECS?

I want to set up dynamic port mapping for my container instance in Amazon Elastic Container Service (Amazon ECS).

Short descriptionThe Classic Load Balancer doesn't allow you to run multiple copies of a task on the same instance. Instead, when using the Classic Load Balancer, you must statically map port numbers on a container instance. However, an Application Load Balancer uses dynamic port mapping, so you can run multiple tasks from a single service on the same container instance.Note: Network Load Balancers can also be used for dynamic port mapping. See Network Load Balancer and Creating a Network Load Balancer.

ResolutionTo set up dynamic port mapping, follow these steps:Create an Application Load Balancer and a target group.Important: To route health check traffic correctly when you create a target group, expand Advanced health check settings. For Port, select traffic port.Open the Amazon ECS console, and then set the host port to 0 for the task definition that you're creating or updating. Be sure to set the container port mappings for your application.Important: The host and awsvpc network modes do not support dynamic host port mapping.Add a rule to allow inbound traffic from your load balancer to your container instances. The security group and network access control list (network ACL) must allow traffic from the load balancer to the instances over the ephemeral port range.Note: For more information about ephemeral port ranges, see PortMapping.Open the Amazon ECS console, and then configure your service to use the Application Load Balancer that you created.Important: You can add a load balancer only during the creation of the service. After service creation, you can't change the target group's Amazon Resource Name (ARN), container name, or the container port specified in the service definition. You can't add, remove, or change the load balancer configuration of an existing service. If you update the service task definition, then the container name and container port specified at service creation must remain in the task definition. For more information, see Service load balancing.Open the Amazon Elastic Compute Cloud (Amazon EC2) console, and then choose Target Groups from the navigation pane. Choose the hyperlinked name for your target group. From the Targets tab, you can see what port is used for the task in the service that you created.If dynamic port mapping is set up correctly, then you see the registered targets in the target group and the assigned port for the task. You also see the task in the registered targets for the following ephemeral port ranges: 49153–65535 and 32768–61000.

Related informationWhat is Elastic Load Balancing?What is an Application Load Balancer?
Question: How do I allow domain users RDP access to an EC2 Windows instance using group policy in AWS Managed Microsoft AD or Simple AD?

My Amazon Elastic Compute Cloud (Amazon EC2) Windows instance is joined to AWS Directory Service for Microsoft Active Directory or Simple Active Directory (Simple AD). I want to allow domain users Remote Desktop Protocol (RDP) access for the instance. When I try to connect using the built-in Remote Desktop Users group as a domain user, I receive the following message: "The connection was denied because the user account is not authorized for remote login." How can I fix this?Short descriptionAWS Managed Microsoft AD and Simple AD don't allow you to add domain users to the built-in Remote Desktop Users domain group. Instead, create a Group Policy Object (GPO) using the built-in Admin account, and then apply the policy to the delegated computers.Note: The GPO applies to all computers in the organizational unit (OU) that the policy is linked to. Any users that you add to the group using the following procedure will have RDP access to any computer in the OU.

ResolutionBefore you start:Join an EC2 Windows instance (Windows Server 2012 R2 or later) to a Simple AD or AWS Managed Microsoft AD directory.Install the Remote Server Administration Tools (RSAT) and Group Policy Management console on the instance.To allow domain users RDP access to the domain joined Windows instances, follow these steps:Connect to your Windows EC2 instance using RDP.Create a user. Repeat this step if you need more than one user.Create a security group. Note the security group name for a later step.Add the new users to the new security group.Open Group Policy Management. Select your domain’s Forest, expand Domains, and then expand your domain name.Expand your delegated OU (NetBIOS name of the directory). Open the context (right-click) menu for Computers, and then choose Create a GPO in this domain, and Link it here.For Name, enter a name, and then choose Ok.In the navigation pane, expand Computers. Open the context (right-click) menu for the policy, and then choose Edit.In the Computer Configuration section of the navigation pane, expand Preferences, Control Panel Settings.Open the context (right-click) menu for Local Users and Groups, and then choose New, Local Group.For Group name, choose Remote Desktop Users (built-in), and then choose Add.For Name, enter the name of the security group that you created in step 3, and then choose Ok.This policy updates your environment at the next policy refresh interval. To force the policy to apply immediately, run the gpupdate /force command on the target server.

Related informationAWS Managed Microsoft ADSimply Active DirectoryManage users and groups in AWS Managed Microsoft AD
Question: How do I enable MFA for AWS Managed Microsoft AD?

I want to enable multi-factor authentication (MFA) for AWS Directory Service for Microsoft Active Directory (AWS Managed Microsoft AD). How do I do this?ResolutionBefore enabling MFA for AWS Managed Microsoft AD, review the AWS Managed Microsoft AD prerequisites.For instructions on enabling MFA for AWS Managed Microsoft AD, see Enable multi-factor authentication for AWS Managed Microsoft AD.

Related informationWhat is AWS Directory Service?AWS Managed Microsoft ADSecure your AWS Managed Microsoft AD directoryHow do I enable MFA for AD Connector?Why is MFA failing on my AWS Managed Microsoft AD directory or my AD Connector?
Question: Can I use AWS Directory Service for Microsoft Active Directory to authenticate users in Amazon QuickSight?

I want to use AWS Directory Service for Microsoft Active Directory to authenticate users in Amazon QuickSight. How can I use Directory Service to authenticate users in Amazon QuickSight?ResolutionAmazon QuickSight supports both AWS Managed Microsoft AD and Active Directory Connector. AWS Managed Microsoft AD creates a fully managed Microsoft Active Directory in the AWS Cloud. For more information, see Getting started with AWS Managed Microsoft AD.To use AWS Managed Microsoft AD to authenticate users in Amazon QuickSight, perform the following:1.    Create an AWS Managed Microsoft AD directory.2.    Create an Amazon QuickSight Enterprise edition account.3.    Manage the user accounts in Amazon QuickSight Enterprise edition.When you use Directory Service to launch a directory, AWS creates an organizational unit (OU) with the same name as your domain. AWS also creates an administrative account with delegated administrative rights for the OU. You can create user accounts, groups, and policies within the OU using Active Directory users and groups. For more information, see Best practices for AWS Managed Microsoft AD.

Related informationSharing analysesWorking with data sources in Amazon QuickSightCreate an analysis using your own database dataUsing Active Directory with Amazon QuickSight Enterprise Edition
Question: Why am I getting the error “The request contained an unsupported argument.” when I add an account to my Amazon S3 bucket’s ACL?

I'm trying to add an account to my Amazon Simple Storage Service (Amazon S3) bucket's access control list (ACL). I receive the error "The request contained an unsupported argument." How do I troubleshoot this?Short descriptionI'm trying to add another AWS account to the access control list (ACL) of my Amazon S3 bucket by specifying an email address. I'm getting the error "The request contained an unsupported argument", even though I verified that the email address is correct. How can I fix this?ResolutionEnter the account's canonical user ID instead of the account email address. For instructions on retrieving the canonical ID, see Finding your account canonical user ID.If your bucket is in a newer AWS Region, then enter the canonical user ID to be sure that it maps to the account. For more information about AWS Regions, see Global Infrastructure.Note: It's a best practice to keep ACL deactivated and to use S3 Object Ownership to take ownership of every object in your bucket.

Related informationConfiguring ACLs