Question: Why did the running task count change in my Amazon ECS service?

The running task count in my Amazon Elastic Container Service (Amazon ECS) service has increased or decreased.

Short descriptionThe running task count for your Amazon ECS service might increase or decrease due to multiple reasons. Some of the common causes are the following:Your Amazon ECS tasks are failing health checks.AWS CloudFormation or AWS Cloud Development Kit (AWS CDK) updated the desired count in the ECS service.You manually updated the desired count for your ECS service.The Amazon ECS Service Auto Scaling updated the number of tasks based on the scaling policy.You updated the minimum or maximum count in Application Auto Scaling using the RegisterScalableTarget API.The deployment configuration of your ECS service modified the running task count during deployment.

ResolutionYour ECS tasks are failing health checksIf your tasks are using the Amazon EC2 launch type, see How can I get my Amazon ECS tasks running using the Amazon EC2 launch type to pass the Application Load Balancer health check in Amazon ECS?If the tasks are using the AWS Fargate launch type, see How do I troubleshoot health check failures for Amazon ECS tasks on Fargate?CloudFormation or AWS CDK updated the desired count in your ECS serviceIf you created your ECS service with CloudFormation or CDK without specifying the DesiredCount field, then the desired count is set to the default value of 1. However, when the service is updated through CloudFormation or CDK without specifying the DesiredCount field, the existing desired count in the current deployment is used for the new deployment.To troubleshoot this issue, review AWS CloudTrail events to check if the UpdateService API call was made by CloudFormation. You can verify this same by checking that the userAgent field in the Event record section specifies the following:"userAgent": "cloudformation.amazonaws.com"To resolve this issue, remove the desiredCount parameter from the CloudFormation or CDK code during the update.You manually updated the desired count for your ECS serviceIf you manually updated the ECS Service using the UpdateService API, Amazon ECS console, or AWS Command Line Interface (AWS CLI), then update the service again to change the desired count to an appropriate value. For more information, see Updating a service and update-service.To troubleshoot this issue, review CloudTrail events to check whether the UpdateService API call was made from the Amazon ECS console or AWS CLI. You can view this information in the userAgent field in the CloudTrail record.Example:"userAgent": "console.amazonaws.com"The Amazon ECS Service Auto Scaling updated the number of tasks based on the scaling policyThe Amazon ECS Service Auto Scaling might have increased or decreased the desired count based on the scaling policy. Review the service and check for any recent scaling activity.To view the scaling activities, do the following:Open the Amazon ECS console.In the navigation pane, choose Clusters.Choose the cluster that you want to check.Choose the Services tab.Choose the service that you want to check.Choose the Events tab.If there was a scaling event, then you might view informational messages, such as the following, in the Message column:Successfully set desired count to 1. Change successfully fulfilled by ecs. Cause: monitor alarm TargetTracking-service/service-autoscaling/sample-webapp-AlarmLow-fcd80aef-5161-4890-aeb4-35dde11ff42c in state ALARM triggered policy TargetTrackingPolicy.If the tasks are stopped due to a scaling activity, review the target tracking or step scaling policies. Then, update the desired count according to your application’s requirements.To view the Application Auto Scaling configuration, do the following:Open the Amazon ECS console.In the navigation pane, choose Clusters.Choose the cluster that you want to check.Choose the Services tab.Choose the service that you want to check.Choose the Auto Scaling tab.Note: If you're using the new Amazon ECS console, choose Configuration and tasks, and then view the information under Service configuration.You updated the minimum and maximum count in Application Auto Scaling using the RegisterScalableTarget APIIf you used the RegisterScalableTarget API to update the scalable target, then check the MinCapacity property value specified in the API. If you specified a new minimum value, then Application Auto Scaling uses this value as the new minimum value that the tasks can be scaled down to.Note: Application Auto Scaling doesn't scale down the desired count to less than the specified minimum value.The deployment configuration of your ECS service modified the running task count during deploymentReview CloudTrail events to check if the UpdateService API was called recently. You can view the deployment configuration of your ECS service by checking the deploymentConfigration request parameter in the UpdateService API.-or-Review the deployment configuration of your ECS service in the Amazon ECS console by doing the following:Open the Amazon ECS console.In the navigation pane, choose Clusters.Choose the cluster that you want to check.Choose the Services tab.Choose the service that you want to check.Choose the Deployments tab.The deployment configuration parameters control how many tasks are run during a deployment and the sequence of stopping and starting tasks. If the deployment configuration is set incorrectly, then the number of tasks running during the deployment is reduced. Therefore, it's a best practice to configure the value of the minimumHealthyPercent parameter to higher than 0%.The minimumHealthyPercent parameter represents the lower limit on either of the following as a percentage of the desired number of tasks for the service:Number of tasks that should be running for a service during a deploymentNumber of tasks that should be running when a container instance is draining.The value of this parameter is rounded up. For example, if minimumHealthyPercent is 50% and the desired number of tasks is four, then the scheduler can stop two existing tasks before starting two new tasks. If minimumHealthyPercent is 75% and the desired number of tasks is two, then the scheduler can't stop any tasks due to the resulting value also being two.The maximumPercent parameter represents the upper limit on either of the following as a percentage of the desired number of tasks for a service:Number of tasks that should be running for a service during a deploymentNumber of tasks that should be running when a container instance is draining.The value of this parameter is rounded down. For example, if maximumPercent is 200% and the desired number of tasks is four, then the scheduler can start four new tasks before stopping four existing tasks. If maximumPercent is 125% and the desired number of tasks is three, then the scheduler can't start any tasks due to the resulting value also being three.For more information, see Rolling update.

Related informationService auto scalingHow do I resolve the "[AWS service] was unable to place a task because no container instance met all of its requirements" error in Amazon ECS?