Repository: ansible/ansible

Issue:
Have the ec2_asg module stop overshooting instances.
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report


##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
ec2_asg Module
##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```

```
ansible 2.2.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
RHEL7 and AWS
##### SUMMARY
<!--- Explain the problem briefly -->
Prevent the ec2_asg module from overshooting when either creating or updating an autoscaling group.
##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
1.-Call the ec2_asg module for AWS using the replace_all_instances and replace_batch_size set to all and 1 respectively.
2.-Watch how AWS launches the ASG when Boto talks to the AWS API, it wastes resources and time. See behavior below.
<!--- Paste example playbooks or commands between quotes below -->
```yaml
  ec2_asg:
    name: "{{ app_name }}"
    load_balancers: "{{ app_name }}"
    launch_config_name: "{{ launch_config.name }}"
    min_size:  4
    max_size: 8
    desired_capacity: 4
    region: "{{ region }}"
    vpc_zone_identifier: "{{ subnets | join(',') }}"
    health_check_type: EC2
    replace_all_instances: yes
    replace_batch_size: 1
    lc_check: false
    default_cooldown: 1200
    health_check_period: 1200
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
An ASG is created with min, desired and max values set as indicated by the Ansible YAML code. The amount of instances created only reaches the desired amount.
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
The ASG launches the MAXIMUM amount of instances: 8 then scales down by terminating the instances until it reaches the desired amount. If the ASG health check is based on an ELB this might cause the ec2_asg module to time out and fail. Not to mention customers have to OVERPAY for instances they'll never get to use.
<!--- Paste verbatim command output between quotes below -->
```

```

Have the ec2_asg module stop overshooting instances.
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report


##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
ec2_asg Module
##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```

```
ansible 2.2.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
RHEL7 and AWS
##### SUMMARY
<!--- Explain the problem briefly -->
Prevent the ec2_asg module from overshooting when either creating or updating an autoscaling group.
##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
1.-Call the ec2_asg module for AWS using the replace_all_instances and replace_batch_size set to all and 1 respectively.
2.-Watch how AWS launches the ASG when Boto talks to the AWS API, it wastes resources and time. See behavior below.
<!--- Paste example playbooks or commands between quotes below -->
```yaml
  ec2_asg:
    name: "{{ app_name }}"
    load_balancers: "{{ app_name }}"
    launch_config_name: "{{ launch_config.name }}"
    min_size:  4
    max_size: 8
    desired_capacity: 4
    region: "{{ region }}"
    vpc_zone_identifier: "{{ subnets | join(',') }}"
    health_check_type: EC2
    replace_all_instances: yes
    replace_batch_size: 1
    lc_check: false
    default_cooldown: 1200
    health_check_period: 1200
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
An ASG is created with min, desired and max values set as indicated by the Ansible YAML code. The amount of instances created only reaches the desired amount.
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
The ASG launches the MAXIMUM amount of instances: 8 then scales down by terminating the instances until it reaches the desired amount. If the ASG health check is based on an ELB this might cause the ec2_asg module to time out and fail. Not to mention customers have to OVERPAY for instances they'll never get to use.
<!--- Paste verbatim command output between quotes below -->
```

```



Discussion:
cc @garethr
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Here's an example of how the module fails when it takes longer than the healthcheck period (1200 seconds mind in this example):

`{
    "_ansible_parsed": true,
    "_ansible_no_log": false,
    "changed": false,
    "failed": true,
    "invocation": {
        "module_name": "ec2_asg",
        "module_args": {
            "default_cooldown": 1200,
            "health_check_period": 1200,
            "ec2_url": null,
            "aws_secret_key": null,
            "lc_check": false,
            "availability_zones": null,
            "replace_batch_size": 1,
            "replace_instances": [],
            "desired_capacity": 1,
            "state": "present",
            "health_check_type": "ELB",
            "max_size": 1,
            "profile": null,
            "tags": [
                {
                    "Name": "redeem-dev"
                },
                {
                    "Application": "redeem"
                },
                {
                    "Originating_ECM": "ECM-1736"
                },
                {
                    "Product_Owner": "bshamsian@boingo.com"
                },
                {
                    "Resource_Owner": "ecoutin@boingo.com"
                },
                {
                    "Role": "Comes-with-Boingo"
                },
                {
                    "Service_Category": "Platform Systems - Wordpress - redeem.boingo.com"
                }
            ],
            "min_size": 1,
            "vpc_zone_identifier": [
                "subnet-6673d410",
                "subnet-7cfe7218"
            ],
            "load_balancers": [
                "redeem-dev"
            ],
            "launch_config_name": "redeem-dev-copy-1502491816",
            "aws_access_key": null,
            "name": "redeem-dev",
            "security_token": null,
            "replace_all_instances": true,
            "wait_for_instances": true,
            "region": "us-west-2",
            "termination_policies": [
                "Default"
            ],
            "wait_timeout": 300,
            "notification_topic": null,
            "notification_types": [
                "autoscaling:EC2_INSTANCE_LAUNCH",
                "autoscaling:EC2_INSTANCE_LAUNCH_ERROR",
                "autoscaling:EC2_INSTANCE_TERMINATE",
                "autoscaling:EC2_INSTANCE_TERMINATE_ERROR"
            ],
            "validate_certs": true
        }
    },
    "msg": "Waited too long for ELB instances to be healthy. Fri Aug 11 18:58:07 2017"
} `

Code is the same as above.
@einarc Could you test this with ansible-2.3 or the current ansible devel branch from git? There have been a lot of changes to the ec2_asg.py since ansible-2.2.

If possible, paste a copy of the 'ansible-playbook' output with the '-vvvv' verbosity level enabled for a run that shows this error.
@alikins  Version 2.3.2 passes Unit Testing, unfortunately we still have to do regression and integration with the rest of our code to make sure the update did not break things or introduced new bugs. Not to mention we're using Tower so Integration is also required.
I cannot update to 2.3.2 because it breaks another module, aws s3:

"
![image](https://user-images.githubusercontent.com/6519208/29736598-a642bd02-89b7-11e7-9d6f-48df2012acc2.png)

"

Code:
- name: Inject SQL - Get dump
  s3: bucket="{{ s3_bucket }}" object="{{ s3_key }}" dest="{{ s3_sql_file }}" mode=get overwrite=always
  when: create_rds
  tags: rds_database
  when: create_rds

Case:
https://access.redhat.com/support/cases/#/case/01904604

On the other hand 2.1.2 handles the same file on the same DB instance with success, the only thing that changed was the ansible core version. Here's output:
![image](https://user-images.githubusercontent.com/6519208/29736625-ef8e57aa-89b7-11e7-949f-e421eae81624.png)


![image](https://user-images.githubusercontent.com/6519208/29736617-e06636b2-89b7-11e7-8294-aaf139326d72.png)

[2.2.1-Ansible-Core-Inject-DB-Success.txt](https://github.com/ansible/ansible/files/1253262/2.2.1-Ansible-Core-Inject-DB-Success.txt)


[2.3.2-Ansible-Core_Failed.txt](https://github.com/ansible/ansible/files/1253263/2.3.2-Ansible-Core_Failed.txt)

Reducing file size to about 630 MB, still no success.
Hello,

Actually we're still seeing the same behavior on 2.3.2 as in 2.2.1,
![image](https://user-images.githubusercontent.com/6519208/29897810-8c92dd22-8d97-11e7-8fbf-d2c1e514a49c.png)

![image](https://user-images.githubusercontent.com/6519208/29897840-a4599fe0-8d97-11e7-814e-c99247579ff8.png)


![image](https://user-images.githubusercontent.com/6519208/29897887-f2f51f9e-8d97-11e7-954e-e8f9bd8f4908.png)


And btw, this ALSO affects 2.3.2, so please go ahead and add that tag
cc @willthames
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Still affecting ansible 2.4.3.0.


Edit. Well, perhaps not. I asked for max_cap 3 desired_cap 3, have two running, asked for 1 to be killed, 3 started and then one was terminated a minute later.
I ran into this issue as well.  Looking into it I think the issue is related to `lc_check: false` in combination with `replace_all_instances: yes`. I believe I understand where the issue is and will try to submit a pull request for a fix this weekend.

Also, giving the specific replace_instances list worked for me instead of using `replace_all_instances` as a temporary workaround. e.g.
```
---

- hosts: localhost
  gather_facts: false
  tasks:
    - ec2_asg_facts:
        name: "{{asg_name}}"
      register: ec2_facts

    # If asg doesn't exist yet results won't be defined
    - set_fact:
        rinst: "{{ ec2_facts.results.0.instances|map(attribute='instance_id')|list }}"
      when:
          ec2_facts.results

    - ec2_asg:
        name: "{{ asg_name }}"
        launch_config_name: test-lc
        replace_instances: "{{ rinst|default( [] ) }}"
        wait_for_instances: true
        lc_check: false
        min_size: 2
        max_size: 2
        desired_capacity: 2
        replace_batch_size: 2
```

cc @garethr
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Here's an example of how the module fails when it takes longer than the healthcheck period (1200 seconds mind in this example):

`{
    "_ansible_parsed": true,
    "_ansible_no_log": false,
    "changed": false,
    "failed": true,
    "invocation": {
        "module_name": "ec2_asg",
        "module_args": {
            "default_cooldown": 1200,
            "health_check_period": 1200,
            "ec2_url": null,
            "aws_secret_key": null,
            "lc_check": false,
            "availability_zones": null,
            "replace_batch_size": 1,
            "replace_instances": [],
            "desired_capacity": 1,
            "state": "present",
            "health_check_type": "ELB",
            "max_size": 1,
            "profile": null,
            "tags": [
                {
                    "Name": "redeem-dev"
                },
                {
                    "Application": "redeem"
                },
                {
                    "Originating_ECM": "ECM-1736"
                },
                {
                    "Product_Owner": "bshamsian@boingo.com"
                },
                {
                    "Resource_Owner": "ecoutin@boingo.com"
                },
                {
                    "Role": "Comes-with-Boingo"
                },
                {
                    "Service_Category": "Platform Systems - Wordpress - redeem.boingo.com"
                }
            ],
            "min_size": 1,
            "vpc_zone_identifier": [
                "subnet-6673d410",
                "subnet-7cfe7218"
            ],
            "load_balancers": [
                "redeem-dev"
            ],
            "launch_config_name": "redeem-dev-copy-1502491816",
            "aws_access_key": null,
            "name": "redeem-dev",
            "security_token": null,
            "replace_all_instances": true,
            "wait_for_instances": true,
            "region": "us-west-2",
            "termination_policies": [
                "Default"
            ],
            "wait_timeout": 300,
            "notification_topic": null,
            "notification_types": [
                "autoscaling:EC2_INSTANCE_LAUNCH",
                "autoscaling:EC2_INSTANCE_LAUNCH_ERROR",
                "autoscaling:EC2_INSTANCE_TERMINATE",
                "autoscaling:EC2_INSTANCE_TERMINATE_ERROR"
            ],
            "validate_certs": true
        }
    },
    "msg": "Waited too long for ELB instances to be healthy. Fri Aug 11 18:58:07 2017"
} `

Code is the same as above.
@einarc Could you test this with ansible-2.3 or the current ansible devel branch from git? There have been a lot of changes to the ec2_asg.py since ansible-2.2.

If possible, paste a copy of the 'ansible-playbook' output with the '-vvvv' verbosity level enabled for a run that shows this error.
@alikins  Version 2.3.2 passes Unit Testing, unfortunately we still have to do regression and integration with the rest of our code to make sure the update did not break things or introduced new bugs. Not to mention we're using Tower so Integration is also required.
I cannot update to 2.3.2 because it breaks another module, aws s3:

"
![image](https://user-images.githubusercontent.com/6519208/29736598-a642bd02-89b7-11e7-9d6f-48df2012acc2.png)

"

Code:
- name: Inject SQL - Get dump
  s3: bucket="{{ s3_bucket }}" object="{{ s3_key }}" dest="{{ s3_sql_file }}" mode=get overwrite=always
  when: create_rds
  tags: rds_database
  when: create_rds

Case:
https://access.redhat.com/support/cases/#/case/01904604

On the other hand 2.1.2 handles the same file on the same DB instance with success, the only thing that changed was the ansible core version. Here's output:
![image](https://user-images.githubusercontent.com/6519208/29736625-ef8e57aa-89b7-11e7-949f-e421eae81624.png)


![image](https://user-images.githubusercontent.com/6519208/29736617-e06636b2-89b7-11e7-8294-aaf139326d72.png)

[2.2.1-Ansible-Core-Inject-DB-Success.txt](https://github.com/ansible/ansible/files/1253262/2.2.1-Ansible-Core-Inject-DB-Success.txt)


[2.3.2-Ansible-Core_Failed.txt](https://github.com/ansible/ansible/files/1253263/2.3.2-Ansible-Core_Failed.txt)

Reducing file size to about 630 MB, still no success.
Hello,

Actually we're still seeing the same behavior on 2.3.2 as in 2.2.1,
![image](https://user-images.githubusercontent.com/6519208/29897810-8c92dd22-8d97-11e7-8fbf-d2c1e514a49c.png)

![image](https://user-images.githubusercontent.com/6519208/29897840-a4599fe0-8d97-11e7-814e-c99247579ff8.png)


![image](https://user-images.githubusercontent.com/6519208/29897887-f2f51f9e-8d97-11e7-954e-e8f9bd8f4908.png)


And btw, this ALSO affects 2.3.2, so please go ahead and add that tag
cc @willthames
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Still affecting ansible 2.4.3.0.


Edit. Well, perhaps not. I asked for max_cap 3 desired_cap 3, have two running, asked for 1 to be killed, 3 started and then one was terminated a minute later.
I ran into this issue as well.  Looking into it I think the issue is related to `lc_check: false` in combination with `replace_all_instances: yes`. I believe I understand where the issue is and will try to submit a pull request for a fix this weekend.

Also, giving the specific replace_instances list worked for me instead of using `replace_all_instances` as a temporary workaround. e.g.
```
---

- hosts: localhost
  gather_facts: false
  tasks:
    - ec2_asg_facts:
        name: "{{asg_name}}"
      register: ec2_facts

    # If asg doesn't exist yet results won't be defined
    - set_fact:
        rinst: "{{ ec2_facts.results.0.instances|map(attribute='instance_id')|list }}"
      when:
          ec2_facts.results

    - ec2_asg:
        name: "{{ asg_name }}"
        launch_config_name: test-lc
        replace_instances: "{{ rinst|default( [] ) }}"
        wait_for_instances: true
        lc_check: false
        min_size: 2
        max_size: 2
        desired_capacity: 2
        replace_batch_size: 2
```


Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/ec2_asg.py b/lib/ansible/modules/cloud/amazon/ec2_asg.py
--- a/lib/ansible/modules/cloud/amazon/ec2_asg.py
+++ b/lib/ansible/modules/cloud/amazon/ec2_asg.py
@@ -1191,6 +1191,7 @@ def replace(connection):
     desired_capacity = module.params.get('desired_capacity')
     lc_check = module.params.get('lc_check')
     replace_instances = module.params.get('replace_instances')
+    replace_all_instances = module.params.get('replace_all_instances')
 
     as_group = describe_autoscaling_groups(connection, group_name)[0]
     if desired_capacity is None:
@@ -1199,6 +1200,10 @@ def replace(connection):
     wait_for_new_inst(connection, group_name, wait_timeout, as_group['MinSize'], 'viable_instances')
     props = get_properties(as_group)
     instances = props['instances']
+    if replace_all_instances:
+        # If replacing all instances, then set replace_instances to current set
+        # This allows replace_instances and replace_all_instances to behave same
+        replace_instances = instances
     if replace_instances:
         instances = replace_instances
     # check to see if instances are replaceable if checking launch configs
@@ -1235,7 +1240,7 @@ def replace(connection):
 
     as_group = describe_autoscaling_groups(connection, group_name)[0]
     update_size(connection, as_group, max_size + batch_size, min_size + batch_size, desired_capacity + batch_size)
-    wait_for_new_inst(connection, group_name, wait_timeout, as_group['MinSize'], 'viable_instances')
+    wait_for_new_inst(connection, group_name, wait_timeout, as_group['MinSize'] + batch_size, 'viable_instances')
     wait_for_elb(connection, group_name)
     wait_for_target_group(connection, group_name)
     as_group = describe_autoscaling_groups(connection, group_name)[0]
@@ -1414,6 +1419,12 @@ def wait_for_new_inst(connection, group_name, wait_timeout, desired_size, prop):
     return props
 
 
+def asg_exists(connection):
+    group_name = module.params.get('name')
+    as_group = describe_autoscaling_groups(connection, group_name)
+    return bool(len(as_group))
+
+
 def main():
     argument_spec = ec2_argument_spec()
     argument_spec.update(
@@ -1483,13 +1494,16 @@ def main():
                             endpoint=ec2_url,
                             **aws_connect_params)
     changed = create_changed = replace_changed = False
+    exists = asg_exists(connection)
 
     if state == 'present':
         create_changed, asg_properties = create_autoscaling_group(connection)
     elif state == 'absent':
         changed = delete_autoscaling_group(connection)
         module.exit_json(changed=changed)
-    if replace_all_instances or replace_instances:
+
+    # Only replace instances if asg existed at start of call
+    if exists and (replace_all_instances or replace_instances):
         replace_changed, asg_properties = replace(connection)
     if create_changed or replace_changed:
         changed = True


Test patch:
diff --git a/test/integration/targets/ec2_asg/tasks/main.yml b/test/integration/targets/ec2_asg/tasks/main.yml
--- a/test/integration/targets/ec2_asg/tasks/main.yml
+++ b/test/integration/targets/ec2_asg/tasks/main.yml
@@ -387,6 +387,122 @@
 
     # ============================================================
 
+    # perform rolling replace with new launch configuration and lc_check:false
+
+    # Note - this is done async so we can query asg_facts during
+    # the execution. Issues #28087 and #35993 result in correct
+    # end result, but spin up extraneous instances during execution.
+    - name: "perform rolling update to new AMI with lc_check: false"
+      ec2_asg:
+        name: "{{ resource_prefix }}-asg"
+        launch_config_name: "{{ resource_prefix }}-lc-2"
+        health_check_type: EC2
+        desired_capacity: 3
+        min_size: 1
+        max_size: 5
+        health_check_period: 900
+        load_balancers: []
+        vpc_zone_identifier: "{{ testing_subnet.subnet.id }}"
+        wait_for_instances: yes
+        replace_all_instances: yes
+        replace_batch_size: 3
+        lc_check: false
+        wait_timeout: 1800
+        state: present
+        <<: *aws_connection_info
+      async: 1800
+      poll: 0
+      register: asg_job
+
+    - name: get ec2_asg facts for 3 minutes
+      ec2_asg_facts:
+        name: "{{ resource_prefix }}-asg"
+        <<: *aws_connection_info
+      register: output
+      loop_control:
+          pause: 15
+      with_sequence: count=12
+
+    - set_fact:
+        inst_id_json_query: 'results[*].results[*].instances[*].instance_id'
+
+    # Since we started with 3 servers and replace all of them.
+    # We should see 6 servers total.
+    - assert:
+        that:
+          - "lookup('flattened',output|json_query(inst_id_json_query)).split(',')|unique|length == 6"
+
+    - name: Ensure ec2_asg task completes
+      async_status: jid="{{ asg_job.ansible_job_id }}"
+      register: status
+      until: status.finished
+      retries: 200
+      delay: 15
+
+    # ============================================================
+
+    - name: kill asg
+      ec2_asg:
+        name: "{{ resource_prefix }}-asg"
+        state: absent
+        <<: *aws_connection_info
+      async: 300
+
+    # Create new asg with replace_all_instances and lc_check:false
+
+    # Note - this is done async so we can query asg_facts during
+    # the execution. Issues #28087 results in correct
+    # end result, but spin up extraneous instances during execution.
+    - name: "new asg with lc_check: false"
+      ec2_asg:
+        name: "{{ resource_prefix }}-asg"
+        launch_config_name: "{{ resource_prefix }}-lc"
+        health_check_type: EC2
+        desired_capacity: 3
+        min_size: 1
+        max_size: 5
+        health_check_period: 900
+        load_balancers: []
+        vpc_zone_identifier: "{{ testing_subnet.subnet.id }}"
+        wait_for_instances: yes
+        replace_all_instances: yes
+        replace_batch_size: 3
+        lc_check: false
+        wait_timeout: 1800
+        state: present
+        <<: *aws_connection_info
+      async: 1800
+      poll: 0
+      register: asg_job
+
+    # Collect ec2_asg_facts for 3 minutes
+    - name: get ec2_asg facts
+      ec2_asg_facts:
+        name: "{{ resource_prefix }}-asg"
+        <<: *aws_connection_info
+      register: output
+      loop_control:
+          pause: 15
+      with_sequence: count=12
+
+    - set_fact:
+        inst_id_json_query: 'results[*].results[*].instances[*].instance_id'
+
+    # Get all instance_ids we saw and assert we saw number expected
+    # Should only see 3 (don't replace instances we just created)
+    - assert:
+        that:
+          - "lookup('flattened',output|json_query(inst_id_json_query)).split(',')|unique|length == 3"
+
+    - name: Ensure ec2_asg task completes
+      async_status: jid="{{ asg_job.ansible_job_id }}"
+      register: status
+      until: status.finished
+      retries: 200
+      delay: 15
+
+# ============================================================
+
   always:
 
     - name: kill asg