Repository: ansible/ansible

Issue:
ec2_vpc_route_table does not support vpc_endpoint in 2.5.0rc1
##### ISSUE TYPE
 - Bug Report


##### COMPONENT NAME
ec2_vpc_route_table

##### ANSIBLE VERSION
```
ansible 2.5.0rc1
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ubuntu/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/cloud_supervisor/releases/20180305095234/venv/lib/python3.5/site-packages/ansible
  executable location = /opt/cloud_supervisor/releases/20180305095234/venv/bin/ansible
  python version = 3.5.3 (default, Apr 22 2017, 00:00:00) [GCC 4.8.4]
```

##### CONFIGURATION
```
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False
```

##### OS / ENVIRONMENT
Ubuntu 16.04

##### SUMMARY
In my AWS route table definition, I have routes related to a VPC endpoint. These routes have no CIDR block definition.
In the ec2_vpc_route_table code, to see if a route is the same (in order possibly to delete them, or update rather than create), the DestinationCidrBlock attribute is used. On VPC endpoint routes, this attribute is not set.

In ansible < 2.5 I think these lines were simply ignored (which was fine by me since ec2_vpc_route_table does not handle vpc endpoint routes AFAIK)

I tracked it down to the following change
```patch
 def route_spec_matches_route_cidr(route_spec, route):
-    cidr_attr = 'destination_cidr_block'
-    return route_spec[cidr_attr] == getattr(route, cidr_attr)
+    return route_spec['DestinationCidrBlock'] == route['DestinationCidrBlock']
```

oddly enough my git log points to commit a685b621cdf  but this commit does not have this change when I browse it on github

##### STEPS TO REPRODUCE

##### EXPECTED RESULTS

##### ACTUAL RESULTS

ec2_vpc_route_table does not support vpc_endpoint in 2.5.0rc1
##### ISSUE TYPE
 - Bug Report


##### COMPONENT NAME
ec2_vpc_route_table

##### ANSIBLE VERSION
```
ansible 2.5.0rc1
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ubuntu/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/cloud_supervisor/releases/20180305095234/venv/lib/python3.5/site-packages/ansible
  executable location = /opt/cloud_supervisor/releases/20180305095234/venv/bin/ansible
  python version = 3.5.3 (default, Apr 22 2017, 00:00:00) [GCC 4.8.4]
```

##### CONFIGURATION
```
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False
```

##### OS / ENVIRONMENT
Ubuntu 16.04

##### SUMMARY
In my AWS route table definition, I have routes related to a VPC endpoint. These routes have no CIDR block definition.
In the ec2_vpc_route_table code, to see if a route is the same (in order possibly to delete them, or update rather than create), the DestinationCidrBlock attribute is used. On VPC endpoint routes, this attribute is not set.

In ansible < 2.5 I think these lines were simply ignored (which was fine by me since ec2_vpc_route_table does not handle vpc endpoint routes AFAIK)

I tracked it down to the following change
```patch
 def route_spec_matches_route_cidr(route_spec, route):
-    cidr_attr = 'destination_cidr_block'
-    return route_spec[cidr_attr] == getattr(route, cidr_attr)
+    return route_spec['DestinationCidrBlock'] == route['DestinationCidrBlock']
```

oddly enough my git log points to commit a685b621cdf  but this commit does not have this change when I browse it on github

##### STEPS TO REPRODUCE

##### EXPECTED RESULTS

##### ACTUAL RESULTS



Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/ec2_vpc_route_table.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2_vpc_route_table.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @willthames @wimnat
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@gbataille Do you specify or omit vpc endpoint routes in your task? (and if not, did purge_routes ignore those prior to 2.5?)
```
- dest: pl-xxxxxxxx
  gateway_id: vpce-xxxxxxxx
```
ouh... I did not know I could use the pl-xxxx as a destination.
But no I'm not specifying those. But I guess it does not change anything.

purge_route does not change anything although I understand that I should set it to false.

Before, those routes were ignored because boto returns no "CidrBlock" and therefore the "match" method returns nothing on those existing AWS rules. They become transparent to ansible. (i.e. they are left untouched whatever purge_route)

Now, because of the dict access (without .get) ansible will crash if such a route exists because of the matching.

Does that make sense?
Yeah, pl-xxxx as a destination doesn't work. I'm working on fixing it and trying to figure out what the behavior should be. I asked so I could replicate your task. Thank you for the extra info.
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/ec2_vpc_route_table.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2_vpc_route_table.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @willthames @wimnat
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@gbataille Do you specify or omit vpc endpoint routes in your task? (and if not, did purge_routes ignore those prior to 2.5?)
```
- dest: pl-xxxxxxxx
  gateway_id: vpce-xxxxxxxx
```
ouh... I did not know I could use the pl-xxxx as a destination.
But no I'm not specifying those. But I guess it does not change anything.

purge_route does not change anything although I understand that I should set it to false.

Before, those routes were ignored because boto returns no "CidrBlock" and therefore the "match" method returns nothing on those existing AWS rules. They become transparent to ansible. (i.e. they are left untouched whatever purge_route)

Now, because of the dict access (without .get) ansible will crash if such a route exists because of the matching.

Does that make sense?
Yeah, pl-xxxx as a destination doesn't work. I'm working on fixing it and trying to figure out what the behavior should be. I asked so I could replicate your task. Thank you for the extra info.

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/ec2_vpc_route_table.py b/lib/ansible/modules/cloud/amazon/ec2_vpc_route_table.py
--- a/lib/ansible/modules/cloud/amazon/ec2_vpc_route_table.py
+++ b/lib/ansible/modules/cloud/amazon/ec2_vpc_route_table.py
@@ -412,12 +412,15 @@ def get_route_table_by_tags(connection, module, vpc_id, tags):
 def route_spec_matches_route(route_spec, route):
     if route_spec.get('GatewayId') and 'nat-' in route_spec['GatewayId']:
         route_spec['NatGatewayId'] = route_spec.pop('GatewayId')
+    if route_spec.get('GatewayId') and 'vpce-' in route_spec['GatewayId']:
+        if route_spec.get('DestinationCidrBlock', '').startswith('pl-'):
+            route_spec['DestinationPrefixListId'] = route_spec.pop('DestinationCidrBlock')
 
     return set(route_spec.items()).issubset(route.items())
 
 
 def route_spec_matches_route_cidr(route_spec, route):
-    return route_spec['DestinationCidrBlock'] == route['DestinationCidrBlock']
+    return route_spec['DestinationCidrBlock'] == route.get('DestinationCidrBlock')
 
 
 def rename_key(d, old_key, new_key):
@@ -440,15 +443,26 @@ def ensure_routes(connection=None, module=None, route_table=None, route_specs=No
     for route_spec in route_specs:
         match = index_of_matching_route(route_spec, routes_to_match)
         if match is None:
-            route_specs_to_create.append(route_spec)
+            if route_spec.get('DestinationCidrBlock'):
+                route_specs_to_create.append(route_spec)
+            else:
+                module.warn("Skipping creating {0} because it has no destination cidr block. "
+                            "To add VPC endpoints to route tables use the ec2_vpc_endpoint module.".format(route_spec))
         else:
             if match[0] == "replace":
-                route_specs_to_recreate.append(route_spec)
+                if route_spec.get('DestinationCidrBlock'):
+                    route_specs_to_recreate.append(route_spec)
+                else:
+                    module.warn("Skipping recreating route {0} because it has no destination cidr block.".format(route_spec))
             del routes_to_match[match[1]]
 
     routes_to_delete = []
     if purge_routes:
         for r in routes_to_match:
+            if not r.get('DestinationCidrBlock'):
+                module.warn("Skipping purging route {0} because it has no destination cidr block. "
+                            "To remove VPC endpoints from route tables use the ec2_vpc_endpoint module.".format(r))
+                continue
             if r['Origin'] == 'CreateRoute':
                 routes_to_delete.append(r)
 


Test patch:
diff --git a/test/integration/targets/ec2_vpc_route_table/tasks/main.yml b/test/integration/targets/ec2_vpc_route_table/tasks/main.yml
--- a/test/integration/targets/ec2_vpc_route_table/tasks/main.yml
+++ b/test/integration/targets/ec2_vpc_route_table/tasks/main.yml
@@ -545,10 +545,55 @@
         - recreate_private_table.changed
         - recreate_private_table.route_table.id != create_public_table.route_table.id
 
+  - name: create a VPC endpoint to test ec2_vpc_route_table ignores it
+    ec2_vpc_endpoint:
+      state: present
+      vpc_id: "{{ vpc.vpc.id }}"
+      service: "com.amazonaws.{{ aws_region }}.s3"
+      route_table_ids:
+        - "{{ recreate_private_table.route_table.route_table_id }}"
+      <<: *aws_connection_info
+    register: vpc_endpoint
+
+  - name: purge routes
+    ec2_vpc_route_table:
+      vpc_id: "{{ vpc.vpc.id }}"
+      tags:
+        Public: "false"
+        Name: "Private route table"
+      routes:
+      - nat_gateway_id: "{{ nat_gateway.nat_gateway_id }}"
+        dest: 0.0.0.0/0
+      subnets: "{{ vpc_subnets|json_query('subnets[?tags.Public == `False`].id') }}"
+      purge_routes: true
+      <<: *aws_connection_info
+    register: result
+
+  - name: Get endpoint facts to verify that it wasn't purged from the route table
+    ec2_vpc_endpoint_facts:
+      query: endpoints
+      vpc_endpoint_ids:
+        - "{{ vpc_endpoint.result.vpc_endpoint_id }}"
+      <<: *aws_connection_info
+    register: endpoint_details
+
+  - name: assert the route table is associated with the VPC endpoint
+    assert:
+      that:
+        - endpoint_details.vpc_endpoints[0].route_table_ids[0] == recreate_private_table.route_table.route_table_id
+
   always:
   #############################################################################
   # TEAR DOWN STARTS HERE
   #############################################################################
+  - name: remove the VPC endpoint
+    ec2_vpc_endpoint:
+      state: absent
+      vpc_endpoint_id: "{{ vpc_endpoint.result.vpc_endpoint_id }}"
+      <<: *aws_connection_info
+    when: vpc_endpoint is defined
+    ignore_errors: yes
+
   - name: destroy route tables
     ec2_vpc_route_table:
       route_table_id: "{{ item.route_table.id }}"