Issue 1:
@mpdehaan this is for you buddy

Bump specfile

Review man pages (any new paramaters added?)

For good measure:

$ touch VERSION
$ make docs

Tag 0.7

ping me to start working on the corresponding Fedora update

Issue 2:
ISSUE TYPE

Bug Report

COMPONENT NAME

ansible/modules/cloud/openstack/os_server.py

ansible/modules/cloud/openstack/os_volume.py

ansible/modules/cloud/openstack/os_server_volume.py

ANSIBLE VERSION

2.3.1.0

CONFIGURATION

default ansible.cfg

OS / ENVIRONMENT

ansible installed Ubuntu 14.04

Test public cloud based on Openstack

SUMMARY

STEPS TO REPRODUCE

ansible-playbook test.yml

test.yml
---
- name: Create an instance
hosts: localhost
gather_facts: false

vars:
ipv4_address: 10.32.15.150

tasks:
- name: Create System Volume
os_volume:
state: present
availability_zone: "AZ1"
size: "8"
wait: yes
image: "Standard_CentOS_7.3_latest"
volume_type: "SATA"
display_name: "ops-dummy-0001-volume-system"

- name: Create Data Volume
os_volume:
state: present
availability_zone: "AZ1"
size: "50"
wait: yes
volume_type: "SSD"
display_name: "ops-dummy-0001-volume-data"

- name: Create static IP NIC
os_port:
state: present
name: "ops-dummy-0001-port1"
admin_state_up: true
network: "76cc4d0c-18b8-433d-9dc8-a26b5c3f6508"
security_groups: "ops-sg-tier1"
fixed_ips:
- ip_address: "10.32.15.150"
when: ipv4_address is defined

- name: Create DHCP NIC
os_port:
state: present
name: "ops-dummy-0001-port1"
admin_state_up: true
network: "76cc4d0c-18b8-433d-9dc8-a26b5c3f6508"
security_groups: "ops-sg-tier1"
when: ipv4_address is not defined

- name: Create an instance
os_server:
state: present
name: "ops-dummy-0001"
key_name: "ops"
boot_volume: "ops-dummy-0001-volume-system"
timeout: 200
flavor: "c1.medium"
availability_zone: "AZ1"
auto_ip: false
reuse_ips: false # for now
nics:
- port-name: "ops-dummy-0001-port1"
register: osvm

- name: Attach volumes
os_server_volume:
state: present
server: "ops-dummy-0001"
volume: "ops-dummy-0001-volume-data"
device: "/dev/xvdb"

- name: Return configured local ipv4
set_fact:
instance_ipv4: "{{ osvm.openstack.private_v4}}"

- name: Wait for 90 seconds so the instance is ready for the next task
pause:
seconds: 90

EXPECTED RESULTS

1.The whole job completes within a very short time.

2. No performance difference between openstack plugin and aws plugin

ACTUAL RESULTS

===============================================================================

Create an instance ---------------------------------------------------- 218.52s

Attach volumes -------------------------------------------------------- 173.52s

Create System Volume -------------------------------------------------- 116.32s

Wait for 90 seconds so the instance is ready for the next task --------- 90.01s

Create static IP NIC --------------------------------------------------- 16.40s

Create Data Volume ----------------------------------------------------- 13.59s

Return configured local ipv4 -------------------------------------------- 0.02s

Create DHCP NIC --------------------------------------------------------- 0.01s

real	10m29.258s

user	0m44.327s

sys	0m6.167s

Analyse:

From the API log, we find some unreasonable parts , which means unreasonable api calls.

1.When we create system volume, the playbook specify volume name and image id, the plugin should use the list API with filter for volume and get image by id, rather than List all Volumes with detail and List all images with detail.

2.The API like this list all images and all volumes cost much time that the API with some filter parameters.

Since the playbook specify flavor name already, the OpenStack plugin shouldn’t get all flavors with detail and loop to query every flavor detail

When we create data volume, the playbook specify volume name, the plugin should use the list API with filter for volume rather than List all Volumes with detail, the same to the port, floatingip, network, subnet, security-group, image.

5.After the VM create successful, the plugin should call get detail by instance id rather that loop to get all of the resources, such as port, floatingip, network, security-group, image, subnet etc.

6.The playbook specify image id, the plugin should use get image by id API, rather than List all images with detail.

7.For the step attaching volume for instance, there is no reason to list all of the resource about the tenant, from the process, we find that this step call lots of list all resource for port, servers, floatingips, networks, subnets, security-groups, volumes etc.

In all,

Every get_${resource} function in OpenStack plugin has the below logic:

a.	List all of the resources for the specify tenant.

b.	Loop all resource to find the one which the user specify.

The resource maybe image, flavor, volume, server, port, subnet, floating-ip, security-group, network etc.

These functions are called very frequently during the task, for example, after create instance request sent, the plugin will call get_server(server_id) function to check if the VM status is ACTIVE.

Every query function to check the resource status in Amazon plugin is not similar with the implementation in OpenStack plugin, and we can find the amazon plugin have better performance and specialized optimization in the Plugin.

Above all,

a.	The result of callback plugin isn’t very proper, it includes lots of unreasonable API calls lead to more time cost of the testing, and we can confirm that all the API calls during the testing is normal. For example, the time of creating instances cost 23s, but the ansible count 218s totally.

For create instance task, it include the following API calls:

List all flavors

Get every flavors detail

List all flavors with detail

List all volumes with detail

Create Instances

List all servers

List all networks

List all subnets

List all servers to check if servers ACTIVE

List all ports

List all floatingips

List all ports

List all floatingips

List all flavors with detail

Get Server security-groups

List all images with detail

List all volumes with detail

b.	The logic of OpenStack plugin of Ansible has poor performance, if the plugin would get a resource by id, it will get all the resources of the tenant, the loop to find the specify resource.

We use a temporary method to improve a little the performance, but it is really not the best way. We hope ansible can fix this issue. Thanks~

Our steps are as follows:

1. Modify these three modules

ansible/modules/cloud/openstack/os_server.py
ansible/modules/cloud/openstack/os_volume.py
ansible/modules/cloud/openstack/os_server_volume.py

a. Edit the os_server.py and append the below code before the line with content “cloud = shade.openstack_cloud(**cloud_params)”

from shade import os_monkey_patch
os_monkey_patch.monkey_patch_os_server()

b.Edit the os_volume.py and append the below code before the line with content “cloud = shade.openstack_cloud(**module.params)”

from shade import os_monkey_patch
os_monkey_patch.monkey_patch_os_volume()

c.Edit the os_server_volume.py and append the below code before the line with content “cloud = shade.openstack_cloud(**module.params)”

from shade import os_monkey_patch
os_monkey_patch.monkey_patch_os_server_volume()

2. Touch a new file called **os_monkey_patch.py**
python2.7/dist-packages/shade/os_monkey_patch.py

The file is as follows:
import shade
import uuid
import keystoneauth1.exceptions
import warnings
from urllib import urlencode

from shade import _utils
from shade import meta
from shade.openstackcloud import OpenStackCloud

# Patch the os_server module in ansible.modules.cloud.openstack
def patch_list_flavors(self, get_extra=None, filters=None):
if get_extra is None:
get_extra = self._extra_config['get_flavor_extra_specs']
flavors = self._normalize_flavors(
self._compute_client.get(
'/flavors/detail', params=dict(is_public='None'),
error_message="Error fetching flavor list"))

filters_values = filters.values() if filters else []
for flavor in flavors:
if flavor.id not in filters_values and flavor.name not in filters_values:
continue
if not flavor.extra_specs and get_extra:
endpoint = "/flavors/{id}/os-extra_specs".format(
id=flavor.id)
try:
flavor.extra_specs = self._compute_client.get(
endpoint,
error_message="Error fetching flavor extra specs")
except shade.exc.OpenStackCloudHTTPError as e:
flavor.extra_specs = {}
self.log.debug(
'Fetching extra specs for flavor failed:'
' %(msg)s', {'msg': str(e)})
return flavors

def patch_search_flavors(self, name_or_id=None, filters=None, get_extra=True):
flavors = self.list_flavors(get_extra=get_extra, filters={'name_or_id': name_or_id})
return _utils._filter_list(flavors, name_or_id, filters)

def _no_pending_images(images):
"""If there are any images not in a steady state, don't cache"""
for image in images:
if image.status not in ('active', 'deleted', 'killed'):
return False
return True

@_utils.cache_on_arguments(should_cache_fn=_no_pending_images)
def patch_list_images(self, filter_deleted=True, filters=None):
images = []
image_list = []
try:
if self.cloud_config.get_api_version('image') == '2':
endpoint = '/images'
else:
endpoint = '/images/detail'
params = {}
try:
uuid.UUID(filters['name_or_id'])
params['id'] = filters['name_or_id']
except ValueError:
params['name'] = filters['name_or_id']
if params:
response = self._image_client.get(endpoint + "?" + urlencode(params))
if len(response['images']) == 0 and 'id' in params:
params['name'] = params.pop('id')
response = self._image_client.get(endpoint + "?" + urlencode(params))
else:
response = self._image_client.get(endpoint)

except keystoneauth1.exceptions.catalog.EndpointNotFound:
response = self._compute_client.get('/images/detail')
while 'next' in response:
image_list.extend(meta.obj_list_to_munch(response['images']))
endpoint = response['next']
response = self._raw_image_client.get(endpoint)
if 'images' in response:
image_list.extend(meta.obj_list_to_munch(response['images']))
else:
image_list.extend(response)

for image in image_list:
if not filter_deleted:
images.append(image)
elif image.status.lower() != 'deleted':
images.append(image)
return self._normalize_images(images)

def patch_search_images(self, name_or_id=None, filters=None):
images = self.list_images(filters={'name_or_id': name_or_id})
return _utils._filter_list(images, name_or_id, filters)

def _no_pending_volumes(volumes):
for volume in volumes:
if volume['status'] not in ('available', 'error', 'in-use'):
return False
return True

def patch_search_volumes(self, name_or_id=None, filters=None):
volumes = self.list_volumes(filters={'name_or_id': name_or_id})
return _utils._filter_list(
volumes, name_or_id, filters)

@_utils.cache_on_arguments(should_cache_fn=_no_pending_volumes)
def patch_list_volumes(self, cache=True, filters=None):
def _list(data):
volumes.extend(data.get('volumes', []))
endpoint = None
for l in data.get('volumes_links', []):
if 'rel' in l and 'next' == l['rel']:
endpoint = l['href']
break
if endpoint:
try:
_list(self._volume_client.get(endpoint))
except shade.exc.OpenStackCloudURINotFound:
self.log.debug(
"While listing volumes, could not find next link"
" {link}.".format(link=data))
raise

if not cache:
warnings.warn('cache argument to list_volumes is deprecated. Use '
'invalidate instead.')
attempts = 5
for _ in range(attempts):
volumes = []
params = {}
if not filters:
data = self._volume_client.get('/volumes/detail')
else:
try:
uuid.UUID(filters['name_or_id'])
params['id'] = filters['name_or_id']
except ValueError:
params['name'] = filters['name_or_id']
if params:
data = self._volume_client.get('/volumes/detail', params=params)
if len(data.get('volumes', [])) == 0 and 'id' in params:
params['name'] = params.pop('id')
data = self._volume_client.get('/volumes/detail', params=params)
else:
data = self._volume_client.get('/volumes/detail')

if 'volumes_links' not in data:
volumes.extend(data.get('volumes', []))
break
try:
_list(data)
break
except shade.exc.OpenStackCloudURINotFound:
pass
else:
self.log.debug(
"List volumes failed to retrieve all volumes after"
" {attempts} attempts. Returning what we found.".format(
attempts=attempts))
return self._normalize_volumes(
meta.get_and_munchify(key=None, data=volumes))

def monkey_patch_os_server():
setattr(OpenStackCloud, 'list_flavors', patch_list_flavors)
setattr(OpenStackCloud, 'search_flavors', patch_search_flavors)
setattr(OpenStackCloud, 'list_images', patch_list_images)
setattr(OpenStackCloud, 'search_images', patch_search_images)
setattr(OpenStackCloud, 'list_volumes', patch_list_volumes)
setattr(OpenStackCloud, 'search_volumes', patch_search_volumes)

def monkey_patch_os_volume():
setattr(OpenStackCloud, 'list_images', patch_list_images)
setattr(OpenStackCloud, 'search_images', patch_search_images)
setattr(OpenStackCloud, 'list_volumes', patch_list_volumes)
setattr(OpenStackCloud, 'search_volumes', patch_search_volumes)

def monkey_patch_os_server_volume():
setattr(OpenStackCloud, 'list_volumes', patch_list_volumes)
setattr(OpenStackCloud, 'search_volumes', patch_search_volumes)