OpenStack Neutron API Testの実行方法

# OpenStack Neutron API Test

Tempestに居たNeutronのAPIテストがNeutronのリポジトリに移動したらしい。

## やってみる

環境はUbuntu 14.04 server, devstackでmaster(2015/05/08)。

https://github.com/openstack/neutron/blob/2015.1.0/TESTING.rst

によると、以下のようにtoxでAPIテストが実行出来るっぽい。

```tox -e api```

とりあえずやってみた。

```
ubuntu@ubuntu:/opt/stack/neutron$ tox -e api
api create: /opt/stack/neutron/.tox/api
api installdeps: -r/opt/stack/neutron/requirements.txt, -r/opt/stack/neutron/test-requirements.txt
api develop-inst: /opt/stack/neutron
api runtests: PYTHONHASHSEED='497646445'
api runtests: commands[0] | sh tools/pretty_tox.sh
running testr
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --list
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit}  --load-list /tmp/tmpYb3hnU
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit}  --load-list /tmp/tmpsXlaPP
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit}  --load-list /tmp/tmplVajJY
{0} neutron.tests.api.admin.test_agent_management.AgentManagementTestJSON.test_update_agent_description [0.711050s] ... ok
{2} neutron.tests.api.admin.test_agent_management.AgentManagementTestJSON.test_list_agent [0.061274s] ... ok
{2} neutron.tests.api.admin.test_agent_management.AgentManagementTestJSON.test_list_agents_non_admin [0.173823s] ... ok
{2} neutron.tests.api.admin.test_agent_management.AgentManagementTestJSON.test_show_agent [0.014137s] ... ok
{2} neutron.tests.api.admin.test_agent_management.AgentManagementTestJSON.test_update_agent_status [0.019427s] ... ok
{1} neutron.tests.api.admin.test_dhcp_agent_scheduler.DHCPAgentSchedulersTestJSON.test_add_remove_network_from_dhcp_agent [1.060659s] ... ok
{0} neutron.tests.api.admin.test_external_network_extension.ExternalNetworksTestJSON.test_delete_external_networks_with_floating_ip [1.655746s] ... ok
{2} neutron.tests.api.admin.test_dhcp_agent_scheduler.DHCPAgentSchedulersTestJSON.test_list_dhcp_agent_hosting_network [0.288019s] ... ok
{0} neutron.tests.api.admin.test_external_network_extension.ExternalNetworksTestJSON.test_list_external_networks [1.378430s] ... ok
{2} neutron.tests.api.admin.test_dhcp_agent_scheduler.DHCPAgentSchedulersTestJSON.test_list_networks_hosted_by_one_dhcp [0.113022s] ... ok
{0} neutron.tests.api.admin.test_external_network_extension.ExternalNetworksTestJSON.test_show_external_networks_attribute [0.353835s] ... ok
{1} neutron.tests.api.admin.test_floating_ips_admin_actions.FloatingIPAdminTestJSON.test_associate_floating_ip_with_port_from_another_tenant [2.283913s] ... ok
{1} neutron.tests.api.admin.test_floating_ips_admin_actions.FloatingIPAdminTestJSON.test_create_list_show_floating_ip_with_tenant_id_by_admin [0.410622s] ... ok
{0} neutron.tests.api.admin.test_l3_agent_scheduler.L3AgentSchedulerTestJSON.test_add_list_remove_router_on_l3_agent [0.455953s] ... ok
{2} neutron.tests.api.admin.test_external_network_extension.ExternalNetworksTestJSON.test_create_external_network [0.624782s] ... ok
{2} neutron.tests.api.admin.test_external_network_extension.ExternalNetworksTestJSON.test_update_external_network [0.856571s] ... ok
{0} neutron.tests.api.admin.test_lbaas_agent_scheduler.LBaaSAgentSchedulerTestJSON.test_show_lbaas_agent_hosting_pool [0.351017s] ... ok
{2} neutron.tests.api.admin.test_external_networks_negative.ExternalNetworksAdminNegativeTestJSON.test_create_port_with_precreated_floatingip_as_fixed_ip [0.687538s] ... ok


...(略)



======
Totals
======
Ran: 344 tests in 513.0000 sec.
 - Passed: 335
 - Skipped: 1
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 8
Sum of execute time for each test: 653.5867 sec.

==============
Worker Balance
==============
 - Worker 0 (114 tests) => 0:07:14.391424s
 - Worker 1 (114 tests) => 0:08:25.051351s
 - Worker 2 (116 tests) => 0:07:38.931830s
ERROR: InvocationError: '/bin/sh tools/pretty_tox.sh '
________________________________________________________________________ summary ________________________________________________________________________
ERROR:   api: commands failed
ubuntu@ubuntu:/opt/stack/neutron$
```

すこし失敗している模様。
本当にapiに対してテストをしているか確認したいので、neutron-serverを止めてもう一度実行。（Captured Tracebackの下の行は-じゃなく~なのですが、markdownのコードブロックが切れてしまうので書き換えた）

```
ubuntu@ubuntu:/opt/stack/neutron$ tox -e api
api develop-inst-noop: /opt/stack/neutron
api runtests: PYTHONHASHSEED='1848685913'
api runtests: commands[0] | sh tools/pretty_tox.sh
running testr
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --list
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit}  --load-list /tmp/tmphNLZMd
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit}  --load-list /tmp/tmpKwnfsf
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit}  --load-list /tmp/tmpBOWAjL
{0} setUpClass (neutron.tests.api.admin.test_agent_management.AgentManagementTestJSON) [0.000000s] ... FAILED

Captured traceback:
-------------------
    Traceback (most recent call last):
      File "neutron/tests/tempest/test.py", line 260, in setUpClass
        cls.resource_setup()
      File "neutron/tests/api/admin/test_agent_management.py", line 28, in resource_setup
        body = cls.admin_client.list_agents()
      File "neutron/tests/tempest/services/network/json/network_client.py", line 102, in _list
        resp, body = self.get(uri)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 267, in get
        return self.request('GET', url, extra_headers, headers)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 616, in request
        headers=headers, body=body)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 516, in _request
        req_url, method, headers=req_headers, body=req_body)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 547, in raw_request
        headers=headers, body=body)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/tempest_lib/common/http.py", line 25, in request
        return super(ClosingHttp, self).request(*args, **new_kwargs)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1608, in request
        (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1350, in _request
        (response, content) = self._conn_request(conn, request_uri, method, body, headers)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1272, in _conn_request
        conn.connect()
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/httplib2/__init__.py", line 928, in connect
        raise socket.error, msg
    socket.error: [Errno 111] ECONNREFUSED

{2} setUpClass (neutron.tests.api.admin.test_agent_management.AgentManagementTestJSON) [0.000000s] ... FAILED

Captured traceback:
-------------------
    Traceback (most recent call last):
      File "neutron/tests/tempest/test.py", line 260, in setUpClass
        cls.resource_setup()
      File "neutron/tests/api/admin/test_agent_management.py", line 28, in resource_setup
        body = cls.admin_client.list_agents()
      File "neutron/tests/tempest/services/network/json/network_client.py", line 102, in _list
        resp, body = self.get(uri)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 267, in get
        return self.request('GET', url, extra_headers, headers)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 616, in request
        headers=headers, body=body)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 516, in _request
        req_url, method, headers=req_headers, body=req_body)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 547, in raw_request
        headers=headers, body=body)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/tempest_lib/common/http.py", line 25, in request
        return super(ClosingHttp, self).request(*args, **new_kwargs)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1608, in request
        (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1350, in _request
        (response, content) = self._conn_request(conn, request_uri, method, body, headers)
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1272, in _conn_request
        conn.connect()
      File "/opt/stack/neutron/.tox/api/local/lib/python2.7/site-packages/httplib2/__init__.py", line 928, in connect
        raise socket.error, msg
    socket.error: [Errno 111] ECONNREFUSED


...(略)


======
Totals
======
Ran: 231 tests in 109.0000 sec.
 - Passed: 0
 - Skipped: 1
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 230
Sum of execute time for each test: 9.4285 sec.

==============
Worker Balance
==============
 - Worker 0 (78 tests) => 0:01:37.483229s
 - Worker 1 (82 tests) => 0:01:44.444588s
 - Worker 2 (71 tests) => 0:01:26.966016s
ERROR: InvocationError: '/bin/sh tools/pretty_tox.sh '
________________________________________________________________________ summary ________________________________________________________________________
ERROR:   api: commands failed
ubuntu@ubuntu:/opt/stack/neutron$
```

すべて失敗な上、原因はconnectionを貼れないことっぽいので、ちゃんとAPIを実行していると思われる。

## 見る

### 設定ファイル

https://github.com/openstack/neutron/blob/2015.1.0/tox.ini#L28-L31

```tox.ini``` を見たところ、テストがらみの設定は ```/opt/stack/tempest/etc``` を参照しているっぽい。
環境変数に ```TEMPEST_CONFIG_DIR``` を設定すると、設定ファイルの参照パスを変えられるもよう。

### テストコード

適当に以下のファイルを見てみる。
https://github.com/openstack/neutron/blob/2015.1.0/neutron/tests/api/test_floating_ips.py

tempestにいた頃と大した違いは内容に思えた。tempest_libを使っていたり、親クラスが ```neutron.tests.api.base``` のクラスになっているくらい？

親クラスも見てみる。
https://github.com/openstack/neutron/blob/2015.1.0/neutron/tests/api/base.py

特に変化は無いんじゃないかと思う。よく見ていない。
API Clientがtempestにあるクライアントじゃないのかと思い以下のモジュールを見てみると...
https://github.com/openstack/neutron/blob/2015.1.0/neutron/tests/api/clients.py

```neutron.tests.tempest.services.network.json.network_client``` をimportしている。
ちなみにtempestでのNeutron API Clientは```tempest.services.network.json.network_client```。

## コメント

ほぼ移動、コピーしただけっぽい。
toxで実行できる分、個人的にはAPIテストがやりやすくなったと思う。
テストを作る分には今までと同じように作れば良さそう。

## おまけ

```
tox -e api -- neutron.tests.api.admin
```

こんなかんじで、テスト対象をフィルタリング出来る。