Openstack Yogaを1から構築してみる ~Neutron編~

こんにちは。
株式会社クラスアクト インフラストラクチャ事業部の大塚です。

以前からOpenstack環境を1から構築することにチャレンジ中です。
なかなか大変だなぁと思う一方で、導入方法についてはGlance,Novaともあまり変わらず、
1. MariaDBにそれぞれのコンポーネントのデータベースを作成
1. Openstackにそれぞれのコンポーネントのuserやservice、endpointの作成
1. それぞれのコンポーネントをインストールしてconfigをいじる
1. MariaDBと同期をとる
1. 動くか確認する

みたいな流れで基本的には動作させることができるなと個人的な気づきを得ており、ぼちぼち充実した構築ライフを送れているような気がしています。
今回はNova/Placementの続き、Neutronの導入をしていきたいと思います。


https://qiita.com/ohtsuka-shota/items/de7d3b10c24091a1e29e

https://qiita.com/ohtsuka-shota/items/152e88e55d733e5a939e

https://qiita.com/ohtsuka-shota/items/4fc6106dd3a6497dd0f5

# HW構成

|HW  |値  |
|---|---|
|CPU  |4sockets 4cores |
|Memory  |40GB  |
|Disk  |100G  |
|NIC1(ens18)  |192.168.2.10  |
|NIC2(ens19)  |192.168.2.11　|

# 用語
## Neutronとは？
公式サイトより引用します
>OpenStack Networking (neutron) は、 OpenStack 環境で、仮想ネットワークインフラストラクチャ (VNI) のすべての面と、物理ネットワークインフラストラクチャ (PNI) のアクセス層の側面を管理します。

https://docs.openstack.org/liberty/ja/install-guide-ubuntu/neutron-concepts.html


## Open vSwitchとは？
※OVS,ML2,L3は同じURLからの参照です。

>Open vSwitch は、レガシーの Linux ソフトウェアブリッジと同様の、ソフトウェア定義ネットワーク (SDN: Software-Defined Networking) の仮想スイッチです。OVS は業界標準の NetFlow、OpenFlow、および sFlow をサポートする仮想ネットワークへのスイッチングサービスを提供します。Open vSwitch は、STP、LACP、802.1Q VLAN タグ付け などのレイヤー 2 (L2) 機能を使用することで物理スイッチとの統合も可能です。

## ML2とは？
※OVS,ML2,L3は同じURLからの参照です。

>ML2 とは、OpenStack Havana リリースで導入された OpenStack Networking コアプラグインです。以前のモノリシックなプラグインのモデルに置き換わる、ML2 のモジュラー型設計により、複数のネットワーク技術を組み合わせた操作を同時に実行できます。モノリシックな Open vSwitch および Linux Bridge プラグインは、非推奨となり、削除されました。これらの機能は、代わりに ML2 メカニズムドライバーとして再実装されています。

これがあることでVLANやVXLAN、flat等異なる種類のネットワークを同時に作成・接続することがかのうになるようです

## L3とは？
※OVS,ML2,L3は同じURLからの参照です。

>L3 エージェントは openstack-neutron パッケージに含まれています。ネットワーク名前空間は、各プロジェクトに独自の分離されたレイヤー 3 ルーターを提供するのに使用されます。レイヤー 3 ルーターは、トラフィックを誘導し、レイヤー 2 ネットワーク向けのゲートウェイサービスを提供します。L3 エージェントはこれらのルーターの管理を支援します。L3 エージェントをホストするノードでは、外部ネットワークに接続されたネットワークインターフェースに手動で IP アドレスを設定することはできません。代わりに、OpenStack Networking で利用可能な外部ネットワークの IP アドレスの範囲内で指定する必要があります。これらの IP アドレスは、内部ネットワークと外部ネットワークの間を接続するルーターに割り当てられます。選択した範囲は、デプロイメント内の各ルーターに一意の IP アドレスと、必要な各 Floating IP を指定するのに十分な大きさである必要があります。

https://access.redhat.com/documentation/ja-jp/red_hat_openstack_platform/9/html/networking_guide/openstack_networking_concepts

# 参考サイト

https://docs.openstack.org/neutron/2023.1/install/controller-install-ubuntu.html

https://docs.openstack.org/neutron/2023.1/install/controller-install-option2-ubuntu.html

# Config等

追ってGitLab上にまとめていきます。

# 構築
## MariaDBにNeutronのデータベースを作成
以下のコマンドを実行して、Neutron用のデータベースを作成します。
```
root@openstack:~# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 107615
Server version: 10.6.12-MariaDB-0ubuntu0.22.04.1 Ubuntu 22.04
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> CREATE DATABASE neutron;
Query OK, 1 row affected (0.000 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.002 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.009 sec)

MariaDB [(none)]>  
Query OK, 0 rows affected (0.003 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.002 sec)

MariaDB [(none)]> exit;
Bye
```

## OpenstackにNeutronのユーザを登録したりエンドポイントを作成したりする
openstackコマンドを使ってユーザを作成し、そのユーザにadmin権限を与えたり、エンドポイントを作成したりします。最後のcatalog listでサービスとエンドポイントが問題なく作成されているかを確認しています。
```
root@openstack:~# openstack user create --domain default --project test_project --password password neutron
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| default_project_id  | 94f179ab12a24facaf5001a1365700c2 |
| domain_id           | default                          |
| enabled             | True                             |
| id                  | d025c107fa2e4377a70c96be1cd68346 |
| name                | neutron                          |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+
root@openstack:~# openstack role add --project test_project --user neutron admin
root@openstack:~# openstack service create --name neutron --description "OpenStack Networking" network
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | OpenStack Networking             |
| enabled     | True                             |
| id          | 2d01b57f1efa4966aa45af708b5be5f8 |
| name        | neutron                          |
| type        | network                          |
+-------------+----------------------------------+
root@openstack:~# openstack endpoint create --region RegionOne network public http://openstack:9696
+--------------+----------------------------------+
| Field        | Value                            |
+--------------+----------------------------------+
| enabled      | True                             |
| id           | 703d9a422aa44807af6aa13988716542 |
| interface    | public                           |
| region       | RegionOne                        |
| region_id    | RegionOne                        |
| service_id   | 2d01b57f1efa4966aa45af708b5be5f8 |
| service_name | neutron                          |
| service_type | network                          |
| url          | http://openstack:9696            |
+--------------+----------------------------------+
root@openstack:~# openstack endpoint create --region RegionOne network internal http://openstack:
9696
+--------------+----------------------------------+
| Field        | Value                            |
+--------------+----------------------------------+
| enabled      | True                             |
| id           | dd0c663cc082437588a7b51a2b2d8fd8 |
| interface    | internal                         |
| region       | RegionOne                        |
| region_id    | RegionOne                        |
| service_id   | 2d01b57f1efa4966aa45af708b5be5f8 |
| service_name | neutron                          |
| service_type | network                          |
| url          | http://openstack:9696            |
+--------------+----------------------------------+
root@openstack:~# openstack endpoint create --region RegionOne network admin http://openstack:969
6
+--------------+----------------------------------+
| Field        | Value                            |
+--------------+----------------------------------+
| enabled      | True                             |
| id           | 38b5ddd7e831446aa46eba86e25dfd75 |
| interface    | admin                            |
| region       | RegionOne                        |
| region_id    | RegionOne                        |
| service_id   | 2d01b57f1efa4966aa45af708b5be5f8 |
| service_name | neutron                          |
| service_type | network                          |
| url          | http://openstack:9696            |
+--------------+----------------------------------+
root@openstack:~# openstack catalog list
+-----------+-----------+-------------------------------------------------------------------------+
| Name      | Type      | Endpoints                                                               |
+-----------+-----------+-------------------------------------------------------------------------+
| neutron   | network   | RegionOne                                                               |
|           |           |   admin: http://openstack:9696                                          |
|           |           | RegionOne                                                               |
|           |           |   public: http://openstack:9696                                         |
|           |           | RegionOne                                                               |
|           |           |   internal: http://openstack:9696                                       |
|           |           |                                                                         |
| glance    | image     | RegionOne                                                               |
|           |           |   public: http://openstack:9292                                         |
|           |           | RegionOne                                                               |
|           |           |   admin: http://openstack:9292                                          |
|           |           | RegionOne                                                               |
|           |           |   internal: http://openstack:9292                                       |
|           |           |                                                                         |
| placement | placement | RegionOne                                                               |
|           |           |   admin: http://openstack:8778                                          |
|           |           | RegionOne                                                               |
|           |           |   public: http://openstack:8778                                         |
|           |           | RegionOne                                                               |
|           |           |   internal: http://openstack:8778                                       |
|           |           |                                                                         |
| keystone  | identity  | RegionOne                                                               |
|           |           |   admin: http://openstack:5000/v3/                                      |
|           |           | RegionOne                                                               |
|           |           |   public: http://openstack:5000/v3/                                     |
|           |           | RegionOne                                                               |
|           |           |   internal: http://openstack:5000/v3/                                   |
|           |           |                                                                         |
| nova      | compute   | RegionOne                                                               |
|           |           |   public: http://openstack:8774/v2.1/5ea984608bdf4a338bddfc4e9f14bec7   |
|           |           | RegionOne                                                               |
|           |           |   admin: http://openstack:8774/v2.1/5ea984608bdf4a338bddfc4e9f14bec7    |
|           |           | RegionOne                                                               |
|           |           |   internal: http://openstack:8774/v2.1/5ea984608bdf4a338bddfc4e9f14bec7 |
|           |           |                                                                         |
+-----------+-----------+-------------------------------------------------------------------------+
```

## 必要なパッケージ等のインストール
```
root@openstack:~# apt install neutron-server neutron-plugin-ml2 neutron-openvswitch-agent neutron-l3-agent neutron-dhcp-agent neutron-metadata-agent -y
```

## Neutronのconfigをいじる
ここら辺は公式サイトのまま設定を入れてあげればOKです。
```
root@openstack:~# cp -p /etc/neutron/neutron.conf /etc/neutron/neutron.conf.org
root@openstack:~# vi /etc/neutron/neutron.conf
root@openstack:~# diff /etc/neutron/neutron.conf /etc/neutron/neutron.conf.org
3,7d2
< service_plugins = router
< transport_url = rabbit://openstack:password@openstack
< auth_strategy = keystone
< notify_nova_on_port_status_changes = true
< notify_nova_on_port_data_changes = true
829c824
< connection = mysql+pymysql://neutron:password@controller/neutron
---
> connection = sqlite:////var/lib/neutron/neutron.sqlite
1069,1077d1063
< www_authenticate_uri = http://openstack:5000
< auth_url = http://openstack:5000
< memcached_servers = openstack:11211
< auth_type = password
< project_domain_name = Default
< user_domain_name = Default
< project_name = test_project
< username = neutron
< password = password
1240,1247d1225
< auth_url = http://openstack:5000
< auth_type = password
< project_domain_name = Default
< user_domain_name = Default
< region_name = RegionOne
< project_name = test_project
< username = nova
< password = password
1385c1363
< lock_path = /var/lib/neutron/tmp
---
> #lock_path = <None>
```

```
root@openstack:~# cp -p /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugins/ml2/ml2_conf.ini.org
root@openstack:~# vi /etc/neutron/plugins/ml2/ml2_conf.ini
root@openstack:~# diff /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugins/ml2/ml2_conf.ini.org
154,159d153
< type_drivers = flat,vlan,vxlan
< tenant_network_types = vxlan
< mechanism_drivers = openvswitch,l2population
< extension_drivers = port_security
< flat_networks = provider
< vni_ranges = 1:1000
```

以下は少し気を付けないといけないかもです。検証しきれいているわけではないですが、私はTeratermでSSHしていないほうのPortを指定しました。devstackやpackstackでデプロイするときに、よくSSHしていないほうのPortを指定してデプロイをしていたので、それと同じかな？と思いこのような設定をしました。
実際にインスタンスを立ち上げてみたり、仮想ネットワークをデプロイしてみないとこのあたりの指定が正解なのかは正直わかりかねます。が、いったんNeutronが立ち上がったことは確認しているのでQiitaに投げておきます。
```
root@openstack:~# cp -p /etc/neutron/plugins/ml2/openvswitch_agent.ini /etc/neutron/plugins/ml2/openvswitch_agent.ini.org
root@openstack:~# vi /etc/neutron/plugins/ml2/openvswitch_agent.ini
root@openstack:~# diff /etc/neutron/plugins/ml2/openvswitch_agent.ini /etc/neutron/plugins/ml2/openvswitch_agent.ini.org
264,268d263
< bridge_mappings = provider:ens19
<
< [vxlan]
< local_ip = 192.168.2.11
< l2_population = true
415,416d409
< enable_security_group = true
< firewall_driver = openvswitch
```

```
root@openstack:~# cp -p /etc/neutron/l3_agent.ini /etc/neutron/l3_agent.ini.org
root@openstack:~# vi /etc/neutron/l3_agent.ini
root@openstack:~# diff /etc/neutron/l3_agent.ini /etc/neutron/l3_agent.ini.org
2d1
< interface_driver = openvswitch
```

```
root@openstack:~# cp -p /etc/neutron/dhcp_agent.ini /etc/neutron/dhcp_agent.ini.org
root@openstack:~# vi /etc/neutron/dhcp_agent.ini
root@openstack:~# diff /etc/neutron/dhcp_agent.ini /etc/neutron/dhcp_agent.ini.org
2,4d1
< interface_driver = openvswitch
< dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq
< enable_isolated_metadata = true
```

```
root@openstack:~# cp -p /etc/neutron/metadata_agent.ini /etc/neutron/metadata_agent.ini.org
root@openstack:~# vi /etc/neutron/metadata_agent.ini
root@openstack:~# diff /etc/neutron/metadata_agent.ini /etc/neutron/metadata_agent.ini.org
2,3d1
< nova_metadata_host = openstack
< metadata_proxy_shared_secret = METADATA_SECRET
```

```
root@openstack:~# vi /etc/nova/nova.conf
root@openstack:~# diff /etc/nova/nova.conf /etc/nova/nova.conf.org
5,7d4
< my_ip = 192.168.2.10
< enabled_apis = osapi_compute,metadata
< transport_url = rabbit://openstack:password@192.168.2.10
886d882
< auth_strategy = keystone
1099,1100d1094
< connection = mysql+pymysql://nova:password@192.168.2.10/nova_api
<
1797d1790
< connection = mysql+pymysql://nova:password@192.168.2.10/nova
2091d2083
< api_servers = http://192.168.2.10:9292
2730,2739d2721
< www_authenticate_uri = http://192.168.2.10:5000
< auth_url = http://192.168.2.10:5000
< memcached_servers = 192.168.2.10:11211
< auth_type = password
< project_domain_name = default
< user_domain_name = default
< project_name = test_project
< username = nova
< password = password
<
3553,3563d3534
< auth_url = http://openstack:5000
< auth_type = password
< project_domain_name = Default
< user_domain_name = Default
< region_name = RegionOne
< project_name = test_project
< username = neutron
< password = password
< service_metadata_proxy = true
< metadata_proxy_shared_secret = METADATA_SECRET
<
3801d3771
< lock_path = $state_path/tmp
4370,4377d4339
< auth_url = http://192.168.2.10:5000
< os_region_name = RegionOne
< auth_type = password
< project_domain_name = default
< user_domain_name = default
< project_name = test_project
< username = placement
< password = password
5434c5396
< server_listen = 0.0.0.0
---
> #server_listen = 127.0.0.1
5439c5401
< server_proxyclient_address = 0.0.0.0
---
> #server_proxyclient_address = 127.0.0.1
5617d5578
< api_paste_config = /etc/nova/api-paste.ini
```

## データベースと同期をする
```
root@openstack:~# su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
  Running upgrade for neutron ...
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> kilo
INFO  [alembic.runtime.migration] Running upgrade kilo -> 354db87e3225
INFO  [alembic.runtime.migration] Running upgrade 354db87e3225 -> 599c6a226151
INFO  [alembic.runtime.migration] Running upgrade 599c6a226151 -> 52c5312f6baf
INFO  [alembic.runtime.migration] Running upgrade 52c5312f6baf -> 313373c0ffee
INFO  [alembic.runtime.migration] Running upgrade 313373c0ffee -> 8675309a5c4f
INFO  [alembic.runtime.migration] Running upgrade 8675309a5c4f -> 45f955889773
INFO  [alembic.runtime.migration] Running upgrade 45f955889773 -> 26c371498592
INFO  [alembic.runtime.migration] Running upgrade 26c371498592 -> 1c844d1677f7
INFO  [alembic.runtime.migration] Running upgrade 1c844d1677f7 -> 1b4c6e320f79
INFO  [alembic.runtime.migration] Running upgrade 1b4c6e320f79 -> 48153cb5f051
INFO  [alembic.runtime.migration] Running upgrade 48153cb5f051 -> 9859ac9c136
INFO  [alembic.runtime.migration] Running upgrade 9859ac9c136 -> 34af2b5c5a59
INFO  [alembic.runtime.migration] Running upgrade 34af2b5c5a59 -> 59cb5b6cf4d
INFO  [alembic.runtime.migration] Running upgrade 59cb5b6cf4d -> 13cfb89f881a
INFO  [alembic.runtime.migration] Running upgrade 13cfb89f881a -> 32e5974ada25
INFO  [alembic.runtime.migration] Running upgrade 32e5974ada25 -> ec7fcfbf72ee
INFO  [alembic.runtime.migration] Running upgrade ec7fcfbf72ee -> dce3ec7a25c9
INFO  [alembic.runtime.migration] Running upgrade dce3ec7a25c9 -> c3a73f615e4
INFO  [alembic.runtime.migration] Running upgrade c3a73f615e4 -> 659bf3d90664
INFO  [alembic.runtime.migration] Running upgrade 659bf3d90664 -> 1df244e556f5
INFO  [alembic.runtime.migration] Running upgrade 1df244e556f5 -> 19f26505c74f
INFO  [alembic.runtime.migration] Running upgrade 19f26505c74f -> 15be73214821
INFO  [alembic.runtime.migration] Running upgrade 15be73214821 -> b4caf27aae4
INFO  [alembic.runtime.migration] Running upgrade b4caf27aae4 -> 15e43b934f81
INFO  [alembic.runtime.migration] Running upgrade 15e43b934f81 -> 31ed664953e6
INFO  [alembic.runtime.migration] Running upgrade 31ed664953e6 -> 2f9e956e7532
INFO  [alembic.runtime.migration] Running upgrade 2f9e956e7532 -> 3894bccad37f
INFO  [alembic.runtime.migration] Running upgrade 3894bccad37f -> 0e66c5227a8a
INFO  [alembic.runtime.migration] Running upgrade 0e66c5227a8a -> 45f8dd33480b
INFO  [alembic.runtime.migration] Running upgrade 45f8dd33480b -> 5abc0278ca73
INFO  [alembic.runtime.migration] Running upgrade 5abc0278ca73 -> d3435b514502
INFO  [alembic.runtime.migration] Running upgrade d3435b514502 -> 30107ab6a3ee
INFO  [alembic.runtime.migration] Running upgrade 30107ab6a3ee -> c415aab1c048
INFO  [alembic.runtime.migration] Running upgrade c415aab1c048 -> a963b38d82f4
INFO  [alembic.runtime.migration] Running upgrade kilo -> 30018084ec99
INFO  [alembic.runtime.migration] Running upgrade 30018084ec99 -> 4ffceebfada
INFO  [alembic.runtime.migration] Running upgrade 4ffceebfada -> 5498d17be016
INFO  [alembic.runtime.migration] Running upgrade 5498d17be016 -> 2a16083502f3
INFO  [alembic.runtime.migration] Running upgrade 2a16083502f3 -> 2e5352a0ad4d
INFO  [alembic.runtime.migration] Running upgrade 2e5352a0ad4d -> 11926bcfe72d
INFO  [alembic.runtime.migration] Running upgrade 11926bcfe72d -> 4af11ca47297
INFO  [alembic.runtime.migration] Running upgrade 4af11ca47297 -> 1b294093239c
INFO  [alembic.runtime.migration] Running upgrade 1b294093239c -> 8a6d8bdae39
INFO  [alembic.runtime.migration] Running upgrade 8a6d8bdae39 -> 2b4c2465d44b
INFO  [alembic.runtime.migration] Running upgrade 2b4c2465d44b -> e3278ee65050
INFO  [alembic.runtime.migration] Running upgrade e3278ee65050 -> c6c112992c9
INFO  [alembic.runtime.migration] Running upgrade c6c112992c9 -> 5ffceebfada
INFO  [alembic.runtime.migration] Running upgrade 5ffceebfada -> 4ffceebfcdc
INFO  [alembic.runtime.migration] Running upgrade 4ffceebfcdc -> 7bbb25278f53
INFO  [alembic.runtime.migration] Running upgrade 7bbb25278f53 -> 89ab9a816d70
INFO  [alembic.runtime.migration] Running upgrade 89ab9a816d70 -> c879c5e1ee90
INFO  [alembic.runtime.migration] Running upgrade c879c5e1ee90 -> 8fd3918ef6f4
INFO  [alembic.runtime.migration] Running upgrade 8fd3918ef6f4 -> 4bcd4df1f426
INFO  [alembic.runtime.migration] Running upgrade 4bcd4df1f426 -> b67e765a3524
INFO  [alembic.runtime.migration] Running upgrade a963b38d82f4 -> 3d0e74aa7d37
INFO  [alembic.runtime.migration] Running upgrade 3d0e74aa7d37 -> 030a959ceafa
INFO  [alembic.runtime.migration] Running upgrade 030a959ceafa -> a5648cfeeadf
INFO  [alembic.runtime.migration] Running upgrade a5648cfeeadf -> 0f5bef0f87d4
INFO  [alembic.runtime.migration] Running upgrade 0f5bef0f87d4 -> 67daae611b6e
INFO  [alembic.runtime.migration] Running upgrade b67e765a3524 -> a84ccf28f06a
INFO  [alembic.runtime.migration] Running upgrade a84ccf28f06a -> 7d9d8eeec6ad
INFO  [alembic.runtime.migration] Running upgrade 67daae611b6e -> 6b461a21bcfc
INFO  [alembic.runtime.migration] Running upgrade 6b461a21bcfc -> 5cd92597d11d
INFO  [alembic.runtime.migration] Running upgrade 5cd92597d11d -> 929c968efe70
INFO  [alembic.runtime.migration] Running upgrade 929c968efe70 -> a9c43481023c
INFO  [alembic.runtime.migration] Running upgrade a9c43481023c -> 804a3c76314c
INFO  [alembic.runtime.migration] Running upgrade 804a3c76314c -> 2b42d90729da
INFO  [alembic.runtime.migration] Running upgrade 2b42d90729da -> 62c781cb6192
INFO  [alembic.runtime.migration] Running upgrade 62c781cb6192 -> c8c222d42aa9
INFO  [alembic.runtime.migration] Running upgrade c8c222d42aa9 -> 349b6fd605a6
INFO  [alembic.runtime.migration] Running upgrade 349b6fd605a6 -> 7d32f979895f
INFO  [alembic.runtime.migration] Running upgrade 7d32f979895f -> 594422d373ee
INFO  [alembic.runtime.migration] Running upgrade 594422d373ee -> 61663558142c
INFO  [alembic.runtime.migration] Running upgrade 61663558142c -> 867d39095bf4, port forwarding
INFO  [alembic.runtime.migration] Running upgrade 867d39095bf4 -> d72db3e25539, modify uniq port forwarding
INFO  [alembic.runtime.migration] Running upgrade d72db3e25539 -> cada2437bf41
INFO  [alembic.runtime.migration] Running upgrade cada2437bf41 -> 195176fb410d, router gateway IP QoS
INFO  [alembic.runtime.migration] Running upgrade 195176fb410d -> fb0167bd9639
INFO  [alembic.runtime.migration] Running upgrade fb0167bd9639 -> 0ff9e3881597
INFO  [alembic.runtime.migration] Running upgrade 0ff9e3881597 -> 9bfad3f1e780
INFO  [alembic.runtime.migration] Running upgrade 9bfad3f1e780 -> 63fd95af7dcd
INFO  [alembic.runtime.migration] Running upgrade 63fd95af7dcd -> c613d0b82681
INFO  [alembic.runtime.migration] Running upgrade c613d0b82681 -> c3e9d13c4367
INFO  [alembic.runtime.migration] Running upgrade c3e9d13c4367 -> 86274d77933e
INFO  [alembic.runtime.migration] Running upgrade 86274d77933e -> f4b9654dd40c
INFO  [alembic.runtime.migration] Running upgrade f4b9654dd40c -> a010322604bc
INFO  [alembic.runtime.migration] Running upgrade a010322604bc -> 263d454a9655
INFO  [alembic.runtime.migration] Running upgrade 263d454a9655 -> Ibac91d24da2
INFO  [alembic.runtime.migration] Running upgrade Ibac91d24da2 -> 2217c4222de6
INFO  [alembic.runtime.migration] Running upgrade 2217c4222de6 -> 18a7e90ae768
INFO  [alembic.runtime.migration] Running upgrade 18a7e90ae768 -> e4e236b0e1ff
INFO  [alembic.runtime.migration] Running upgrade e4e236b0e1ff -> e88badaa9591
INFO  [alembic.runtime.migration] Running upgrade e88badaa9591 -> d8bdf05313f4
INFO  [alembic.runtime.migration] Running upgrade d8bdf05313f4 -> dfe425060830
INFO  [alembic.runtime.migration] Running upgrade dfe425060830 -> fd6107509ccd
INFO  [alembic.runtime.migration] Running upgrade fd6107509ccd -> 1ea5dab0897a
INFO  [alembic.runtime.migration] Running upgrade 1ea5dab0897a -> 49d8622c5221
INFO  [alembic.runtime.migration] Running upgrade 49d8622c5221 -> I38991de2b4
INFO  [alembic.runtime.migration] Running upgrade I38991de2b4 -> 532aa95457e2
INFO  [alembic.runtime.migration] Running upgrade 532aa95457e2 -> f010820fc498
INFO  [alembic.runtime.migration] Running upgrade f010820fc498 -> a964d94b4677
INFO  [alembic.runtime.migration] Running upgrade a964d94b4677 -> 26d1e9f5c766
INFO  [alembic.runtime.migration] Running upgrade 26d1e9f5c766 -> 1e0744e4ffea
INFO  [alembic.runtime.migration] Running upgrade 1e0744e4ffea -> 6135a7bd4425
INFO  [alembic.runtime.migration] Running upgrade 6135a7bd4425 -> 8df53b0d2c0e
INFO  [alembic.runtime.migration] Running upgrade 8df53b0d2c0e -> 1bb3393de75d, add qos policy rule Packet Rate Limit
INFO  [alembic.runtime.migration] Running upgrade 1bb3393de75d -> c181bb1d89e4
INFO  [alembic.runtime.migration] Running upgrade c181bb1d89e4 -> ba859d649675
INFO  [alembic.runtime.migration] Running upgrade ba859d649675 -> e981acd076d3
INFO  [alembic.runtime.migration] Running upgrade e981acd076d3 -> 76df7844a8c6, add Local IP tables
INFO  [alembic.runtime.migration] Running upgrade 76df7844a8c6 -> 1ffef8d6f371, migrate RBAC registers from "target_tenant" to "target_project"
INFO  [alembic.runtime.migration] Running upgrade 1ffef8d6f371 -> 8160f7a9cebb, drop portbindingports table
INFO  [alembic.runtime.migration] Running upgrade 8160f7a9cebb -> cd9ef14ccf87
INFO  [alembic.runtime.migration] Running upgrade cd9ef14ccf87 -> 34cf8b009713
INFO  [alembic.runtime.migration] Running upgrade 7d9d8eeec6ad -> a8b517cff8ab
INFO  [alembic.runtime.migration] Running upgrade a8b517cff8ab -> 3b935b28e7a0
INFO  [alembic.runtime.migration] Running upgrade 3b935b28e7a0 -> b12a3ef66e62
INFO  [alembic.runtime.migration] Running upgrade b12a3ef66e62 -> 97c25b0d2353
INFO  [alembic.runtime.migration] Running upgrade 97c25b0d2353 -> 2e0d7a8a1586
INFO  [alembic.runtime.migration] Running upgrade 2e0d7a8a1586 -> 5c85685d616d
  OK
```

## サービスの再起動
```
root@openstack:~# service nova-api restart
root@openstack:~# service neutron-server restart
root@openstack:~# service neutron-openvswitch-agent restart
root@openstack:~# service neutron-dhcp-agent restart
root@openstack:~# service neutron-metadata-agent restart
root@openstack:~# service neutron-l3-agent restart
```

## 試験
openstack network agent listコマンドを実行してみて、何らかの結果が返ってくれば問題ないのかな？と思います。
細かい修正は、今後実際にインスタンスを立ち上げてみたりして都度修正になるかと思います。
```
root@openstack:~# openstack network agent list
+--------------------------------------+----------------+-----------+-------------------+-------+-------+------------------------+
| ID                                   | Agent Type     | Host      | Availability Zone | Alive | State | Binary                 |
+--------------------------------------+----------------+-----------+-------------------+-------+-------+------------------------+
| b6edbada-e28c-4763-8a25-5dca827012dd | DHCP agent     | openstack | nova              | :-)   | UP    | neutron-dhcp-agent     |
| ca3050cb-3cec-4b4f-b4bd-d4ca569e4a2a | L3 agent       | openstack | nova              | :-)   | UP    | neutron-l3-agent       |
| f7c2d944-de06-42e0-acc6-340151bca9f7 | Metadata agent | openstack | None              | :-)   | UP    | neutron-metadata-agent |
+--------------------------------------+----------------+-----------+-------------------+-------+-------+------------------------+
```