GNS3環境でpyATSを使って自動試験する

## 概要
GNS3でNexus9000v×4台＋vJunos-switch×3台を使った環境を構築している。結構リソースを詰め込んで使っていることもあって時々IFがerrdisableになっているときがあり、正常な状態でないことを検知できるようにpyATSを使って自動試験しようと思うに至った。

使用するトポロジー
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3359374/7e55e297-8ab2-0c32-0fcf-f738cc775dec.png)

## 環境
- ThinkCentre M75q Gen2（8CPUs, Mem64GB）
- Ubuntu 20.04.1 LTS
- GNS3：2.2.42
- pyATS 23.7

GNS3上のNW機器とGNSの外側にいるpyATSを接続できるようにするために、GNS3でCloud Nodeを置いてvirbr0と接続している。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3359374/81cd7939-2325-03f8-c3df-f5d5e024bcb2.png)

## pyATSセットアップ
### Install
[公式ドキュメント | Getting Started](https://pubhub.devnetcloud.com/media/pyats/docs/getting_started/index.html#installation) のとおり行うのみ。

### testbed.yaml
ググって出てきたサンプルを参考に、踏み台経由でNW機器へ接続するようにproxyを記載した。
```
devices:
  jump_host:
    os: linux
    type: linux
    connections:
      ssh:
        ip: 192.168.11.55
        port: 22
        protocol: ssh
    credentials:
      default:
        password: "%ENV{JUMP_HOST_PASS}"
        username: ubuntu
  LEAF-01:
    os: nxos
    platform: nxosv
    type: NX-OSv
    alias: nx-osv-1
    connections:
      ssh:
        ip: 192.168.122.101
        protocol: ssh
        proxy: jump_host
    credentials:
      default:
        password: "%ENV{DEVICE_PASS}"
        username: admin
  LEAF-02:
    os: nxos
    platform: nxosv
    type: NX-OSv
    alias: nx-osv-2
    connections:
      ssh:
        ip: 192.168.122.102
        protocol: ssh
        proxy: jump_host
    credentials:
      default:
        password: "%ENV{DEVICE_PASS}"
        username: admin
  LEAF-03:
    os: nxos
    platform: nxosv
    type: NX-OSv
    alias: nx-osv-3
    connections:
      ssh:
        ip: 192.168.122.103
        protocol: ssh
        proxy: jump_host
    credentials:
      default:
        password: "%ENV{DEVICE_PASS}"
        username: admin
  SPINE-01:
    os: nxos
    platform: nxosv
    type: NX-OSv
    alias: nx-osv-4
    connections:
      ssh:
        ip: 192.168.122.105
        protocol: ssh
        proxy: jump_host
    credentials:
      default:
        password: "%ENV{DEVICE_PASS}"
        username: admin
```

testbed.yamlのバリデーションをチェックする。interfaceに関するWarning messageが出ているがこれは無視してOK。
```
(ubuntu) ubuntu@pyats:~$ pyats validate testbed ~/yaml/testbed.yaml
/home/ubuntu/bin/pyats:5: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pyats.cli.__main__ import main
Loading testbed file: /home/ubuntu/yaml/testbed.yaml
--------------------------------------------------------------------------------

Testbed Name:
    testbed

Testbed Devices:
.
|-- LEAF-01 [nxos/nxosv]
|-- LEAF-02 [nxos/nxosv]
|-- LEAF-03 [nxos/nxosv]
|-- SPINE-01 [nxos/nxosv]
`-- jump_host [linux/linux]

YAML Lint Messages
------------------

Warning Messages
----------------
 - Device 'LEAF-01' has no interface definitions
 - Device 'LEAF-02' has no interface definitions
 - Device 'LEAF-03' has no interface definitions
 - Device 'SPINE-01' has no interface definitions
 - Device 'jump_host' has no interface definitions

(ubuntu) ubuntu@pyats:~$
```

testbed.yamlに書いている装置へ接続してみる。このように機器に接続したり、showコマンドの結果をparseしたりして使うようだ。
```
(ubuntu) ubuntu@pyats:~$ pyats shell --testbed ~/yaml/testbed.yaml
/home/ubuntu/bin/pyats:5: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pyats.cli.__main__ import main
Welcome to pyATS Interactive Shell
==================================
Python 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0]

>>> from pyats.topology.loader import load
>>> testbed = load('/home/ubuntu/yaml/testbed.yaml')
-------------------------------------------------------------------------------
>>>
>>> testbed.devices['LEAF-01'].connect()

2023-08-17 11:45:45,551: %UNICON-INFO: +++ LEAF-01 logfile /tmp/LEAF-01-cli-20230817T114545550.log +++
2023-08-17 11:45:45,557: %UNICON-INFO: connection via proxy jump_host
2023-08-17 11:45:45,562: %UNICON-INFO: +++ connection to spawn: ssh -l ubuntu 192.168.11.55 -p 22, id: 139745503361728 +++
2023-08-17 11:45:45,564: %UNICON-INFO: connection to jump_host
ubuntu@192.168.11.55's password:

ubuntu@gns3:~$

2023-08-17 11:45:45,866: %UNICON-INFO: +++ initializing handle +++
2023-08-17 11:45:45,868: %UNICON-INFO: connection to LEAF-01
ssh -l admin 192.168.122.101
User Access Verification
Password:

Cisco NX-OS Software
Copyright (c) 2002-2021, Cisco Systems, Inc. All rights reserved.
Nexus 9000v software ("Nexus 9000v Software") and related documentation,
files or other reference materials ("Documentation") are
the proprietary property and confidential information of Cisco
Systems, Inc. ("Cisco") and are protected, without limitation,
pursuant to United States and International copyright and trademark
laws in the applicable jurisdiction which provide civil and criminal
penalties for copying or distribution without Cisco's authorization.

Any use or disclosure, in whole or in part, of the Nexus 9000v Software
or Documentation to any third party for any purposes is expressly
prohibited except as otherwise authorized by Cisco in writing.
The copyrights to certain works contained herein are owned by other
third parties and are used and distributed under license. Some parts
of this software may be covered under the GNU Public License or the
GNU Lesser General Public License. A copy of each such license is
available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
***************************************************************************
*  Nexus 9000v is strictly limited to use for evaluation, demonstration   *
*  and NX-OS education. Any use or disclosure, in whole or in part of     *
*  the Nexus 9000v Software or Documentation to any third party for any   *
*  purposes is expressly prohibited except as otherwise authorized by     *
*  Cisco in writing.                                                      *
***************************************************************************
LEAF-01#
```

## 自動試験
pythonで書けば何でもできるようだが、それだとpyATSを使う意味がないように思ったので標準の機能で試験できるようにしたい。→pyATS learnというものを使えばよさそうだ。

#### まず通常状態をlearnしておく
learnの対象はさまざまあるようだが今回はroutingを対象とした。learnした結果をv1として保存しておく。
```
(ubuntu) ubuntu@pyats:~$ pyats learn routing --testbed-file ~/yaml/testbed.yaml --output v1
```

#### NW機器で障害を発生させる
LEAF-01でSPINE-01向けIFをshutする。これによりLEAF-01とSPINE-01間のospfネイバーが切れる。
```
LEAF-01# conf t
Enter configuration commands, one per line. End with CNTL/Z.
LEAF-01(config)# int po2
LEAF-01(config-if)# shut
```

#### 現在の状態をlearnする
learnした結果をv2として保存する。
```
(ubuntu) ubuntu@pyats:~$ pyats learn routing --testbed-file ~/yaml/testbed.yaml --output v2
```

#### 通常状態と現在をdiffする
diffした結果のファイルが作成される。ファイルを見ると差分が表示されており、ネイバーが切れていたりnext-hopが変わったことが分かる。
```
(ubuntu) ubuntu@pyats:~$ pyats diff v1 v2 --output diff-file
(ubuntu) ubuntu@pyats:~$ ls ./diff-file/
diff_routing_nxos_LEAF-01_ops.txt  diff_routing_nxos_LEAF-02_ops.txt  diff_routing_nxos_LEAF-03_ops.txt  diff_routing_nxos_SPINE-01_ops.txt

//
(ubuntu) ubuntu@pyats:~$ cat ./diff-file/diff_routing_nxos_LEAF-01_ops.txt
--- v1/routing_nxos_LEAF-01_ops.txt
+++ v2/routing_nxos_LEAF-01_ops.txt
 info:
  vrf:
   default:
    address_family:
     ipv4:
      routes:
       192.168.1.4/32:
        next_hop:
         next_hop_list:
          1:
-          next_hop: 192.168.2.21
+          next_hop: 192.168.2.38
-          outgoing_interface: Port-channel2
+          outgoing_interface: Vlan102
-         2:
-          index: 2
-          next_hop: 192.168.2.38
-          outgoing_interface: Vlan102
-          updated: 00:56:55
       192.168.2.24/30:
-       metric: 40
+       metric: 60
        next_hop:
         next_hop_list:
          1:
-          next_hop: 192.168.2.21
+          next_hop: 192.168.2.38
-          outgoing_interface: Port-channel2
+          outgoing_interface: Vlan102
       2.2.2.2/32:
        next_hop:
         next_hop_list:
          1:
-          next_hop: 192.168.2.21
+          next_hop: 192.168.2.38
-          outgoing_interface: Port-channel2
+          outgoing_interface: Vlan102
-         2:
-          index: 2
-          next_hop: 192.168.2.38
-          outgoing_interface: Vlan102
-          updated: 00:59:02
-      192.168.2.20/30:
-       active: True
-       metric: 0
-       next_hop:
-        next_hop_list:
-         1:
-          index: 1
-          next_hop: 192.168.2.22
-          outgoing_interface: Port-channel2
-          updated: 00:01:36
-       route: 192.168.2.20/30
-       route_preference: 0
-       source_protocol: direct
-      192.168.2.22/32:
-       active: True
-       metric: 0
-       next_hop:
-        next_hop_list:
-         1:
-          index: 1
-          next_hop: 192.168.2.22
-          outgoing_interface: Port-channel2
-          updated: 00:01:36
-       route: 192.168.2.22/32
-       route_preference: 0
-       source_protocol: local
-      192.168.2.28/30:
-       active: True
-       metric: 40
-       next_hop:
-        next_hop_list:
-         1:
-          index: 1
-          next_hop: 192.168.2.21
-          outgoing_interface: Port-channel2
-          updated: 00:00:41
-       route: 192.168.2.28/30
-       route_preference: 110
-       source_protocol: ospf
-      3.3.3.3/32:
-       active: True
-       metric: 41
-       next_hop:
-        next_hop_list:
-         1:
-          index: 1
-          next_hop: 192.168.2.21
-          outgoing_interface: Port-channel2
-          updated: 00:00:34
-       route: 3.3.3.3/32
-       route_preference: 110
-       source_protocol: ospf
-      5.5.5.5/32:
-       active: True
-       metric: 21
-       next_hop:
-        next_hop_list:
-         1:
-          index: 1
-          next_hop: 192.168.2.21
-          outgoing_interface: Port-channel2
-          updated: 00:01:21
-       route: 5.5.5.5/32
-       route_preference: 110
-       source_protocol: ospf
```

## まとめ
すべてpythonで書くのであれば、paramikoやnetmikoを使ってssh接続して情報取得したものをparseしたほうが早い気がする。ただ、pyATSで出来ることは他にもいろいろあるようなので、使いこなしていくと使い入れが出てくるかもしれない。