AIX の監視：Njmon / infulxDB / Grafana 導入


##### はじめに
Nmon の進化系、NJMON を使用して、監視メトリクスを Grafana で視覚化する方法を確認しました。IBM Cloud Power Systems Virtual Server にも適用可能です。


(参考) Njmon についてはこちら：
http://nmon.sourceforge.net/pmwiki.php?n=Site.Njmon

以下の文書を参考に、AIX を監視サーバーとして、Njmon, influxDB, Grafana を導入します。
・AIX: Installing InfluxDB 1.8 and Grafana 7
https://www.ibm.com/support/pages/aix-installing-influxdb-18-and-grafana-7

---
##### 環境
・OS : AIX 7.2 TL5 SP1
・HW : S922 (Power Systems Virtual Server)

---

##### 事前準備

```
# oslevel -s
7200-05-01-2038
# lspv
hdisk0          00fa00d6b552f41b                    rootvg          active
# lspv
hdisk0          00fa00d6b552f41b                    rootvg          active
# cfgmgr
# lspv
hdisk0          00fa00d6b552f41b                    rootvg          active
hdisk1          none                                None
#
```

実行は省略しますが、influxDB用に hdisk1 に Volume Group,Logical Volume, FileSystem を作成しました。

```
# lspv
hdisk0          00faxxxxxxx2f41b                    rootvg          active
hdisk1          00ca784xxxxxxx98                    testvg          active
#
# lsvg -l testvg
testvg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1       1       1    closed/syncd  N/A
fslv00              jfs2       576     576     1    closed/syncd  /var/lib/influxdb
#
# lsfs | grep influxdb
/dev/fslv00     --         /var/lib/influxdb      jfs2  18874368 rw         no   no
#
```

AIX Toolbox が yum repository に定義されていることを確認

```
# yum repolist
AIX_Toolbox                                                                                                | 2.6 kB  00:00:00
AIX_Toolbox/primary_db                                                                                     | 1.8 MB  00:00:01
AIX_Toolbox_72                                                                                             | 2.5 kB  00:00:00
AIX_Toolbox_72/primary_db                                                                                  | 278 kB  00:00:00
AIX_Toolbox_noarch                                                                                         | 2.5 kB  00:00:00
AIX_Toolbox_noarch/primary_db                                                                              | 100 kB  00:00:00
repo id                                                  repo name                                                          status
AIX_Toolbox                                              AIX generic repository                                             2,661
AIX_Toolbox_72                                           AIX 7.2 specific repository                                          400
AIX_Toolbox_noarch                                       AIX noarch repository                                                281
repolist: 3,342
```


・wget を導入

```
# yum install wget
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package wget.ppc 0:1.21.1-1 will be installed
--> Processing Dependency: libiconv >= 1.16 for package: wget-1.21.1-1.ppc

〜省略〜

---> Package sqlite.ppc 0:3.35.5-1 will be an update
--> Finished Dependency Resolution


Dependencies Resolved


==================================================================================================================================
 Package                          Arch                    Version                        Repository                          Size
==================================================================================================================================
Installing:
 wget                             ppc                     1.21.1-1                       AIX_Toolbox                        703 k
Installing for dependencies:
 libgcc8                          ppc                     8.3.0-5                        AIX_Toolbox_72                     974 k

〜省略〜

Transaction Summary
==================================================================================================================================
Install      10 Packages
Upgrade      15 Packages


Total size: 106 M
Is this ok [y/N]: y
Downloading Packages:
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : libgcc8-8.3.0-5.ppc                                                                                           1/40 
〜省略〜
  Cleanup    : db-4.8.24-3.ppc                                                                                              40/40


Installed:
  wget.ppc 0:1.21.1-1


Dependency Installed:
  libgcc8.ppc 0:8.3.0-5       libgomp.ppc 1:8-1           libgomp8.ppc 0:8.3.0-5 libiconv.ppc 0:1.16-1 libstdc++8.ppc 0:8.3.0-5
  libtextstyle.ppc 0:0.20.2-1 libunistring.ppc 0:0.9.10-1 libxml2.ppc 0:2.9.10-1 xz-libs.ppc 0:5.2.5-1


Dependency Updated:
  bzip2.ppc 0:1.0.8-2    db.ppc 1:5.3.28-1            expat.ppc 0:2.2.9-2          gdbm.ppc 0:1.19-1     gettext.ppc 0:0.20.2-1
  glib2.ppc 0:2.56.1-3   libffi.ppc 0:3.2.1-3         libgcc.ppc 1:8-1             libstdc++.ppc 1:8-1   ncurses.ppc 0:6.2-2
  python.ppc 0:2.7.18-3  python-devel.ppc 0:2.7.18-3  python-tools.ppc 0:2.7.18-3  readline.ppc 0:8.1-1  sqlite.ppc 0:3.35.5-1


Complete!
#
```

PATH が切られていないので、PATH (/opt/freeware/bin) を設定します。

```
# which wget
no wget in /usr/bin /etc /usr/sbin /usr/ucb /usr/bin/X11 /sbin /usr/java8_64/jre/bin /usr/java8_64/bin
# ls -l /opt/freeware/bin | grep wget
lrwxrwxrwx    1 root     system            7 Aug 20 01:56 wget -> wget_64
-rwxr-xr-x    1 root     system       620878 Mar 03 22:43 wget_32
-rwxr-xr-x    1 root     system       620688 Mar 03 22:43 wget_64

# export PATH=$PATH:/opt/freeware/bin
# echo $PATH
/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin

# which wget
/opt/freeware/bin/wget
```


unzip を導入

```
# yum install unzip
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package unzip.ppc 0:6.0-3 will be installed
--> Finished Dependency Resolution


Dependencies Resolved


==================================================================================================================================
 Package                      Arch                       Version                         Repository                          Size
==================================================================================================================================
Installing:
 unzip                        ppc                        6.0-3                           AIX_Toolbox                        353 k


Transaction Summary
==================================================================================================================================
Install       1 Package


Total download size: 353 k
Installed size: 353 k
Is this ok [y/N]: y
Downloading Packages:
unzip-6.0-3.aix6.1.ppc.rpm                                                                                 | 353 kB  00:00:00
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : unzip-6.0-3.ppc                                                                                                1/1


Installed:
  unzip.ppc 0:6.0-3


Complete!
#
```

unzip導入確認

```
# which unzip
/usr/bin/unzip
```

----

##### influxDB 導入

influxDB 1.8.3.1 をダウンロード


```
# wget https://dl.power-devops.com/powerdevops.influxdb.1.8.3.1.bff.gz
--2021-08-20 02:00:09--  https://dl.power-devops.com/powerdevops.influxdb.1.8.3.1.bff.gz
Resolving dl.power-devops.com... 159.69.86.175
Connecting to dl.power-devops.com|159.69.86.175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 56685356 (54M) [application/x-gzip]
Saving to: 'powerdevops.influxdb.1.8.3.1.bff.gz'


powerdevops.influxdb.1.8.3.1.bff 100%[========================================================>]  54.06M  1.08MB/s    in 51s


2021-08-20 02:01:02 (1.05 MB/s) - 'powerdevops.influxdb.1.8.3.1.bff.gz' saved [56685356/56685356]
```

gz ファイルを解凍

```
# gunzip powerdevops.influxdb.1.8.3.1.bff.gz
# ls -l
total 210040
-rw-r--r--    1 root     system    107540480 Oct 15 2020  powerdevops.influxdb.1.8.3.1.bff
#
```

influxDB 1.8.3.1 の インストールを実行

```
# installp -acgXYd powerdevops.influxdb.1.8.3.1.bff all
+-----------------------------------------------------------------------------+
                    Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...


SUCCESSES
---------
  Filesets listed in this section passed pre-installation verification
  and will be installed.


  Selected Filesets
  -----------------
  powerdevops.influxdb.rte 1.8.3.1            # Distributed time-series data...


  << End of Success Section >>


+-----------------------------------------------------------------------------+
                   BUILDDATE Verification ...
+-----------------------------------------------------------------------------+
Verifying build dates...done
FILESET STATISTICS
------------------
    1  Selected to be installed, of which:
        1  Passed pre-installation verification
  ----
    1  Total to be installed


+-----------------------------------------------------------------------------+
                         Installing Software...
+-----------------------------------------------------------------------------+


installp: APPLYING software for:
        powerdevops.influxdb.rte 1.8.3.1


Finished processing all filesets.  (Total time:  6 secs).


+-----------------------------------------------------------------------------+
                                Summaries:
+-----------------------------------------------------------------------------+


Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
powerdevops.influxdb.rte    1.8.3.1         USR         APPLY       SUCCESS
powerdevops.influxdb.rte    1.8.3.1         ROOT        APPLY       SUCCESS
#
```

influxDB 起動

```
# /etc/rc.d/init.d/influxdb start
Starting influxdb...
influxdb process was started [ OK ]
```

起動確認

```
# ps -ef | grep influx | grep -v grep
influxdb 13697510        1   0 02:02:10  pts/0  0:00 /usr/bin/influxd -pidfile /var/run/influxdb/influxd.pid -config /etc/influxdb/influxdb.conf
#
```

influxDB に "njmon"データベースを作成

```
# influx <<EOF
> create database njmon
> EOF
#
# influx <<EOF
> show databases
> EOF
name: databases
name
----
_internal
njmon
#
```

----

##### Grafana 導入

Grafana をダウンロード

```
# wget https://dl.power-devops.com/powerdevops.grafana.7.3.1.1.bff.gz
--2021-08-20 02:03:21--  https://dl.power-devops.com/powerdevops.grafana.7.3.1.1.bff.gz
Resolving dl.power-devops.com... 159.69.86.175
Connecting to dl.power-devops.com|159.69.86.175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 69270506 (66M) [application/x-gzip]
Saving to: 'powerdevops.grafana.7.3.1.1.bff.gz'


powerdevops.grafana.7.3.1.1.bff. 100%[========================================================>]  66.06M  1.08MB/s    in 62s


2021-08-20 02:04:24 (1.06 MB/s) - 'powerdevops.grafana.7.3.1.1.bff.gz' saved [69270506/69270506]
```

gzファイルを解凍

```
# gunzip powerdevops.grafana.7.3.1.1.bff.gz
```

Grafana 7.3.1.1 のインストールを実行

```
# installp -acgXYd powerdevops.grafana.7.3.1.1.bff all
+-----------------------------------------------------------------------------+
                    Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...


SUCCESSES
---------
  Filesets listed in this section passed pre-installation verification
  and will be installed.


  Selected Filesets
  -----------------
  powerdevops.grafana.rte 7.3.1.1             # feature rich metrics dashboa...


  << End of Success Section >>


+-----------------------------------------------------------------------------+
                   BUILDDATE Verification ...
+-----------------------------------------------------------------------------+
Verifying build dates...done
FILESET STATISTICS
------------------
    1  Selected to be installed, of which:
        1  Passed pre-installation verification
  ----
    1  Total to be installed


+-----------------------------------------------------------------------------+
                         Installing Software...
+-----------------------------------------------------------------------------+


installp: APPLYING software for:
        powerdevops.grafana.rte 7.3.1.1


Finished processing all filesets.  (Total time:  14 secs).


+-----------------------------------------------------------------------------+
                                Summaries:
+-----------------------------------------------------------------------------+


Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
powerdevops.grafana.rte     7.3.1.1         USR         APPLY       SUCCESS
powerdevops.grafana.rte     7.3.1.1         ROOT        APPLY       SUCCESS
#
```

Grafana 起動

```
# /etc/rc.d/init.d/grafana start
Starting grafana:
# Sending output to nohup.out
```

Grafana 起動確認

```
# ps -ef | grep graf
 grafana 10682768        1   0 02:05:42  pts/0  0:00 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/log/grafana/grafana-server.pid cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
    root 18153810 12976542   0 02:05:53  pts/0  0:00 grep graf
#
```

----

##### Njmon 導入


Njmon をダウンロード

```
# wget http://sourceforge.net/projects/nmon/files/njmon_aix_binaries_v69.zip
--2021-08-20 02:06:10--  http://sourceforge.net/projects/nmon/files/njmon_aix_binaries_v69.zip
Resolving sourceforge.net... 204.68.111.105
Connecting to sourceforge.net|204.68.111.105|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://sourceforge.net/projects/nmon/files/njmon_aix_binaries_v69.zip [following]
--2021-08-20 02:06:11--  https://sourceforge.net/projects/nmon/files/njmon_aix_binaries_v69.zip
Connecting to sourceforge.net|204.68.111.105|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://sourceforge.net/projects/nmon/files/njmon_aix_binaries_v69.zip/ [following]
--2021-08-20 02:06:11--  https://sourceforge.net/projects/nmon/files/njmon_aix_binaries_v69.zip/
Reusing existing connection to sourceforge.net:443.
HTTP request sent, awaiting response... 302 Found
Location: https://sourceforge.net/projects/nmon/files/njmon_aix_binaries_v69.zip/download [following]
--2021-08-20 02:06:11--  https://sourceforge.net/projects/nmon/files/njmon_aix_binaries_v69.zip/download
Reusing existing connection to sourceforge.net:443.
HTTP request sent, awaiting response... 302 Found
Location: https://downloads.sourceforge.net/project/nmon/njmon_aix_binaries_v69.zip?ts=gAAAAABhH1RkzhqmQ4oc0vC9UdeplBBFC86leX5BMrtkcUxfLA-TTwd5QnUC03CQ3PFgQ1qO1vK3J3RMXW9TnINtV04qXSIT6Q%3D%3D&use_mirror=master&r= [following]
--2021-08-20 02:06:12--  https://downloads.sourceforge.net/project/nmon/njmon_aix_binaries_v69.zip?ts=gAAAAABhH1RkzhqmQ4oc0vC9UdeplBBFC86leX5BMrtkcUxfLA-TTwd5QnUC03CQ3PFgQ1qO1vK3J3RMXW9TnINtV04qXSIT6Q%3D%3D&use_mirror=master&r=
Resolving downloads.sourceforge.net... 204.68.111.105
Connecting to downloads.sourceforge.net|204.68.111.105|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://master.dl.sourceforge.net/project/nmon/njmon_aix_binaries_v69.zip?viasf=1 [following]
--2021-08-20 02:06:12--  https://master.dl.sourceforge.net/project/nmon/njmon_aix_binaries_v69.zip?viasf=1
Resolving master.dl.sourceforge.net... 216.105.38.12
Connecting to master.dl.sourceforge.net|216.105.38.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2829066 (2.7M) [application/octet-stream]
Saving to: 'njmon_aix_binaries_v69.zip'


njmon_aix_binaries_v69.zip       100%[========================================================>]   2.70M  1.73MB/s    in 1.6s


2021-08-20 02:06:15 (1.73 MB/s) - 'njmon_aix_binaries_v69.zip' saved [2829066/2829066]
```

zip ファイルを解凍

```
# unzip njmon_aix_binaries_v69.zip
Archive:  njmon_aix_binaries_v69.zip
  inflating: nimon.1
  inflating: nimon_aix61_v69
  inflating: nimon_aix71_v69
  inflating: nimon_aix722_v69
  inflating: nimon_aix724_v69
  inflating: nimon_vios2_v69
  inflating: nimon_vios3_v69
  inflating: ninstall
  inflating: njmon.1
  inflating: njmon_aix61_v69
  inflating: njmon_aix71_v69
  inflating: njmon_aix722_v69
  inflating: njmon_aix724_v69
  inflating: njmon_vios2_v69
  inflating: njmon_vios3_v69
#
```

導入スクリプトの権限変更(実行権限を追加)

```
# chmod 755 ninstall
#
```

Njmon のインストールを実行

```
# ./ninstall
Operating Systems AIX7200-05 installing into /usr/lbin version=v69
Setting file permissions
-rwxrwxr-x    1 root     system       962288 Aug 20 02:08 /usr/lbin/nimon
-rwxrwxr-x    1 root     system       878530 Aug 20 02:08 /usr/lbin/njmon
Installing manual pages for njmon and nimon in /usr/share/man/man1
#
```

パスを追加 (/usr/lbin/njmon)

```
export PATH=$PATH:/usr/lbin
# echo $PATH
/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin:/opt/freeware/bin:/usr/lbin

# which nimon
/usr/lbin/nimon
# which njmon
/usr/lbin/njmon
#
```

nimon 起動

```
# nimon -s 30 -k -i localhost -p 8086 -x njmon -n
#
# ps -ef | grep nimon | grep -v grep
    root 19136784        1   0 05:08:26      -  0:00 nimon -s 30 -k -i localhost -p 8086 -x njmon -n
#
```

---

(optional) パスの設定
パスを .profile に設定する

```
# vi /.profile


"/.profile" 3 lines, 61 characters
#!/bin/sh
PATH=$PATH:/opt/freeware/bin:/usr/lbin
export PATH
```

OS起動時にinfluxDB, Grafana 起動設定

```
# ln -s /etc/rc.d/init.d/influxdb /etc/rc.d/rc2.d/Sinfluxdb
# ln -s /etc/rc.d/init.d/influxdb /etc/rc.d/rc2.d/Kinfluxdb
# ln -s /etc/rc.d/init.d/grafana /etc/rc.d/rc2.d/Sgrafana
# ln -s /etc/rc.d/init.d/grafana /etc/rc.d/rc2.d/Kgrafana
#
```


---


##### Grafana GUI 確認

* IBM Cloud PowerVS では、Public Network で 3000 ポートが外に出ていないようでしたので、プロキシー・サーバー経由でアクセスできるようにしています。


https://xx.xx.xx.xx:3000  にアクセス

Grafana の画面
![スクリーンショット 2021-08-20 19.15.39.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/29f02244-b46e-0458-f641-e84182ee1c68.png)

デフォルト・ユーザーは admin

![スクリーンショット 2021-08-20 19.19.22.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/6a3ff488-35e3-075e-5c98-5a4224ad2bf8.png)

データ・ソースの構成
![スクリーンショット 2021-08-20 19.19.41.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/3b362f0f-f206-3e91-b38f-9ffddbdf6b1a.png)

Add data souce
![スクリーンショット 2021-08-20 19.19.55.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/31c1fc67-aca9-1823-571d-d07badb8bf9c.png)

influxDB 設定
![スクリーンショット 2021-08-20 19.20.21.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/4013c466-cb15-521c-eda8-60d90918106e.png)

![スクリーンショット 2021-08-20 19.21.16.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/e2067a87-4b6d-1edd-aecf-67b8e5573e7b.png)

Database に 作成したデータベース njmon を設定し 「Save & Test」

![スクリーンショット 2021-08-20 19.21.22.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/8d682587-0324-91c9-8363-e260d0effa6e.png)

テストで「Data source is working」が表示されることを確認

![スクリーンショット 2021-08-20 19.21.35.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/e77e28a2-8834-3759-496c-347fa4611675.png)

ダッシュボードのインポート
![スクリーンショット 2021-08-20 19.25.55.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/663ed044-a172-efb5-5e9b-170c77ff3f04.png)


カタログ 10891 (njmon for AIX Simple Six v64) を入力し「Load」
https://grafana.com/grafana/dashboards/10891
![スクリーンショット 2021-08-20 19.26.27.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/abbd81e2-3d08-7689-d406-66627cde4116.png)

influxDB njmon を指定し、「Import」

![スクリーンショット 2021-08-20 19.26.56.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/8a0187b7-b6d3-913e-56f6-cc673ec10925.png)

hostname「aixgrafana」のデータが表示されるようになりました!

![スクリーンショット 2021-08-23 11.18.55.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/e2bf3681-f136-33db-d32f-37a79872d7e9.png)

シンプルに 6 つのグラフです。

---

・別のダッシュボードをインポート (njmon for AIX v40b (一番ダウンロードされている))
https://grafana.com/grafana/dashboards/10829

![図1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/e354fbfb-1baa-97a2-afbe-b2e82834c27a.png)

このダッシュボードは表示が多いですね。

![スクリーンショット 2021-08-23 11.14.52.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/2ce4b02b-8697-3e0d-b4f6-fc6301bd1e0a.png)

Disk, Network も確認できます。

![スクリーンショット 2021-08-23 11.15.06.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/d889d862-4127-a4fe-3e89-92385966b191.png)


---

##### おわりに

・今回は、手順通りのバージョンで実施しましたが、新しいバージョンの導入も試したいと思います。
・Grafana ダッシュボードの Njmon 対応版は色々あるようなので、表示を確認してみたいと思います。



以上です。