ZABBIX -構築（H/A・zabbix-agent(active)による監視ノード自動登録）-

# 目的
* AZを跨いだ冗長構成（Active-Standby）
* 浮動ipでの運用でなく、ELBのヘルスチェックに依存させる
    * 最近Updateされた機能により、運用が可能
        * [Register or De-Register EC2 Instances for Your Load Balancer](http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-deregister-register-instances.html)
* オートディスカバリは無効にし、Zabbix-agent（アクティブ）にInternal-ELBを記述し、ノードの自動登録を行う


# イメージ
![cloudcraft - Zabbix_H-A 1.png](https://qiita-image-store.s3.amazonaws.com/0/58788/a906266d-2ec6-ab76-a859-2d45f19d4526.png)

# 環境
* t2-micro(運用していくにはC4以上が好ましい)
* Active/Standby構成（Standbyはサーバは起動しているが、F/Oのトリガーとなるプロセスを停止させている）
* F/Oのトリガーはzabbix-serverの停止
* クラスタ制御にはCorosync、リソース制御はPacemakerを使用
    * [Rundeck -構築（H/A）-](http://qiita.com/digitalpeak/items/836c79c00fc8289f372e)

# 構築
## インストール
```bash
$ sudo rpm -ivh http://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-2.4-1.el6.noarch.rpm
$ sudo yum install zabbix-server-mysql zabbix-web-mysql zabbix-web-japanese zabbix-agent
```
## 設定
### MySQL
文字化けしないように事前にパラメータグループを作り、character_set_clientとcharacter_set_serverの値をutf-8にしておく。
![Screenshot from 2016-01-27 13:25:04.png](https://qiita-image-store.s3.amazonaws.com/0/58788/790f9ff5-dbf1-36ae-4a06-baac0bddc2b3.png)

```bash
$ mysql -h <rds-endpoint> -u <user> -p
mysql> grant all privileges on <dbname>.* to <user>@localhost identified by '<password>';

#初期スキーマとデータをインポート
$ cd /usr/share/doc/zabbix-server-mysql-2.4.7/create
$ mysql -h <rds-endpoint> <db> -u <user> -p < schema.sql
$ mysql -h <rds-endpoint> <db> -u <user> -p < images.sql
$ mysql -h <rds-endpoint> <db> -u <user> -p < data.sql
```

### zabbix-server
```bash:zabbix_server.conf
### Option: ListenPort
#       Listen port for trapper.
#
# Mandatory: no
# Range: 1024-32767
# Default
# ListenPort=10051

### Option: DBHost
#       Database host name.
#       If set to localhost, socket is used for MySQL.
#       If set to empty string, socket is used for PostgreSQL.
#
# Mandatory: no
# Default:
# DBHost=localhost
DBHost=<rds-endpoint>
### Option: DBName
#       Database name.
#       For SQLite3 path to database file must be provided. DBUser and DBPassword are ignored.
#
# Mandatory: yes
# Default:
# DBName=

DBName=<DBName>

### Option: DBSchema
#       Schema name. Used for IBM DB2 and PostgreSQL.
#
# Mandatory: no
# Default:
# DBSchema=

### Option: DBUser
#       Database user. Ignored for SQLite.
#
# Mandatory: no
# Default:
# DBUser=

DBUser=<user>

### Option: DBPassword
#       Database password. Ignored for SQLite.
#       Comment this line if no password is used.
#
# Mandatory: no
# Default:
# DBPassword=
DBPassword=********
### Option: DBSocket
#       Path to MySQL socket.
#
# Mandatory: no
# Default:
# DBSocket=/tmp/mysql.sock

DBSocket=/var/lib/mysql/mysql.sock
```
```bash:zabbix_agent.conf
### Option: Server
#       List of comma delimited IP addresses (or hostnames) of Zabbix servers.
#       Incoming connections will be accepted only from the hosts listed here.
#       If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally.
#
# Mandatory: no
# Default:
# Server=

Server=10.0.1.23,10.0.3.60

### Option: ListenPort
#       Agent will listen on this port for connections from the server.
#
# Mandatory: no
# Range: 1024-32767
# Default:
ListenPort=10050

### Option: ServerActive
#       List of comma delimited IP:port (or hostname:port) pairs of Zabbix servers for active checks.
#       If port is not specified, default port is used.
#       IPv6 addresses must be enclosed in square brackets if port for that host is specified.
#       If port is not specified, square brackets for IPv6 addresses are optional.
#       If this parameter is not specified, active checks are disabled.
#       Example: ServerActive=127.0.0.1:20051,zabbix.domain,[::1]:30051,::1,[12fc::1]
#
# Mandatory: no
# Default:
# ServerActive=

ServerActive=<internal ELB Aレコード>or<internal ELB CNAME>

### Option: Hostname
#       Unique, case sensitive hostname.
#       Required for active checks and must match hostname as configured on the server.
#       Value is acquired from HostnameItem if undefined.
#
# Mandatory: no
# Default:
# Hostname=

#Hostname=Zabbix server

### Option: HostnameItem
#       Item used for generating Hostname if it is undefined. Ignored if Hostname is defined.
#       Does not support UserParameters or aliases.
#
# Mandatory: no
# Default:
HostnameItem=system.hostname
```

### Apache/PHP
```bash:httpd.conf
$ sudo vim /etc/httpd/conf/httpd.conf
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/www/zabbix"
```
```bash
$ cd /var/www
$ sudo ln -s /usr/share/zabbix zabbix

$ sudo vim /etc/php.ini
post_max_size = 16M
max_execution_time = 300
max_input_time = 300
date.timezone = Asia/Tokyo
always_populate_raw_post_data = -1
```

### フロントエンド

```bash
$ sudo service httpd start&&sudo chkconfig httpd on
$ sudo service zabbix-server start&&sudo chkconfig zabbix-server on
$ sudo service zabbix-agent start&&sudo chkconfig zabbix-agent on
```
![Screenshot from 2015-12-28 17:27:16.png](https://qiita-image-store.s3.amazonaws.com/0/58788/3df788f7-7af0-0a16-34a2-2d454d042846.png)
![Screenshot from 2015-12-28 17:27:53.png](https://qiita-image-store.s3.amazonaws.com/0/58788/96dd276a-eca1-4009-61b8-7b6bfd6ec479.png)
![Screenshot from 2015-12-28 17:29:44.png](https://qiita-image-store.s3.amazonaws.com/0/58788/873f62eb-0bfa-8ecd-b4a8-e4408f479b44.png)
![Screenshot from 2015-12-28 17:30:40.png](https://qiita-image-store.s3.amazonaws.com/0/58788/c31f4eb3-6e0d-8264-474a-af7728f100b0.png)
![Screenshot from 2015-12-28 17:31:13.png](https://qiita-image-store.s3.amazonaws.com/0/58788/411a7d8c-51e1-433c-c891-1871276a2178.png)

#### 自動登録アクション
* Actionにのみ表示させるホスト情報を記述
    * Conditions記述不要。Operationsは「ホスト追加」「テンプレートとリンク」

![Screenshot from 2015-12-28 23:48:34.png](https://qiita-image-store.s3.amazonaws.com/0/58788/b8800320-aeaf-56ca-9884-d5b778eee3fb.png)
![Screenshot from 2015-12-29 13:30:06.png](https://qiita-image-store.s3.amazonaws.com/0/58788/14320531-ce6b-00aa-1342-272f9f10ca0d.png)
![Screenshot from 2015-12-29 13:30:49.png](https://qiita-image-store.s3.amazonaws.com/0/58788/6cda057d-8d24-4a95-6627-2ad2e7fbdbf4.png)
![Screenshot from 2015-12-29 13:29:09.png](https://qiita-image-store.s3.amazonaws.com/0/58788/750a74c9-391d-79cb-282a-64752e903791.png)


### zabbix-agent
* 監視対象ノードに設定

#### インストール
```bash
$ sudo rpm -ivh http://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-2.4-1.el6.noarch.rpm
$ sudo yum -y install zabbix-agent
```

```bash:zabbix_agent.conf
### Option: SourceIP
#	Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
# SourceIP=
SourceIP=10.0.3.6

### Option: Server
#       List of comma delimited IP addresses (or hostnames) of Zabbix servers.
#       Incoming connections will be accepted only from the hosts listed here.
#       If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally.
#
# Mandatory: no
# Default:
# Server=

Server=10.0.1.23,10.0.3.60

### Option: ListenPort
#       Agent will listen on this port for connections from the server.
#
# Mandatory: no
# Range: 1024-32767
# Default:
ListenPort=10050

### Option: ServerActive
#       List of comma delimited IP:port (or hostname:port) pairs of Zabbix servers for active checks.
#       If port is not specified, default port is used.
#       IPv6 addresses must be enclosed in square brackets if port for that host is specified.
#       If port is not specified, square brackets for IPv6 addresses are optional.
#       If this parameter is not specified, active checks are disabled.
#       Example: ServerActive=127.0.0.1:20051,zabbix.domain,[::1]:30051,::1,[12fc::1]
#
# Mandatory: no
# Default:
# ServerActive=

ServerActive=<internal ELB Aレコード>or<internal ELB CNAME>

### Option: Hostname
#       Unique, case sensitive hostname.
#       Required for active checks and must match hostname as configured on the server.
#       Value is acquired from HostnameItem if undefined.
#
# Mandatory: no
# Default:
# Hostname=

#Hostname=Zabbix server

### Option: HostnameItem
#       Item used for generating Hostname if it is undefined. Ignored if Hostname is defined.
#       Does not support UserParameters or aliases.
#
# Mandatory: no
# Default:
HostnameItem=system.hostname
```
##### エラー

```bash
$ sudo service zabbix-agent start
Starting Zabbix agent:                                     [  OK  ]
$ sudo tail -f /var/log/zabbix/zabbix_agentd.log
  2709:20151228:150932.183 Starting Zabbix Agent [ip-10-0-3-6]. Zabbix 2.4.7 (revision 56694).
  2709:20151228:150932.183 using configuration file: /etc/zabbix/zabbix_agentd.conf
  2709:20151228:150932.183 agent #0 started [main process]
  2714:20151228:150932.183 agent #5 started [active checks #1]
  2713:20151228:150932.183 agent #4 started [listener #3]
  2712:20151228:150932.184 agent #3 started [listener #2]
  2711:20151228:150932.184 agent #2 started [listener #1]
  2710:20151228:150932.184 agent #1 started [collector]
  2714:20151228:150932.217 no active checks on server [******.elb.amazonaws.com:10051]: host [ip-10-0-3-6] not found
```

* フロントエンドではノードの自動登録が成功しているように見えるが、Zabbix-serverとZabbix-agentとの通信が失敗している。
![Screenshot from 2015-12-29 00:13:00.png](https://qiita-image-store.s3.amazonaws.com/0/58788/bcb147e2-fb23-cab0-b464-315daf7c788f.png)
* Agent interfacesで取得しているIPが不明
![Screenshot from 2015-12-28 22:55:10.png](https://qiita-image-store.s3.amazonaws.com/0/58788/b61a2c2f-f3b4-20e9-e0d0-2c8aaee7ebf6.png)

* Internal-ELBのIPだった

```bash
$ dig internal-******.elb.amazonaws.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.42.amzn1 <<>> internal-******.elb.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38512
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;internal-******.elb.amazonaws.com. IN A

;; ANSWER SECTION:
internal-******.elb.amazonaws.com. 60 IN A 10.0.2.62
internal-******.elb.amazonaws.com. 60 IN A 10.0.2.58

;; Query time: 5 msec
;; SERVER: 10.0.0.2#53(10.0.0.2)
;; WHEN: Mon Dec 28 15:13:25 2015
;; MSG SIZE  rcvd: 116
```
* 対策としては、zabbix-agent.confのListenIPに自身のIPを記述することで、Agent interfacesで取得できるようになる。

```bash
### Option: ListenIP
#       List of comma delimited IP addresses that the agent should listen on.
#       First IP address is sent to Zabbix server if connecting to it to retrieve list of active checks.
#
# Mandatory: no
# Default:
ListenIP=10.0.3.6
```