[Oracle Database] CRS-5017 CRS-2674 and CRS-5008 発生したので対処してみてみた

今日は Oracle Real Application Clusters (RAC) の VIP、Scan Listnerが起動できず、クライアント接続ができない事象に遭遇したので対処してみてみます。

# ■ Clusterware ステータス 確認
Oracle Grid Infrastructure(GI) の Clusterwareリソースのステータスおよび構成情報を取得し確認すると VIP が起動せず、それに依存する SCAN Listner なども OFFLINE の状態です。

```
[grid@db-tokyo1 ~]$ crsctl stat res -t
	--------------------------------------------------------------------------------
	Name           Target  State        Server                   State details
	--------------------------------------------------------------------------------
	Local Resources
	--------------------------------------------------------------------------------
	ora.DATA.COMMONSTORE.advm
				ONLINE  ONLINE       db-tokyo1                STABLE
				ONLINE  ONLINE       db-tokyo2                STABLE
	ora.LISTENER.lsnr
				ONLINE  OFFLINE      db-tokyo1                STABLE
				ONLINE  OFFLINE      db-tokyo2                STABLE
	ora.chad
				ONLINE  ONLINE       db-tokyo1                STABLE
				OFFLINE OFFLINE      db-tokyo2                STABLE
	ora.data.commonstore.acfs
				ONLINE  ONLINE       db-tokyo1                mounted on /opt/orac
																le/dcs/commonstore,S
																TABLE
				ONLINE  ONLINE       db-tokyo2                mounted on /opt/orac
																le/dcs/commonstore,S
																TABLE
	ora.net1.network
				ONLINE  OFFLINE      db-tokyo1                STABLE
				ONLINE  OFFLINE      db-tokyo2                STABLE
	ora.ons
				ONLINE  OFFLINE      db-tokyo1                STABLE
				ONLINE  OFFLINE      db-tokyo2                STABLE
	ora.proxy_advm
				ONLINE  ONLINE       db-tokyo1                STABLE
				ONLINE  ONLINE       db-tokyo2                STABLE
	--------------------------------------------------------------------------------
	Cluster Resources
	--------------------------------------------------------------------------------
	ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
		1        ONLINE  ONLINE       db-tokyo1                STABLE
		2        ONLINE  ONLINE       db-tokyo2                STABLE
	ora.DATA.dg(ora.asmgroup)
		1        ONLINE  ONLINE       db-tokyo1                STABLE
		2        ONLINE  ONLINE       db-tokyo2                STABLE
	ora.LISTENER_SCAN1.lsnr
		1        ONLINE  OFFLINE                               STABLE
	ora.LISTENER_SCAN2.lsnr
		1        ONLINE  OFFLINE                               STABLE
	ora.LISTENER_SCAN3.lsnr
		1        ONLINE  OFFLINE                               STABLE
	ora.RECO.dg(ora.asmgroup)
		1        ONLINE  ONLINE       db-tokyo1                STABLE
		2        ONLINE  ONLINE       db-tokyo2                STABLE
	ora.asm(ora.asmgroup)
		1        ONLINE  ONLINE       db-tokyo1                Started,STABLE
		2        ONLINE  ONLINE       db-tokyo2                Started,STABLE
	ora.asmnet1.asmnetwork(ora.asmgroup)
		1        ONLINE  ONLINE       db-tokyo1                STABLE
		2        ONLINE  ONLINE       db-tokyo2                STABLE
	ora.cdb_tokyo.db
		1        OFFLINE OFFLINE                               Instance Shutdown,ST
																ABLE
		2        OFFLINE OFFLINE                               Instance Shutdown,ST
																ABLE
	ora.cvu
		1        ONLINE  OFFLINE                               STABLE
	ora.db-tokyo1.vip
		1        ONLINE  OFFLINE                               STABLE
	ora.db-tokyo2.vip
		1        ONLINE  OFFLINE                               STABLE
	ora.qosmserver
		1        ONLINE  OFFLINE                               STABLE
	ora.scan1.vip
		1        ONLINE  OFFLINE                               STABLE
	ora.scan2.vip
		1        ONLINE  OFFLINE                               STABLE
	ora.scan3.vip
		1        ONLINE  OFFLINE                               STABLE
	--------------------------------------------------------------------------------
```

# ■ VIPリソース起動
srvctl start vip で起動してみてみます。
CRS-5008: Invalid attribute value: ens3 for the network interface というエラーが出ています。

## ● Node1

```
[grid@db-tokyo1 ~]$ srvctl start vip -vip db-tokyo1
	PRCR-1079 : Failed to start resource ora.db-tokyo1.vip
	CRS-5017: The resource action "ora.net1.network start" encountered the following error:
	CRS-5008: Invalid attribute value: ens3 for the network interface
	. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/db-tokyo1/crs/trace/crsd_orarootagent_root.trc".

	CRS-2674: Start of 'ora.net1.network' on 'db-tokyo1' failed
	CRS-5017: The resource action "ora.net1.network start" encountered the following error:
	CRS-5008: Invalid attribute value: ens3 for the network interface
	. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/db-tokyo2/crs/trace/crsd_orarootagent_root.trc".

	CRS-2674: Start of 'ora.net1.network' on 'db-tokyo2' failed
	CRS-2632: There are no more servers to try to place resource 'ora.db-tokyo1.vip' on that would satisfy its placement policy
```

## ● Node2
```
[grid@db-tokyo2 ~]$ srvctl start vip -vip db-tokyo2
	PRCR-1079 : Failed to start resource ora.db-tokyo2.vip
	CRS-5017: The resource action "ora.net1.network start" encountered the following error:
	CRS-5008: Invalid attribute value: ens3 for the network interface
	. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/db-tokyo2/crs/trace/crsd_orarootagent_root.trc".

	CRS-2674: Start of 'ora.net1.network' on 'db-tokyo2' failed
	CRS-2632: There are no more servers to try to place resource 'ora.db-tokyo2.vip' on that would satisfy its placement policy
```


# ■ OS Network Interface確認
ifconfigで確認してみてみます。
VIP起動するサービス用IFの ens3は起動していて ssh接続もできるので、特に問題なさそうです。

## ● Node1
```
[grid@db-tokyo1 ~]$ ifconfig -a
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 10.10.1.10  netmask 255.255.255.224  broadcast 10.10.1.31
        ether 02:00:17:03:12  txqueuelen 1000  (Ethernet)
        RX packets 170230  bytes 1116295467 (1.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 100517  bytes 140217187 (133.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 192.168.16.18  netmask 255.255.255.0  broadcast 192.168.16.255
        ether 02:00:17:00:55  txqueuelen 1000  (Ethernet)
        RX packets 52669  bytes 74312772 (70.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 58455  bytes 145003999 (138.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 11955  bytes 5919795 (5.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11955  bytes 5919795 (5.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
```

## ● Node2
```
[grid@db-tokyo2 ~]$ ifconfig -a
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 10.10.1.4  netmask 255.255.255.224  broadcast 10.10.1.31
        ether 02:00:17:02:48  txqueuelen 1000  (Ethernet)
        RX packets 396946  bytes 1472978705 (1.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 249472  bytes 741036999 (706.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 192.168.16.19  netmask 255.255.255.0  broadcast 192.168.16.255
        ether 02:00:17:02:a4  txqueuelen 1000  (Ethernet)
        RX packets 60078  bytes 149204245 (142.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 54843  bytes 76570483 (73.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 55376  bytes 225465544 (215.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 55376  bytes 225465544 (215.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
```

# ■ ora.net1.networkリソース設定確認
CRS-5008: Invalid attribute value: ens3 for the network interface というエラーをたよりに、GIのネットワーク関連が設定されてる ora.net1.networkリソースのパラメータ内容を確認してみようと思います。
crsctl status resourceで確認してみると ens3インターフェースのNETMASKが ifconfig と異なる値設定されています。

	・ifconfig: netmask 255.255.255.224
	・ora.net1.network: USR_ORA_NETMASK=255.255.255.240

```
[root@db-tokyo1 ~]# su - grid
[grid@db-tokyo1 ~]$ crsctl status resource ora.net1.network -p | grep USR_ORA
	USR_ORA_AUTO=static
	USR_ORA_ENV=
	USR_ORA_IF=ens3
	USR_ORA_NETMASK=255.255.255.240
	USR_ORA_SUBNET=10.10.1.0

[grid@db-tokyo1 ~]$ crsctl status resource ora.net1.network -p
	NAME=ora.net1.network
	TYPE=ora.network.type
	ACL=owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:grid:r-x
	ACTIONS=ipsec_setup,user:"grid"
	ACTION_SCRIPT=
	ACTION_TIMEOUT=60
	ADDRESS_TYPE=IPV4
	AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%
	AUTO_START=restore
	CHECK_INTERVAL=1
	CHECK_TIMEOUT=0
	CLEAN_TIMEOUT=60
	CSS_CRITICAL=no
	DELETE_TIMEOUT=60
	DESCRIPTION=Oracle Network resource
	ENABLED=1
	IGNORE_TARGET_ON_FAILURE=no
	INSTANCE_FAILOVER=1
	INTERMEDIATE_TIMEOUT=0
	LOAD=1
	LOGGING_LEVEL=1
	MODIFY_TIMEOUT=60
	NLS_LANG=
	NODE_VIPLESS=false
	OFFLINE_CHECK_INTERVAL=60
	PING_TARGET=
	RESOURCE_GROUP=
	RESTART_ATTEMPTS=5
	RESTART_DELAY=0
	SCRIPT_TIMEOUT=60
	SERVER_CATEGORY=ora.hub.category
	START_CONCURRENCY=0
	START_DEPENDENCIES=
	START_TIMEOUT=0
	STOP_CONCURRENCY=0
	STOP_DEPENDENCIES=
	STOP_TIMEOUT=0
	TARGET_DEFAULT=atleastone
	TYPE_VERSION=3.3
	UPTIME_THRESHOLD=1d
	USER_WORKLOAD=no
	USR_ORA_AUTO=static
	USR_ORA_ENV=
	USR_ORA_IF=ens3
	USR_ORA_NETMASK=255.255.255.240
	USR_ORA_SUBNET=10.10.1.0
	WORKLOAD_CPU=0
	WORKLOAD_CPU_CAP=0
	WORKLOAD_MEMORY_MAX=0
	WORKLOAD_MEMORY_TARGET=0
```

# ■ ora.net1.network修正
GIのNetowrkリソース ora.net1.network を ens3インターフェース の NETMASK 255.255.255.224 と同じ設定に修正してみてみます。
srvctl modify network は rootユーザーで実行します。

```:構文
# srvctl modify network -k <network_number> -S <subnet>/<netmask>/<interface1>
```

```
[root@db-tokyo1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl modify network -k 1 -S 10.10.1.0/255.255.255.224/ens3
[root@db-tokyo1 ~]# su - grid
[grid@db-tokyo1 ~]$ crsctl status resource ora.net1.network -p | grep USR_ORA
	USR_ORA_AUTO=static
	USR_ORA_ENV=
	USR_ORA_IF=ens3
	USR_ORA_NETMASK=255.255.255.224
	USR_ORA_SUBNET=10.10.1.0
```


# ■ OS再起動
全Node再起動

```:Node1
[root@db-tokyo1 ~]# shutdown -r now
```

```:Node2
[root@db-tokyo2 ~]# shutdown -r now
```

# ■ 正常稼働確認
正常に起動しました ＼(^o^)／

```
[grid@db-tokyo1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.COMMONSTORE.advm
               ONLINE  ONLINE       db-tokyo1                STABLE
               ONLINE  ONLINE       db-tokyo2                STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       db-tokyo1                STABLE
               ONLINE  ONLINE       db-tokyo2                STABLE
ora.chad
               ONLINE  ONLINE       db-tokyo1                STABLE
               ONLINE  ONLINE       db-tokyo2                STABLE
ora.data.commonstore.acfs
               ONLINE  ONLINE       db-tokyo1                mounted on /opt/orac
                                                             le/dcs/commonstore,S
                                                             TABLE
               ONLINE  ONLINE       db-tokyo2                mounted on /opt/orac
                                                             le/dcs/commonstore,S
                                                             TABLE
ora.net1.network
               ONLINE  ONLINE       db-tokyo1                STABLE
               ONLINE  ONLINE       db-tokyo2                STABLE
ora.ons
               ONLINE  ONLINE       db-tokyo1                STABLE
               ONLINE  ONLINE       db-tokyo2                STABLE
ora.proxy_advm
               ONLINE  ONLINE       db-tokyo1                STABLE
               ONLINE  ONLINE       db-tokyo2                STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       db-tokyo1                STABLE
      2        ONLINE  ONLINE       db-tokyo2                STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       db-tokyo1                STABLE
      2        ONLINE  ONLINE       db-tokyo2                STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       db-tokyo1                STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       db-tokyo2                STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       db-tokyo2                STABLE
ora.RECO.dg(ora.asmgroup)
      1        ONLINE  ONLINE       db-tokyo1                STABLE
      2        ONLINE  ONLINE       db-tokyo2                STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       db-tokyo1                Started,STABLE
      2        ONLINE  ONLINE       db-tokyo2                Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       db-tokyo1                STABLE
      2        ONLINE  ONLINE       db-tokyo2                STABLE
ora.cdb_tokyo.db
      1        ONLINE  ONLINE       db-tokyo1                Open,HOME=/u01/app/o
                                                             racle/product/19.0.0
                                                             .0/dbhome_1,STABLE
      2        ONLINE  ONLINE       db-tokyo2                Open,HOME=/u01/app/o
                                                             racle/product/19.0.0
                                                             .0/dbhome_1,STABLE
ora.cvu
      1        ONLINE  ONLINE       db-tokyo2                STABLE
ora.db-tokyo1.vip
      1        ONLINE  ONLINE       db-tokyo1                STABLE
ora.db-tokyo2.vip
      1        ONLINE  ONLINE       db-tokyo2                STABLE
ora.qosmserver
      1        ONLINE  ONLINE       db-tokyo2                STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       db-tokyo1                STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       db-tokyo2                STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       db-tokyo2                STABLE
--------------------------------------------------------------------------------
```


# ■ 原因
VCN Subnet CIDRサイズを 255.255.255.240(/28)から 255.255.255.224(/27)へ拡張変更したのだそうな、
OSのens3インターフェースのNetowrk設定は更新されたが、GIの Networkリソースは更新しなかったので、GIのVIPが起動できなくなった、ということでした。


# ■ 参考
・キュメント: [Oracle Clusterware制御(CRSCTL)ユーティリティ・リファレンス](https://docs.oracle.com/cd/F19136_01/cwadd/oracle-clusterware-control-crsctl-utility-reference.html#GUID-1A6774CF-FD5F-463E-817F-34E4466443C7)
・My Oracle Support: [Unable To Start ora.net2.network with CRS-2672 CRS-2674 and CRS-5008: Invalid attribute value: en4 for the network interface (Doc ID 1548049.1)](https://support.oracle.com/knowledge/Oracle%20Database%20Products/1548049_1.html)