【Oracle Database】サービスのフェイルオーバーの挙動

基本的なところをおさらいしよう

全ノードにサービスがある場合はあまり考えなくてもよいけど、シングルサービスなどを構築している場合に、フェイルオーバーどうなるんだっけ？となるので検証して再確認

###環境
DB：12.1
GI：12.1

###１．preferredとavailableを設定しているサービスのフェイルオーバー
こんな感じで優先度が設定されたサービスを作成

```
[12:20:33 oracle@test2 ~]$ srvctl add service -db orcl -service test -preferred orcl1 -available orcl2
[12:20:43 oracle@test2 ~]$ su - grid
Password:
[grid@test2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.asm
               ONLINE  ONLINE       test2              Started,STABLE
               ONLINE  ONLINE       test3              Started,STABLE
ora.net1.network
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.ons
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       test2              STABLE
ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.67.149 192.1
                                                             68.0.71,STABLE
ora.cvu
      1        ONLINE  ONLINE       test2              STABLE
ora.dsdb.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.dsdb.dsdb_sv.svc
      1        OFFLINE OFFLINE                               STABLE
ora.dsdb.singleton.svc
      1        OFFLINE OFFLINE                               STABLE
ora.mgmtdb
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.oc4j
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.db
      1        ONLINE  ONLINE       test2              Open,STABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.orcl.manage20.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.manage80.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.srv_test1.svc
      1        ONLINE  ONLINE       test2              STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.orcl.srv_test2.svc
      1        ONLINE  ONLINE       test2              STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.orcl.srv_test3.svc
      1        OFFLINE OFFLINE                               STABLE
      2        ONLINE  ONLINE       test2              STABLE
ora.orcl.test.svc
      1        OFFLINE OFFLINE                               STABLE
ora.orclmulti.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.testcdb1.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.test2.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.test3.vip
      1        ONLINE  ONLINE       test3              STABLE
xag.gg_1.goldengate
      1        OFFLINE OFFLINE                               STABLE
xag.test7-vip.vip
      1        ONLINE  ONLINE       test2              STABLE
--------------------------------------------------------------------------------


[12:21:42 oracle@test2 ~]$ srvctl start service -db orcl -service test
[12:21:52 oracle@test2 ~]$ srvctl status service -db orcl -service test
Service test is running on instance(s) orcl1

[12:22:14 oracle@test2 ~]$ srvctl config service -db orcl -service test
Service name: test
Server pool:
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
TAF failover retries:
TAF failover delay:
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Session State Consistency:
GSM Flags: 0
Service is enabled
Preferred instances: orcl1
Available instances: orcl2


[grid@test2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.asm
               ONLINE  ONLINE       test2              Started,STABLE
               ONLINE  ONLINE       test3              Started,STABLE
ora.net1.network
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.ons
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       test2              STABLE
ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.67.149 192.1
                                                             68.0.71,STABLE
ora.cvu
      1        ONLINE  ONLINE       test2              STABLE
ora.dsdb.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.dsdb.dsdb_sv.svc
      1        OFFLINE OFFLINE                               STABLE
ora.dsdb.singleton.svc
      1        OFFLINE OFFLINE                               STABLE
ora.mgmtdb
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.oc4j
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.db
      1        ONLINE  ONLINE       test2              Open,STABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.orcl.manage20.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.manage80.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.srv_test1.svc
      1        ONLINE  ONLINE       test2              STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.orcl.srv_test2.svc
      1        ONLINE  ONLINE       test2              STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.orcl.srv_test3.svc
      1        OFFLINE OFFLINE                               STABLE
      2        ONLINE  ONLINE       test2              STABLE
ora.orcl.test.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orclmulti.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.testcdb1.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.test2.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.test3.vip
      1        ONLINE  ONLINE       test3              STABLE
xag.gg_1.goldengate
      1        OFFLINE OFFLINE                               STABLE
xag.test7-vip.vip
      1        ONLINE  ONLINE       test2              STABLE
--------------------------------------------------------------------------------
```

無事優先インスタンスorcl1(node test2)で起動している

ではpmonをkill

```
[12:27:55 oracle@test2 ~]$ ps -ef  |grep pmon
grid      3979     1  0 Jan21 ?        00:11:10 asm_pmon_+ASM1
oracle   20780     1  0 12:26 ?        00:00:00 ora_pmon_orcl1
oracle   21948 16238  0 12:27 pts/2    00:00:00 grep pmon
[12:27:59 oracle@test2 ~]$ kill -9 20780
[12:28:05 oracle@test2 ~]$ ps -ef  |grep pmon
grid      3979     1  0 Jan21 ?        00:11:10 asm_pmon_+ASM1
oracle   22096 16238  0 12:28 pts/2    00:00:00 grep pmon
```

crsctlでstatus確認

```
[grid@test2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.asm
               ONLINE  ONLINE       test2              Started,STABLE
               ONLINE  ONLINE       test3              Started,STABLE
ora.net1.network
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.ons
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       test2              STABLE
ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.67.149 192.1
                                                             68.0.71,STABLE
ora.cvu
      1        ONLINE  ONLINE       test2              STABLE
ora.dsdb.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.dsdb.dsdb_sv.svc
      1        OFFLINE OFFLINE                               STABLE
ora.dsdb.singleton.svc
      1        OFFLINE OFFLINE                               STABLE
ora.mgmtdb
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.oc4j
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.db
      1        ONLINE  OFFLINE      test2              Instance Shutdown,ST
                                                             ARTING
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.orcl.manage20.svc
      1        ONLINE  OFFLINE                               STABLE
ora.orcl.manage80.svc
      1        ONLINE  OFFLINE                               STABLE
ora.orcl.srv_test1.svc
      1        ONLINE  ONLINE       test3              STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.orcl.srv_test2.svc
      1        ONLINE  ONLINE       test3              STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.orcl.srv_test3.svc
      1        OFFLINE OFFLINE                               STABLE
      2        ONLINE  ONLINE       test3              STABLE
ora.orcl.test.svc
      1        ONLINE  ONLINE       test3              STABLE
ora.orclmulti.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.testcdb1.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.test2.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.test3.vip
      1        ONLINE  ONLINE       test3              STABLE
xag.gg_1.goldengate
      1        OFFLINE OFFLINE                               STABLE
xag.test7-vip.vip
      1        ONLINE  ONLINE       test2              STABLE
--------------------------------------------------------------------------------

```

testサービスがフェイルオーバーして、orcl1インスタンスが再起動中なのがわかる

じゃあ再起動したらどうなるか

```
[grid@test2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.asm
               ONLINE  ONLINE       test2              Started,STABLE
               ONLINE  ONLINE       test3              Started,STABLE
ora.net1.network
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.ons
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       test2              STABLE
ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.67.149 192.1
                                                             68.0.71,STABLE
ora.cvu
      1        ONLINE  ONLINE       test2              STABLE
ora.dsdb.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.dsdb.dsdb_sv.svc
      1        OFFLINE OFFLINE                               STABLE
ora.dsdb.singleton.svc
      1        OFFLINE OFFLINE                               STABLE
ora.mgmtdb
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.oc4j
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.db
      1        ONLINE  ONLINE       test2              Open,STABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.orcl.manage20.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.manage80.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.srv_test1.svc
      1        ONLINE  ONLINE       test3              STABLE
      2        ONLINE  ONLINE       test2              STABLE
ora.orcl.srv_test2.svc
      1        ONLINE  ONLINE       test3              STABLE
      2        ONLINE  ONLINE       test2              STABLE
ora.orcl.srv_test3.svc
      1        ONLINE  ONLINE       test2              STABLE
      2        ONLINE  ONLINE       test3              STABLE
ora.orcl.test.svc
      1        ONLINE  ONLINE       test3              STABLE
ora.orclmulti.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.testcdb1.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.test2.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.test3.vip
      1        ONLINE  ONLINE       test3              STABLE
xag.gg_1.goldengate
      1        OFFLINE OFFLINE                               STABLE
xag.test7-vip.vip
      1        ONLINE  ONLINE       test2              STABLE
--------------------------------------------------------------------------------

```

testサービスはフェイルバックしない

これは仕様動作でマニュアルにもしっかり記述がある

>管理者管理データベースのサービス・フェイルオーバーについて
>管理者管理データベースにサービス・フェイルオーバーを構成する場合は、優先インスタンスおよび使用可能なインスタンスを構成する必要があります。
>通常の操作でサービスは、サービスのカーディナリティ(定義されたPREFERREDインスタンスの数)に応じて、優先インスタンスと使用可能インスタンスの任意の組合せにおいて実行できます。サービスの最初の起動時のみ、Oracle ClusterwareはPREFERREDインスタンスでサービスを起動しようとします。インスタンスで障害が発生した場合、サービスを提供していない優先インスタンスおよび使用可能インスタンスの組合せリスト内のいずれかにサービスがフェイルオーバーされます。サービスを提供しない優先インスタンスおよび使用可能インスタンスの組合せリスト内のインスタンスの1つに対して、サービスを手動で再配置することもできます。
>サービスが使用可能インスタンスにフェイルオーバーした場合、そのサービスが優先インスタンスに自動的に戻ることはありません。ただし、FANコールアウトを使用すると、優先インスタンスへのサービスの再配置を自動化できます。
>サービスの優先インスタンスを構成する際にそのサービスに対して使用可能インスタンスを1つ以上指定しない場合、優先インスタンスが失敗してもサービスは他のインスタンスにフェイルオーバーされません。
>Enterprise Managerを使用して、インスタンスを「未使用」として指定することもできます。この設定により、サービスの優先インスタンスが失敗しても、そのサービスはインスタンスで稼働しません。

ではこの状態でorcl2に障害が発生したらフェイルバックするか

```
[12:58:34 oracle@test3 ~]$ ps -ef | grep pmon
oracle    3060     1  0 Apr18 ?        00:00:45 ora_pmon_orclmult2
grid      3246     1  0 Jan21 ?        00:11:02 asm_pmon_+ASM2
oracle    7176     1  0 Mar26 ?        00:04:26 ora_pmon_orcl2
oracle   15991 15947  0 12:58 pts/0    00:00:00 grep pmon
oracle   22571     1  0 Feb18 ?        00:10:08 ora_pmon_dsdb2
[12:58:49 oracle@test3 ~]$ kill -9 7176
[12:59:01 oracle@test3 ~]$ ps -ef | grep pmon
oracle    3060     1  0 Apr18 ?        00:00:45 ora_pmon_orclmult2
grid      3246     1  0 Jan21 ?        00:11:02 asm_pmon_+ASM2
oracle   16070 15947  0 12:59 pts/0    00:00:00 grep pmon
oracle   22571     1  0 Feb18 ?        00:10:08 ora_pmon_dsdb2
```

```
[grid@test2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.asm
               ONLINE  ONLINE       test2              Started,STABLE
               ONLINE  ONLINE       test3              Started,STABLE
ora.net1.network
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.ons
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       test2              STABLE
ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.67.149 192.1
                                                             68.0.71,STABLE
ora.cvu
      1        ONLINE  ONLINE       test2              STABLE
ora.dsdb.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.dsdb.dsdb_sv.svc
      1        OFFLINE OFFLINE                               STABLE
ora.dsdb.singleton.svc
      1        OFFLINE OFFLINE                               STABLE
ora.mgmtdb
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.oc4j
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.db
      1        ONLINE  ONLINE       test2              Open,STABLE
      2        ONLINE  OFFLINE      test3              Instance Shutdown,ST
                                                             ARTING
ora.orcl.manage20.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.manage80.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.srv_test1.svc
      1        ONLINE  OFFLINE                               STABLE
      2        ONLINE  ONLINE       test2              STABLE
ora.orcl.srv_test2.svc
      1        ONLINE  OFFLINE                               STABLE
      2        ONLINE  ONLINE       test2              STABLE
ora.orcl.srv_test3.svc
      1        ONLINE  ONLINE       test2              STABLE
      2        ONLINE  OFFLINE                               STABLE
ora.orcl.test.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orclmulti.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.testcdb1.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.test2.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.test3.vip
      1        ONLINE  ONLINE       test3              STABLE
xag.gg_1.goldengate
      1        OFFLINE OFFLINE                               STABLE
xag.test7-vip.vip
      1        ONLINE  ONLINE       test2              STABLE
--------------------------------------------------------------------------------
```

フェイルバックした
当然といえば当然の挙動だがあえて確認

###２．availableを設定しないサービスについて
こちらも試してみる

```
[13:04:03 oracle@test2 ~]$ srvctl add service -db orcl -service test -preferred orcl1


[13:04:22 oracle@test2 ~]$ srvctl config service -db orcl -service test
Service name: test
Server pool:
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
TAF failover retries:
TAF failover delay:
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Session State Consistency:
GSM Flags: 0
Service is enabled
Preferred instances: orcl1
Available instances:

[grid@test2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.asm
               ONLINE  ONLINE       test2              Started,STABLE
               ONLINE  ONLINE       test3              Started,STABLE
ora.net1.network
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.ons
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       test2              STABLE
ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.67.149 192.1
                                                             68.0.71,STABLE
ora.cvu
      1        ONLINE  ONLINE       test2              STABLE
ora.dsdb.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.dsdb.dsdb_sv.svc
      1        OFFLINE OFFLINE                               STABLE
ora.dsdb.singleton.svc
      1        OFFLINE OFFLINE                               STABLE
ora.mgmtdb
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.oc4j
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.db
      1        ONLINE  ONLINE       test2              Open,STABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.orcl.manage20.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.manage80.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.srv_test1.svc
      1        ONLINE  ONLINE       test3              STABLE
      2        ONLINE  ONLINE       test2              STABLE
ora.orcl.srv_test2.svc
      1        ONLINE  ONLINE       test3              STABLE
      2        ONLINE  ONLINE       test2              STABLE
ora.orcl.srv_test3.svc
      1        ONLINE  ONLINE       test2              STABLE
      2        ONLINE  ONLINE       test3              STABLE
ora.orcl.test.svc
      1        ONLINE  ONLINE       test2              STABLE
ora.orclmulti.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.testcdb1.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.test2.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.test3.vip
      1        ONLINE  ONLINE       test3              STABLE
xag.gg_1.goldengate
      1        OFFLINE OFFLINE                               STABLE
xag.test7-vip.vip
      1        ONLINE  ONLINE       test2              STABLE
--------------------------------------------------------------------------------

pmon kill

```
[13:05:13 oracle@test2 ~]$ srvctl start service -db orcl -service test
[13:06:11 oracle@test2 ~]$ ps -ef | grep pmon
grid      3979     1  0 Jan21 ?        00:11:10 asm_pmon_+ASM1
oracle    5248 16238  0 13:06 pts/2    00:00:00 grep pmon
oracle   22138     1  0 12:28 ?        00:00:00 ora_pmon_orcl1
[13:06:39 oracle@test2 ~]$ kill -9 22138
[13:06:44 oracle@test2 ~]$ ps -ef | grep pmon
grid      3979     1  0 Jan21 ?        00:11:10 asm_pmon_+ASM1
oracle    5357 16238  0 13:06 pts/2    00:00:00 grep pmon
```

```
[grid@test2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.asm
               ONLINE  ONLINE       test2              Started,STABLE
               ONLINE  ONLINE       test3              Started,STABLE
ora.net1.network
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
ora.ons
               ONLINE  ONLINE       test2              STABLE
               ONLINE  ONLINE       test3              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       test2              STABLE
ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.67.149 192.1
                                                             68.0.71,STABLE
ora.cvu
      1        ONLINE  ONLINE       test2              STABLE
ora.dsdb.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.dsdb.dsdb_sv.svc
      1        OFFLINE OFFLINE                               STABLE
ora.dsdb.singleton.svc
      1        OFFLINE OFFLINE                               STABLE
ora.mgmtdb
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.oc4j
      1        ONLINE  ONLINE       test2              STABLE
ora.orcl.db
      1        ONLINE  OFFLINE                               Abnormal Termination
                                                             ,CLEANING
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.orcl.manage20.svc
      1        ONLINE  OFFLINE                               STABLE
ora.orcl.manage80.svc
      1        ONLINE  OFFLINE                               STABLE
ora.orcl.srv_test1.svc
      1        ONLINE  ONLINE       test3              STABLE
      2        ONLINE  OFFLINE                               STABLE
ora.orcl.srv_test2.svc
      1        ONLINE  ONLINE       test3              STABLE
      2        ONLINE  OFFLINE                               STABLE
ora.orcl.srv_test3.svc
      1        ONLINE  OFFLINE                               STABLE
      2        ONLINE  ONLINE       test3              STABLE
ora.orcl.test.svc
      1        ONLINE  OFFLINE                               STABLE
ora.orclmulti.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       test3              Open,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.testcdb1.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.test2.vip
      1        ONLINE  ONLINE       test2              STABLE
ora.test3.vip
      1        ONLINE  ONLINE       test3              STABLE
xag.gg_1.goldengate
      1        OFFLINE OFFLINE                               STABLE
xag.test7-vip.vip
      1        ONLINE  ONLINE       test2              STABLE
--------------------------------------------------------------------------------
```

ですよね、って感じの結果

###おまけ
ちなみにクラスタリソースには下記のように確認できるRESTART_ATTEMPTSがある

```
[grid@test2 ~]$ crsctl stat res ora.orcl.test.svc -p
NAME=ora.orcl.test.svc
TYPE=ora.service.type
ACL=owner:oracle:rwx,pgrp:oinstall:r--,other::r--,group:dba:r-x,group:oper:r-x,user:grid:r-x
ACTIONS=
ACTION_SCRIPT=
ACTION_TIMEOUT=60
ACTIVE_PLACEMENT=0
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
AGENT_PARAMETERS=
AQ_HA_NOTIFICATION=0
AUTO_START=restore
CARDINALITY=1
CHECK_INTERVAL=600
CHECK_TIMEOUT=30
CLB_GOAL=LONG
CLEAN_TIMEOUT=60
COMMIT_OUTCOME=0
DEGREE=1
DELETE_TIMEOUT=60
DESCRIPTION=Oracle Service resource
DTP=0
EDITION=
ENABLED=1
FAILOVER_DELAY=0
FAILOVER_METHOD=
FAILOVER_RETRIES=
FAILOVER_TYPE=
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
GEN_SERVICE_NAME=test
GLOBAL=false
GSM_FLAGS=0
HOSTING_MEMBERS=
INSTANCE_FAILOVER=1
INTERMEDIATE_TIMEOUT=0
LOAD=1
LOGGING_LEVEL=1
MANAGEMENT_POLICY=AUTOMATIC
MAX_LAG_TIME=ANY
MODIFY_TIMEOUT=60
NLS_LANG=
OFFLINE_CHECK_INTERVAL=0
PLACEMENT=restricted
PLUGGABLE_DATABASE=
RELOCATE_BY_DEPENDENCY=1
REPLAY_INITIATION_TIME=300
RESTART_ATTEMPTS=0
RETENTION=86400
RLB_GOAL=NONE
ROLE=PRIMARY
SCRIPT_TIMEOUT=60
SERVER_CATEGORY=
SERVER_POOLS=ora.orcl_test
SERVICE_NAME=test
SERVICE_NAME_PQ=
SERVICE_TYPE=MAIN
SESSION_NOREPLAY=false
SESSION_STATE_CONSISTENCY=
SQL_TRANSLATION_PROFILE=
START_CONCURRENCY=0
START_DEPENDENCIES=hard(ora.orcl.db,type:ora.cluster_vip_net1.type) weak(type:ora.listener.type) pullup(type:ora.cluster_vip_net1.type) pullup:always(ora.orcl.db)
START_TIMEOUT=600
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(intermediate:ora.orcl.db,type:ora.cluster_vip_net1.type)
STOP_TIMEOUT=600
TAF_FAILOVER_DELAY=
TAF_POLICY=NONE
TYPE_VERSION=3.2
UPTIME_THRESHOLD=1h
USER_WORKLOAD=yes
USE_STICKINESS=0
USR_ORA_DISCONNECT=false
USR_ORA_ENV=
USR_ORA_FLAGS=
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_STOP_MODE=
```

これはあくまで再起動回数でFAILOVER試行回数ではないのでご注意を