Newsgroup: it.comp.os.linux.sys
Titolo: [ PER ESPERTI ] Analisi Split Brain Cluster Pacemaker Corosync DRBD

Ciao,
ho un clusterino a due nodi configurato
in modalit� Attivo/Passivo con Debian Squeeze + Corosync + DRBD.
DRBD sincronizza la partizione /opt.
 
Dopo tre messetti di attivit�, mi segnala uno SplitBrain.
 
La mia analisi � che il nodo "slave" (nodo1) ha prodotto errori al FS,
e per questo Corosync ha escluso il "nodo1" dal cluster...
 
Ma ho un dubbio:
Dal dmesg l'esclusione dal cluster avviene prima della rilevazione
"EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode".
 
Quindi non vorrei fosse un problema di timeout troppo brevi .. ma non mi 
sembra, ed il server � molto scarico.
 
di seguito la papaprdella dei log e delle config per chi avesse pazienza :-)
 
---------------------------------------------------
CRM_MON
============
Last updated: Wed Dec  5 15:08:36 2012
Last change: Tue Oct  9 18:18:13 2012 via cibadmin on nodo1
Stack: openais
Current DC: nodo1 - partition with quorum
Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14
2 Nodes configured, 2 expected votes
5 Resources configured.
============
 
Online: [ nodo1 nodo2 ]
 
ClusterIP       (ocf::heartbeat:IPaddr2):       Started nodo2
res_Filesystem_1        (ocf::heartbeat:Filesystem):    Started nodo2
  Master/Slave Set: ms_drbd_1 [res_drbd_1]
      Masters: [ nodo2 ]
      Slaves: [ nodo1 ]
res_zimbra_ZimbraSrv    (lsb:ZimbraCluster):    Started nodo2
 
Failed actions:
     res_zimbra_ZimbraSrv_monitor_15000 (node=nodo2, call=22, rc=7, 
status=complete): not running
 
--------------------------------------------------------
 
Situazione DRBD nodo On-Line
 
[root@nodo2 ~]# cat /proc/drbd
version: 8.3.13 (api:88/proto:86-96)
GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by 
dag@Build64R6, 2012-05-07 11:52:13
  0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r-----
     ns:0 nr:0 dw:259622876 dr:39862761 al:54214 bm:7255 lo:0 pe:0 ua:0 
ap:0 ep:1 wo:b oos:32065524
 
 
Situazione DRBD nodo in Fail
--------------------------------------------------------
[root@nodo1 ~]# cat /proc/drbd
version: 8.3.13 (api:88/proto:86-96)
GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by 
dag@Build64R6, 2012-05-07 11:52:13
  0: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown   r-----
     ns:0 nr:0 dw:158244 dr:404269 al:71 bm:11 lo:0 pe:0 ua:0 ap:0 ep:1 
wo:b oos:154932
--------------------------------------------------------
 
Corosync Log
--------------------------------------------------------
Nov 08 09:41:02 [7974] nodo1        cib:     info: cib_stats: 
Processed 31 operations (0.00us average, 0% utilization) in the last
  10min
Nov 08 09:47:37 [7979] nodo1       crmd:     info: process_graph_event: 
         Detected action res_zimbra_ZimbraSrv_monitor_15000 f
rom a different transition: 5 vs. 2856
Nov 08 09:47:37 [7979] nodo1       crmd:     info: 
abort_transition_graph:      process_graph_event:476 - Triggered 
transition abort
  (complete=1, tag=lrm_rsc_op, id=res_zimbra_ZimbraSrv_last_failure_0, 
magic=0:7;43:5:0:9ab720aa-74bf-4289-8022-4081ace3a7e2, cib=0.2
5.8) : Old event
Nov 08 09:47:37 [7979] nodo1       crmd:  warning: update_failcount: 
Updating failcount for res_zimbra_ZimbraSrv on nodo2 after f
ailed monitor: rc=7 (update=value++, time=1352364457)
Nov 08 09:47:37 [7979] nodo1       crmd:   notice: do_state_transition: 
         State transition S_IDLE -> S_POLICY_ENGINE [ input=I
_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Nov 08 09:47:37 [7978] nodo1    pengine:   notice: unpack_config: 
On loss of CCM Quorum: Ignore
Nov 08 09:47:37 [7978] nodo1    pengine:  warning: unpack_rsc_op: 
Processing failed op res_zimbra_ZimbraSrv_last_failure_0 on nodo2: not 
running (7)
Nov 08 09:47:37 [7979] nodo1       crmd:     info: 
abort_transition_graph:      te_update_diff:176 - Triggered transition 
abort (complete=1, tag=nvpair, 
id=status-nodo2-fail-count-res_zimbra_ZimbraSrv, 
name=fail-count-res_zimbra_ZimbraSrv, value=1, magic=NA, cib=0.25.9) : 
Transient attribute: update
Nov 08 09:47:37 [7978] nodo1    pengine:   notice: LogActions:  Recover 
res_zimbra_ZimbraSrv    (Started nodo2)
Nov 08 09:47:37 [7979] nodo1       crmd:     info: handle_response: 
pe_calc calculation pe_calc-dc-1352364457-2930 is obsolete
----------------------------------------------------------
 
DMESG Nodo Master attualmente online
----------------------------------------------------------
block drbd0: Starting worker thread (from cqueue [2463])
block drbd0: disk( Diskless -> Attaching )
block drbd0: Found 4 transactions (192 active extents) in activity log.
block drbd0: Method to ensure write ordering: barrier
block drbd0: drbd_bm_resize called with capacity == 1843143672
block drbd0: resync bitmap: bits=230392959 words=3599890 pages=7032
block drbd0: size = 879 GB (921571836 KB)
block drbd0: bitmap READ of 7032 pages took 265 jiffies
block drbd0: recounting of set bits took additional 25 jiffies
block drbd0: 2660 KB (665 bits) marked out-of-sync by on disk bit-map.
block drbd0: disk( Attaching -> UpToDate )
block drbd0: attached to UUIDs 
449500BE8BAE122F:B755F85F6DE0399F:3FE6BB5136D1E808:3FE5BB5136D1E808
block drbd0: conn( StandAlone -> Unconnected )
block drbd0: Starting receiver thread (from drbd0_worker [25097])
block drbd0: receiver (re)started
block drbd0: conn( Unconnected -> WFConnection )
block drbd0: Handshake successful: Agreed network protocol version 96
block drbd0: Peer authenticated using 20 bytes of 'sha1' HMAC
block drbd0: conn( WFConnection -> WFReportParams )
block drbd0: Starting asender thread (from drbd0_receiver [25112])
block drbd0: data-integrity-alg: <not-used>
block drbd0: drbd_sync_handshake:
block drbd0: self 
449500BE8BAE122E:B755F85F6DE0399F:3FE6BB5136D1E808:3FE5BB5136D1E808 
bits:665 flags:0
block drbd0: peer 
224EB8A06A2F6965:B755F85F6DE0399E:3FE6BB5136D1E808:3FE5BB5136D1E808 
bits:0 flags:0
block drbd0: uuid_compare()=100 by rule 90
block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0
block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 
exit code 0 (0x0)
block drbd0: Split-Brain detected but unresolved, dropping connection!
block drbd0: helper command: /sbin/drbdadm split-brain minor-0
block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 
0 (0x0)
block drbd0: conn( WFReportParams -> Disconnecting )
block drbd0: error receiving ReportState, l: 4!
block drbd0: asender terminated
block drbd0: Terminating asender thread
block drbd0: Connection closed
block drbd0: conn( Disconnecting -> StandAlone )
block drbd0: receiver terminated
block drbd0: Terminating receiver thread
block drbd0: role( Secondary -> Primary )
EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts:
----------------------------------------------------------
 
DMESG su nodo espulso dal cluster:
----------------------------------------------------------
block drbd0: Starting worker thread (from cqueue [1317])
block drbd0: disk( Diskless -> Attaching )
block drbd0: Found 4 transactions (115 active extents) in activity log.
block drbd0: Method to ensure write ordering: barrier
block drbd0: drbd_bm_resize called with capacity == 1843143672
block drbd0: resync bitmap: bits=230392959 words=3599890 pages=7032
block drbd0: size = 879 GB (921571836 KB)
block drbd0: bitmap READ of 7032 pages took 331 jiffies
block drbd0: recounting of set bits took additional 25 jiffies
block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
block drbd0: disk( Attaching -> UpToDate )
block drbd0: attached to UUIDs 
B755F85F6DE0399E:0000000000000000:3FE6BB5136D1E808:3FE5BB5136D1E808
block drbd0: conn( StandAlone -> Unconnected )
block drbd0: Starting receiver thread (from drbd0_worker [8148])
block drbd0: receiver (re)started
block drbd0: conn( Unconnected -> WFConnection )
block drbd0: role( Secondary -> Primary )
block drbd0: new current UUID 
224EB8A06A2F6965:B755F85F6DE0399E:3FE6BB5136D1E808:3FE5BB5136D1E808
block drbd0: Handshake successful: Agreed network protocol version 96
block drbd0: Peer authenticated using 20 bytes of 'sha1' HMAC
block drbd0: conn( WFConnection -> WFReportParams )
block drbd0: Starting asender thread (from drbd0_receiver [8163])
block drbd0: data-integrity-alg: <not-used>
block drbd0: drbd_sync_handshake:
block drbd0: self 
224EB8A06A2F6965:B755F85F6DE0399E:3FE6BB5136D1E808:3FE5BB5136D1E808 
bits:0 flags:0
block drbd0: peer 
449500BE8BAE122E:B755F85F6DE0399F:3FE6BB5136D1E808:3FE5BB5136D1E808 
bits:665 flags:0
block drbd0: uuid_compare()=100 by rule 90
block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0
block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 
exit code 0 (0x0)
block drbd0: Split-Brain detected but unresolved, dropping connection!
block drbd0: helper command: /sbin/drbdadm split-brain minor-0
block drbd0: meta connection shut down by peer.
block drbd0: conn( WFReportParams -> NetworkFailure )
block drbd0: asender terminated
block drbd0: Terminating asender thread
block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 
0 (0x0)
block drbd0: conn( NetworkFailure -> Disconnecting )
block drbd0: error receiving ReportState, l: 4!
block drbd0: Connection closed
block drbd0: conn( Disconnecting -> StandAlone )
block drbd0: receiver terminated
block drbd0: Terminating receiver thread
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 35784082
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 35783762
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 35783942
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 35783201
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 35782774
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 35783029
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 35784039
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 35782853
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 35783974
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 35782852
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 34866045
EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode 34866072
......
.....
EXT4-fs (drbd0): 33 orphan inodes deleted
EXT4-fs (drbd0): recovery complete
EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts:
block drbd0: role( Primary -> Secondary )
block drbd0: bitmap WRITE of 32 pages took 1 jiffies
block drbd0: 151 MB (38733 bits) marked out-of-sync by on disk bit-map.
----------------------------------------------------------
 
Config DRBD:
----------------------------------------------------------
/etc/drbd.d/global_common.conf
## generated by drbd-gui
 
global {
                usage-count     yes;
}
 
common {
        startup {
                degr-wfc-timeout        0;
        }
 
        net {
                cram-hmac-alg   sha1;
                shared-secret   FaXXXXXXXXXXXXXXXXXXm9;
        }
 
        disk {
                on-io-error     detach;
        }
 
        syncer {
                rate    30M;
        }
 
}
----------------------------------------------------------
cat /etc/drbd.d/r0.res
resource r0 {
        protocol        C;
 
        on nodo1 {
                device          /dev/drbd0;
                disk            /dev/vgZimbra/lvol0;
                flexible-meta-disk      internal;
                address         
192.168.50.110:7788
;
        }
        on nodo2 {
                device          /dev/drbd0;
                disk            /dev/vgZimbra/lvol0;
                flexible-meta-disk      internal;
                address         
192.168.50.111:7788
;
        }
}
----------------------------------------------------------
 
Configurazione Pacemaker crm
----------------------------------------------------------
crm(live)configure# show
node nodo1 \
         attributes standby="off"
node nodo2 \
         attributes standby="off"
primitive ClusterIP ocf:heartbeat:IPaddr2 \
         params ip="10.8.10.12" nic="eth0" cidr_netmask="24" \
         operations $id="ClusterIP-operations" \
         op monitor interval="10" timeout="20" start-delay="0" \
         meta target-role="started"
primitive res_Filesystem_1 ocf:heartbeat:Filesystem \
         params device="/dev/drbd0" directory="/opt" fstype="ext4" \
         operations $id="res_Filesystem_1-operations" \
         op start interval="0" timeout="60" \
         op stop interval="0" timeout="60" \
         op monitor interval="20" timeout="40" start-delay="0" \
         op notify interval="0" timeout="60" \
         meta target-role="started"
primitive res_drbd_1 ocf:linbit:drbd \
         params drbd_resource="r0" \
         operations $id="res_drbd_1-operations" \
         op start interval="0" timeout="240" \
         op promote interval="0" timeout="90" \
         op demote interval="0" timeout="90" \
         op stop interval="0" timeout="100" \
         op monitor interval="20" role="Slave" timeout="20" depth="0" \
         op monitor interval="10" role="Master" timeout="20" depth="0" \
         op notify interval="0" timeout="90" \
         meta allow-migrate="true" target-role="started"
primitive res_zimbra_ZimbraSrv lsb:ZimbraCluster \
         operations $id="res_zimbra_ZimbraSrv-operations" \
         op start interval="0" timeout="6min" \
         op stop interval="0" timeout="5min" \
         op monitor interval="15" timeout="15" start-delay="15" \
         meta target-role="started"
ms ms_drbd_1 res_drbd_1 \
         meta clone-max="2" notify="true" interleave="true" 
allow-migrate="true"
colocation col_ms_drbd_1_ClusterIP inf: ClusterIP ms_drbd_1:Master
colocation col_res_Filesystem_1_ms_drbd_1 inf: res_Filesystem_1 
ms_drbd_1:Master
colocation col_res_zimbra_ZimbraSrv_res_Filesystem_1 inf: 
res_zimbra_ZimbraSrv res_Filesystem_1
order ord_ms_drbd_1_ClusterIP inf: ms_drbd_1:promote ClusterIP:start
order ord_ms_drbd_1_res_Filesystem_1 inf: ms_drbd_1:promote 
res_Filesystem_1:start
order ord_res_Filesystem_1_res_zimbra_ZimbraSrv inf: res_Filesystem_1 
res_zimbra_ZimbraSrv
property $id="cib-bootstrap-options" \
         dc-version="1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14" \
         cluster-infrastructure="openais" \
         expected-quorum-votes="2" \
         stonith-enabled="false" \
         no-quorum-policy="ignore" \
----------------------------------------------------------
 
Grazie !!
 
-- Yena --

 stonith-enabled="false"
 
… male ...
 

 "EXT4-fs (drbd0): ext4_orphan_cleanup: deleting unreferenced inode".
 
Se interpreto correttamente il cleanup dovrebbe farlo l'fsck. Se č cosě 
magari l'fsck l'ha fatto l'altro nodo che volendo diventare attivo si č 
accorto degli errori sull'fs prima di montarlo. Poi qualcosa č andato 
storto e ti sei ritrovato con lo split-brain.
 


Male fare un HA con due soli nodi...


Purtroppo non ho una periferica di fencing, quindi da quanto ho letto, è meglio impostarlo a False .. sbaglio ?
 
-- Yena --

 storto e ti sei ritrovato con lo split-brain.
 
Si mi sa che hai ragione ..
 
-- Yena --

 meglio impostarlo a False .. sbaglio ?
 
Beh se non hai fence device l'impostazione è corretta. Quando si fa un 
cluster stile linux basato sul concetto di stonith spesso si tralascia 
la parte di fencing, che però è fondamentale per l'integrità dei dati.
 
Non so che hardware hai a disposizione, ma nel caso potresti usare come 
fence device bmc (Dell) o ILO (HP) o similari, anche se una PDU 
gestibile tramite IP non dovrebbe costare follie.
 
Ad ogni modo dalle mie prove all'epoca con DRBD mi ricordo che c'erano 
vari parametri di configurazione riguardo alla gestione dello split 
brain, cose del tipo: per prima cosa prova così, altrimenti prova cosà, 
altrimenti chiama l'admin :-)
 
    after-sb-0pri discard-zero-changes;
    after-sb-1pri discard-secondary;
    after-sb-2pri disconnect;
 
 

     after-sb-2pri disconnect;
 
 
Hardware Supermicro con IPMI integrata .. potrei forse usare quella .. mi documento.
Come PDU tu cosa consigli, a prezzo contenuto ?
 
Invece per lo splitbrain, si hai ragione.. solo che senza fancing, ho preferito un approccio manuale .. non si sa mai ..
 
-- Yena --

 preferito un approccio manuale .. non si sa mai ..
 
E qui sta il problema: "non si sa mai". In parte hai ragione perché 
simulare condizioni di guasto in fase di test non è mai banale, per 
quanto uno ci possa pensare in produzione capiteranno sicuramente dei 
casi non previsti. Mi vengono in menete però alcune prove prima di 
andare in produzione (su entrambi i nodi alternativamente):
 
- spegnere brutalmente
- scoleggare dalla rete
- togliere brutalemnte i dischi
 
Inoltre considera, se già non è così, di ridondare i path di rete, 
quindi usando il bonding sulle interfacce, e magari usando 2 switch 
incrociati. Buona norma potrebbe essere quella di separare fisicamente 
le reti:
 
- una per il servzio verso gli utenti/client, sui cui è attestato il 
vip
- una per la sincronia DRBD (questa da ridondare con il bonding quasi 
per certo)
- una eventuale per il fencing tramtite IPMI
 
Come vedi il progetto si complica molto, e devi fare un'analisi per la 
gestione del rapporto complessità/benefici. Potresti prendere anche in 
considerazione un cluster a 3 nodi, ma qui DRBD potrebbe andarti 
stretto.
 
Ad ogni modo invito i luminari di queste parti a dare indicazioni più 
precise, ed a corregermi.