Newsgroup: it.comp.os.linux.sys
Titolo: Split brain con DRBD

Sto ancora facendo dei test con DRBD.
Normalmente, se riavvio uno dei nodi o butto giů drbd, quando ricollego i
due nodi, la ricostruzione č veloce, nel senso che invia solo il delta delle
modifiche.
Se invece stacco i cavi di rete di una delle due macchine, continuo a
lavorare tranquillamente sull'altra, il contenuto della partizione in drbd
viene modificato, ma quando ricollego la macchina che prima avevo staccato,
mi va in split brain e sono costretto a dare un invalidate con conseguente
resync completo. Ho provato a fare disconnect/connect up/down ecc, ma finché
non do l'invalidate non c'č verso di resicronizzare le due partizioni.
Qualche idea?
Grazie

 
Quando stacchi i cavi di rete cosa fanno le due macchine "separate"? 
Diventano entrambe "primary"? Se lo fanno, poi devi invalidare una delle 
due.
Leandro

 due.
Quella che è in stand by giustamente diventa "primary", sull'altra gira
heartbeat con ipfail che la trasforma in "secondary".
Le due macchine sono comunque collegate con il cavo seriale per heartbeat.

Sam ha scritto:

Ti devi essere confuso. Secondo me ti diventano entrambi "primary". Se 
uno rimanesse "primary" e l'altro "secondary" non avresti lo split 
brain, ma sarebbe il modo "normale" di lavorare di drbd.
Fai in modo che heartbeat non ti trasformi il secondary in primary e 
vedrai che le potrai ricollegare in qualsiasi momento.
Leandro

 brain, ma sarebbe il modo "normale" di lavorare di drbd.
Ho riprovato, ti allego il log. Asterix è la macchina che era primary e hp è
il backup che è diventato primary quando ho staccato i cavi di rete da
asterix.
Poi ho ricollegato asterix in rete.
Quello che so che può succedere dopo che ho staccato il cavo di rete è che
in fase di stop postgres modifichi qualcosa sul file system che sta per
diventare secondary; però non vedo modi per evitarlo.
Qui ho drbd0 che è esportato tramite iscsi, mentre drbd1 che è usato
internamente da posgres che gira in locale sul cluster.
Oct 22 21:59:55 localhost kernel: bonding: bond0: link status down for
interface eth0, disabling it in 200 ms.
Oct 22 21:59:55 localhost kernel: bonding: bond0: link status definitely
down for interface eth0, disabling it
Oct 22 22:00:00 localhost kernel: eth1: link down
Oct 22 22:00:00 localhost kernel: bonding: bond0: link status down for idle
interface eth1, disabling it in 200 ms.
Oct 22 22:00:01 localhost kernel: bonding: bond0: link status definitely
down for interface eth1, disabling it
Oct 22 22:00:05 localhost heartbeat: [10047]: WARN: node 
10.100.0.151
: is
dead
Oct 22 22:00:05 localhost heartbeat: [10047]: info: Link
10.100.0.151:10.100.0.151 dead.
Oct 22 22:00:06 localhost harc[10822]: [10825]: info: Running
/etc/ha.d/rc.d/status status
Oct 22 22:00:07 localhost kernel: drbd1: peer( Secondary -> Unknown )
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Oct 22 22:00:07 localhost kernel: drbd1: asender terminated
Oct 22 22:00:07 localhost kernel: drbd1: Terminating asender thread
Oct 22 22:00:07 localhost kernel: drbd1: Creating new current UUID
Oct 22 22:00:07 localhost kernel: drbd1: Writing meta data super block now.
Oct 22 22:00:07 localhost kernel: drbd1: tl_clear()
Oct 22 22:00:07 localhost kernel: drbd1: Connection closed
Oct 22 22:00:07 localhost kernel: drbd1: conn( NetworkFailure ->
Unconnected )
Oct 22 22:00:07 localhost kernel: drbd1: receiver terminated
Oct 22 22:00:07 localhost kernel: drbd1: receiver (re)started
Oct 22 22:00:07 localhost kernel: drbd1: conn( Unconnected -> WFConnection )
Oct 22 22:00:11 localhost kernel: drbd0: peer( Secondary -> Unknown )
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Oct 22 22:00:11 localhost kernel: drbd0: asender terminated
Oct 22 22:00:11 localhost kernel: drbd0: Terminating asender thread
Oct 22 22:00:11 localhost kernel: drbd0: Creating new current UUID
Oct 22 22:00:11 localhost kernel: drbd0: Writing meta data super block now.
Oct 22 22:00:11 localhost kernel: drbd0: tl_clear()
Oct 22 22:00:11 localhost kernel: drbd0: Connection closed
Oct 22 22:00:11 localhost kernel: drbd0: conn( NetworkFailure ->
Unconnected )
Oct 22 22:00:11 localhost kernel: drbd0: receiver terminated
Oct 22 22:00:11 localhost kernel: drbd0: receiver (re)started
Oct 22 22:00:11 localhost kernel: drbd0: conn( Unconnected -> WFConnection )
Oct 22 22:00:13 localhost heartbeat: [10047]: info: asterix wants to go
standby [all]
Oct 22 22:00:14 localhost heartbeat: [10047]: info: standby: hp can take our
all resources
Oct 22 22:00:14 localhost heartbeat: [10838]: info: give up all HA resources
(standby).
Oct 22 22:00:14 localhost ResourceManager[10848]: [10856]: info: Releasing
resource group: hp IPaddr::10.100.0.100 drbddisk::all scst iscsi-scst
Filesystem::/dev/drbd1::/usr2::ext3 postgresql-8.1
Oct 22 22:00:14 localhost ResourceManager[10848]: [10866]: info: Running
/etc/init.d/postgresql-8.1  stop
Oct 22 22:00:16 localhost ResourceManager[10848]: [10894]: info: Running
/etc/ha.d/resource.d/Filesystem /dev/drbd1 /usr2 ext3 stop
Oct 22 22:00:17 localhost Filesystem[10961]: [10967]: INFO: Running stop for
/dev/drbd1 on /usr2
Oct 22 22:00:17 localhost Filesystem[10961]: [10977]: INFO: Trying to
unmount /usr2
Oct 22 22:00:17 localhost Filesystem[10961]: [10980]: INFO: unmounted /usr2
successfully
Oct 22 22:00:17 localhost Filesystem[10896]: [10982]: INFO: Filesystem
Success
Oct 22 22:00:17 localhost ResourceManager[10848]: [10992]: info: Running
/etc/init.d/iscsi-scst  stop
Oct 22 22:00:17 localhost kernel: iscsi-scst: Releasing allocated resources
Oct 22 22:00:18 localhost kernel: scst: Target d9a3fc80 for template iscsi
unregistered successfully
Oct 22 22:00:18 localhost kernel: iscsi-scst: Write thread PID 10434
finished
Oct 22 22:00:18 localhost kernel: iscsi-scst: Write thread PID 10433
finished
Oct 22 22:00:18 localhost kernel: iscsi-scst: Read thread PID 10432 finished
Oct 22 22:00:18 localhost kernel: iscsi-scst: Read thread PID 10431 finished
Oct 22 22:00:18 localhost kernel: scst: Target template iscsi unregistered
successfully
Oct 22 22:00:18 localhost ResourceManager[10848]: [11018]: info: Running
/etc/init.d/scst  stop
Oct 22 22:00:18 localhost kernel: scst: Removed device drbd0 from group
Default
Oct 22 22:00:18 localhost kernel: scst: Removed device drbd0 from group
iqn.2008-08.com.cluster:storage.drbd0
Oct 22 22:00:18 localhost kernel: dev_vdisk: Detached SCSI target virtual
device drbd0 ("/dev/drbd0")
Oct 22 22:00:18 localhost kernel: scst: Detached SCSI target mid-level from
virtual device drbd0 (id 1)
Oct 22 22:00:18 localhost kernel: dev_vdisk: Virtual device drbd0
unregistered
Oct 22 22:00:18 localhost kernel: scst: Device handler "vdisk_blk" unloaded
Oct 22 22:00:18 localhost kernel: scst: Device handler "vdisk" unloaded
Oct 22 22:00:18 localhost kernel: scst: Device handler "vcdrom" unloaded
Oct 22 22:00:18 localhost kernel: scst: Device handler "dev_disk_perf" for
type 0 unloaded
Oct 22 22:00:18 localhost kernel: scst: Device handler "dev_disk" for type 0
unloaded
Oct 22 22:00:18 localhost kernel: scst: Processing thread PID 10393 finished
Oct 22 22:00:18 localhost kernel: scst: Task management thread PID 10395
finished
Oct 22 22:00:18 localhost kernel: scst: Management thread PID 10396 finished
Oct 22 22:00:18 localhost kernel: scst: Init thread PID 10394 finished
Oct 22 22:00:18 localhost kernel: scst: SCST unloaded
Oct 22 22:00:18 localhost ResourceManager[10848]: [11047]: info: Running
/etc/ha.d/resource.d/drbddisk all stop
Oct 22 22:00:18 localhost kernel: drbd0: role( Primary -> Secondary )
Oct 22 22:00:18 localhost kernel: drbd0: Writing meta data super block now.
Oct 22 22:00:18 localhost kernel: drbd1: role( Primary -> Secondary )
Oct 22 22:00:18 localhost kernel: drbd1: Writing meta data super block now.
Oct 22 22:00:18 localhost ResourceManager[10848]: [11067]: info: Running
/etc/ha.d/resource.d/IPaddr 10.100.0.100 stop
Oct 22 22:00:19 localhost IPaddr[11139]: [11148]: INFO: /sbin/route -n
del -host 10.100.0.100
Oct 22 22:00:19 localhost IPaddr[11139]: [11150]: INFO: /sbin/ifconfig
bond0:0 10.100.0.100 down
Oct 22 22:00:19 localhost IPaddr[11139]: [11153]: INFO: IP Address
10.100.0.100 released
Oct 22 22:00:19 localhost IPaddr[11069]: [11154]: INFO: IPaddr Success
Oct 22 22:00:19 localhost heartbeat: [10838]: info: all HA resource release
completed (standby).
Oct 22 22:00:19 localhost heartbeat: [10047]: info: Local standby process
completed [all].
Oct 22 22:00:29 localhost heartbeat: [10047]: WARN: 1 lost packet(s) for
[hp] [91:93]
Oct 22 22:00:29 localhost heartbeat: [10047]: info: remote resource
transition completed.
Oct 22 22:00:29 localhost heartbeat: [10047]: info: No pkts missing from hp!
Oct 22 22:00:29 localhost heartbeat: [10047]: info: Other node completed
standby takeover of all resources.
Oct 22 22:02:13 localhost kernel: eth1: link up, 100Mbps, half-duplex, lpa
0x40A1
Oct 22 22:02:13 localhost kernel: bonding: bond0: link status up for
interface eth1, enabling it in 200 ms.
Oct 22 22:02:13 localhost kernel: bonding: bond0: link status definitely up
for interface eth1.
Oct 22 22:02:13 localhost kernel: bonding: bond0: first active interface up!
Oct 22 22:02:14 localhost kernel: eth1: link down
Oct 22 22:02:14 localhost kernel: bonding: bond0: link status down for idle
interface eth1, disabling it in 200 ms.
Oct 22 22:02:15 localhost kernel: bonding: bond0: link status definitely
down for interface eth1, disabling it
Oct 22 22:02:21 localhost kernel: eth1: link up, 100Mbps, half-duplex, lpa
0x40A1
Oct 22 22:02:21 localhost kernel: bonding: bond0: link status up for
interface eth1, enabling it in 200 ms.
Oct 22 22:02:22 localhost kernel: bonding: bond0: link status definitely up
for interface eth1.
Oct 22 22:02:22 localhost kernel: bonding: bond0: first active interface up!
Oct 22 22:02:23 localhost heartbeat: [10047]: info: Link
10.100.0.151:10.100.0.151 up.
Oct 22 22:02:23 localhost heartbeat: [10047]: WARN: Late heartbeat: Node
10.100.0.151
: interval 143010 ms
Oct 22 22:02:23 localhost heartbeat: [10047]: info: Status update for node
10.100.0.151
: status ping
Oct 22 22:02:28 localhost kernel: drbd1: Handshake successful: DRBD Network
Protocol version 86
Oct 22 22:02:28 localhost kernel: drbd1: conn( WFConnection ->
WFReportParams )
Oct 22 22:02:28 localhost kernel: drbd0: Handshake successful: DRBD Network
Protocol version 86
Oct 22 22:02:28 localhost kernel: drbd0: conn( WFConnection ->
WFReportParams )
Oct 22 22:02:28 localhost kernel: drbd1: Starting asender thread (from
drbd1_receiver [2475])
Oct 22 22:02:28 localhost kernel: drbd0: Starting asender thread (from
drbd0_receiver [2483])
Oct 22 22:02:28 localhost kernel: drbd1: Considerable difference in lower
level device sizes: 8594432s vs. 6302336s
Oct 22 22:02:28 localhost kernel: drbd1: self
F93482B9DE4D1C48:4494F18F9E7BCF89:CE3175C07501EF5E:39D2AC625DBAAA5D
Oct 22 22:02:28 localhost kernel: drbd1: peer
9302ADA9C1D42311:4494F18F9E7BCF88:CE3175C07501EF5E:39D2AC625DBAAA5D
Oct 22 22:02:28 localhost kernel: drbd1: helper command: /sbin/drbdadm
split-brain
Oct 22 22:02:28 localhost kernel: drbd1: conn( WFReportParams ->
NetworkFailure )
Oct 22 22:02:28 localhost kernel: drbd1: asender terminated
Oct 22 22:02:28 localhost kernel: drbd1: Terminating asender thread
Oct 22 22:02:28 localhost kernel: drbd1: tl_clear()
Oct 22 22:02:28 localhost kernel: drbd1: Connection closed
Oct 22 22:02:28 localhost kernel: drbd1: conn( NetworkFailure ->
Unconnected )
Oct 22 22:02:28 localhost kernel: drbd1: receiver terminated
Oct 22 22:02:28 localhost kernel: drbd1: receiver (re)started
Oct 22 22:02:28 localhost kernel: drbd1: conn( Unconnected -> WFConnection )
Oct 22 22:02:28 localhost kernel: drbd0: Considerable difference in lower
level device sizes: 8578304s vs. 6297088s
Oct 22 22:02:28 localhost kernel: drbd0: self
7E3C6089256D9D7C:85AF11A2B80DC19F:6F13C2F1DDDE01A6:296EF6F3B5B0E485
Oct 22 22:02:28 localhost kernel: drbd0: peer
DE14B9C54E90FAFF:85AF11A2B80DC19E:6F13C2F1DDDE01A6:296EF6F3B5B0E485
Oct 22 22:02:28 localhost kernel: drbd0: helper command: /sbin/drbdadm
split-brain
Oct 22 22:02:28 localhost kernel: drbd0: conn( WFReportParams ->
Disconnecting )
Oct 22 22:02:28 localhost kernel: drbd0: asender terminated
Oct 22 22:02:28 localhost kernel: drbd0: Terminating asender thread
Oct 22 22:02:28 localhost kernel: drbd0: tl_clear()
Oct 22 22:02:28 localhost kernel: drbd0: Connection closed
Oct 22 22:02:28 localhost kernel: drbd0: conn( Disconnecting -> StandAlone )
Oct 22 22:02:28 localhost kernel: drbd0: receiver terminated
Oct 22 22:02:28 localhost kernel: drbd0: Terminating receiver thread
Oct 22 22:02:45 localhost kernel: drbd0: conn( StandAlone -> Unconnected )
Oct 22 22:02:45 localhost kernel: drbd0: Starting receiver thread (from
drbd0_worker [2465])
Oct 22 22:02:45 localhost kernel: drbd0: receiver (re)started
Oct 22 22:02:45 localhost kernel: drbd0: conn( Unconnected -> WFConnection )
Oct 22 22:02:45 localhost kernel: drbd0: Handshake successful: DRBD Network
Protocol version 86
Oct 22 22:02:45 localhost kernel: drbd0: conn( WFConnection ->
WFReportParams )
Oct 22 22:02:45 localhost kernel: drbd0: Starting asender thread (from
drbd0_receiver [11224])
Oct 22 22:02:45 localhost kernel: drbd0: Considerable difference in lower
level device sizes: 8578304s vs. 6297088s
Oct 22 22:02:45 localhost kernel: drbd0: self
7E3C6089256D9D7C:85AF11A2B80DC19F:6F13C2F1DDDE01A6:296EF6F3B5B0E485
Oct 22 22:02:45 localhost kernel: drbd0: peer
DE14B9C54E90FAFF:85AF11A2B80DC19E:6F13C2F1DDDE01A6:296EF6F3B5B0E485
Oct 22 22:02:45 localhost kernel: drbd0: helper command: /sbin/drbdadm
split-brain
Oct 22 22:02:45 localhost kernel: drbd0: conn( WFReportParams ->
NetworkFailure )
Oct 22 22:02:45 localhost kernel: drbd0: asender terminated
Oct 22 22:02:45 localhost kernel: drbd0: Terminating asender thread
Oct 22 22:02:45 localhost kernel: drbd0: tl_clear()
Oct 22 22:02:45 localhost kernel: drbd0: Connection closed
Oct 22 22:02:45 localhost kernel: drbd0: conn( NetworkFailure ->
Unconnected )
Oct 22 22:02:45 localhost kernel: drbd0: receiver terminated
Oct 22 22:02:45 localhost kernel: drbd0: receiver (re)started
Oct 22 22:02:45 localhost kernel: drbd0: conn( Unconnected -> WFConnection )

Sam ha scritto:

Torna tutto. Quando stacchi i cavi, "asterisk" vede l'altro come 
"unknown", mentre lui è primary. "Asterisk" ha quindi fatto delle 
modifiche al device che non è riuscito a spedire a "hp". Poi asterisk 
passa "secondary" e quelle modifiche le ha sempre in pancia: non può 
spedirle a "hp". "hp" passerà "primary" per via dell'heartbeat, "hp" non 
sà niente delle modifiche che "asterisk" ha fatto al device e che non è 
riuscito a comunicargli. "hp" lavora un po', poi quando riconnetti i 
cavi, si accorgano che hanno lavorato ciascuno in modo indipendente e 
quindi => split brain.
Ergo, se stacchi i cavi, fai in modo che chi è primary rimanga primary e 
chi è secondary rimanga secondary.
Leandro