antirez/redis — Issue #145: Server crash on failed master/slave sync (refused connection)
Author: janoberst
Created: 2011-10-17 17:30:26+00:00
State: open

Three of my instances (on server 1 and 2) just crashed when I restarted server 3. Instance A was a master to an instance on server 3. Instances B and C were slaves of instances on server 3.

I'm sorry that I have to be so vague about this. I just found these stackdumps in my logfiles a few minutes after it happened. I believe, instances A,B,C were trying to do some kind of syncing with server 3, which was being restarted at the time. I assume server 3 refused the connections when the instances were in a state where these exceptions weren't handled.

I'm using the fresh 2.4.1 from this morning on Ubuntu Oneiric. Let me know if you need more input.

instance 1:

[2710] 17 Oct 16:56:25 * MASTER MODE enabled (user request)
[2710] 17 Oct 16:56:25 * Slave ask for synchronization
[2710] 17 Oct 16:56:25 * Starting BGSAVE for SYNC
[2710] 17 Oct 16:56:25 * Background saving started by pid 4019
[2710] 17 Oct 16:56:27 - DB 0: 91 keys (0 volatile) in 128 slots HT.
[2710] 17 Oct 16:56:27 - 0 clients connected (1 slaves), 92076368 bytes in use
[2710] 17 Oct 16:56:30 * Non blocking connect for SYNC fired the event.
[2710] 17 Oct 16:56:30 # I/O error writing to MASTER: Connection refused
[2710] 17 Oct 16:56:30 * Connecting to MASTER...
[2710] 17 Oct 16:56:30 # ======= Ooops! Redis 2.4.1 got signal: -11- =======
[2710] 17 Oct 16:56:30 # redis_version:2.4.1
redis_git_sha1:ab648d97
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:2710
uptime_in_seconds:4920
uptime_in_days:0
lru_clock:1863635
used_cpu_sys:16.91
used_cpu_user:6.16
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
connected_clients:0
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:92076368
used_memory_human:87.81M
used_memory_rss:96280576
used_memory_peak:1022465704
used_memory_peak_human:975.10M
mem_fragmentation_ratio:1.05
mem_allocator:jemalloc-2.2.1
loading:0
aof_enabled:1
changes_since_last_save:16060
bgsave_in_progress:1
last_save_time:1318865670
bgrewriteaof_in_progress:0
total_connections_received:2475
total_commands_processed:13985
expired_keys:0
evicted_keys:0
keyspace_hits:1204
keyspace_misses:9139
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:28578
vm_enabled:0
role:master
aof_current_size:91031534
aof_base_size:0
aof_pending_rewrite:0
db0
[2710] 17 Oct 16:56:30 # /lib/x86_64-linux-gnu/libc.so.6(inet_aton+0x35) [0x7f3fc4b64215]
[2710] 17 Oct 16:56:30 # /lib/x86_64-linux-gnu/libc.so.6(inet_aton+0x35) [0x7f3fc4b64215]
[2710] 17 Oct 16:56:30 # /usr/local/bin/redis-server() [0x40ca99]
[2710] 17 Oct 16:56:30 # /usr/local/bin/redis-server(connectWithMaster+0x1a) [0x41c14a]
[2710] 17 Oct 16:56:30 # /usr/local/bin/redis-server(replicationCron+0xe6) [0x41c416]
[2710] 17 Oct 16:56:30 # /usr/local/bin/redis-server(serverCron+0x1c8) [0x40fa78]
[2710] 17 Oct 16:56:30 # /usr/local/bin/redis-server(aeProcessEvents+0x1db) [0x40c45b]
[2710] 17 Oct 16:56:30 # /usr/local/bin/redis-server(aeMain+0x2e) [0x40c70e]
[2710] 17 Oct 16:56:30 # /usr/local/bin/redis-server(main+0x106) [0x40b606]
[2710] 17 Oct 16:56:30 # /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f3fc4a9530d]
[2710] 17 Oct 16:56:30 # /usr/local/bin/redis-server() [0x40b72d]
[4019] 17 Oct 16:56:32 * DB saved on disk

instance 2:

[2698] 17 Oct 16:56:30 * Non blocking connect for SYNC fired the event.
[2698] 17 Oct 16:56:30 # I/O error writing to MASTER: Connection refused
[2698] 17 Oct 16:56:30 * Connecting to MASTER...
[2698] 17 Oct 16:56:30 # ======= Ooops! Redis 2.4.1 got signal: -11- =======
[2698] 17 Oct 16:56:30 # redis_version:2.4.1
redis_git_sha1:ab648d97
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:2698
uptime_in_seconds:4920
uptime_in_days:0
lru_clock:1863635
used_cpu_sys:17.05
used_cpu_user:6.90
used_cpu_sys_children:0.14
used_cpu_user_children:0.62
connected_clients:0
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:87061824
used_memory_human:83.03M
used_memory_rss:92336128
used_memory_peak:1018799560
used_memory_peak_human:971.60M
mem_fragmentation_ratio:1.06
mem_allocator:jemalloc-2.2.1
loading:0
aof_enabled:1
changes_since_last_save:0
bgsave_in_progress:0
last_save_time:1318870572
bgrewriteaof_in_progress:0
total_connections_received:2565
total_commands_processed:13986
expired_keys:0
evicted_keys:0
keyspace_hits:1034
keyspace_misses:8965
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:26442
vm_enabled:0
role:master
aof_current_size:86031167
aof_base_size:0
aof_pending_rewrite:0
db0:key
[2698] 17 Oct 16:56:30 # /lib/x86_64-linux-gnu/libc.so.6(inet_aton+0x35) [0x7f402ca77215]
[2698] 17 Oct 16:56:30 # /lib/x86_64-linux-gnu/libc.so.6(inet_aton+0x35) [0x7f402ca77215]
[2698] 17 Oct 16:56:30 # /usr/local/bin/redis-server() [0x40ca99]
[2698] 17 Oct 16:56:30 # /usr/local/bin/redis-server(connectWithMaster+0x1a) [0x41c14a]
[2698] 17 Oct 16:56:30 # /usr/local/bin/redis-server(replicationCron+0xe6) [0x41c416]
[2698] 17 Oct 16:56:30 # /usr/local/bin/redis-server(serverCron+0x1c8) [0x40fa78]
[2698] 17 Oct 16:56:30 # /usr/local/bin/redis-server(aeProcessEvents+0x1db) [0x40c45b]
[2698] 17 Oct 16:56:30 # /usr/local/bin/redis-server(aeMain+0x2e) [0x40c70e]
[2698] 17 Oct 16:56:30 # /usr/local/bin/redis-server(main+0x106) [0x40b606]
[2698] 17 Oct 16:56:30 # /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f402c9a830d]
[2698] 17 Oct 16:56:30 # /usr/local/bin/redis-server() [0x40b72d]

instance 3:

[2684] 17 Oct 16:56:31 * Non blocking connect for SYNC fired the event.
[2684] 17 Oct 16:56:31 # I/O error writing to MASTER: Connection refused
[2684] 17 Oct 16:56:31 * Connecting to MASTER...
[2684] 17 Oct 16:56:31 # ======= Ooops! Redis 2.4.1 got signal: -11- =======
[2684] 17 Oct 16:56:31 # redis_version:2.4.1
redis_git_sha1:ab648d97
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:2684
uptime_in_seconds:4916
uptime_in_days:0
lru_clock:1863635
used_cpu_sys:28.23
used_cpu_user:8.24
used_cpu_sys_children:1.48
used_cpu_user_children:6.89
connected_clients:0
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:87061960
used_memory_human:83.03M
used_memory_rss:91430912
used_memory_peak:1018693576
used_memory_peak_human:971.50M
mem_fragmentation_ratio:1.05
mem_allocator:jemalloc-2.2.1
loading:0
aof_enabled:1
changes_since_last_save:0
bgsave_in_progress:0
last_save_time:1318870582
bgrewriteaof_in_progress:0
total_connections_received:2710
total_commands_processed:14067
expired_keys:0
evicted_keys:0
keyspace_hits:1023
keyspace_misses:8950
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:27831
vm_enabled:0
role:master
aof_current_size:86031166
aof_base_size:0
aof_pending_rewrite:0
db0:key
[2684] 17 Oct 16:56:31 # /lib/x86_64-linux-gnu/libc.so.6(inet_aton+0x35) [0x7fd1abedb215]
[2684] 17 Oct 16:56:31 # /lib/x86_64-linux-gnu/libc.so.6(inet_aton+0x35) [0x7fd1abedb215]
[2684] 17 Oct 16:56:31 # /usr/local/bin/redis-server() [0x40ca99]
[2684] 17 Oct 16:56:31 # /usr/local/bin/redis-server(connectWithMaster+0x1a) [0x41c14a]
[2684] 17 Oct 16:56:31 # /usr/local/bin/redis-server(replicationCron+0xe6) [0x41c416]
[2684] 17 Oct 16:56:31 # /usr/local/bin/redis-server(serverCron+0x1c8) [0x40fa78]
[2684] 17 Oct 16:56:31 # /usr/local/bin/redis-server(aeProcessEvents+0x1db) [0x40c45b]
[2684] 17 Oct 16:56:31 # /usr/local/bin/redis-server(aeMain+0x2e) [0x40c70e]
[2684] 17 Oct 16:56:31 # /usr/local/bin/redis-server(main+0x106) [0x40b606]
[2684] 17 Oct 16:56:31 # /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7fd1abe0c30d]
[2684] 17 Oct 16:56:31 # /usr/local/bin/redis-server() [0x40b72d]