ClusterLabs/pacemaker — pull request #331 by kskmori (2013-07-25 09:01:52+00:00)
Title: Low: ipc: retry to connect for EAGAIN in cib clients

crm_* commands may fail when they are invoked many instances at the same time.
connect() to UNIX domain sockets may return EAGAIN when the listen backlog overflowed on the server side.

It is reproduceable by running a test script like this:
```
for i in `seq 1 10`; do
strace -tt -o /tmp/crmmon${i}.strace crm_mon -n1 -VVVVV 2>&1 >/tmp/crmmon${i}.txt &
done
```

As a result, some of crm_mon commands fail with this error:
```
Connection to cluster failed: Transport endpoint is not connected
```

An excerpt from the strace output look like this:
```
14:35:40.897523 write(2, " info: main: \tStarting crm_mo"..., 34) = 34
14:35:40.901632 mmap(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f622be8c000
14:35:40.901681 socket(PF_FILE, SOCK_STREAM, 0) = 4
14:35:40.901716 fcntl(4, F_GETFD) = 0
14:35:40.901739 fcntl(4, F_SETFD, FD_CLOEXEC) = 0
14:35:40.901759 fcntl(4, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
14:35:40.901794 connect(4, {sa_family=AF_FILE, path=@"cib_ro"}, 110) = -1 EAGAIN (Resource temporarily unavailable)
14:35:40.901861 close(4) = 0
14:35:40.901894 shutdown(4294967295, 2 /* send and receive */) = -1 EBADF (Bad file descriptor)
14:35:40.901916 close(4294967295) = -1 EBADF (Bad file descriptor)
14:35:40.901950 write(2, "Could not establish cib_ro conne"..., 77) = 77
14:35:40.902009 write(2, " info: crm_ipc_connect: \tCoul"..., 105) = 105
```

Thanks,