frsyuki/serverengine — pull request #15 by hoshi-sano (2014-04-21 16:37:44+00:00)
Title: Fix multi process log rotation

This PR fixes three issues which cause a log rotation problem.

### background

Multi process log rotation fails on the following conditions.

* [Multi process worker](https://github.com/frsyuki/serverengine#multiprocess-server)
* Each worker write logs intensely

You can see the details and reproduce -> https://github.com/hoshi-sano/daemon_logger_test

### 1. bad condition to execute rotation

https://github.com/frsyuki/serverengine/blob/c7aa1a55702b64c34a328d7f1565b20f939a3a1f/lib/serverengine/daemon_logger.rb#L213

Suppose that Process-A and Process-B is running.

log/server.log # <= pointed by A-logger's @file, A-logger's @path,
# B-logger's @file, B-logger's @path

If `server.log` reached `log_rotate_size`, A-logger tries to rotate `server.log`.
After A-logger's rotation succeeded:

log/server.log # <= pointed by A's @file, A's @path, B's @path
log/server.log.0 # <= pointed by B's @file,

Then, B-logger will behave like below:

1. check `@file`(`server.log.0`) size: [L198](https://github.com/frsyuki/serverengine/blob/c7aa1a55702b64c34a328d7f1565b20f939a3a1f/lib/serverengine/daemon_logger.rb#L198)
2. detect that `@file` has reached `log_rotate_size`
3. try to lock the file of `@path`(`server.log`): [L209-211](https://github.com/frsyuki/serverengine/blob/c7aa1a55702b64c34a328d7f1565b20f939a3a1f/lib/serverengine/daemon_logger.rb#L209-L211)
4. check the inode-number (this will detect that log-file has already rotated by other processes while executing L209-213)
5. try to rotate `server.log` and `server.log.0`

After B-logger's rotation succeeded:

log/server.log # <= pointed by B's @file, A's @path, B's @path
log/server.log.0 # <= pointed by A's @file
log/server.log.1

Then, A-looger and B-logger writes logs to separate files in parallel.

### 2. `reopen!` causes daedlock

https://github.com/frsyuki/serverengine/blob/c7aa1a55702b64c34a328d7f1565b20f939a3a1f/lib/serverengine/daemon_logger.rb#L217

* `#reopen!` uses `@mutex.synchronize`
* `#reopen!` called in `#log_rotate_or_reopen`
* `#log_rotate_or_reopen` called in `@mutex.synchronize` of `#write`

Double `@mutex.synchronize` causes deadlock, and an error occurs.

### 3. `flock` is not released

* https://github.com/frsyuki/serverengine/commit/dfe03a624f7e322c30057c929cfb5b1fdb274e77#commitcomment-4374588
* https://github.com/frsyuki/serverengine/commit/dfe03a624f7e322c30057c929cfb5b1fdb274e77#commitcomment-4374690

This causes loss of some process's logger, and whenever log rotation is repeated, the lost loggers increase in number.
(it may continue beging blocked ?)