dCache/dcache — pull request #110 by alrossi (2013-06-26 12:59:02+00:00)
Title: dcache: (remote logging) change default server port to be outside epheme...

...ral port range

module: dcache (alarms)

This patch is an added precaution in connection with http://rb.dcache.org/r/5664. It was discovered that the default port suggested by ch.qos.logback for the SocketAppender and server, 60001, is in fact not a good choice because it lies within the ephemeral port range for some OS's. When there is no endpoint listening (I believe this only occurs when the server is supposed to be running local to the client domain), that port can be chosen as source for establishing a connection, such that the appender then ends up connecting to itself. If events are actually sent over this connection, it quickly fills up the buffer and blocks. This causes all threads which do logging to block, effectively freezing the system.

The above-mentioned patch sets remote logging by default to off, so that events are not sent unless it is explicitly activated. The self-connected appender, however, is still potentially present (though at the 10 second reconnect timeout -- which is what the logback.xml is configured for -- it will take a while for it to find the 60001 port).

Setting the default port outside most ephemeral port ranges will guarantee that this connect-back does not happen.

Testing done:

Using a small test class with the remote logger reconnect delay set to 1 ms, and no server endpoint running, port 60001 was first chosen. After 30 seconds,

[root@otfrid dcache]# netstat -anp | grep 60001
tcp 4 0 127.0.0.1:60001 127.0.0.1:60001 ESTABLISHED 6051/java

was seen. The test waits 60 seconds, then begins to log continuously. The system freezes after 5446 events:

6001 25 Jun 2013 08:03:24 [] error message 1
...
6001 25 Jun 2013 08:03:24 [] error message 5446

Rerunning with port set to 9876 shows that after 10 minutes and more than 17M events, no connection is established and hence no freeze.

Target: 2.6
Require-notes: yes
Require-book: no
Acked-by: Gerd
Bug: http://rt.dcache.org/Ticket/Display.html?id=7856
Merge-req:

RELEASE NOTES:

The default port for remote logging has been changed from 60001 to 9867; the latter lies outside most OS ephemeral port ranges. This avoids a situation in which the connection host is localhost (the default), but there is no server listening on that port, thus permitting that port to be chosen as source for the appender connection and consequently establishing a connection to itself. When this happens, the buffer quickly fills and blocks further logging, freezing all threads that do logging (effectively the entire domain).