facebook/scribe — Issue #47: Scribe stops writing logs
Author: sitano
Created: 2011-11-06 12:37:21+00:00
State: open

Scribe server starts as daemon configured for one file store for 10 categories, one thread per category -> 10 different files.
There is also one default* and one ignore* category stores.

Client is a long-lived server that uses scribe.AsyncClient (it impl.s framed transport as is and it is keep alived (long lived client connection)) to push messages. Thrift 0.5 as required.

Scribe server is binded on localhost to default port. Client sits at same machine. Server is Ubuntu EC2 Server, large instance.

It all starts to work properly, until at some unexpected moment (about 10 minutes after client connected) scribe stops to write/accept messages from client. netstat shows that connection is alive. tcpdump shows that scribe respond by ack for each pushed message, but no answer sent back (live ok or fail). Counters for accepted good messages are not incrementing.

Log rotation configured for one day. Log placed on mdadm raid1 ext4. It shouldn't bring much diff.

Mostly scribe log is ok. But sometimes i saw TNonBlockingServer client died: no data to read. Sometime i saw - you are not using tframedprotocol? And all other time there is nothing strange there, it just stop writing.

If i reconnect client than it starts to write messages. But after some time later problem take place again.

I can't reproduce this on other machine. No ideas what's wrong.