nimbusproject/nimbus — Issue #81: lantorrent corruption
Author: bc-umigs
Created: 2011-11-21 20:15:13+00:00
State: open

We have noticed an issue with lantorrent that causes it to hang and become unresponsive. The situation that causes this can be reproduced by repeating the following steps.

-Start up an instance/cluster either using the cloud-client or ec2 api.
-Issue a terminate before the propagation has completed.

After this occurs the database is left with the entries as stale records, which never seem to get updated. If this occurs either after a large termination or after repeated executions of the above process, lantorrent stops copying any new requests. Looking at the lantorrent daemon log shows this error:

2011-11-21 11:37:27,422 - WARNING - Stack trace
2011-11-21 11:37:27,423 - WARNING - ===========
2011-11-21 11:37:27,423 - WARNING - Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/pylantorrent/ltConnection.py", line 114, in send
self._write_to_socket(data)
File "build/bdist.linux-x86_64/egg/pylantorrent/ltConnection.py", line 99, in _write_to_socket
self.socket.sendall(data)
File "", line 1, in sendall
error: [Errno 104] Connection reset by peer

2011-11-21 11:37:27,423 - WARNING - ===========
2011-11-21 11:37:27,423 - WARNING -

If we clear out the req.db file and restart lantorrent, all seems to be well again. This is occurring in our production environment of 2.7, as well as devel which is at 2.8.

Please let me know what other information you need in order to help troubleshoot this.

Thanks,
Brian