Repository: filecoin-project/lotus

Issue #4063: RPC client error: sendRequest failed: websocket routine exiting
Created: 2020-09-27T18:49:20Z
Describe the bug
This issue is about the connection between lotus-miner and lotus-worker.
websocket i/o timeout after completing PreCommit2 causes the following error repeatedly:
ERROR	advmgr	sector-storage/sched.go:370	trySched(1) req.sel.Ok error: getting supported worker task types:
    github.com/filecoin-project/lotus/extern/sector-storage.(*taskSelector).Ok
        /home/downloads/lotus/extern/sector-storage/selector_task.go:25
  - RPC client error: sendRequest failed: websocket routine exiting


No errors on lotus-worker's side.
No known hardware issues.
Occurs on random servers.
Recovery requires restarting the server.
Not a one time incident. It's been happening dozens of times everyday since SR1.

To Reproduce

Wait until a worker goes idle (after completing PreCommit2 and waiting for seed for example)
Get the following errors:


I've once been spammed by this error (over 10,000 identical logs occurred consecutively).
When it happened, lotus-miner went idle and missed windowed posts.
I had to restart lotus-miner to get it working again.
Version (run lotus version):
Daemon:  0.8.0+git.2c1d96bc+api0.15.0
Local: lotus-miner version 0.8.0+git.2c1d96bc
Additional context
API timeout has been increased from 30s (default) to 60s in miner's config.

Duplicate issue #4021: sendRequest failed: websocket routine exiting
Created: 2020-09-25T13:42:04Z
Describe the bug
It seems lotus-miner always lost the connection with lotus-node
To Reproduce
Sep 25 21:28:03 m001 lotus-miner[23452]: 2020-09-25T21:28:03.554+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:03 m001 lotus-miner[23452]: 2020-09-25T21:28:03.554+0800        ERROR        events        events/events.go:95        listen head changes errored: listenHeadChanges ChainNotify call failed: RPC client error: sendRequest failed: websocket routine exiting
Sep 25 21:28:03 m001 lotus-miner[23452]: 2020-09-25T21:28:03.554+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:03 m001 lotus-miner[23452]: 2020-09-25T21:28:03.554+0800        ERROR        events        events/events.go:95        listen head changes errored: listenHeadChanges ChainNotify call failed: RPC client error: sendRequest failed: websocket routine exiting
Sep 25 21:28:04 m001 lotus-miner[23452]: 2020-09-25T21:28:04.554+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:04 m001 lotus-miner[23452]: 2020-09-25T21:28:04.554+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:04 m001 lotus-miner[23452]: 2020-09-25T21:28:04.554+0800        ERROR        events        events/events.go:95        listen head changes errored: listenHeadChanges ChainNotify call failed: RPC client error: sendRequest failed: websocket routine exiting
Sep 25 21:28:05 m001 lotus-miner[23452]: 2020-09-25T21:28:05.555+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:05 m001 lotus-miner[23452]: 2020-09-25T21:28:05.555+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:05 m001 lotus-miner[23452]: 2020-09-25T21:28:05.555+0800        ERROR        events        events/events.go:95        listen head changes errored: listenHeadChanges ChainNotify call failed: RPC client error: sendRequest failed: websocket routine exiting
Sep 25 21:28:06 m001 lotus-miner[23452]: 2020-09-25T21:28:06.434+0800        ERROR        storageminer        storage/wdpost_sched.go:101        ChainNotify error: RPC client error: sendRequest failed: websocket routine exiting
Sep 25 21:28:06 m001 lotus-miner[23452]: 2020-09-25T21:28:06.555+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:06 m001 lotus-miner[23452]: 2020-09-25T21:28:06.555+0800        ERROR        events        events/events.go:95        listen head changes errored: listenHeadChanges ChainNotify call failed: RPC client error: sendRequest failed: websocket routine exiting
Sep 25 21:28:07 m001 lotus-miner[23452]: 2020-09-25T21:28:07.534+0800        ERROR        miner        miner/miner.go:172        failed to get best mining candidate: RPC client error: sendRequest failed: websocket routine exiting
Sep 25 21:28:07 m001 lotus-miner[23452]: 2020-09-25T21:28:07.555+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:07 m001 lotus-miner[23452]: 2020-09-25T21:28:07.555+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:07 m001 lotus-miner[23452]: 2020-09-25T21:28:07.555+0800        ERROR        events        events/events.go:95        listen head changes errored: listenHeadChanges ChainNotify call failed: RPC client error: sendRequest failed: websocket routine exiting
Sep 25 21:28:08 m001 lotus-miner[23452]: 2020-09-25T21:28:08.555+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:08 m001 lotus-miner[23452]: 2020-09-25T21:28:08.555+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:08 m001 lotus-miner[23452]: 2020-09-25T21:28:08.555+0800        ERROR        events        events/events.go:95        listen head changes errored: listenHeadChanges ChainNotify call failed: RPC client error: sendRequest failed: websocket routine exiting
Sep 25 21:28:09 m001 lotus-miner[23452]: 2020-09-25T21:28:09.555+0800        INFO        events        events/events.go:104        restarting listenHeadChanges
Sep 25 21:28:09 m001 lotus-miner[23452]: 2020-09-25T21:28:09.555+0800        ERROR        events        events/events.go:95        listen head changes errored: listenHeadChanges ChainNotify call failed: RPC client error: sendRequest failed: websocket routine exiting

Expected behavior
Screenshots
Version (run lotus version):
Daemon:  0.7.1+git.4121063c+api0.15.0
Local: lotus version 0.7.1+git.4121063c
Additional context

Closing comment (2020-09-28T04:25:35Z): duplicate #4063, closing this one since the other ticket contains more info