Repository: Chia-Network/chia-blockchain
Pull request: Add a daemon heartbeat setting to config.yaml
This might not be worthwhile, but sometimes it seems having a config setting for this is useful for some troubleshooting.
Also change the default for the Daemon to be 300 seconds.
Rationale: The daemon connections are typically almost always localhost connections. The heartbeat is only useful for detecting connections that were closed. I'll note that there is already a manual ping task running in the daemon server that will also detect closed connections since the send will fail. Therefore, the heartbeat is mostly useful for clients to detect that the daemon connection has closed when there is minimal traffic and I don't believe that there is any practical scenario where this causes substantial user issues.
Note that other WS connections (farmer<->harvester for example) continue to use a 60 seconds heartbeat setting. (although similar arguments can be made this isn't that useful, since these connections tend to be quite chatty anyway)

Problem statement:
[Bug]  Harvester raise asyncio.exceptions.TimeoutError exception
### What happened?

Recently, one of my harvester started to raise asyncio.exceptions.TimeoutError after running for couple hours. I have to reboot the machine to start chia harvester again.

I have attached the log.



### Version

1.5.1

### What platform are you using?

Linux

### What ui mode are you using?

CLI

### Relevant log output

```shell
2022-09-20T03:12:25.000 harvester harvester_server        : DEBUG    <- new_signage_point_harvester from peer 2daa464f784aca4d84aa6adc2f6ab67f2539b43ea9cf8832bc6d146889e9593f 192.168.100.11
2022-09-20T03:12:30.308 harvester harvester_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '192.168.100.11', 'port': 8447}. Traceback (most recent call last):
  File "/home/lxu/chia-blockchain/chia/server/server.py", line 607, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-09-20T03:12:30.308 harvester harvester_server        : DEBUG    Inbound_handler task cancelled
2022-09-20T03:12:30.310 harvester harvester_server        : INFO     Connection closed: 192.168.100.11, node id: 2daa464f784aca4d84aa6adc2f6ab67f2539b43ea9cf8832bc6d146889e9593f
2022-09-20T03:12:30.310 harvester harvester_server        : WARNING  Banning 192.168.100.11 for 10 seconds
2022-09-20T03:12:30.311 harvester chia.harvester.harvester: INFO     peer disconnected {'host': '192.168.100.11', 'port': 8447}
2022-09-20T03:12:46.686 daemon chia.daemon.server         : DEBUG    About to ping: chia_harvester
2022-09-20T03:13:16.698 daemon chia.daemon.server         : DEBUG    About to ping: chia_harvester
2022-09-20T03:13:16.698 daemon chia.daemon.server         : ERROR    Ping error
Traceback (most recent call last):
  File "/home/lxu/chia-blockchain/chia/daemon/server.py", line 275, in ping_task
    await socket.ping()
  File "/home/lxu/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/web_ws.py", line 295, in ping
    await self._writer.ping(message)
  File "/home/lxu/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 676, in ping
    await self._send_frame(message, WSMsgType.PING)
  File "/home/lxu/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 653, in _send_frame
    self._write(header + message)
  File "/home/lxu/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 663, in _write
    raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport
2022-09-20T03:13:16.701 daemon chia.daemon.server         : ERROR    Ping failed, connection closed.
```


[Bug] Ping errors in Daemon
### What happened?

2022-12-22T20:10:21.497 daemon __main__                   : ERROR    Ping error
Traceback (most recent call last):
  File "chia/daemon/server.py", line 283, in ping_task
  File "aiohttp/web_ws.py", line 295, in ping
  File "aiohttp/http_websocket.py", line 676, in ping
  File "aiohttp/http_websocket.py", line 653, in _send_frame
  File "aiohttp/http_websocket.py", line 663, in _write
ConnectionResetError: Cannot write to closing transport
2022-12-22T20:11:38.996 harvester harvester_server        : WARNING  Cannot write to closing transport 127.0.0.1
2022-12-22T20:11:39.027 daemon __main__                   : ERROR    Ping failed, connection closed.
2022-12-22T20:11:39.317 daemon aiohttp.server             : ERROR    Error handling request
Traceback (most recent call last):
  File "aiohttp/web_protocol.py", line 433, in _handle_request
  File "aiohttp/web_app.py", line 504, in _handle
  File "chia/daemon/server.py", line 224, in incoming_connection
  File "aiohttp/web_ws.py", line 444, in receive
  File "aiohttp/web_ws.py", line 301, in pong
  File "aiohttp/http_websocket.py", line 670, in pong
  File "aiohttp/http_websocket.py", line 653, in _send_frame
  File "aiohttp/http_websocket.py", line 663, in _write
ConnectionResetError: Cannot write to closing transport

### Version

1.7.0b4

### What platform are you using?

Linux

### What ui mode are you using?

GUI

### Relevant log output

```shell
2022-12-22T20:10:21.497 daemon __main__                   : ERROR    Ping error
Traceback (most recent call last):
  File "chia/daemon/server.py", line 283, in ping_task
  File "aiohttp/web_ws.py", line 295, in ping
  File "aiohttp/http_websocket.py", line 676, in ping
  File "aiohttp/http_websocket.py", line 653, in _send_frame
  File "aiohttp/http_websocket.py", line 663, in _write
ConnectionResetError: Cannot write to closing transport
2022-12-22T20:11:38.996 harvester harvester_server        : WARNING  Cannot write to closing transport 127.0.0.1
2022-12-22T20:11:39.027 daemon __main__                   : ERROR    Ping failed, connection closed.
2022-12-22T20:11:39.317 daemon aiohttp.server             : ERROR    Error handling request
Traceback (most recent call last):
  File "aiohttp/web_protocol.py", line 433, in _handle_request
  File "aiohttp/web_app.py", line 504, in _handle
  File "chia/daemon/server.py", line 224, in incoming_connection
  File "aiohttp/web_ws.py", line 444, in receive
  File "aiohttp/web_ws.py", line 301, in pong
  File "aiohttp/http_websocket.py", line 670, in pong
  File "aiohttp/http_websocket.py", line 653, in _send_frame
  File "aiohttp/http_websocket.py", line 663, in _write
ConnectionResetError: Cannot write to closing transport
```

Hints from issue discussion:

Commit message: Various updates from feedback

Patch to review:
diff --git a/chia/daemon/client.py b/chia/daemon/client.py
index 99df01a12efe..9870c1a0fe2d 100644
--- a/chia/daemon/client.py
+++ b/chia/daemon/client.py
@@ -18,12 +18,14 @@ def __init__(
         self,
         uri: str,
         ssl_context: Optional[ssl.SSLContext],
+        heartbeat: int,
         max_message_size: int = 50 * 1000 * 1000,
     ):
         self._uri = uri
         self._request_dict: Dict[str, asyncio.Event] = {}
         self.response_dict: Dict[str, WsRpcMessage] = {}
         self.ssl_context = ssl_context
+        self.heartbeat = heartbeat
         self.client_session: Optional[aiohttp.ClientSession] = None
         self.websocket: Optional[aiohttp.ClientWebSocketResponse] = None
         self.max_message_size = max_message_size
@@ -39,7 +41,7 @@ async def start(self) -> None:
                 self._uri,
                 autoclose=True,
                 autoping=True,
-                heartbeat=60,
+                heartbeat=self.heartbeat,
                 ssl_context=self.ssl_context,
                 max_msg_size=self.max_message_size,
             )
@@ -151,13 +153,18 @@ async def exit(self) -> WsRpcMessage:
 
 
 async def connect_to_daemon(
-    self_hostname: str, daemon_port: int, max_message_size: int, ssl_context: ssl.SSLContext
+    self_hostname: str, daemon_port: int, max_message_size: int, ssl_context: ssl.SSLContext, heartbeat: int
 ) -> DaemonProxy:
     """
     Connect to the local daemon.
     """
 
-    client = DaemonProxy(f"wss://{self_hostname}:{daemon_port}", ssl_context, max_message_size)
+    client = DaemonProxy(
+        f"wss://{self_hostname}:{daemon_port}",
+        ssl_context=ssl_context,
+        max_message_size=max_message_size,
+        heartbeat=heartbeat,
+    )
     await client.start()
     return client
 
@@ -173,13 +180,18 @@ async def connect_to_daemon_and_validate(
 
     try:
         daemon_max_message_size = config.get("daemon_max_message_size", 50 * 1000 * 1000)
+        daemon_heartbeat = config.get("daemon_heartbeat", 300)
         crt_path = root_path / config["daemon_ssl"]["private_crt"]
         key_path = root_path / config["daemon_ssl"]["private_key"]
         ca_crt_path = root_path / config["private_ssl_ca"]["crt"]
         ca_key_path = root_path / config["private_ssl_ca"]["key"]
         ssl_context = ssl_context_for_client(ca_crt_path, ca_key_path, crt_path, key_path)
         connection = await connect_to_daemon(
-            config["self_hostname"], config["daemon_port"], daemon_max_message_size, ssl_context
+            config["self_hostname"],
+            config["daemon_port"],
+            max_message_size=daemon_max_message_size,
+            ssl_context=ssl_context,
+            heartbeat=daemon_heartbeat,
         )
         r = await connection.ping()
 
diff --git a/chia/daemon/keychain_proxy.py b/chia/daemon/keychain_proxy.py
index dabc06fdcd0f..e4a55caf564d 100644
--- a/chia/daemon/keychain_proxy.py
+++ b/chia/daemon/keychain_proxy.py
@@ -50,8 +50,9 @@ def __init__(
         local_keychain: Optional[Keychain] = None,
         user: Optional[str] = None,
         service: Optional[str] = None,
+        heartbeat: int = 300,
     ):
-        super().__init__(uri, ssl_context)
+        super().__init__(uri, ssl_context, heartbeat=heartbeat)
         self.log = log
         if local_keychain:
             self.keychain = local_keychain
@@ -109,7 +110,7 @@ async def connect_to_keychain(self) -> None:
                     self._uri,
                     autoclose=True,
                     autoping=True,
-                    heartbeat=60,
+                    heartbeat=self.heartbeat,
                     ssl_context=self.ssl_context,
                     max_msg_size=self.max_message_size,
                 )
@@ -358,6 +359,7 @@ def wrap_local_keychain(keychain: Keychain, log: logging.Logger) -> KeychainProx
 async def connect_to_keychain(
     self_hostname: str,
     daemon_port: int,
+    daemon_heartbeat: int,
     ssl_context: Optional[ssl.SSLContext],
     log: logging.Logger,
     user: Optional[str] = None,
@@ -368,7 +370,12 @@ async def connect_to_keychain(
     """
 
     client = KeychainProxy(
-        uri=f"wss://{self_hostname}:{daemon_port}", ssl_context=ssl_context, log=log, user=user, service=service
+        uri=f"wss://{self_hostname}:{daemon_port}",
+        heartbeat=daemon_heartbeat,
+        ssl_context=ssl_context,
+        log=log,
+        user=user,
+        service=service,
     )
     # Connect to the service if the proxy isn't using a local keychain
     if not client.use_local_keychain():
@@ -393,8 +400,9 @@ async def connect_to_keychain_and_validate(
         ca_crt_path = root_path / net_config["private_ssl_ca"]["crt"]
         ca_key_path = root_path / net_config["private_ssl_ca"]["key"]
         ssl_context = ssl_context_for_client(ca_crt_path, ca_key_path, crt_path, key_path, log=log)
+        daemon_heartbeat = net_config.get("daemon_heartbeat", 300)
         connection = await connect_to_keychain(
-            net_config["self_hostname"], net_config["daemon_port"], ssl_context, log, user, service
+            net_config["self_hostname"], net_config["daemon_port"], daemon_heartbeat, ssl_context, log, user, service
         )
 
         # If proxying to a local keychain, don't attempt to ping
diff --git a/chia/daemon/server.py b/chia/daemon/server.py
index af1bcae89616..b1aefe3df3f1 100644
--- a/chia/daemon/server.py
+++ b/chia/daemon/server.py
@@ -144,6 +144,7 @@ def __init__(
         self.self_hostname = self.net_config["self_hostname"]
         self.daemon_port = self.net_config["daemon_port"]
         self.daemon_max_message_size = self.net_config.get("daemon_max_message_size", 50 * 1000 * 1000)
+        self.heartbeat = self.net_config.get("daemon_heartbeat", 300)
         self.webserver: Optional[WebServer] = None
         self.ssl_context = ssl_context_for_server(ca_crt_path, ca_key_path, crt_path, key_path, log=self.log)
         self.keychain_server = KeychainServer()
@@ -219,7 +220,9 @@ async def stop(self) -> Dict[str, Any]:
         return {"success": True, "services_stopped": service_names}
 
     async def incoming_connection(self, request):
-        ws: WebSocketResponse = web.WebSocketResponse(max_msg_size=self.daemon_max_message_size, heartbeat=30)
+        ws: WebSocketResponse = web.WebSocketResponse(
+            max_msg_size=self.daemon_max_message_size, heartbeat=self.heartbeat
+        )
         await ws.prepare(request)
 
         while True:
diff --git a/chia/rpc/rpc_server.py b/chia/rpc/rpc_server.py
index ff0ad194dc7f..ba5c3ff16fa4 100644
--- a/chia/rpc/rpc_server.py
+++ b/chia/rpc/rpc_server.py
@@ -147,6 +147,7 @@ class RpcServer:
     ssl_context: SSLContext
     ssl_client_context: SSLContext
     webserver: Optional[WebServer] = None
+    daemon_heartbeat: int = 300
     daemon_connection_task: Optional[asyncio.Task[None]] = None
     shut_down: bool = False
     websocket: Optional[ClientWebSocketResponse] = None
@@ -160,9 +161,10 @@ def create(
         key_path = root_path / net_config["daemon_ssl"]["private_key"]
         ca_cert_path = root_path / net_config["private_ssl_ca"]["crt"]
         ca_key_path = root_path / net_config["private_ssl_ca"]["key"]
+        daemon_heartbeat = net_config.get("daemon_heartbeat", 300)
         ssl_context = ssl_context_for_server(ca_cert_path, ca_key_path, crt_path, key_path, log=log)
         ssl_client_context = ssl_context_for_client(ca_cert_path, ca_key_path, crt_path, key_path, log=log)
-        return cls(rpc_api, stop_cb, service_name, ssl_context, ssl_client_context)
+        return cls(rpc_api, stop_cb, service_name, ssl_context, ssl_client_context, daemon_heartbeat=daemon_heartbeat)
 
     async def start(self, self_hostname: str, rpc_port: uint16, max_request_body_size: int, prefer_ipv6: bool) -> None:
         if self.webserver is not None:
@@ -373,7 +375,7 @@ async def inner() -> None:
                         f"wss://{self_hostname}:{daemon_port}",
                         autoclose=True,
                         autoping=True,
-                        heartbeat=60,
+                        heartbeat=self.daemon_heartbeat,
                         ssl_context=self.ssl_client_context,
                         max_msg_size=max_message_size,
                     )
diff --git a/chia/util/initial-config.yaml b/chia/util/initial-config.yaml
index f46bb6dd4e11..fdde119cbbe1 100644
--- a/chia/util/initial-config.yaml
+++ b/chia/util/initial-config.yaml
@@ -6,6 +6,7 @@ self_hostname: &self_hostname "localhost"
 prefer_ipv6: False
 daemon_port: 55400
 daemon_max_message_size: 50000000 # maximum size of RPC message in bytes
+daemon_heartbeat: 300 # sets the heartbeat for ping/ping interval and timeouts
 inbound_rate_limit_percent: 100
 outbound_rate_limit_percent: 30
 
diff --git a/tests/conftest.py b/tests/conftest.py
index 336d08861089..875059283ce9 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -620,7 +620,6 @@ async def daemon_connection_and_temp_keychain(get_b_tools):
                 f"wss://127.0.0.1:{get_b_tools._config['daemon_port']}",
                 autoclose=True,
                 autoping=True,
-                heartbeat=60,
                 ssl=get_b_tools.get_daemon_ssl_context(),
                 max_msg_size=52428800,
             ) as ws:
diff --git a/tests/core/daemon/test_daemon.py b/tests/core/daemon/test_daemon.py
index c2803d9d3cd5..3bf6fb621b99 100644
--- a/tests/core/daemon/test_daemon.py
+++ b/tests/core/daemon/test_daemon.py
@@ -207,7 +207,6 @@ async def num_connections():
         f"wss://127.0.0.1:{daemon1.daemon_port}",
         autoclose=True,
         autoping=True,
-        heartbeat=60,
         ssl_context=get_b_tools.get_daemon_ssl_context(),
         max_msg_size=100 * 1024 * 1024,
     )
diff --git a/tests/core/server/test_dos.py b/tests/core/server/test_dos.py
index 74dec0ad78b2..0ee6b0a959a3 100644
--- a/tests/core/server/test_dos.py
+++ b/tests/core/server/test_dos.py
@@ -49,14 +49,14 @@ async def test_large_message_disconnect_and_ban(self, setup_two_nodes_fixture, s
 
         ssl_context = server_2.ssl_client_context
         ws = await session.ws_connect(
-            url, autoclose=True, autoping=True, heartbeat=60, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
+            url, autoclose=True, autoping=True, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
         )
         assert not ws.closed
         await ws.close()
         assert ws.closed
 
         ws = await session.ws_connect(
-            url, autoclose=True, autoping=True, heartbeat=60, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
+            url, autoclose=True, autoping=True, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
         )
         assert not ws.closed
 
@@ -74,7 +74,7 @@ async def test_large_message_disconnect_and_ban(self, setup_two_nodes_fixture, s
         assert ws.closed
         try:
             ws = await session.ws_connect(
-                url, autoclose=True, autoping=True, heartbeat=60, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
+                url, autoclose=True, autoping=True, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
             )
             response: WSMessage = await ws.receive()
             assert response.type == WSMsgType.CLOSE
@@ -96,7 +96,7 @@ async def test_bad_handshake_and_ban(self, setup_two_nodes_fixture, self_hostnam
 
         ssl_context = server_2.ssl_client_context
         ws = await session.ws_connect(
-            url, autoclose=True, autoping=True, heartbeat=60, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
+            url, autoclose=True, autoping=True, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
         )
         await ws.send_bytes(bytes([1] * 1024))
 
@@ -111,7 +111,7 @@ async def test_bad_handshake_and_ban(self, setup_two_nodes_fixture, self_hostnam
         assert ws.closed
         try:
             ws = await session.ws_connect(
-                url, autoclose=True, autoping=True, heartbeat=60, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
+                url, autoclose=True, autoping=True, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
             )
             response: WSMessage = await ws.receive()
             assert response.type == WSMsgType.CLOSE
@@ -120,9 +120,7 @@ async def test_bad_handshake_and_ban(self, setup_two_nodes_fixture, self_hostnam
         await asyncio.sleep(6)
 
         # Ban expired
-        await session.ws_connect(
-            url, autoclose=True, autoping=True, heartbeat=60, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
-        )
+        await session.ws_connect(url, autoclose=True, autoping=True, ssl=ssl_context, max_msg_size=100 * 1024 * 1024)
 
         await session.close()
 
@@ -140,7 +138,7 @@ async def test_invalid_protocol_handshake(self, setup_two_nodes_fixture, self_ho
 
         ssl_context = server_2.ssl_client_context
         ws = await session.ws_connect(
-            url, autoclose=True, autoping=True, heartbeat=60, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
+            url, autoclose=True, autoping=True, ssl=ssl_context, max_msg_size=100 * 1024 * 1024
         )
 
         # Construct an otherwise valid handshake message
diff --git a/tests/core/test_daemon_rpc.py b/tests/core/test_daemon_rpc.py
index cec32ef33e44..935f0cc40205 100644
--- a/tests/core/test_daemon_rpc.py
+++ b/tests/core/test_daemon_rpc.py
@@ -10,7 +10,7 @@ async def test_get_version_rpc(self, get_daemon, bt):
         ws_server = get_daemon
         config = bt.config
         client = await connect_to_daemon(
-            config["self_hostname"], config["daemon_port"], 50 * 1000 * 1000, bt.get_daemon_ssl_context()
+            config["self_hostname"], config["daemon_port"], 50 * 1000 * 1000, bt.get_daemon_ssl_context(), heartbeat=300
         )
         response = await client.get_version()
 


Code review comments:
diff_hunk: @@ -10,7 +10,7 @@ async def test_get_version_rpc(self, get_daemon, bt):
         ws_server = get_daemon
         config = bt.config
         client = await connect_to_daemon(
-            config["self_hostname"], config["daemon_port"], 50 * 1000 * 1000, bt.get_daemon_ssl_context()
+            config["self_hostname"], config["daemon_port"], 50 * 1000 * 1000, bt.get_daemon_ssl_context(), heartbeat=300
path: tests/core/test_daemon_rpc.py
text: @user1:
May as well?

```suggestion
            config["self_hostname"], config["daemon_port"], 50 * 1000 * 1000, bt.get_daemon_ssl_context(), heartbeat=config["daemon_heartbeat"]
```