Repository: hyperium/tonic (Rust)

Issue:
Tonic Transport Errors not mapped to Unavailable
## Bug Report
<!--
Thank you for reporting an issue.

Please fill in as much of the template below as you're able.
-->

### Version
For completeness, all of the potentially relevant deps in our Cargo.toml are:
```toml
[dependencies]
async-stream = "=0.3.1"
futures = "=0.3.14"
futures-util = "=0.3.14"
http = "=0.2.4"
hyper = "=0.14.5"
prost = "=0.7.0"
prost-derive = "=0.7.0"
prost-types = "=0.7.0"
prost-build = "=0.7.0"
ring = "=0.16.20"
rustls = { version = "=0.19.0", features = ["default", "dangerous_configuration"] }
rustls-native-certs = "=0.5.0"
tokio = { version = "=1.5.0", features = ["full"] }
tokio-stream = "=0.1.5"
tokio-util = "=0.6.6"
tonic = { version = "=0.4.2", features = ["default", "tls", "tls-roots", "transport"] }
tonic-build = "=0.4.2"
tower = "=0.4.6"
```

### Platform
> Linux <hostname> 5.8.0-38-generic #43~20.04.1-Ubuntu SMP Tue Jan 12 16:39:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

### Description
We were doing robustness testing of our connections.  We start with a basic `tonic::transport::let channel = tonic::transport::Channel::from_static("http://some-endpoint.com").connect_lazy()`.  We will either have the network disconnected at the start, or disconnect it during the use of the client.  We've seen a copy different errors related to the underlying connection, but they all show up as transport errors.

> Failed to upload file "4": Tonic gRPC Error: status: Unknown, message: "transport error: connection error: broken pipe", details: [], metadata: MetadataMap { headers: {} }

> Failed to upload file "4": Tonic gRPC Error: status: Unknown, message: "transport error: error trying to connect: dns error: failed to lookup address information: Name or service not known", details: [], metadata: MetadataMap { headers: {} }

I believe Unavailable is the more correct code here, so that the client knows it's valid to retry.  I'm relatively sure that is consistent with gRPC's implementation.  It's been a little bit, so I'll need to put together a specific example.


Discussion:
Perhaps tonic::transport::Error could be handled, like TimeoutExpired?
https://github.com/hyperium/tonic/blob/9ff4f7b8e418278a923a86bb925e3f3e189ca7e0/tonic/src/status.rs#L329-L338
Could any of errors actually be fixed by retrying? If so then yeah it does seem appropriate. Though I'm no expert on the spec.
Yeah.  So what we did for testing was:
1. Unplug the network cable
2. Construct the client lazily
3. Attempt to make a unary call -> Unknown dns error right away  (This step can be repeated many times with the same result)
4. Reconnect network cable
5. Attempt to make a unary call -> succeeds

Though, because of how Connection and Reconnect interact, the behavior is slightly different depending on if things have succeeded previously or not 
1. Construct the client lazily
2. Attempt to make a unary call -> succeeds
3. Unplug the network cable
4. Attempt to make a unary call -> hangs or succeeds depending on length of time between this step and next
5. Reconnect network cable
6. Previous unary call errors out (assuming step 4 hung)
7. Attempt to make a unary call -> succeeds

Step 4 won't hang indefinitely if [timeout](https://docs.rs/tonic/0.4.2/tonic/transport/channel/struct.Endpoint.html#method.timeout) is set.  In that case it returns a:
> Tonic gRPC Error: status: Unknown, message: "transport error: request timed out", details: [], metadata: MetadataMap { headers: {} }

I put up #629 as a draft approach to this.



Patch:
diff --git a/tonic/src/status.rs b/tonic/src/status.rs
index e4e1b32a2..ee560c9f3 100644
--- a/tonic/src/status.rs
+++ b/tonic/src/status.rs
@@ -330,7 +330,8 @@ impl Status {
             Err(err) => err,
         };
 
-        if let Some(status) = find_status_in_source_chain(&*err) {
+        if let Some(mut status) = find_status_in_source_chain(&*err) {
+            status.source = Some(err);
             return Ok(status);
         }
 
@@ -377,6 +378,31 @@ impl Status {
         reason.into()
     }
 
+    /// Handles hyper errors specifically, which expose a number of different parameters about the
+    /// http stream's error: https://docs.rs/hyper/0.14.11/hyper/struct.Error.html.
+    ///
+    /// Returns Some if there's a way to handle the error, or None if the information from this
+    /// hyper error, but perhaps not its source, should be ignored.
+    #[cfg(feature = "transport")]
+    fn from_hyper_error(err: &hyper::Error) -> Option<Status> {
+        // is_timeout results from hyper's keep-alive logic
+        // (https://docs.rs/hyper/0.14.11/src/hyper/error.rs.html#192-194).  Per the grpc spec
+        // > An expired client initiated PING will cause all calls to be closed with an UNAVAILABLE
+        // > status. Note that the frequency of PINGs is highly dependent on the network
+        // > environment, implementations are free to adjust PING frequency based on network and
+        // > application requirements, which is why it's mapped to unavailable here.
+        //
+        // Likewise, if we are unable to connect to the server, map this to UNAVAILABLE.  This is
+        // consistent with the behavior of a C++ gRPC client when the server is not running, and
+        // matches the spec of:
+        // > The service is currently unavailable. This is most likely a transient condition that
+        // > can be corrected if retried with a backoff.
+        if err.is_timeout() || err.is_connect() {
+            return Some(Status::unavailable(err.to_string()));
+        }
+        None
+    }
+
     pub(crate) fn map_error<E>(err: E) -> Status
     where
         E: Into<Box<dyn Error + Send + Sync>>,
@@ -556,11 +582,20 @@ fn find_status_in_source_chain(err: &(dyn Error + 'static)) -> Option<Status> {
             return Some(Status::cancelled(timeout.to_string()));
         }
 
+        #[cfg(feature = "transport")]
+        if let Some(hyper) = err
+            .downcast_ref::<hyper::Error>()
+            .and_then(Status::from_hyper_error)
+        {
+            return Some(hyper);
+        }
+
         source = err.source();
     }
 
     None
 }
+
 impl fmt::Debug for Status {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         // A manual impl to reduce the noise of frequently empty fields.


Test patch:
diff --git a/tests/integration_tests/tests/connection.rs b/tests/integration_tests/tests/connection.rs
index c345a35d9..27294278b 100644
--- a/tests/integration_tests/tests/connection.rs
+++ b/tests/integration_tests/tests/connection.rs
@@ -5,7 +5,7 @@ use std::time::Duration;
 use tokio::sync::oneshot;
 use tonic::{
     transport::{Endpoint, Server},
-    Request, Response, Status,
+    Code, Request, Response, Status,
 };
 
 struct Svc(Arc<Mutex<Option<oneshot::Sender<()>>>>);
@@ -52,7 +52,8 @@ async fn connect_returns_err_via_call_after_connected() {
 
     let res = client.unary_call(Request::new(Input {})).await;
 
-    assert!(res.is_err());
+    let err = res.unwrap_err();
+    assert_eq!(err.code(), Code::Unavailable);
 
     jh.await.unwrap();
 }
@@ -86,7 +87,9 @@ async fn connect_lazy_reconnects_after_first_failure() {
 
     // The server shut down, third call should fail
     tokio::time::sleep(Duration::from_millis(100)).await;
-    client.unary_call(Request::new(Input {})).await.unwrap_err();
+    let err = client.unary_call(Request::new(Input {})).await.unwrap_err();
+
+    assert_eq!(err.code(), Code::Unavailable);
 
     jh.await.unwrap();
 }