Repository: awslabs/aws-lambda-rust-runtime (language: rust)

Issue:
thread 'main' panicked at 'lambda runtime failed: Error("failed to convert header to a str", line: 0, column: 0)'
We use the [AWS Lambda Web Adapter](https://github.com/awslabs/aws-lambda-web-adapter) in combination with a Docker container image to resize images on the fly. This works great, but occasionally (though very rarely) we observe the following error in CloudWatch:

```
thread 'main' panicked at 'lambda runtime failed: Error("failed to convert header to a str", line: 0, column: 0)', src/main.rs:25:25
```

![image](https://github.com/awslabs/aws-lambda-rust-runtime/assets/4801917/7b2e8a14-f285-4210-9c59-5e91f0ac7a2c)

Any pointers on how we might debug this would be appreciated!


Discussion / hints:
if you set `RUST_LOG=trace` in the environment, you'll be able to see the payload that the function receives in the log. These kind of errors are caused because the runtime cannot deserialize they payload into the struct that your function receives.
Actually, now that I look at it, it might be something caused by Lambda sending the runtime an unexpected internal value 🤔 

I might take a look at surfacing all that information better.
I cannot find anything in the runtime that could have thrown that panic.
Thanks for your comments.

So you're suggestion would be to set `RUST_LOG=trace` for our Lambda function and check the payload the function receives when these errors occur? 
Is there anything performance related we should be aware of when enabling the tracing log level? Or does this just increase the log volume?
It just increases the log volume, there is no difference performance wise. We don't print payloads be default because there might be sensitive information in them, and we don't want to be responsible for what your cloudwatch logs include.
Had to wait for almost 12 hours until another event occurred. I exported the CloudWatch logs to a gist https://gist.github.com/andreas-venturini/d68aa1266795b17f2ec623b16757e622

I anonymized the user IP and our AWS account id and replaced the Lambda domain name w/ XXX, other than that the logs are unchanged.

We have a CloudFront origin group w/ automatic failover to GCP Cloud Run (as we're still trialing Lambda for our use case) and Cloud Run was able to process the request without error (only difference is that Cloud Run uses x86_64 architecture). 

So it seems to me the problem is either caused by the ARM Docker container service or the Lambda Rust runtime/AWS Lambda web adapter?
I can get this to fail consistently w/ Lambda.

Maybe the UTF char in the following header trips up the Rust runtime @calavera 

```
Schillers sch\\xc3\\xb6nste Szenenanweisungen -Kabale und Liebe.mp4.avif
```
@andreas-venturini I will take a look. Could you share the version of Lambda Web Adapter you are using? 
@bnusunny we are using [v0.7.2](https://github.com/awslabs/aws-lambda-web-adapter/releases/tag/v0.7.2)
It might be related to this: https://github.com/awslabs/aws-lambda-rust-runtime/issues/509

See this specific comment: https://github.com/awslabs/aws-lambda-rust-runtime/issues/509#issuecomment-1279128551
@andreas-venturini I couldn't reproduce the error with that filename. 

`.header('content-disposition', "inline; filename=\"Schillers sch\\xc3\\xb6nste Szenenanweisungen -Kabale und Liebe.mp4.avif\"")`

Could you share the original filename? It would be best to share the original file if possible. 


@bnusunny here is a [link to the source file](https://filmmakers-eu-west-1.s3.eu-west-1.amazonaws.com/production/nfs4/media/workspace/53262/Schillers%20sch%C3%B6nste%20Szenenanweisungen%20-Kabale%20und%20Liebe.mp4.jpg?response-content-disposition=inline&X-Amz-Security-Token=IQoJb3JpZ2luX2VjENP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDGV1LWNlbnRyYWwtMSJIMEYCIQCm3%2BMQOUNYdJjGwO4ToEAB07jRgGeJyWm%2FM1L4GiKc5gIhANWlz6lSnQdP%2B%2BGYO2%2BPXxU9OYSOch0n2SvZ8ImFP7IyKoUECIz%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQARoMNzgyMTU0NTQ5NjQ0IgxDfHZpaYRQDBKYAuoq2QM8xEet8ElgLIuxYBS0ft9mmkLBnEcXmwhIlogNA1P1Qh2O2X2Jq3BcZIa7kOgaqsF0r8AizOA93NG5Oe7lkStOEuETqPf2MWwlfaOI2OEftUXm0SE4PlhU%2FWftHK077F9Epd3xlw%2FPKLvL5KfpWHp2fznFuMp6NGUjAtXJl%2F4hKD38x6QFL1zIkZPUlAwwTcub8iGvX7bhyCy7jhP6%2F%2B7%2FNSL%2FrxVA0zoVi0R76u3iKCLiqObCxTINzTM6%2BGm9KrT86TrN9tY6GwiwtO88Ry7n7S5FqMVsUGhaVyXfmfgFRT1IoMR%2Fy%2F8dzsfldyLbiqVXodkQdIS0X%2BSIeE35AMrMrLzx6hxl1EderOqF1A%2Fqjd4dNUhX%2Be0GLcAPHdd8Y%2Bm3PG%2BptujrAwiArLX3Gn4dIlDyMqCgyfTIWXhUvbJozZs1UCvJdnS6wVF%2BlCeWagdlcwtxpCzhr9J%2BtRVCFKaI7O3Eo41z0swfzRx4ekMT3inWkVxfAeYY6Co%2BkbxTZjDhq%2BYyFM%2B5TArPO4DwAeX5N6ABV02nX118glEq03oaeIYy%2Flt9C%2FsWeTwcesGgExewH2xJDW8QXh%2FFR%2FHitLzz2QP2kHpJU%2BktExT9Fc6zd%2FhhxsB%2BxAgHITDn58itBjqTAifap2B%2Bb3eJ8vgTIgXf0fuNX3aL5PY8MHVv7zUcIotkgxozeupaaaLLajcthT3dY%2Fyo6V4QG48paJg6NOhoNJZV3WHRefIKHu4m3P6NaFomB0VYavUyEGhMyDKKxHF2HkOL4O8m2YDHU%2FJtHrr%2Fl%2F5lYcYaP%2BNx3X448zN%2BKSrMHWMJZibsTh2AlDEM6BfVMq8Z%2F3DI64rRIKWwPrpyl21LeeFC%2F8XRSJ%2Fpm5lx9skyIsPmQZeVo3OG8QdpjROZPQekjEes5ogvjoI9sFiMNunsppayssAx%2FcWIlmgOphXT%2Bm7jj4H84bShtuAktA54c59tO8HD34ud8U3Ex1av5oUTPnRfAGqIlXsBohnO0LSA8%2Fj1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240125T103629Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43200&X-Amz-Credential=ASIA3MHALJGGADEET6WB%2F20240125%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Signature=0eac1b07f7712173bdf487dd83df371bde24fe5c58053cfe34d1bb43aea29e2b) (12 hours valid) 
Got it. Thanks!
I reproduced the issue with `BUFFERED` invoke mode. But it works with `RESPONSE_STREAM` invoke mode. I will dig deeper. 
The issue happend at [this line](https://github.com/awslabs/aws-lambda-rust-runtime/blob/9f8f2500c99b45f50a4ea866851c484485bce5a2/lambda-runtime/src/requests.rs#L121). 

```rust
let body = serde_json::to_vec(&body)?;
```

Runtime uses `serde_json::to_vec()` to serialize the response object to a JSON byte vector. The filename in the `content-disposition` header contains non UTF-8 charactores. So the operation failed. 

@calavera any suggestions on how to handle non UTF-8 charactores in the headers? 


Nope. Serde just doesn't support it, as explained in https://github.com/awslabs/aws-lambda-rust-runtime/issues/509#issuecomment-1279128551
@andreas-venturini Can you switch on response streaming mode? It should work. I need to figure out why. 
Wait, `Schillers schönste Szenenanweisungen -Kabale und Liebe.mp4.avif` is actually valid UTF-8. I need to dive deeper. 

Fix patch:
diff --git a/lambda-events/src/custom_serde/headers.rs b/lambda-events/src/custom_serde/headers.rs
index 9cb89e40..2ef3050e 100644
--- a/lambda-events/src/custom_serde/headers.rs
+++ b/lambda-events/src/custom_serde/headers.rs
@@ -13,7 +13,7 @@ where
     for key in headers.keys() {
         let mut map_values = Vec::new();
         for value in headers.get_all(key) {
-            map_values.push(value.to_str().map_err(S::Error::custom)?)
+            map_values.push(String::from_utf8(value.as_bytes().to_vec()).map_err(S::Error::custom)?)
         }
         map.serialize_entry(key.as_str(), &map_values)?;
     }
@@ -27,8 +27,8 @@ where
 {
     let mut map = serializer.serialize_map(Some(headers.keys_len()))?;
     for key in headers.keys() {
-        let map_value = headers[key].to_str().map_err(S::Error::custom)?;
-        map.serialize_entry(key.as_str(), map_value)?;
+        let map_value = String::from_utf8(headers[key].as_bytes().to_vec()).map_err(S::Error::custom)?;
+        map.serialize_entry(key.as_str(), &map_value)?;
     }
     map.end()
 }
@@ -187,4 +187,42 @@ mod tests {
         let decoded: Test = serde_json::from_value(data).unwrap();
         assert!(decoded.headers.is_empty());
     }
+
+    #[test]
+    fn test_serialize_utf8_headers() {
+        #[derive(Deserialize, Serialize)]
+        struct Test {
+            #[serde(deserialize_with = "deserialize_headers", default)]
+            #[serde(serialize_with = "serialize_headers")]
+            pub headers: HeaderMap,
+            #[serde(deserialize_with = "deserialize_headers", default)]
+            #[serde(serialize_with = "serialize_multi_value_headers")]
+            pub multi_value_headers: HeaderMap,
+        }
+
+        let content_disposition =
+            "inline; filename=\"Schillers schönste Szenenanweisungen -Kabale und Liebe.mp4.avif\"";
+        let data = serde_json::json!({
+            "headers": {
+                "Content-Disposition": content_disposition
+            },
+            "multi_value_headers": {
+                "Content-Disposition": content_disposition
+            }
+        });
+        let decoded: Test = serde_json::from_value(data).unwrap();
+        assert_eq!(content_disposition, decoded.headers.get("Content-Disposition").unwrap());
+        assert_eq!(
+            content_disposition,
+            decoded.multi_value_headers.get("Content-Disposition").unwrap()
+        );
+
+        let recoded = serde_json::to_value(decoded).unwrap();
+        let decoded: Test = serde_json::from_value(recoded).unwrap();
+        assert_eq!(content_disposition, decoded.headers.get("Content-Disposition").unwrap());
+        assert_eq!(
+            content_disposition,
+            decoded.multi_value_headers.get("Content-Disposition").unwrap()
+        );
+    }
 }


Test patch: