Repository: Azure/iot-operations-sdks
Pull request #676: [.NET + ADR] Error modeling in headers
Author: login: timtay-microsoft
Created: 2025-04-03T22:06:37Z | Merged: 2025-04-30T17:49:54Z



Diff:
diff --git a/doc/dev/adr/0021-error-modeling-headers.md b/doc/dev/adr/0021-error-modeling-headers.md
new file mode 100644
index 0000000000..03b80b9442
--- /dev/null
+++ b/doc/dev/adr/0021-error-modeling-headers.md
@@ -0,0 +1,99 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications must route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined string (that is conventionally, but not necessarily, a stringified JSON object/value/array) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling the error codes in DTDL. This modeling will be detailed in a separate ADR.
+
+## Code Example
+
+### Command executor side
+
+```csharp
+public ExtendedResponse<IncrementResponsePayload> Increment(IncrementRequestPayload request)
+{
+    if (request.IncrementValue < 0)
+    {
+        var response = new ExtendedResponse<IncrementResponsePayload>()
+        {
+            Response = new IncrementResponsePayload { CounterResponse = _counter },
+        };
+
+        // Specify error code, but no error payload  
+        response.WithApplicationError("negativeValue");
+
+        // Or you can specify error code and error payload. The error payload can be any JSON primitive or a JSON object or a JSON array
+        JsonNode? jsonPayload = JsonArray.Parse("[\"1\",\"2\"]");
+        jsonPayload = JsonObject.Parse("{\"key\":\"value\"}");
+        jsonPayload = JsonValue.Create(false);
+        jsonPayload = JsonValue.Create(10);
+        response.WithApplicationError("negativeValue", jsonPayload.toString());
+
+        return response;
+    }
+
+    // happy path omitted
+}
+```
+
+### Command invoker side
+
+```csharp
+public void main()
+{
+    MqttSessionClient mqttClient = ...;();
+    CounterClient counterClient = new CounterClient(mqttClient);
+
+    IncrementRequestPayload payload = new IncrementRequestPayload
+    {
+        IncrementValue = -1
+    };
+
+    var response = counterClient.Increment(executorId, payload).WithMetadata();
+    
+    // Check the RPC response for an application error
+    if (response.TryGetApplicationError(out string? errorCode, out string? errorPayload))
+    {
+        // use the error code and strongly typed custom error payload type as wanted.
+        JsonObject errorPayloadObject = JsonObject.Parse(errorPayload);
+    }
+
+    // Alternatively just fetch the error code if you don't want to deserialize the payload yet.
+    if (response.TryGetApplicationError(out string? errorCode))
+    {
+        // ...
+    }
+}
+```
+
+### Enforcement
+
+Note that, while this decision creates a standardized way of communicating application errors, our SDK will **not** enforce that users communicate application errors with this standard. Users will still be able to model application errors in response payloads or in custom user properties if they prefer. There is no way to force users to communicate errors with this pattern.
+
+## Samples
+
+We want to demonstrate this pattern in at least one sample per-language.
+
+For the sake of demonstrating this in all languages similarly, we will make the counter service return an application error if the invocation specifies a negative value to increment by. This change won't include any DTDL level changes yet.
+
+## Extra considerations
+
+- This ADR does not change the decision made in (ADR 19)[./0019-codegen-user-errs.md] which allows users to model errors in payloads.
+   - For applications where users could model errors in either headers or the payload, we should encourage them to model them in the payload.
+
+## Open Questions
+
+- MQTT user property value size limit considerations? Users may provide very large payload objects and MQTT as a protocol may not be built for that?
\ No newline at end of file
diff --git a/dotnet/samples/Protocol/Codegen/SampleClient/RpcCommandRunner.cs b/dotnet/samples/Protocol/Codegen/SampleClient/RpcCommandRunner.cs
index dfbc079c25..c879709c53 100644
--- a/dotnet/samples/Protocol/Codegen/SampleClient/RpcCommandRunner.cs
+++ b/dotnet/samples/Protocol/Codegen/SampleClient/RpcCommandRunner.cs
@@ -8,6 +8,9 @@
 using TestEnvoys.Math;
 using TestEnvoys.Greeter;
 using TestEnvoys.CustomTopicTokens;
+using SampleServer;
+using System.Text.Json;
+using System.Text.Json.Nodes;
 
 namespace SampleClient;
 
@@ -124,7 +127,7 @@ private async Task RunCounterCommands(string executorId)
         {
 
             CommandRequestMetadata reqMd = new();
-
+            
             logger.LogInformation("Calling ReadCounter with {c}", reqMd.CorrelationId);
             ExtendedResponse<ReadCounterResponsePayload> respCounter = await counterClient.ReadCounterAsync(executorId, reqMd).WithMetadata();
             logger.LogInformation("called read {c} with id {id}", respCounter.Response!.CounterResponse, respCounter.ResponseMetadata!.CorrelationId);
@@ -154,6 +157,20 @@ private async Task RunCounterCommands(string executorId)
             ExtendedResponse<ReadCounterResponsePayload> respCounter4 = await counterClient.ReadCounterAsync(executorId).WithMetadata();
             logger.LogInformation("counter {c} with id {id}", respCounter4.Response!.CounterResponse, respCounter4.ResponseMetadata!.CorrelationId);
 
+            // This increment request will deliberately trigger an application level error in the counter service for demonstration purposes
+            IncrementRequestPayload invalidPayload = new IncrementRequestPayload
+            {
+                IncrementValue = -1
+            };
+
+            ExtendedResponse<IncrementResponsePayload> responseWithApplicationError = await counterClient.IncrementAsync(executorId, invalidPayload).WithMetadata();
+
+            // Upon receiving a response, you should check for any application level errors like this. Only the above Increment request should trigger an application error in this sample, though.
+            if (responseWithApplicationError.TryGetApplicationError(out string? errorCode, out string? errorPayloadJson) && errorPayloadJson != null)
+            {
+                var applicationErrorPayload = JsonSerializer.Deserialize<CounterServiceApplicationError>(errorPayloadJson);
+                logger.LogInformation("counter {c} with id {id} responded to an invalid request with application level error code {code} and payload {payload}", responseWithApplicationError!.Response.CounterResponse, responseWithApplicationError.ResponseMetadata!.CorrelationId, errorCode, errorPayloadJson);
+            }
         }
         catch (Exception ex)
         {
diff --git a/dotnet/samples/Protocol/Codegen/SampleClient/SampleClient.csproj b/dotnet/samples/Protocol/Codegen/SampleClient/SampleClient.csproj
index b11a1393e7..e0dbfdd64e 100644
--- a/dotnet/samples/Protocol/Codegen/SampleClient/SampleClient.csproj
+++ b/dotnet/samples/Protocol/Codegen/SampleClient/SampleClient.csproj
@@ -13,6 +13,7 @@
 
 	<ItemGroup>
 	  <ProjectReference Include="..\..\TestEnvoys\TestEnvoys.csproj" />
+	  <ProjectReference Include="..\SampleServer\SampleServer.csproj" />
 	</ItemGroup>
 
 </Project>
diff --git a/dotnet/samples/Protocol/Codegen/SampleServer/CounterService.cs b/dotnet/samples/Protocol/Codegen/SampleServer/CounterService.cs
index ab990e7155..643d809f5d 100644
--- a/dotnet/samples/Protocol/Codegen/SampleServer/CounterService.cs
+++ b/dotnet/samples/Protocol/Codegen/SampleServer/CounterService.cs
@@ -5,6 +5,8 @@
 using Azure.Iot.Operations.Mqtt.Session;
 using TestEnvoys.Counter;
 using Azure.Iot.Operations.Protocol;
+using System.Text.Json.Nodes;
+using System.Text.Json;
 
 namespace SampleServer;
 
@@ -17,6 +19,21 @@ public CounterService(ApplicationContext applicationContext, MqttSessionClient m
     public override Task<ExtendedResponse<IncrementResponsePayload>> IncrementAsync(IncrementRequestPayload request, CommandRequestMetadata requestMetadata, CancellationToken cancellationToken)
     {
         Console.WriteLine($"--> Executing Counter.Increment with id {requestMetadata.CorrelationId} for {requestMetadata.InvokerClientId}");
+
+        if (request.IncrementValue < 0)
+        {
+            var response =
+                new ExtendedResponse<IncrementResponsePayload>()
+                {
+                    Response = new IncrementResponsePayload { CounterResponse = _counter },
+                }
+                .WithApplicationError(
+                    "negativeValue",
+                    JsonSerializer.Serialize(new CounterServiceApplicationError() { InvalidRequestArgumentValue = request.IncrementValue }));
+
+            return Task.FromResult(response);
+        }
+
         Interlocked.Add(ref _counter, request.IncrementValue);
         Console.WriteLine($"--> Executed Counter.Increment with id {requestMetadata.CorrelationId} for {requestMetadata.InvokerClientId}");
         return Task.FromResult(new ExtendedResponse<IncrementResponsePayload>
diff --git a/dotnet/samples/Protocol/Codegen/SampleServer/CounterServiceApplicationError.cs b/dotnet/samples/Protocol/Codegen/SampleServer/CounterServiceApplicationError.cs
new file mode 100644
index 0000000000..044df25bc4
--- /dev/null
+++ b/dotnet/samples/Protocol/Codegen/SampleServer/CounterServiceApplicationError.cs
@@ -0,0 +1,16 @@
+﻿// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+
+using System.Text.Json.Serialization;
+
+namespace SampleServer
+{
+    public class CounterServiceApplicationError
+    {
+        [JsonPropertyName("InvalidRequestArgumentValue")]
+        [JsonIgnore(Condition = JsonIgnoreCondition.Never)]
+        [JsonRequired]
+        public int InvalidRequestArgumentValue { get; set; } = default!;
+    }
+}
diff --git a/dotnet/src/Azure.Iot.Operations.Protocol/RPC/CommandExecutor.cs b/dotnet/src/Azure.Iot.Operations.Protocol/RPC/CommandExecutor.cs
index 4da46ad3bb..bf1d9c75e5 100644
--- a/dotnet/src/Azure.Iot.Operations.Protocol/RPC/CommandExecutor.cs
+++ b/dotnet/src/Azure.Iot.Operations.Protocol/RPC/CommandExecutor.cs
@@ -8,7 +8,6 @@
 using System.Collections.Generic;
 using System.Diagnostics;
 using System.Globalization;
-using System.IO;
 using System.Linq;
 using System.Text;
 using System.Threading;
@@ -29,6 +28,7 @@ public abstract class CommandExecutor<TReq, TResp> : IAsyncDisposable
 
         private readonly IMqttPubSubClient _mqttClient;
         private readonly string _commandName;
+
         private readonly IPayloadSerializer _serializer;
 
         private readonly ApplicationContext _applicationContext;
diff --git a/dotnet/src/Azure.Iot.Operations.Protocol/RPC/CommandResponseMetadata.cs b/dotnet/src/Azure.Iot.Operations.Protocol/RPC/CommandResponseMetadata.cs
index 0859e10ce1..496b024695 100644
--- a/dotnet/src/Azure.Iot.Operations.Protocol/RPC/CommandResponseMetadata.cs
+++ b/dotnet/src/Azure.Iot.Operations.Protocol/RPC/CommandResponseMetadata.cs
@@ -65,11 +65,6 @@ public CommandResponseMetadata()
             UserData = [];
         }
 
-        //// Constructor for testing
-        //internal CommandResponseMetadata(HybridLogicalClock timestamp)
-        //{
-        //    Timestamp = timestamp;
-        //}
         internal CommandResponseMetadata(MqttApplicationMessage message)
         {
             CorrelationId = message.CorrelationData != null && GuidExtensions.TryParseBytes(message.CorrelationData, out Guid? correlationId)
diff --git a/dotnet/src/Azure.Iot.Operations.Protocol/RPC/ExtendedResponse.cs b/dotnet/src/Azure.Iot.Operations.Protocol/RPC/ExtendedResponse.cs
index 69ddcccab1..b38907b249 100644
--- a/dotnet/src/Azure.Iot.Operations.Protocol/RPC/ExtendedResponse.cs
+++ b/dotnet/src/Azure.Iot.Operations.Protocol/RPC/ExtendedResponse.cs
@@ -1,24 +1,84 @@
 ﻿// Copyright (c) Microsoft Corporation.
 // Licensed under the MIT License.
 
+using System;
+using System.Text.Json;
+using System.Text.Json.Nodes;
+
 namespace Azure.Iot.Operations.Protocol.RPC
 {
     public struct ExtendedResponse<TResp>
         where TResp : class
     {
+        // These two user properties are used to communicate application level errors in an RPC response message. Code is mandatory, but data is optional.
+        public const string ApplicationErrorCodeUserDataKey = "AppErrCode";
+        public const string ApplicationErrorPayloadUserDataKey = "AppErrPayload";
+
         public TResp Response { get; set; }
 
         public CommandResponseMetadata? ResponseMetadata { get; set; }
 
-#pragma warning disable CA1000 // Do not declare static members on generic types
-        public static ExtendedResponse<TResp> CreateFromResponse(TResp response)
+        public ExtendedResponse<TResp> WithApplicationError(string errorCode)
+        {
+            ResponseMetadata ??= new();
+            SetApplicationError(errorCode, null);
+            return this;
+        }
+
+        public ExtendedResponse<TResp> WithApplicationError(string errorCode, string? errorPayload)
+        {           
+            ResponseMetadata ??= new();
+            SetApplicationError(errorCode, errorPayload);
+            return this;
+        }
+
+        public bool TryGetApplicationError(out string? errorCode, out string? errorPayload)
+        {
+            errorCode = null;
+            errorPayload = null;
+
+            if (ResponseMetadata == null || ResponseMetadata.UserData == null || !ResponseMetadata.UserData.TryGetValue(ApplicationErrorCodeUserDataKey, out string? code) || code == null)
+            {
+                return false;
+            }
+
+            errorCode = code;
+
+            if (ResponseMetadata != null && ResponseMetadata.UserData != null && ResponseMetadata.UserData.TryGetValue(ApplicationErrorPayloadUserDataKey, out string? errorPayloadString) && errorPayloadString != null)
+            {
+                errorPayload = errorPayloadString;
+            }
+
+            return true;
+        }
+
+        public bool IsApplicationError()
         {
-            return new()
+            return ResponseMetadata?.UserData != null && ResponseMetadata != null && ResponseMetadata.UserData.ContainsKey(ApplicationErrorCodeUserDataKey);
+        }
+
+        private void SetApplicationError(string applicationErrorCode, string? errorPayload)
+        {
+            ResponseMetadata ??= new();
+            ResponseMetadata.UserData ??= new();
+            ResponseMetadata.UserData[ApplicationErrorCodeUserDataKey] = applicationErrorCode;
+
+            if (errorPayload != null)
             {
-                Response = response,
-                ResponseMetadata = null,
-            };
+                try
+                {
+                    ResponseMetadata.UserData[ApplicationErrorPayloadUserDataKey] = errorPayload;
+                }
+                catch (Exception e)
+                {
+                    throw new AkriMqttException("Failed to serialize the application error", e)
+                    {
+                        IsRemote = true,
+                        Kind = AkriMqttErrorKind.PayloadInvalid,
+                        IsShallow = false,
+                    };
+                }
+            }
         }
-#pragma warning restore CA1000 // Do not declare static members on generic types
     }
 }
diff --git a/dotnet/test/Azure.Iot.Operations.Protocol.IntegrationTests/CounterEnvoyTests.cs b/dotnet/test/Azure.Iot.Operations.Protocol.IntegrationTests/CounterEnvoyTests.cs
index a1f25034ea..2fb6de2c21 100644
--- a/dotnet/test/Azure.Iot.Operations.Protocol.IntegrationTests/CounterEnvoyTests.cs
+++ b/dotnet/test/Azure.Iot.Operations.Protocol.IntegrationTests/CounterEnvoyTests.cs
@@ -7,6 +7,8 @@
 using Azure.Iot.Operations.Protocol.RPC;
 using TestEnvoys.Counter;
 using System.Diagnostics;
+using System.Text.Json.Nodes;
+using System.Text.Json;
 
 namespace Azure.Iot.Operations.Protocol.IntegrationTests;
 
@@ -121,4 +123,34 @@ public async Task WrongCorrelationIDThrows()
         Assert.Equal("Correlation Data", userProps.Where(p => p.Name == "__propName").First().Value);
     }
 
+    [Fact]
+    public async Task CanReceiveApplicationErrorResponseInHeaders()
+    {
+        ApplicationContext applicationContext = new ApplicationContext();
+        string executorId = "counter-server-" + Guid.NewGuid();
+        await using MqttSessionClient mqttExecutor = await ClientFactory.CreateSessionClientFromEnvAsync(executorId);
+
+        await using CounterService counterService = new CounterService(applicationContext, mqttExecutor);
+        await using MqttSessionClient mqttInvoker = await ClientFactory.CreateSessionClientFromEnvAsync();
+        await using CounterClient counterClient = new CounterClient(applicationContext, mqttInvoker);
+
+        await counterService.StartAsync(null, cancellationToken: CancellationToken.None);
+
+        int expectedNegativeValue = -1;
+        IncrementRequestPayload payload = new IncrementRequestPayload
+        {
+            IncrementValue = expectedNegativeValue
+        };
+
+        var resp = await counterClient.IncrementAsync(executorId, payload, commandTimeout: TimeSpan.FromSeconds(30)).WithMetadata();
+        Assert.Equal(0, resp.Response.CounterResponse);
+        Assert.NotNull(resp.ResponseMetadata);
+        Assert.True(resp.TryGetApplicationError(out string? errorCode, out string? errorPayload));
+        Assert.NotNull(errorCode);
+        Assert.Equal(CounterService.NegativeValueArgumentErrorCode, errorCode);
+        Assert.NotNull(errorPayload);
+        CounterServiceApplicationError? deserializedErrorPayload = JsonSerializer.Deserialize<CounterServiceApplicationError>(errorPayload);
+        Assert.NotNull(deserializedErrorPayload);
+        Assert.Equal(expectedNegativeValue, deserializedErrorPayload.InvalidRequestArgumentValue);
+    }
 }
diff --git a/dotnet/test/Azure.Iot.Operations.Protocol.IntegrationTests/CounterService.cs b/dotnet/test/Azure.Iot.Operations.Protocol.IntegrationTests/CounterService.cs
index 7698f8910b..875fa0631d 100644
--- a/dotnet/test/Azure.Iot.Operations.Protocol.IntegrationTests/CounterService.cs
+++ b/dotnet/test/Azure.Iot.Operations.Protocol.IntegrationTests/CounterService.cs
@@ -3,6 +3,8 @@
 
 using TestEnvoys.Counter;
 using Azure.Iot.Operations.Protocol.RPC;
+using System.Text.Json;
+using System.Text.Json.Nodes;
 
 namespace Azure.Iot.Operations.Protocol.IntegrationTests;
 
@@ -10,6 +12,8 @@ public class CounterService : Counter.Service
 {
     private int _counter = 0;
 
+    public const string NegativeValueArgumentErrorCode = "NegativeValue";
+
     public CounterService(ApplicationContext applicationContext, IMqttPubSubClient mqttClient) : base(applicationContext, mqttClient) 
     {
         ReadCounterCommandExecutor.ExecutionTimeout = TimeSpan.FromSeconds(30);
@@ -19,6 +23,19 @@ public CounterService(ApplicationContext applicationContext, IMqttPubSubClient m
 
     public override Task<ExtendedResponse<IncrementResponsePayload>> IncrementAsync(IncrementRequestPayload request, CommandRequestMetadata requestMetadata, CancellationToken cancellationToken)
     {
+        if (request.IncrementValue < 0)
+        {
+            var response =
+                new ExtendedResponse<IncrementResponsePayload>()
+                {
+                    Response = new IncrementResponsePayload { CounterResponse = _counter },
+                }.WithApplicationError(
+                    NegativeValueArgumentErrorCode,
+                    JsonSerializer.Serialize(new CounterServiceApplicationError() { InvalidRequestArgumentValue = request.IncrementValue }));
+
+            return Task.FromResult(response);
+        }
+
         Console.WriteLine($"--> Executing Counter.Increment with id {requestMetadata.CorrelationId} for {requestMetadata.InvokerClientId}");
         Interlocked.Increment(ref _counter);
         Console.WriteLine($"--> Executed Counter.Increment with id {requestMetadata.CorrelationId} for {requestMetadata.InvokerClientId}");
diff --git a/dotnet/test/Azure.Iot.Operations.Protocol.IntegrationTests/CounterServiceApplicationError.cs b/dotnet/test/Azure.Iot.Operations.Protocol.IntegrationTests/CounterServiceApplicationError.cs
new file mode 100644
index 0000000000..667dd10011
--- /dev/null
+++ b/dotnet/test/Azure.Iot.Operations.Protocol.IntegrationTests/CounterServiceApplicationError.cs
@@ -0,0 +1,16 @@
+﻿// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+
+using System.Text.Json.Serialization;
+
+namespace Azure.Iot.Operations.Protocol.IntegrationTests
+{
+    public class CounterServiceApplicationError
+    {
+        [JsonPropertyName("InvalidRequestArgumentValue")]
+        [JsonIgnore(Condition = JsonIgnoreCondition.Never)]
+        [JsonRequired]
+        public int InvalidRequestArgumentValue { get; set; } = default!;
+    }
+}
diff --git a/dotnet/test/Azure.Iot.Operations.Protocol.MetlTests/CommandExecutorTester.cs b/dotnet/test/Azure.Iot.Operations.Protocol.MetlTests/CommandExecutorTester.cs
index 391625f61b..09fb1766fb 100644
--- a/dotnet/test/Azure.Iot.Operations.Protocol.MetlTests/CommandExecutorTester.cs
+++ b/dotnet/test/Azure.Iot.Operations.Protocol.MetlTests/CommandExecutorTester.cs
@@ -394,7 +394,7 @@ private async Task TestCommandExecutorProtocolAsync(string testCaseName, Concurr
                     {
                         await commandExecutor.TrackAsync().ConfigureAwait(false);
                         string response = await ProcessRequestAsync(extReq, testCaseExecutor, countdownEvents, requestResponseSequencer, ct).ConfigureAwait(false);
-                        return ExtendedResponse<string>.CreateFromResponse(response);
+                        return new ExtendedResponse<string>() { Response = response };
                     });
 
                 await commandExecutor.StartAsync(preferredDispatchConcurrency: testCaseExecutor.ExecutionConcurrency).WaitAsync(TestTimeout).ConfigureAwait(false);


Reviews:
user: login: jrdouceur
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T23:43:41Z
user: login: jrdouceur
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T12:26:56Z
user: login: timtay-microsoft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T21:55:50Z
user: login: jrdouceur
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T11:53:42Z
user: login: koepalex
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T19:19:50Z
user: login: koepalex
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T19:22:18Z
user: login: koepalex
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T19:23:43Z
user: login: koepalex
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T19:25:35Z
user: login: koepalex
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T19:26:58Z
user: login: koepalex
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T19:35:51Z
user: login: koepalex
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T19:43:34Z
user: login: timtay-microsoft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T20:57:08Z
user: login: timtay-microsoft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T20:59:28Z
user: login: timtay-microsoft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T21:02:33Z
user: login: timtay-microsoft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T21:07:47Z
user: login: timtay-microsoft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T21:12:37Z
user: login: timtay-microsoft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T21:14:05Z
user: login: jrdouceur
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T21:38:55Z
user: login: clecompt-msft
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T21:29:08Z
user: login: jrdouceur
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T19:20:08Z
user: login: jrdouceur
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T19:24:25Z
user: login: timtay-microsoft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-29T21:04:14Z
user: login: clecompt-msft
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T23:37:45Z

Review comments:
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
+
+By convention, the value of the AppErrPayload object should be a UTF-8 encoded JSON string since MQTT user property values must be UTF-8 encoded strings.
+
+## Code Example
+
+### Command executor side
+
+```csharp
+public ExtendedResponse<IncrementResponsePayload> Increment(IncrementRequestPayload request)
+{
+    if (request.IncrementValue < 0)
+    {
+        var response = new ExtendedResponse<IncrementResponsePayload>()
+        {
+            Response = new IncrementResponsePayload { CounterResponse = _counter },
+        };
+
+        // Specify error code, but no error payload  
+        response.WithApplicationError("negativeValue");
+
+        // Or you can specify error code and error payload
+        response.WithApplicationError(
+            "negativeValue",
+            new CustomErrorPayload() { ArgumentValue = request.IncrementValue },
+            new ErrorPayloadJsonSerializer(new TestEnvoys.Utf8JsonSerializer()));
+
+        return response;
+    }
+
+    // happy path omitted
+}
+```
+
+### Command invoker side
+
+```csharp
+public void main()
+{
+    await using MqttSessionClient mqttInvoker = await ClientFactory.CreateSessionClientFromEnvAsync();
+    await using CounterClient counterClient = new CounterClient(applicationContext, mqttInvoker);
+
+    IncrementRequestPayload payload = new IncrementRequestPayload
+    {
+        IncrementValue = -1
+    };
+
+    var response = await counterClient.Increment(executorId, payload).WithMetadata();
+    
+    // Check the RPC response for an application error
+    if (response.TryGetApplicationError(SomeJsonSerializer, out string? errorCode, out CustomErrorPayload? errorPayload))
+    {
+        // use the error code and strongly typed custom error payload type as wanted.
+    }
+
+    // Alternatively just fetch the error code if you don't want to deserialize the payload yet.
+    if (response.TryGetApplicationError(out string? errorCode))
+    {
+        // ...
+    }
+}
+```
+
+### Enforcement
+
+Note that, while this decision creates a standardized way of communicating application errors, our SDK will **not** enforce that users communicate application errors with this standard. Users will still be able to model application errors in response payloads or in custom user properties if they prefer. There is no way to force users to communicate errors with this standard pattern.
+
+## Samples
+
+We want to establish this pattern as the standard for communicating RPC application errors. To do so, we must either update or add a sample per-language that demonstrates handling at least one application error.
+
+For instance, we would make the counter service return an application error if the invocation specifies a negative value to increment by. This change won't include any DTDL level changes yet.
+
+## Open Questions
+
+- MQTT user properties are strings, not bytes, so is there an issue with using UTF8 encoding to convert a serialized payload to a string? Does this work for the serializers we already support (protobuf, avro, raw, etc.)
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: jrdouceur
body: The formats of AVRO, Protobuf, and raw are binary, so these would need to be Base64-encoded if we want to pack them into MQTT header values.
created_at: 2025-04-03T23:43:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
+
+By convention, the value of the AppErrPayload object should be a UTF-8 encoded JSON string since MQTT user property values must be UTF-8 encoded strings.
+
+## Code Example
+
+### Command executor side
+
+```csharp
+public ExtendedResponse<IncrementResponsePayload> Increment(IncrementRequestPayload request)
+{
+    if (request.IncrementValue < 0)
+    {
+        var response = new ExtendedResponse<IncrementResponsePayload>()
+        {
+            Response = new IncrementResponsePayload { CounterResponse = _counter },
+        };
+
+        // Specify error code, but no error payload  
+        response.WithApplicationError("negativeValue");
+
+        // Or you can specify error code and error payload
+        response.WithApplicationError(
+            "negativeValue",
+            new CustomErrorPayload() { ArgumentValue = request.IncrementValue },
+            new ErrorPayloadJsonSerializer(new TestEnvoys.Utf8JsonSerializer()));
+
+        return response;
+    }
+
+    // happy path omitted
+}
+```
+
+### Command invoker side
+
+```csharp
+public void main()
+{
+    await using MqttSessionClient mqttInvoker = await ClientFactory.CreateSessionClientFromEnvAsync();
+    await using CounterClient counterClient = new CounterClient(applicationContext, mqttInvoker);
+
+    IncrementRequestPayload payload = new IncrementRequestPayload
+    {
+        IncrementValue = -1
+    };
+
+    var response = await counterClient.Increment(executorId, payload).WithMetadata();
+    
+    // Check the RPC response for an application error
+    if (response.TryGetApplicationError(SomeJsonSerializer, out string? errorCode, out CustomErrorPayload? errorPayload))
+    {
+        // use the error code and strongly typed custom error payload type as wanted.
+    }
+
+    // Alternatively just fetch the error code if you don't want to deserialize the payload yet.
+    if (response.TryGetApplicationError(out string? errorCode))
+    {
+        // ...
+    }
+}
+```
+
+### Enforcement
+
+Note that, while this decision creates a standardized way of communicating application errors, our SDK will **not** enforce that users communicate application errors with this standard. Users will still be able to model application errors in response payloads or in custom user properties if they prefer. There is no way to force users to communicate errors with this standard pattern.
+
+## Samples
+
+We want to establish this pattern as the standard for communicating RPC application errors. To do so, we must either update or add a sample per-language that demonstrates handling at least one application error.
+
+For instance, we would make the counter service return an application error if the invocation specifies a negative value to increment by. This change won't include any DTDL level changes yet.
+
+## Open Questions
+
+- MQTT user properties are strings, not bytes, so is there an issue with using UTF8 encoding to convert a serialized payload to a string? Does this work for the serializers we already support (protobuf, avro, raw, etc.)
+  - Maybe this suggests the serialization interface we use here converts object <-> string rather than object <-> bytes
+
+- MQTT user property value size limit considerations? Users may provide very large payload objects and MQTT as a protocol may not be built for that?
+
+- Do we remove support for modeling application errors in the MQTT message payload as described in (ADR 19)[./0019-codegen-user-errs.md]? No one has onboarded to this model yet
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: jrdouceur
body: I believe @abhipsaMisra has begun using the new DTDL user-error modeling mechanisms in the Mqtt v3 extension.
created_at: 2025-04-04T12:26:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
+
+By convention, the value of the AppErrPayload object should be a UTF-8 encoded JSON string since MQTT user property values must be UTF-8 encoded strings.
+
+## Code Example
+
+### Command executor side
+
+```csharp
+public ExtendedResponse<IncrementResponsePayload> Increment(IncrementRequestPayload request)
+{
+    if (request.IncrementValue < 0)
+    {
+        var response = new ExtendedResponse<IncrementResponsePayload>()
+        {
+            Response = new IncrementResponsePayload { CounterResponse = _counter },
+        };
+
+        // Specify error code, but no error payload  
+        response.WithApplicationError("negativeValue");
+
+        // Or you can specify error code and error payload
+        response.WithApplicationError(
+            "negativeValue",
+            new CustomErrorPayload() { ArgumentValue = request.IncrementValue },
+            new ErrorPayloadJsonSerializer(new TestEnvoys.Utf8JsonSerializer()));
+
+        return response;
+    }
+
+    // happy path omitted
+}
+```
+
+### Command invoker side
+
+```csharp
+public void main()
+{
+    await using MqttSessionClient mqttInvoker = await ClientFactory.CreateSessionClientFromEnvAsync();
+    await using CounterClient counterClient = new CounterClient(applicationContext, mqttInvoker);
+
+    IncrementRequestPayload payload = new IncrementRequestPayload
+    {
+        IncrementValue = -1
+    };
+
+    var response = await counterClient.Increment(executorId, payload).WithMetadata();
+    
+    // Check the RPC response for an application error
+    if (response.TryGetApplicationError(SomeJsonSerializer, out string? errorCode, out CustomErrorPayload? errorPayload))
+    {
+        // use the error code and strongly typed custom error payload type as wanted.
+    }
+
+    // Alternatively just fetch the error code if you don't want to deserialize the payload yet.
+    if (response.TryGetApplicationError(out string? errorCode))
+    {
+        // ...
+    }
+}
+```
+
+### Enforcement
+
+Note that, while this decision creates a standardized way of communicating application errors, our SDK will **not** enforce that users communicate application errors with this standard. Users will still be able to model application errors in response payloads or in custom user properties if they prefer. There is no way to force users to communicate errors with this standard pattern.
+
+## Samples
+
+We want to establish this pattern as the standard for communicating RPC application errors. To do so, we must either update or add a sample per-language that demonstrates handling at least one application error.
+
+For instance, we would make the counter service return an application error if the invocation specifies a negative value to increment by. This change won't include any DTDL level changes yet.
+
+## Open Questions
+
+- MQTT user properties are strings, not bytes, so is there an issue with using UTF8 encoding to convert a serialized payload to a string? Does this work for the serializers we already support (protobuf, avro, raw, etc.)
+  - Maybe this suggests the serialization interface we use here converts object <-> string rather than object <-> bytes
+
+- MQTT user property value size limit considerations? Users may provide very large payload objects and MQTT as a protocol may not be built for that?
+
+- Do we remove support for modeling application errors in the MQTT message payload as described in (ADR 19)[./0019-codegen-user-errs.md]? No one has onboarded to this model yet
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: timtay-microsoft
body: Good point. I do like that her use case allows for modeling of the error in the payload. The only question then becomes "do we care that there are 2 ways to model errors in DTDL?"

I know that we haven't really investigated modeling headers in DTDL land, but do you think it is feasible for us to support DTDL modeling of errors in either (and maybe both?) headers and MQTT payloads?

From the pure command invoker/executor side of things, it's certainly doable and not much overhead.
created_at: 2025-04-04T21:55:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
+
+By convention, the value of the AppErrPayload object should be a UTF-8 encoded JSON string since MQTT user property values must be UTF-8 encoded strings.
+
+## Code Example
+
+### Command executor side
+
+```csharp
+public ExtendedResponse<IncrementResponsePayload> Increment(IncrementRequestPayload request)
+{
+    if (request.IncrementValue < 0)
+    {
+        var response = new ExtendedResponse<IncrementResponsePayload>()
+        {
+            Response = new IncrementResponsePayload { CounterResponse = _counter },
+        };
+
+        // Specify error code, but no error payload  
+        response.WithApplicationError("negativeValue");
+
+        // Or you can specify error code and error payload
+        response.WithApplicationError(
+            "negativeValue",
+            new CustomErrorPayload() { ArgumentValue = request.IncrementValue },
+            new ErrorPayloadJsonSerializer(new TestEnvoys.Utf8JsonSerializer()));
+
+        return response;
+    }
+
+    // happy path omitted
+}
+```
+
+### Command invoker side
+
+```csharp
+public void main()
+{
+    await using MqttSessionClient mqttInvoker = await ClientFactory.CreateSessionClientFromEnvAsync();
+    await using CounterClient counterClient = new CounterClient(applicationContext, mqttInvoker);
+
+    IncrementRequestPayload payload = new IncrementRequestPayload
+    {
+        IncrementValue = -1
+    };
+
+    var response = await counterClient.Increment(executorId, payload).WithMetadata();
+    
+    // Check the RPC response for an application error
+    if (response.TryGetApplicationError(SomeJsonSerializer, out string? errorCode, out CustomErrorPayload? errorPayload))
+    {
+        // use the error code and strongly typed custom error payload type as wanted.
+    }
+
+    // Alternatively just fetch the error code if you don't want to deserialize the payload yet.
+    if (response.TryGetApplicationError(out string? errorCode))
+    {
+        // ...
+    }
+}
+```
+
+### Enforcement
+
+Note that, while this decision creates a standardized way of communicating application errors, our SDK will **not** enforce that users communicate application errors with this standard. Users will still be able to model application errors in response payloads or in custom user properties if they prefer. There is no way to force users to communicate errors with this standard pattern.
+
+## Samples
+
+We want to establish this pattern as the standard for communicating RPC application errors. To do so, we must either update or add a sample per-language that demonstrates handling at least one application error.
+
+For instance, we would make the counter service return an application error if the invocation specifies a negative value to increment by. This change won't include any DTDL level changes yet.
+
+## Open Questions
+
+- MQTT user properties are strings, not bytes, so is there an issue with using UTF8 encoding to convert a serialized payload to a string? Does this work for the serializers we already support (protobuf, avro, raw, etc.)
+  - Maybe this suggests the serialization interface we use here converts object <-> string rather than object <-> bytes
+
+- MQTT user property value size limit considerations? Users may provide very large payload objects and MQTT as a protocol may not be built for that?
+
+- Do we remove support for modeling application errors in the MQTT message payload as described in (ADR 19)[./0019-codegen-user-errs.md]? No one has onboarded to this model yet
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: jrdouceur
body: It is certainly feasible from a technical perspective.  But having multiple ways to express error conditions runs counter to the goal (as I understand it) of ensuring commonality across applications.
created_at: 2025-04-07T11:53:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: koepalex
body: `applications want to route RPC`
-->
`applications have to route RPC`
created_at: 2025-04-08T19:19:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: koepalex
body: Why enforcing a user-defined payload object, serialized to bytes and then encoded as UTF8, to support code gen?
created_at: 2025-04-08T19:22:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: koepalex
body: This is only required, if the AppErrorPayload isn't string
created_at: 2025-04-08T19:23:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: koepalex
body: `both the possible error codes` you mean possible header fields? Because the AppErrCode can have various values...
created_at: 2025-04-08T19:25:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
+
+By convention, the value of the AppErrPayload object should be a UTF-8 encoded JSON string since MQTT user property values must be UTF-8 encoded strings.
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: koepalex
body: Above is state that the Error object is serialized to bytes and encoded as UTF8 string, this is mismatching with the UTF-8 encoded JSON string
created_at: 2025-04-08T19:26:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
+
+By convention, the value of the AppErrPayload object should be a UTF-8 encoded JSON string since MQTT user property values must be UTF-8 encoded strings.
+
+## Code Example
+
+### Command executor side
+
+```csharp
+public ExtendedResponse<IncrementResponsePayload> Increment(IncrementRequestPayload request)
+{
+    if (request.IncrementValue < 0)
+    {
+        var response = new ExtendedResponse<IncrementResponsePayload>()
+        {
+            Response = new IncrementResponsePayload { CounterResponse = _counter },
+        };
+
+        // Specify error code, but no error payload  
+        response.WithApplicationError("negativeValue");
+
+        // Or you can specify error code and error payload
+        response.WithApplicationError(
+            "negativeValue",
+            new CustomErrorPayload() { ArgumentValue = request.IncrementValue },
+            new ErrorPayloadJsonSerializer(new TestEnvoys.Utf8JsonSerializer()));
+
+        return response;
+    }
+
+    // happy path omitted
+}
+```
+
+### Command invoker side
+
+```csharp
+public void main()
+{
+    MqttSessionClient mqttClient = ...;();
+    CounterClient counterClient = new CounterClient(mqttClient);
+
+    IncrementRequestPayload payload = new IncrementRequestPayload
+    {
+        IncrementValue = -1
+    };
+
+    var response = counterClient.Increment(executorId, payload).WithMetadata();
+    
+    // Check the RPC response for an application error
+    if (response.TryGetApplicationError(SomeJsonSerializer, out string? errorCode, out CustomErrorPayload? errorPayload))
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: koepalex
body: What is if the client use different Json(de-)Serializer?
created_at: 2025-04-08T19:35:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
+
+By convention, the value of the AppErrPayload object should be a UTF-8 encoded JSON string since MQTT user property values must be UTF-8 encoded strings.
+
+## Code Example
+
+### Command executor side
+
+```csharp
+public ExtendedResponse<IncrementResponsePayload> Increment(IncrementRequestPayload request)
+{
+    if (request.IncrementValue < 0)
+    {
+        var response = new ExtendedResponse<IncrementResponsePayload>()
+        {
+            Response = new IncrementResponsePayload { CounterResponse = _counter },
+        };
+
+        // Specify error code, but no error payload  
+        response.WithApplicationError("negativeValue");
+
+        // Or you can specify error code and error payload
+        response.WithApplicationError(
+            "negativeValue",
+            new CustomErrorPayload() { ArgumentValue = request.IncrementValue },
+            new ErrorPayloadJsonSerializer(new TestEnvoys.Utf8JsonSerializer()));
+
+        return response;
+    }
+
+    // happy path omitted
+}
+```
+
+### Command invoker side
+
+```csharp
+public void main()
+{
+    MqttSessionClient mqttClient = ...;();
+    CounterClient counterClient = new CounterClient(mqttClient);
+
+    IncrementRequestPayload payload = new IncrementRequestPayload
+    {
+        IncrementValue = -1
+    };
+
+    var response = counterClient.Increment(executorId, payload).WithMetadata();
+    
+    // Check the RPC response for an application error
+    if (response.TryGetApplicationError(SomeJsonSerializer, out string? errorCode, out CustomErrorPayload? errorPayload))
+    {
+        // use the error code and strongly typed custom error payload type as wanted.
+    }
+
+    // Alternatively just fetch the error code if you don't want to deserialize the payload yet.
+    if (response.TryGetApplicationError(out string? errorCode))
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: koepalex
body: Can we think about using something like: https://dotnet.github.io/dotNext/features/core/result.html That no matter how the "user" models errors (payload or header), there is kind of more opinionated error handling. In the current proposal the customer can just ignore the error and access the payload object
created_at: 2025-04-08T19:43:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: timtay-microsoft
body: Yes. We eventually want DTDL to model these headers so that we can provide strong typing to users via codegen
created_at: 2025-04-08T20:57:08Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: timtay-microsoft
body: I'll reword this. I'm trying to say that DTDL will model possible values of ```AppErrCode``` and will model the shape of the ```AppErrPayload``` object
created_at: 2025-04-08T20:59:28Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
+
+By convention, the value of the AppErrPayload object should be a UTF-8 encoded JSON string since MQTT user property values must be UTF-8 encoded strings.
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: timtay-microsoft
body: Good catch. I'll rewrite this.
created_at: 2025-04-08T21:02:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
+
+By convention, the value of the AppErrPayload object should be a UTF-8 encoded JSON string since MQTT user property values must be UTF-8 encoded strings.
+
+## Code Example
+
+### Command executor side
+
+```csharp
+public ExtendedResponse<IncrementResponsePayload> Increment(IncrementRequestPayload request)
+{
+    if (request.IncrementValue < 0)
+    {
+        var response = new ExtendedResponse<IncrementResponsePayload>()
+        {
+            Response = new IncrementResponsePayload { CounterResponse = _counter },
+        };
+
+        // Specify error code, but no error payload  
+        response.WithApplicationError("negativeValue");
+
+        // Or you can specify error code and error payload
+        response.WithApplicationError(
+            "negativeValue",
+            new CustomErrorPayload() { ArgumentValue = request.IncrementValue },
+            new ErrorPayloadJsonSerializer(new TestEnvoys.Utf8JsonSerializer()));
+
+        return response;
+    }
+
+    // happy path omitted
+}
+```
+
+### Command invoker side
+
+```csharp
+public void main()
+{
+    MqttSessionClient mqttClient = ...;();
+    CounterClient counterClient = new CounterClient(mqttClient);
+
+    IncrementRequestPayload payload = new IncrementRequestPayload
+    {
+        IncrementValue = -1
+    };
+
+    var response = counterClient.Increment(executorId, payload).WithMetadata();
+    
+    // Check the RPC response for an application error
+    if (response.TryGetApplicationError(SomeJsonSerializer, out string? errorCode, out CustomErrorPayload? errorPayload))
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: timtay-microsoft
body: This proposal allows for user to have different serializers for handling error models in MQTT message payloads vs error models in MQTT user properties.
created_at: 2025-04-08T21:07:47Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
+
+By convention, the value of the AppErrPayload object should be a UTF-8 encoded JSON string since MQTT user property values must be UTF-8 encoded strings.
+
+## Code Example
+
+### Command executor side
+
+```csharp
+public ExtendedResponse<IncrementResponsePayload> Increment(IncrementRequestPayload request)
+{
+    if (request.IncrementValue < 0)
+    {
+        var response = new ExtendedResponse<IncrementResponsePayload>()
+        {
+            Response = new IncrementResponsePayload { CounterResponse = _counter },
+        };
+
+        // Specify error code, but no error payload  
+        response.WithApplicationError("negativeValue");
+
+        // Or you can specify error code and error payload
+        response.WithApplicationError(
+            "negativeValue",
+            new CustomErrorPayload() { ArgumentValue = request.IncrementValue },
+            new ErrorPayloadJsonSerializer(new TestEnvoys.Utf8JsonSerializer()));
+
+        return response;
+    }
+
+    // happy path omitted
+}
+```
+
+### Command invoker side
+
+```csharp
+public void main()
+{
+    MqttSessionClient mqttClient = ...;();
+    CounterClient counterClient = new CounterClient(mqttClient);
+
+    IncrementRequestPayload payload = new IncrementRequestPayload
+    {
+        IncrementValue = -1
+    };
+
+    var response = counterClient.Increment(executorId, payload).WithMetadata();
+    
+    // Check the RPC response for an application error
+    if (response.TryGetApplicationError(SomeJsonSerializer, out string? errorCode, out CustomErrorPayload? errorPayload))
+    {
+        // use the error code and strongly typed custom error payload type as wanted.
+    }
+
+    // Alternatively just fetch the error code if you don't want to deserialize the payload yet.
+    if (response.TryGetApplicationError(out string? errorCode))
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: timtay-microsoft
body: > no matter how the "user" models errors (payload or header), there is kind of more opinionated error handling

I'm not sure this is feasible. Partly because a user may not have any error modeling (in payload or headers), and partly because our library allows users to define application error payloads outside of our standard. We really don't have a way to force users to do all error modeling in a standard way that our library could then parse out.

I think it is reasonable to assert that the user of an RPC invoker will know where to expect to find an error if one existed (in headers vs in payload), but do you feel differently?
created_at: 2025-04-08T21:12:37Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: timtay-microsoft
body: I envision that we'd do the same thing here that we did for raw byte payloads in MQTT messages where we provide a PassthroughSerializer that takes and returns a string as-is. I'll call this out in the doc, though.
created_at: 2025-04-08T21:14:05Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
+
+By convention, the value of the AppErrPayload object should be a UTF-8 encoded JSON string since MQTT user property values must be UTF-8 encoded strings.
+
+## Code Example
+
+### Command executor side
+
+```csharp
+public ExtendedResponse<IncrementResponsePayload> Increment(IncrementRequestPayload request)
+{
+    if (request.IncrementValue < 0)
+    {
+        var response = new ExtendedResponse<IncrementResponsePayload>()
+        {
+            Response = new IncrementResponsePayload { CounterResponse = _counter },
+        };
+
+        // Specify error code, but no error payload  
+        response.WithApplicationError("negativeValue");
+
+        // Or you can specify error code and error payload
+        response.WithApplicationError(
+            "negativeValue",
+            new CustomErrorPayload() { ArgumentValue = request.IncrementValue },
+            new ErrorPayloadJsonSerializer(new TestEnvoys.Utf8JsonSerializer()));
+
+        return response;
+    }
+
+    // happy path omitted
+}
+```
+
+### Command invoker side
+
+```csharp
+public void main()
+{
+    MqttSessionClient mqttClient = ...;();
+    CounterClient counterClient = new CounterClient(mqttClient);
+
+    IncrementRequestPayload payload = new IncrementRequestPayload
+    {
+        IncrementValue = -1
+    };
+
+    var response = counterClient.Increment(executorId, payload).WithMetadata();
+    
+    // Check the RPC response for an application error
+    if (response.TryGetApplicationError(SomeJsonSerializer, out string? errorCode, out CustomErrorPayload? errorPayload))
+    {
+        // use the error code and strongly typed custom error payload type as wanted.
+    }
+
+    // Alternatively just fetch the error code if you don't want to deserialize the payload yet.
+    if (response.TryGetApplicationError(out string? errorCode))
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: jrdouceur
body: I suspect Alex means that if the SDK understands the meaning of the header, it can determine that the response indicates a user error, and it can throw an exception or return an error value instead of returning the normal result value from the payload.  ADR 19 describes how to do this using codegen support, but if the SDKs are directly aware of user-error headers, this could be done without any need for codegen.

However, on reflection, I'm not sure how this would be compatible with [Alex's comment below](https://github.com/Azure/iot-operations-sdks/pull/676#issuecomment-2787512103) that we might need to support both valid response payloads and reported user errors in the same response message.
created_at: 2025-04-08T21:38:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications want to route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined payload object (serialized to bytes and then encoded as a UTF8 string) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Similar to how our SDKs handle serializing the actual MQTT message payload, our SDKs will require the user provide the serializer for serializing/deserializing the AppErrPayload object.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling both the possible error codes (enum-like list of possible values?) and the type of the error payload in DTDL. This modeling will be detailed in a separate ADR, though.
path: doc/dev/adr/0020-error-modeling-headers.md
user: login: clecompt-msft
body: Might be worth mentioning that we'd ideally want DTDL to model headers in general, with the ability to tag them with error metadata similar to what we support for the payload currently.
created_at: 2025-04-09T21:29:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,99 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications must route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined string (that is conventionally, but not necessarily, a stringified JSON object/value/array) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling the error codes in DTDL. This modeling will be detailed in a separate ADR, though. We will not provide modeling of the application error payload types in DTDL, though.
path: doc/dev/adr/0021-error-modeling-headers.md
user: login: jrdouceur
body: > We will not provide modeling of the application error payload types in DTDL, though.

Actually, we already [do](https://github.com/Azure/opendigitaltwins-dtdl/blob/master/DTDL/v4/DTDL.mqtt.v3.md#error-and-errormessage-adjunct-types).
created_at: 2025-04-29T19:20:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,99 @@
+# ADR 19: Modeling User Errors
+
+## Context
+
+We have received feedback from users that they would like to have RPC responses communicate application level failures in the headers of the response rather than the payload of the response. 
+
+This feedback is, in part, because some applications must route RPC responses without reading/deserializing the entire payload. It is also partly because [some applications cannot extend or change the payload model to accommodate error handling.](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996).
+
+## Decision
+
+Our SDKs will add APIs on the command executor side that mark an RPC response as an "application error" by attaching two new MQTT user properties ("AppErrCode" and "AppErrPayload") whose values are a user-defined string and a user-defined string (that is conventionally, but not necessarily, a stringified JSON object/value/array) respectively. When a user wants to indicate that an RPC call failed with an application error they must provide the "AppErrCode" value, but "AppErrPayload" is optional.
+
+In addition, users must be able to include arbitrary user properties in their command response to support this error reporting in case our standard fields are insufficient. This feature is likely in place already for all languages.
+
+On the command invoker side, we will add APIs for checking if a response was an application error and returning the error code and error data fields parsed from the MQTT message "AppErrCode" and "AppErrPayload" user properties.
+
+Other than these two new user properties, the over-the-wire behavior of our protocol won't change as a result of this decision.
+
+In order to provide a strongly-typed experience, we will also add codegen support for modeling the error codes in DTDL. This modeling will be detailed in a separate ADR, though. We will not provide modeling of the application error payload types in DTDL, though.
path: doc/dev/adr/0021-error-modeling-headers.md
user: login: timtay-microsoft
body: Neat! I'll fix this
created_at: 2025-04-29T21:04:14Z
author_association: MEMBER

PR comments:
user: login: jrdouceur
created_at: 2025-04-03T23:41:21Z
body: Is there ever a need for an application error code in the header of a message that contains normal application payload?  Or can we be confident that the inflexible payload schemas alluded to in [this comment](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996) apply only when there are no errors?  If an error (either protocol-level or application-level) implies that there is no normal payload, then the MQTT payload is available for conveying detailed error information.  This would allow us to forego the second header field and the awkward packing of encoded data into a header string value.  It would also enable a minor (but critical) change to ADR 19 that would avoid modifying the payload in success cases, because the distinction between success and error would come from a header instead of which property in the payload's top-level object contains a value.
author_association: CONTRIBUTOR
user: login: timtay-microsoft
created_at: 2025-04-04T21:51:08Z
body: > Is there ever a need for an application error code in the header of a message that contains normal application payload? Or can we be confident that the inflexible payload schemas alluded to in https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996 apply only when there are no errors? If an error (either protocol-level or application-level) implies that there is no normal payload, then the MQTT payload is available for conveying detailed error information. This would allow us to forego the second header field and the awkward packing of encoded data into a header string value. It would also enable a minor (but critical) change to ADR 19 that would avoid modifying the payload in success cases, because the distinction between success and error would come from a header instead of which property in the payload's top-level object contains a value.

This is an important question. @koepalex in the scenario you highlighted in the above issue thread, will your connector be expected to send both a MQTT message payload (where you cannot add error modeling as discussed in the issue thread) + this proposed header payload?

I would personally prefer we keep payload-like objects in the MQTT message payload, but I gather that isn't feasible for your use case.
author_association: MEMBER
user: login: koepalex
created_at: 2025-04-08T19:52:02Z
body: ` Is there ever a need for an application error code in the header of a message that contains normal application payload?`

Yes, there are scenarios, for example it might be that we can handle a READ request, (because we have all mandatory data in the client-sided cache), but the call itself to the OT device failed (for whatever reason). 

In this case we would return the message payload (build from cache) but also signal the gateway error.
 
author_association: CONTRIBUTOR
user: login: timtay-microsoft
created_at: 2025-04-09T16:03:54Z
body: TODO: AppErrPayload does not need to be modeled by DTDL, and it can just be a utf-8 string (usually json, but not forcibly so)
author_association: MEMBER
user: login: jrdouceur
created_at: 2025-04-09T16:10:03Z
body: > TODO: AppErrPayload does not need to be modeled by DTDL, and it can just be a utf-8 string (usually json, but not forcibly so)

I recommend (open to discussion, of course) that we prescribe JSON specifically, in the interest of cross-application consistency and the ability for an SDK to express the value via a DOM.  I don't think it needs to be a JSON object; it could be a JSON string, JSON number, JSON array, etc., but unless there is a scenario for which JSON encoding would be problematic, it could be helpful to remove a degree of variability here.
author_association: CONTRIBUTOR
user: login: timtay-microsoft
created_at: 2025-04-09T16:11:46Z
body: TODO: TryGetApplicationErrorsFromHeaders
author_association: MEMBER
user: login: timtay-microsoft
created_at: 2025-04-09T18:39:52Z
body: > > TODO: AppErrPayload does not need to be modeled by DTDL, and it can just be a utf-8 string (usually json, but not forcibly so)
> 
> I recommend (open to discussion, of course) that we prescribe JSON specifically, in the interest of cross-application consistency and the ability for an SDK to express the value via a DOM. I don't think it needs to be a JSON object; it could be a JSON string, JSON number, JSON array, etc., but unless there is a scenario for which JSON encoding would be problematic, it could be helpful to remove a degree of variability here.

The only (mild) issue here is that we will then need to take a dependency on a JSON library in our RPC code to make that prescription. This is fine for .NET since we have System.Text.Json as a pretty standard official JSON library. 

@clecompt-msft is there a conventional JSON library for Go that we can take a dependency on in our RPC layer?

@vaavva same question for Rust
author_association: MEMBER
user: login: jrdouceur
created_at: 2025-04-09T18:45:02Z
body: > @clecompt-msft is there a conventional JSON library for Go that we can take a dependency on in our RPC layer?
> 
> @vaavva same question for Rust

I can answer both of these.  JSON is supported natively in Go.  Rust would require taking a dependency on an external crate, but it's a very common one.
author_association: CONTRIBUTOR
user: login: timtay-microsoft
created_at: 2025-04-09T18:46:16Z
body: > > @clecompt-msft is there a conventional JSON library for Go that we can take a dependency on in our RPC layer?
> > @vaavva same question for Rust
> 
> I can answer both of these. JSON is supported natively in Go. Rust would require taking a dependency on an external crate, but it's a very common one.

Then I have no objections on this approach
author_association: MEMBER
user: login: vaavva
created_at: 2025-04-09T19:20:45Z
body: > > > @clecompt-msft is there a conventional JSON library for Go that we can take a dependency on in our RPC layer?
> > > @vaavva same question for Rust
> > 
> > 
> > I can answer both of these. JSON is supported natively in Go. Rust would require taking a dependency on an external crate, but it's a very common one.
> 
> Then I have no objections on this approach

I would strongly prefer to not take a JSON serializer dependency in the SDK unless we really needed to, and I'm not sure this change justifies it. In a generic solution, there's no reason that this needs to be specifically JSON as opposed to anything serialized into UTF-8. JSON should just be a recommendation from us as a convenient option. Cross-application consistency for a free-form field should be enforced by over the wire contracts (whether DTDL or not), not the SDK. Even for our payload, the invoker/executor using the same payload format is enforced by either the DTDL or contracts between the writers of both sides. If stronger enforcement for this solution is required/desired, creating wrapper client implementations that force using this error model would be the way to go.

With the discussion earlier that we wouldn't model the error and just provide a UTF-8 string, I'm not sure why enforcing JSON doesn't fall into the same category. If we want to enforce this at the codegen level, where we already have the JSON dependency, then that would seem more reasonable, although my concern about unnecessarily restricting to JSON I think still applies.
author_association: MEMBER
user: login: jrdouceur
created_at: 2025-04-09T19:43:57Z
body: > Cross-application consistency for a free-form field should be enforced by over the wire contracts (whether DTDL or not), not the SDK.

I think we should get @lt72's input on this.  I know he has fairly strong opinions on having the SDKs enforce consistency directly for some core aspects of the protocol, including the means by which user errors are represented.

> With the discussion earlier that we wouldn't model the error and just provide a UTF-8 string, I'm not sure why enforcing JSON doesn't fall into the same category.

It seems to me it would be reasonable to say that:

* The value of `AppErrCode` is a string, representing an error condition, that can be used as an enumerated value in common languages, so it:
  * must contain only letters and digits
  * must start with a letter
  * (possibly allows underscores, but there are subtleties here we might wish to avoid)
* The value of `AppErrDetail` is a string, representing error detail information, that is a valid JSON element.

Requiring JSON for the detail header ensures that no one does something crazy like Base64-encoding a binary representation that might present inter-operational challenges.

author_association: CONTRIBUTOR
user: login: lt72
created_at: 2025-04-09T20:11:47Z
body: > Is there ever a need for an application error code in the header of a message that contains normal application payload? Or can we be confident that the inflexible payload schemas alluded to in [this comment](https://github.com/Azure/iot-operations-sdks/issues/488#issuecomment-2707496996) apply only when there are no errors? If an error (either protocol-level or application-level) implies that there is no normal payload, then the MQTT payload is available for conveying detailed error information. This would allow us to forego the second header field and the awkward packing of encoded data into a header string value. It would also enable a minor (but critical) change to ADR 19 that would avoid modifying the payload in success cases, because the distinction between success and error would come from a header instead of which property in the payload's top-level object contains a value.

I'm not sure I get this, but I can confidently say that all the times, not matter what, the distinction between success and error must come from a header. No question.

author_association: NONE
user: login: lt72
created_at: 2025-04-09T20:14:34Z
body: > > Cross-application consistency for a free-form field should be enforced by over the wire contracts (whether DTDL or not), not the SDK.
> 
> I think we should get @lt72's input on this. I know he has fairly strong opinions on having the SDKs enforce consistency directly for some core aspects of the protocol, including the means by which user errors are represented.
> 
> > With the discussion earlier that we wouldn't model the error and just provide a UTF-8 string, I'm not sure why enforcing JSON doesn't fall into the same category.
> 
> It seems to me it would be reasonable to say that:
> 
> * The value of `AppErrCode` is a string, representing an error condition, that can be used as an enumerated value in common languages, so it:
>   
>   * must contain only letters and digits
>   * must start with a letter
>   * (possibly allows underscores, but there are subtleties here we might wish to avoid)
> * The value of `AppErrDetail` is a string, representing error detail information, that is a valid JSON element.
> 
> Requiring JSON for the detail header ensures that no one does something crazy like Base64-encoding a binary representation that might present inter-operational challenges.

Absolutely. Criss app consistency should also be enforced by SDK as much as possible, because we will have situation, like OPC UA, where we will have non-codegen server talking to codgen clients. 

author_association: NONE
user: login: clecompt-msft
created_at: 2025-04-09T21:15:48Z
body: I want to add some context for purposes of this discussion that came up in the sync this morning: One of the scenarios described for OPC-UA is having a _mix_ of success and error in a single message (e.g. where you have a list of results in the payload, some of which succeeded and some of which failed). This in itself throws into question the concept of binary success/error results, and there's no way to idiomatically represent that across languages and across error states in a general way.

Given scenarios like that (and others that don't have the SDK on both sides of the equation and so may not follow our error model at all, like DSS), we can't assume that we can "enforce" consistency - the best we can do is encourage it, which is I believe the goal of this ADR. And given _that_, we probably want to make sure we can handle (and ideally model as best as possible) endpoints that don't align with our recommendations.
author_association: CONTRIBUTOR
user: login: timtay-microsoft
created_at: 2025-04-09T21:55:30Z
body: > The value of AppErrCode is a string, representing an error condition, that can be used as an enumerated value in common languages, so it:
must contain only letters and digits
must start with a letter
(possibly allows underscores, but there are subtleties here we might wish to avoid)

Do we even need to impose limits like this? Our SDK gives these values as-is to the user. The only real limitation comes from MQTT user properties. Longer term, we plan on modeling the possible values in DTDL anyways, right?
author_association: MEMBER