Repository: valkey-io/valkey
Pull request #1889: Redact protocol error log when hide-user-data-from-log enabled
Author: login: YueTang-Vanessa
Created: 2025-03-26T22:24:23Z | Merged: 2025-03-31T07:46:20Z

In this code logic: https://github.com/valkey-io/valkey/blob/unstable/src/networking.c#L2767-L2773, `c->querybuf + c->qb_pos` may also include user data.  
Update the log message when config `hide-user-data-from-log` is enabled.


Diff:
diff --git a/src/networking.c b/src/networking.c
index a049936fa90..0723b903b1f 100644
--- a/src/networking.c
+++ b/src/networking.c
@@ -2764,24 +2764,27 @@ static void setProtocolError(const char *errstr, client *c) {
         /* Sample some protocol to given an idea about what was inside. */
         char buf[256];
         buf[0] = '\0';
-        if (c->querybuf && sdslen(c->querybuf) - c->qb_pos < PROTO_DUMP_LEN) {
-            snprintf(buf, sizeof(buf), "Query buffer during protocol error: '%s'", c->querybuf + c->qb_pos);
-        } else if (c->querybuf) {
-            snprintf(buf, sizeof(buf), "Query buffer during protocol error: '%.*s' (... more %zu bytes ...) '%.*s'",
-                     PROTO_DUMP_LEN / 2, c->querybuf + c->qb_pos, sdslen(c->querybuf) - c->qb_pos - PROTO_DUMP_LEN,
-                     PROTO_DUMP_LEN / 2, c->querybuf + sdslen(c->querybuf) - PROTO_DUMP_LEN / 2);
-        }
+        if (server.hide_user_data_from_log) {
+            snprintf(buf, sizeof(buf), "*redacted*");
+        } else {
+            if (c->querybuf && sdslen(c->querybuf) - c->qb_pos < PROTO_DUMP_LEN) {
+                snprintf(buf, sizeof(buf), "'%s'", c->querybuf + c->qb_pos);
+            } else if (c->querybuf) {
+                snprintf(buf, sizeof(buf), "'%.*s' (... more %zu bytes ...) '%.*s'",
+                         PROTO_DUMP_LEN / 2, c->querybuf + c->qb_pos, sdslen(c->querybuf) - c->qb_pos - PROTO_DUMP_LEN,
+                         PROTO_DUMP_LEN / 2, c->querybuf + sdslen(c->querybuf) - PROTO_DUMP_LEN / 2);
+            }
 
-        /* Remove non printable chars. */
-        char *p = buf;
-        while (*p != '\0') {
-            if (!isprint(*p)) *p = '.';
-            p++;
+            /* Remove non printable chars. */
+            char *p = buf;
+            while (*p != '\0') {
+                if (!isprint(*p)) *p = '.';
+                p++;
+            }
         }
-
         /* Log all the client and protocol info. */
         int loglevel = (c->flag.primary) ? LL_WARNING : LL_VERBOSE;
-        serverLog(loglevel, "Protocol error (%s) from client: %s. %s", errstr, client, buf);
+        serverLog(loglevel, "Protocol error (%s) from client: %s. Query buffer: %s", errstr, client, buf);
         sdsfree(client);
     }
     c->flag.close_after_reply = 1;
diff --git a/tests/integration/logging.tcl b/tests/integration/logging.tcl
index 06b2e24e586..72636963c83 100644
--- a/tests/integration/logging.tcl
+++ b/tests/integration/logging.tcl
@@ -109,6 +109,27 @@ if {!$::valgrind} {
             assert_equal "PONG" [r ping]
         }
     }
+
+    # test hide-user-data-from-log
+    set server_path [tmpdir server5.log]
+    start_server [list overrides [list dir $server_path crash-memcheck-enabled no hide-user-data-from-log no]] {
+        test "Config hide-user-data-from-log is off" {
+            r write "*3\r\n\$3\r\nSET\r\n\$1\r\nx\r\n\$blabla\r\n"
+            r flush
+            catch {r debug segfault}
+            wait_for_log_messages 0 {"*Query buffer: *\$blabla*"} 0 10 1000
+        }
+    }
+
+    set server_path [tmpdir server6.log]
+    start_server [list overrides [list dir $server_path crash-memcheck-enabled no hide-user-data-from-log yes]] {
+        test "Config hide-user-data-from-log is on" {
+            r write "*3\r\n\$3\r\nSET\r\n\$1\r\nx\r\n\$blabla\r\n"
+            r flush
+            catch {r debug segfault}
+            wait_for_log_messages 0 {"*Query buffer: *redacted*"} 0 10 1000
+        }
+    }
 }
 
 # test DEBUG ASSERT


Reviews:
user: login: hpatro
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T22:39:48Z
user: login: QuChen88
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T23:11:16Z
user: login: QuChen88
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T23:16:42Z
user: login: hpatro
body: Could you add a test case and assert `*redacted*` on it ?. 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T17:25:51Z
user: login: zuiderkwast
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T19:24:39Z
user: login: YueTang-Vanessa
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T21:37:10Z
user: login: hpatro
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T22:55:58Z
user: login: YueTang-Vanessa
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T00:17:06Z
user: login: QuChen88
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T04:45:30Z
user: login: zuiderkwast
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T11:08:45Z
user: login: zuiderkwast
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T11:35:51Z
user: login: zuiderkwast
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T11:37:02Z
user: login: YueTang-Vanessa
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T19:57:39Z
user: login: madolson
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T23:23:04Z
user: login: hpatro
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T23:57:25Z
user: login: ranshid
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-30T07:48:42Z
user: login: zuiderkwast
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T07:41:01Z

Review comments:
diff_hunk: @@ -2765,11 +2765,15 @@ static void setProtocolError(const char *errstr, client *c) {
         char buf[256];
         buf[0] = '\0';
         if (c->querybuf && sdslen(c->querybuf) - c->qb_pos < PROTO_DUMP_LEN) {
-            snprintf(buf, sizeof(buf), "Query buffer during protocol error: '%s'", c->querybuf + c->qb_pos);
+            snprintf(buf, sizeof(buf), "Query buffer during protocol error: '%s'",
+                     server.hide_user_data_from_log ? "[hide_user_data_from_log]Redacted" : (c->querybuf + c->qb_pos));
path: src/networking.c
user: login: hpatro
body: I think we can just print `*redacted*` like we do in other places.

```suggestion
            snprintf(buf, sizeof(buf), "Query buffer during protocol error: '%s'",
                     server.hide_user_data_from_log ? "*redacted*" : (c->querybuf + c->qb_pos));
```
created_at: 2025-03-26T22:36:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2765,11 +2765,15 @@ static void setProtocolError(const char *errstr, client *c) {
         char buf[256];
         buf[0] = '\0';
         if (c->querybuf && sdslen(c->querybuf) - c->qb_pos < PROTO_DUMP_LEN) {
-            snprintf(buf, sizeof(buf), "Query buffer during protocol error: '%s'", c->querybuf + c->qb_pos);
+            snprintf(buf, sizeof(buf), "Query buffer during protocol error: '%s'",
+                     server.hide_user_data_from_log ? "[hide_user_data_from_log]Redacted" : (c->querybuf + c->qb_pos));
         } else if (c->querybuf) {
-            snprintf(buf, sizeof(buf), "Query buffer during protocol error: '%.*s' (... more %zu bytes ...) '%.*s'",
-                     PROTO_DUMP_LEN / 2, c->querybuf + c->qb_pos, sdslen(c->querybuf) - c->qb_pos - PROTO_DUMP_LEN,
-                     PROTO_DUMP_LEN / 2, c->querybuf + sdslen(c->querybuf) - PROTO_DUMP_LEN / 2);
+            snprintf(buf, sizeof(buf), server.hide_user_data_from_log ?
+                     ("Query buffer during protocol error: [hide_user_data_from_log]Redacted %zu bytes",
+                      sdslen(c->querybuf) - c->qb_pos) :
+                     ("Query buffer during protocol error: '%.*s' (... more %zu bytes ...) '%.*s'",
+                      PROTO_DUMP_LEN / 2, c->querybuf + c->qb_pos, sdslen(c->querybuf) - c->qb_pos - PROTO_DUMP_LEN,
+                      PROTO_DUMP_LEN / 2, c->querybuf + sdslen(c->querybuf) - PROTO_DUMP_LEN / 2));
path: src/networking.c
user: login: hpatro
body: This is still printing out the last few bytes I believe.
created_at: 2025-03-26T22:39:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2765,11 +2765,15 @@ static void setProtocolError(const char *errstr, client *c) {
         char buf[256];
         buf[0] = '\0';
         if (c->querybuf && sdslen(c->querybuf) - c->qb_pos < PROTO_DUMP_LEN) {
-            snprintf(buf, sizeof(buf), "Query buffer during protocol error: '%s'", c->querybuf + c->qb_pos);
+            snprintf(buf, sizeof(buf), "Query buffer during protocol error: '%s'",
+                     server.hide_user_data_from_log ? "[hide_user_data_from_log]Redacted" : (c->querybuf + c->qb_pos));
         } else if (c->querybuf) {
-            snprintf(buf, sizeof(buf), "Query buffer during protocol error: '%.*s' (... more %zu bytes ...) '%.*s'",
-                     PROTO_DUMP_LEN / 2, c->querybuf + c->qb_pos, sdslen(c->querybuf) - c->qb_pos - PROTO_DUMP_LEN,
-                     PROTO_DUMP_LEN / 2, c->querybuf + sdslen(c->querybuf) - PROTO_DUMP_LEN / 2);
+            snprintf(buf, sizeof(buf), server.hide_user_data_from_log ?
+                     ("Query buffer during protocol error: [hide_user_data_from_log]Redacted %zu bytes",
+                      sdslen(c->querybuf) - c->qb_pos) :
+                     ("Query buffer during protocol error: '%.*s' (... more %zu bytes ...) '%.*s'",
+                      PROTO_DUMP_LEN / 2, c->querybuf + c->qb_pos, sdslen(c->querybuf) - c->qb_pos - PROTO_DUMP_LEN,
+                      PROTO_DUMP_LEN / 2, c->querybuf + sdslen(c->querybuf) - PROTO_DUMP_LEN / 2));
         }
 
         /* Remove non printable chars. */
path: src/networking.c
user: login: QuChen88
body: This part can also print out customer sensitive data.
created_at: 2025-03-26T23:11:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2765,11 +2765,15 @@ static void setProtocolError(const char *errstr, client *c) {
         char buf[256];
         buf[0] = '\0';
         if (c->querybuf && sdslen(c->querybuf) - c->qb_pos < PROTO_DUMP_LEN) {
path: src/networking.c
user: login: QuChen88
body: Why not just do the check for `server.hide_user_data_from_log` before getting into all the other logic below? i.e. something like 

```
+     if (server.hide_user_data_from_log) {
+          snprintf(buf, sizeof(buf), "Query buffer during protocol error: <redacted>");
+         goto log_client_and_proto;
+     } 
     ....
     
+ log_client_and_proto:
    /* Log all the client and protocol info. */
    .... 
```               
created_at: 2025-03-26T23:16:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2760,6 +2760,12 @@ void processInlineBuffer(client *c) {
 static void setProtocolError(const char *errstr, client *c) {
     if (server.verbosity <= LL_VERBOSE || c->flag.primary) {
         sds client = catClientInfoString(sdsempty(), c, server.hide_user_data_from_log);
+        int loglevel = (c->flag.primary) ? LL_WARNING : LL_VERBOSE;
+
+        if (server.hide_user_data_from_log) {
+            printf("Query buffer during protocol error: *redacted*");
path: src/networking.c
user: login: zuiderkwast
body: Printf seems wrong. Did you mean snprintf?

```suggestion
            snprintf(buf, sizeof(buf), "Query buffer during protocol error: *redacted*");
```
created_at: 2025-03-27T19:24:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2760,6 +2760,12 @@ void processInlineBuffer(client *c) {
 static void setProtocolError(const char *errstr, client *c) {
     if (server.verbosity <= LL_VERBOSE || c->flag.primary) {
         sds client = catClientInfoString(sdsempty(), c, server.hide_user_data_from_log);
+        int loglevel = (c->flag.primary) ? LL_WARNING : LL_VERBOSE;
+
+        if (server.hide_user_data_from_log) {
+            printf("Query buffer during protocol error: *redacted*");
path: src/networking.c
user: login: YueTang-Vanessa
body: Since using goto after this line, there has no buf declared/initialed, I don't think we should use `snprintf`.
created_at: 2025-03-27T21:37:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2760,6 +2760,12 @@ void processInlineBuffer(client *c) {
 static void setProtocolError(const char *errstr, client *c) {
     if (server.verbosity <= LL_VERBOSE || c->flag.primary) {
         sds client = catClientInfoString(sdsempty(), c, server.hide_user_data_from_log);
+        int loglevel = (c->flag.primary) ? LL_WARNING : LL_VERBOSE;
+
+        if (server.hide_user_data_from_log) {
+            printf("Query buffer during protocol error: *redacted*");
path: src/networking.c
user: login: hpatro
body: Unresolving the conversation. `printf` isn't useful. Do you want to add a statement to logs? Try `serverLog` then.
created_at: 2025-03-27T22:55:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2760,6 +2760,12 @@ void processInlineBuffer(client *c) {
 static void setProtocolError(const char *errstr, client *c) {
     if (server.verbosity <= LL_VERBOSE || c->flag.primary) {
         sds client = catClientInfoString(sdsempty(), c, server.hide_user_data_from_log);
+        int loglevel = (c->flag.primary) ? LL_WARNING : LL_VERBOSE;
+
+        if (server.hide_user_data_from_log) {
+            printf("Query buffer during protocol error: *redacted*");
path: src/networking.c
user: login: YueTang-Vanessa
body: Got it. Will use the serverLog instead then
created_at: 2025-03-28T00:17:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2760,6 +2760,12 @@ void processInlineBuffer(client *c) {
 static void setProtocolError(const char *errstr, client *c) {
     if (server.verbosity <= LL_VERBOSE || c->flag.primary) {
         sds client = catClientInfoString(sdsempty(), c, server.hide_user_data_from_log);
+        int loglevel = (c->flag.primary) ? LL_WARNING : LL_VERBOSE;
+
+        if (server.hide_user_data_from_log) {
+            printf("Query buffer during protocol error: *redacted*");
path: src/networking.c
user: login: zuiderkwast
body: So now we're logging twice? First here, then goto and log again?

Let's log only once.
created_at: 2025-03-28T11:08:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2779,9 +2785,10 @@ static void setProtocolError(const char *errstr, client *c) {
             p++;
         }
 
+    log_client_and_proto_info:
         /* Log all the client and protocol info. */
-        int loglevel = (c->flag.primary) ? LL_WARNING : LL_VERBOSE;
-        serverLog(loglevel, "Protocol error (%s) from client: %s. %s", errstr, client, buf);
+        serverLog(loglevel, "Protocol error (%s) from client: %s. %s", errstr, client,
+                  server.hide_user_data_from_log ? "*redacted*" : buf);
path: src/networking.c
user: login: zuiderkwast
body: When hide-user-data is disabled, the logged messages look like these, depending on buffer length:

* `"Protocol error (errstr) from client: (client). Query buffer during protocol error: (buf)"`

When hide-user-data is enabled, this code logs

* `"Protocol error (errstr) from client: (client). *redacted*"`
 
The message doesn't say what is redacted. It doesn't say that it's the query buffer.

Another thing that bothers me a little is that "protocol error" appears twice in the logged message. I mean in the old messages. This wasn't well formulated.

To align the two variants and also get rid of the duplicated wording, I think we should remove "Query buffer during protocol error" from the `snprintf` code above (where `buf` is constructed) and add "query buffer" here instead, i.e.

```suggestion
        serverLog(loglevel, "Protocol error (%s) from client: %s. Query buffer: %s", errstr, client,
                  server.hide_user_data_from_log ? "*redacted*" : buf);
```
created_at: 2025-03-28T11:35:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2760,6 +2760,12 @@ void processInlineBuffer(client *c) {
 static void setProtocolError(const char *errstr, client *c) {
     if (server.verbosity <= LL_VERBOSE || c->flag.primary) {
         sds client = catClientInfoString(sdsempty(), c, server.hide_user_data_from_log);
+        int loglevel = (c->flag.primary) ? LL_WARNING : LL_VERBOSE;
+
+        if (server.hide_user_data_from_log) {
+            printf("Query buffer during protocol error: *redacted*");
path: src/networking.c
user: login: zuiderkwast
body: And if we don't log it here, there's no reason to move the line `int loglevel = (c->flag.primary) ? LL_WARNING : LL_VERBOSE;`. It can stay where it was.
created_at: 2025-03-28T11:37:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2760,6 +2760,12 @@ void processInlineBuffer(client *c) {
 static void setProtocolError(const char *errstr, client *c) {
     if (server.verbosity <= LL_VERBOSE || c->flag.primary) {
         sds client = catClientInfoString(sdsempty(), c, server.hide_user_data_from_log);
+        int loglevel = (c->flag.primary) ? LL_WARNING : LL_VERBOSE;
+
+        if (server.hide_user_data_from_log) {
+            printf("Query buffer during protocol error: *redacted*");
path: src/networking.c
user: login: YueTang-Vanessa
body: Removed snprint as suggested in the latest version
created_at: 2025-03-28T19:57:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -62,6 +62,29 @@ static fakeConnection *connCreateFake(void) {
     return conn;
 }
 
+static int find_pattern_in_log(char *str) {
path: src/unit/test_networking.c
user: login: madolson
body: I feel like this should be an integration test, in TCL, than in a unit test.
created_at: 2025-03-28T23:22:50Z
author_association: MEMBER
diff_hunk: @@ -62,6 +62,29 @@ static fakeConnection *connCreateFake(void) {
     return conn;
 }
 
+static int find_pattern_in_log(char *str) {
path: src/unit/test_networking.c
user: login: hpatro
body: Yeah, I had the same thought when I requested for it to address Viktor's question. 
created_at: 2025-03-28T23:57:25Z
author_association: CONTRIBUTOR

PR comments:
user: login: zuiderkwast
created_at: 2025-03-27T12:22:04Z
body: When exactly is this triggered? Only when the client sends some incorrectly encoded RESP?
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-03-27T12:35:48Z
body: ## [Codecov](https://app.codecov.io/gh/valkey-io/valkey/pull/1889?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valkey-io) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 71.00%. Comparing base [(`06990c2`)](https://app.codecov.io/gh/valkey-io/valkey/commit/06990c2e6360c90e5e0b1b5aa4e7708b13955851?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valkey-io) to head [(`1b77cf5`)](https://app.codecov.io/gh/valkey-io/valkey/commit/1b77cf58dbd9743fb398a703a7ba25acb7283ed4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valkey-io).
> Report is 5 commits behind head on unstable.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff              @@
##           unstable    #1889      +/-   ##
============================================
- Coverage     71.00%   71.00%   -0.01%     
============================================
  Files           123      123              
  Lines         65671    65675       +4     
============================================
+ Hits          46631    46632       +1     
- Misses        19040    19043       +3     
```

| [Files with missing lines](https://app.codecov.io/gh/valkey-io/valkey/pull/1889?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valkey-io) | Coverage Δ | |
|---|---|---|
| [src/networking.c](https://app.codecov.io/gh/valkey-io/valkey/pull/1889?src=pr&el=tree&filepath=src%2Fnetworking.c&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valkey-io#diff-c3JjL25ldHdvcmtpbmcuYw==) | `88.78% <100.00%> (+0.12%)` | :arrow_up: |

... and [15 files with indirect coverage changes](https://app.codecov.io/gh/valkey-io/valkey/pull/1889/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valkey-io)

</details>
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: QuChen88
created_at: 2025-03-28T04:44:48Z
body: Looks good to me
author_association: CONTRIBUTOR
user: login: QuChen88
created_at: 2025-03-28T04:52:22Z
body: @zuiderkwast Basically any sort of protocol parsing error from the primary to the replica can lead to this condition to get triggered and the query buffer containing potential user data/commands would get logged. This can happen for various reasons including data corruption over the wire etc. i.e. we saw this in a couple of our redis clusters in the past. 

```
3524:S 26 Mar 2025 17:49:50.755 # WARNING: Receiving inline protocol from master, master stream corruption? Closing the master connection and discarding the cached master.
3524:S 26 Mar 2025 17:49:50.755 # Protocol error (Master using the inline protocol. Desync?) from client: ... Query buffer during protocol error: '293..$4..hdel..$31.. <key-name>...
```

The chances of this happening is pretty low, but still non zero.
author_association: CONTRIBUTOR