Repository: minio/minio (language: go)

Issue:
sporadic websocket grid test race
## NOTE
If this case is urgent, please subscribe to [Subnet](https://min.io/pricing) so that our 24/7 support team may help you faster.

<!--- Provide a general summary of the issue in the Title above -->

## Expected Behavior
No race detected
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->

## Current Behavior
```
=== RUN   TestDisconnect
    connection_test.go:86: Started server on 127.0.0.1:33893 URL: http://127.0.0.1:33893/
    connection_test.go:87: Started server on 127.0.0.1:36913 URL: http://127.0.0.1:36913/
    connection_test.go:46: Got a GET request for: /minio/grid/v1
    connection_test.go:136: Roundtrip: sending request
    connection_test.go:138: Roundtrip: 455.721µs [] remote disconnected
    connection_test.go:46: Got a GET request for: /minio/grid/v1
    connection_test.go:153: Resp: {[] remote disconnected} <nil>
    connection_test.go:46: Got a GET request for: /minio/grid/v1
==================
WARNING: DATA RACE
Write at 0x00c0002e3858 by goroutine 10:
  runtime.racewrite()
      <autogenerated>:1 +0x1e
  github.com/minio/minio/internal/grid.(*Connection).debugMsg()
      /home/runner/work/minio/minio/internal/grid/connection.go:1537 +0x2ba
  github.com/minio/minio/internal/grid.(*Manager).debugMsg()
      /home/runner/work/minio/minio/internal/grid/manager.go:323 +0x2a4
  github.com/minio/minio/internal/grid.TestDisconnect.func7()
      /home/runner/work/minio/minio/internal/grid/connection_test.go:96 +0x22f
  runtime.deferreturn()
      /opt/hostedtoolcache/go/1.21.6/x64/src/runtime/panic.go:477 +0x30
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1648 +0x44

Previous read at 0x00c0002e3858 by goroutine 43:
  runtime.raceread()
      <autogenerated>:1 +0x1e
  github.com/minio/minio/internal/grid.(*Connection).handleMessages()
      /home/runner/work/minio/minio/internal/grid/connection.go:858 +0x77
  github.com/minio/minio/internal/grid.(*Connection).connect.func2()
      /home/runner/work/minio/minio/internal/grid/connection.go:709 +0x6b

Goroutine 10 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1648 +0x845
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1595 +0x261
  testing.runTests()
      /opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:2052 +0x8ad
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1925 +0xcd7
  main.main()
      _testmain.go:145 +0x2bd

Goroutine 43 (finished) created at:
  github.com/minio/minio/internal/grid.(*Connection).connect()
      /home/runner/work/minio/minio/internal/grid/connection.go:709 +0x112c
  github.com/minio/minio/internal/grid.newConnection.func1()
      /home/runner/work/minio/minio/internal/grid/connection.go:245 +0x64
==================
    testing.go:1465: race detected during execution of test
--- FAIL: TestDisconnect (0.01s)
```

<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->

## Possible Solution
Not sure
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->

## Steps to Reproduce (for bugs)
Rare and complicated to reproduce via `make test-race`

<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and make sure you have followed https://github.com/minio/minio/tree/release/docs/debugging to capture relevant logs -->


## Context
Races should be removed in the code
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->

## Regression
Not really
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->

## Your Environment
master branch



Discussion / hints:
This one is in the debug system where it injects an error... However, it does seem like it `c.handleMsgWg.Wait()` can overlap with a `c.handleMsgWg.Add()` - where here is harmless. However, I want to make sure this isn't possible for the `reconnected()` where there is a similar wait before I fix this. 

Fix patch:
diff --git a/internal/grid/connection.go b/internal/grid/connection.go
index 40dc000ca9b36..970ec45bf687d 100644
--- a/internal/grid/connection.go
+++ b/internal/grid/connection.go
@@ -831,7 +831,9 @@ func (c *Connection) reconnected() {
 	c.outgoing.Clear()
 
 	// Wait for existing to exit
+	c.connMu.Lock()
 	c.handleMsgWg.Wait()
+	c.connMu.Unlock()
 }
 
 func (c *Connection) updateState(s State) {
@@ -855,7 +857,9 @@ func (c *Connection) updateState(s State) {
 
 func (c *Connection) handleMessages(ctx context.Context, conn net.Conn) {
 	// Read goroutine
+	c.connMu.Lock()
 	c.handleMsgWg.Add(2)
+	c.connMu.Unlock()
 	ctx, cancel := context.WithCancelCause(ctx)
 	go func() {
 		defer func() {
@@ -1534,7 +1538,9 @@ func (c *Connection) debugMsg(d debugMsg, args ...any) {
 			c.debugInConn.Close()
 		}
 	case debugWaitForExit:
+		c.connMu.Lock()
 		c.handleMsgWg.Wait()
+		c.connMu.Unlock()
 	case debugSetConnPingDuration:
 		c.connMu.Lock()
 		defer c.connMu.Unlock()


Test patch: