Repository: minio/minio (language: go)

Issue:
fatal error: concurrent map iteration and map write
## NOTE
If this case is urgent, please subscribe to [Subnet](https://min.io/pricing) so that our 24/7 support team may help you faster.

<!--- Provide a general summary of the issue in the Title above -->

## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->

## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->
```bash
Client 'http://node7.example.com:9000/minio/storage/mnt/disk32/v50' re-connected in 71h30m52.142826629s
Client 'http://node7.example.com:9000/minio/peer-s3/v1' re-connected in 71h31m4.498408213s
fatal error: concurrent map iteration and map write

goroutine 63046000758 [running]:
github.com/minio/minio/cmd.getResourceMetrics.func1({0x2394000?, 0xc000a61548?})
        github.com/minio/minio/cmd/metrics-resource.go:439 +0x23b
github.com/minio/minio/cmd.(*MetricsGroup).RegisterRead.func1.1()
        github.com/minio/minio/cmd/metrics-v2.go:322 +0x25
github.com/minio/minio/cmd.(*timedValue).Get(0xc000a61500)
        github.com/minio/minio/cmd/utils.go:984 +0x3e
github.com/minio/minio/cmd.(*MetricsGroup).Get(0x44f9c0?)
        github.com/minio/minio/cmd/metrics-v2.go:352 +0x25
github.com/minio/minio/cmd.populateAndPublish({0xc00114a3d8, 0x1, 0xc00b94efd0?}, 0xc003637fa8)
        github.com/minio/minio/cmd/metrics-v2.go:3717 +0x5e
github.com/minio/minio/cmd.ReportMetrics.func1()
        github.com/minio/minio/cmd/metrics-v2.go:3682 +0x8d
created by github.com/minio/minio/cmd.ReportMetrics in goroutine 63045614723
        github.com/minio/minio/cmd/metrics-v2.go:3680 +0xe5

goroutine 1 [chan receive, 6185 minutes]:
github.com/minio/minio/cmd.serverMain(0xc000b03860)
        github.com/minio/minio/cmd/server-main.go:959 +0xebc
github.com/minio/cli.HandleAction({0x25c8160?, 0x46b2b68?}, 0x6?)
        github.com/minio/cli@v1.24.2/app.go:492 +0x70
github.com/minio/cli.Command.Run({{0x2b21db6, 0x6}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x2b5f34c, 0x1b}, {0x0, ...}, ...}, ...)
        github.com/minio/cli@v1.24.2/command.go:242 +0x965
github.com/minio/cli.(*App).Run(0xc00124c1c0, {0xc000166000, 0x5, 0x5})
        github.com/minio/cli@v1.24.2/app.go:260 +0xaa7
github.com/minio/minio/cmd.Main({0xc000166000?, 0x5, 0x5})
        github.com/minio/minio/cmd/main.go:215 +0xe8
main.main()
        github.com/minio/minio/main.go:30 +0x28
```
## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->

## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and make sure you have followed https://github.com/minio/minio/tree/release/docs/debugging to capture relevant logs -->

Number of machines: 16
Number of hard drives: 36
Hard drive capacity: 16TiB
I'm using 20 warp pressure measurement clusters, and three machines have been hung up during this period, which is the problem every time.
```
./warp mixed --warp-client=client{1...20} --host=node{1...16}.example.com:9000 --access-key=minioadmin --secret-key=minioadmin   --put-distrib=80 --get-distrib 10 --stat-distrib 10 --delete-distrib 0 --obj.size 32GiB --objects 180000 --noclear --concurrent 16 --bucket warp-benchmark-bucket3
```

## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->

## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio --version`):
```minio --version
minio version RELEASE.2023-10-07T15-07-38Z (commit-id=c27d0583d4d496981140eff822e2abb34a6f0b60)
Runtime: go1.21.1 linux/amd64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-2023 MinIO, Inc.
```
* Server setup and configuration:
* Operating System and version (`uname -a`):
```
Linux DCZ-2007FA103U28-L07-A01 5.15.0-79-generic #86-Ubuntu SMP Mon Jul 10 16:07:21 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
```



Discussion / hints:
Seems like #18057 is updating the maps it has also returned.  I will see if I can fix it up.

Fix patch:
diff --git a/cmd/metrics-resource.go b/cmd/metrics-resource.go
index d008db3402940..171a7caffb558 100644
--- a/cmd/metrics-resource.go
+++ b/cmd/metrics-resource.go
@@ -74,7 +74,8 @@ const (
 var (
 	resourceCollector *minioResourceCollector
 	// resourceMetricsMap is a map of subsystem to its metrics
-	resourceMetricsMap map[MetricSubsystem]ResourceMetrics
+	resourceMetricsMap   map[MetricSubsystem]ResourceMetrics
+	resourceMetricsMapMu sync.RWMutex
 	// resourceMetricsHelpMap maps metric name to its help string
 	resourceMetricsHelpMap map[MetricName]string
 	resourceMetricsGroups  []*MetricsGroup
@@ -159,6 +160,8 @@ func init() {
 }
 
 func updateResourceMetrics(subSys MetricSubsystem, name MetricName, val float64, labels map[string]string, isCumulative bool) {
+	resourceMetricsMapMu.Lock()
+	defer resourceMetricsMapMu.Unlock()
 	subsysMetrics, found := resourceMetricsMap[subSys]
 	if !found {
 		subsysMetrics = ResourceMetrics{}
@@ -320,7 +323,9 @@ func collectLocalResourceMetrics() {
 
 // startResourceMetricsCollection - starts the job for collecting resource metrics
 func startResourceMetricsCollection() {
+	resourceMetricsMapMu.Lock()
 	resourceMetricsMap = map[MetricSubsystem]ResourceMetrics{}
+	resourceMetricsMapMu.Unlock()
 	metricsTimer := time.NewTimer(resourceMetricsCollectionInterval)
 	defer metricsTimer.Stop()
 
@@ -382,12 +387,11 @@ func newMinioResourceCollector(metricsGroups []*MetricsGroup) *minioResourceColl
 func prepareResourceMetrics(rm ResourceMetric, subSys MetricSubsystem, requireAvgMax bool) []Metric {
 	help := resourceMetricsHelpMap[rm.Name]
 	name := rm.Name
-
-	metrics := []Metric{}
+	metrics := make([]Metric, 0, 3)
 	metrics = append(metrics, Metric{
 		Description:    getResourceMetricDescription(subSys, name, help),
 		Value:          rm.Current,
-		VariableLabels: rm.Labels,
+		VariableLabels: cloneMSS(rm.Labels),
 	})
 
 	if requireAvgMax {
@@ -396,7 +400,7 @@ func prepareResourceMetrics(rm ResourceMetric, subSys MetricSubsystem, requireAv
 		metrics = append(metrics, Metric{
 			Description:    getResourceMetricDescription(subSys, avgName, avgHelp),
 			Value:          math.Round(rm.Avg*100) / 100,
-			VariableLabels: rm.Labels,
+			VariableLabels: cloneMSS(rm.Labels),
 		})
 
 		maxName := MetricName(fmt.Sprintf("%s_max", name))
@@ -404,7 +408,7 @@ func prepareResourceMetrics(rm ResourceMetric, subSys MetricSubsystem, requireAv
 		metrics = append(metrics, Metric{
 			Description:    getResourceMetricDescription(subSys, maxName, maxHelp),
 			Value:          rm.Max,
-			VariableLabels: rm.Labels,
+			VariableLabels: cloneMSS(rm.Labels),
 		})
 	}
 
@@ -429,6 +433,8 @@ func getResourceMetrics() *MetricsGroup {
 		metrics := []Metric{}
 
 		subSystems := []MetricSubsystem{interfaceSubsystem, memSubsystem, driveSubsystem, cpuSubsystem}
+		resourceMetricsMapMu.RLock()
+		defer resourceMetricsMapMu.RUnlock()
 		for _, subSys := range subSystems {
 			stats, found := resourceMetricsMap[subSys]
 			if found {


Test patch: