Repository: minio/minio (language: go)

Issue:
"mc admin heal" fails with "internal error" starting with RELEASE.2023-12-09.
## NOTE
If this case is urgent, please subscribe to [Subnet](https://min.io/pricing) so that our 24/7 support team may help you faster.

<!--- Provide a general summary of the issue in the Title above -->
The "mc admin heal" command fails with "internal error" starting with
minio RELEASE.2023-12-09T18-17-51Z. It worked before then. "mc admin
heal --recursive" still works.

## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
There should be no internal error, the heal command should work.

<!--- If you're suggesting a change/improvement, tell us how it should work -->

## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
You get an internal error and the command fails.

<!--- If suggesting a change/improvement, explain the difference from current behavior -->

## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->

## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and make sure you have followed https://github.com/minio/minio/tree/release/docs/debugging to capture relevant logs -->

# minio --version
minio version RELEASE.2024-01-11T07-46-16Z (commit-id=099e88516dd6450ff210606abb06f38051d3bb6a)
Runtime: go1.21.5 linux/amd64

# minio server --address :8399 http://10.1.1.231/minio/disk1 http://10.1.1.231/minio/disk2 http://10.1.1.231/minio/disk3 http://10.1.1.231/minio/disk4 &

# export MC_HOST_s3_8399="http://minioadmin:minioadmin@10.1.1.231:8399"
# mc --no-color stat s3_8399/

# mc --no-color admin heal s3_8399/

API: BackgroundHealStatus()
Time: 19:25:37 UTC 01/12/2024
DeploymentID: 60808010-c04e-4953-9a34-d4d4b13d13ac
RequestID: 17A9B014BBD10B44
RemoteHost: 10.1.1.231
Host: 10.1.1.231:8399
UserAgent: MinIO (linux; amd64) madmin-go/2.0.0 mc-RELEASE.2024-01-11T05-49-32Z/RELEASE.2024-01-11T05-49-32Z
Error: all remote servers failed to report heal status, cluster is unhealthy (*errors.errorString)
       6: internal/logger/logger.go:259:logger.LogIf()
       5: cmd/api-errors.go:2370:cmd.toAPIErrorCode()
       4: cmd/admin-handler-utils.go:232:cmd.toAdminAPIErrCode()
       3: cmd/admin-handler-utils.go:220:cmd.toAdminAPIErr()
       2: cmd/admin-handlers.go:1121:cmd.adminAPIHandlers.BackgroundHealStatusHandler()
       1: net/http/server.go:2136:http.HandlerFunc.ServeHTTP()
mc-RELEASE.2024-01-11T05-49-32Z: <ERROR> Unable to get background heal status. We encountered an internal error, please try again. (all remote servers failed to report heal status, cluster is unhealthy).

# mc --no-color admin info s3_8399/
?  10.1.1.231:8399
   Uptime: 6 minutes
   Version: 2024-01-11T07:46:16Z
   Network: 1/1 OK
   Drives: 4/4 OK
   Pool: 1

Pools:
   1st, Erasure sets: 1, Drives per erasure set: 4

4 drives online, 0 drives offline

# mc --no-color admin heal --recursive s3_8399/
 ?  [bucket-metadata].minio.sys/config/iam/format.json
    0/0 objects; 0 B in 1s
    ????????????????????????????????????
    ? Green  ? 2 ? 100.0% ???????????? ?
    ? Yellow ? 0 ?   0.0%              ?
    ? Red    ? 0 ?   0.0%              ?
    ? Grey   ? 0 ?   0.0%              ?
    ????????????????????????????????????

## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
Trying to check the healing status, if any.

## Regression
<!-- Is this issue a regression? (Yes / No) -->
Yes.
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->
RELEASE.2023-12-09T18-17-51Z

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio --version`):
RELEASE.2024-01-11T07-46-16Z
* Server setup and configuration:
Simple reproducer test env.
* Operating System and version (`uname -a`):
CentOS Linux.


Discussion / hints:
@JeffByers-SF how did you start the server? 
`# minio server --address :8399 http://10.1.1.231/minio/disk1 http://10.1.1.231/minio/disk2 http://10.1.1.231/minio/disk3 http://10.1.1.231/minio/disk4`
All servers are local what is the point of running distributed setup like this? just run it 

```
minio server --address :8399 /minio/disk{1...4}
```

Then `mc admin heal` will work fine. You are trying to setup a distributed cluster where there are no new nodes. 

Fix patch:
diff --git a/cmd/endpoint.go b/cmd/endpoint.go
index 5be80b3dc899d..84a6a61571e2b 100644
--- a/cmd/endpoint.go
+++ b/cmd/endpoint.go
@@ -103,7 +103,7 @@ func (endpoint Endpoint) GridHost() string {
 
 // UpdateIsLocal - resolves the host and updates if it is local or not.
 func (endpoint *Endpoint) UpdateIsLocal() (err error) {
-	if !endpoint.IsLocal {
+	if endpoint.Host != "" {
 		endpoint.IsLocal, err = isLocalHost(endpoint.Hostname(), endpoint.Port(), globalMinioPort)
 		if err != nil {
 			return err
@@ -522,6 +522,16 @@ func (endpoints Endpoints) UpdateIsLocal() error {
 					continue
 				}
 
+				if endpoints[i].Host == "" {
+					resolvedList[i] = true
+					endpoints[i].IsLocal = true
+					epsResolved++
+					if !foundLocal {
+						foundLocal = true
+					}
+					continue
+				}
+
 				// Log the message to console about the host resolving
 				reqInfo := (&logger.ReqInfo{}).AppendTags(
 					"host",
@@ -705,6 +715,17 @@ func (p PoolEndpointList) UpdateIsLocal() error {
 						continue
 					}
 
+					if endpoint.Host == "" {
+						if !foundLocal {
+							foundLocal = true
+						}
+						endpoint.IsLocal = true
+						endpoints[j] = endpoint
+						epsResolved++
+						resolvedList[endpoint] = true
+						continue
+					}
+
 					// Log the message to console about the host resolving
 					reqInfo := (&logger.ReqInfo{}).AppendTags(
 						"host",
@@ -859,6 +880,8 @@ func CreatePoolEndpoints(serverAddr string, poolsLayout ...poolDisksLayout) ([]E
 		return poolEndpoints, setupType, nil
 	}
 
+	uniqueArgs := set.NewStringSet()
+
 	for poolIdx, pool := range poolsLayout {
 		var endpoints Endpoints
 		for setIdx, setLayout := range pool.layout {
@@ -889,26 +912,10 @@ func CreatePoolEndpoints(serverAddr string, poolsLayout ...poolDisksLayout) ([]E
 		poolEndpoints[poolIdx] = endpoints
 	}
 
-	for _, endpoints := range poolEndpoints {
-		// Return Erasure setup when all endpoints are path style.
-		if endpoints[0].Type() == PathEndpointType {
-			setupType = ErasureSetupType
-		}
-		if endpoints[0].Type() == URLEndpointType && setupType != DistErasureSetupType {
-			setupType = DistErasureSetupType
-		}
-	}
-
-	if setupType == ErasureSetupType {
-		return poolEndpoints, setupType, nil
-	}
-
 	if err := poolEndpoints.UpdateIsLocal(); err != nil {
 		return nil, setupType, config.ErrInvalidErasureEndpoints(nil).Msg(err.Error())
 	}
 
-	uniqueArgs := set.NewStringSet()
-
 	for i, endpoints := range poolEndpoints {
 		// Here all endpoints are URL style.
 		endpointPathSet := set.NewStringSet()
@@ -918,7 +925,7 @@ func CreatePoolEndpoints(serverAddr string, poolsLayout ...poolDisksLayout) ([]E
 
 		for _, endpoint := range endpoints {
 			endpointPathSet.Add(endpoint.Path)
-			if endpoint.IsLocal {
+			if endpoint.IsLocal && endpoint.Host != "" {
 				localServerHostSet.Add(endpoint.Hostname())
 
 				_, port, err := net.SplitHostPort(endpoint.Host)
@@ -934,9 +941,8 @@ func CreatePoolEndpoints(serverAddr string, poolsLayout ...poolDisksLayout) ([]E
 		orchestrated := IsKubernetes() || IsDocker()
 		reverseProxy := (env.Get("_MINIO_REVERSE_PROXY", "") != "") && ((env.Get("MINIO_CI_CD", "") != "") || (env.Get("CI", "") != ""))
 		// If not orchestrated
-		if !orchestrated &&
-			// and not setup in reverse proxy
-			!reverseProxy {
+		// and not setup in reverse proxy
+		if !orchestrated && !reverseProxy {
 			// Check whether same path is not used in endpoints of a host on different port.
 			// Only verify this on baremetal setups, DNS is not available in orchestrated
 			// environments so we can't do much here.
@@ -944,14 +950,18 @@ func CreatePoolEndpoints(serverAddr string, poolsLayout ...poolDisksLayout) ([]E
 			hostIPCache := make(map[string]set.StringSet)
 			for _, endpoint := range endpoints {
 				host := endpoint.Hostname()
-				hostIPSet, ok := hostIPCache[host]
-				if !ok {
-					var err error
-					hostIPSet, err = getHostIP(host)
-					if err != nil {
-						return nil, setupType, config.ErrInvalidErasureEndpoints(nil).Msg(fmt.Sprintf("host '%s' cannot resolve: %s", host, err))
+				var hostIPSet set.StringSet
+				if host != "" {
+					var ok bool
+					hostIPSet, ok = hostIPCache[host]
+					if !ok {
+						var err error
+						hostIPSet, err = getHostIP(host)
+						if err != nil {
+							return nil, setupType, config.ErrInvalidErasureEndpoints(nil).Msg(fmt.Sprintf("host '%s' cannot resolve: %s", host, err))
+						}
+						hostIPCache[host] = hostIPSet
 					}
-					hostIPCache[host] = hostIPSet
 				}
 				if IPSet, ok := pathIPMap[endpoint.Path]; ok {
 					if !IPSet.Intersection(hostIPSet).IsEmpty() {
@@ -982,12 +992,14 @@ func CreatePoolEndpoints(serverAddr string, poolsLayout ...poolDisksLayout) ([]E
 
 		// Add missing port in all endpoints.
 		for i := range endpoints {
-			_, port, err := net.SplitHostPort(endpoints[i].Host)
-			if err != nil {
-				endpoints[i].Host = net.JoinHostPort(endpoints[i].Host, serverAddrPort)
-			} else if endpoints[i].IsLocal && serverAddrPort != port {
-				// If endpoint is local, but port is different than serverAddrPort, then make it as remote.
-				endpoints[i].IsLocal = false
+			if endpoints[i].Host != "" {
+				_, port, err := net.SplitHostPort(endpoints[i].Host)
+				if err != nil {
+					endpoints[i].Host = net.JoinHostPort(endpoints[i].Host, serverAddrPort)
+				} else if endpoints[i].IsLocal && serverAddrPort != port {
+					// If endpoint is local, but port is different than serverAddrPort, then make it as remote.
+					endpoints[i].IsLocal = false
+				}
 			}
 		}
 
@@ -1006,10 +1018,12 @@ func CreatePoolEndpoints(serverAddr string, poolsLayout ...poolDisksLayout) ([]E
 			// This means it is DistErasure setup.
 		}
 
-		poolEndpoints[i] = endpoints
-
 		for _, endpoint := range endpoints {
-			uniqueArgs.Add(endpoint.Host)
+			if endpoint.Host != "" {
+				uniqueArgs.Add(endpoint.Host)
+			} else {
+				uniqueArgs.Add(net.JoinHostPort("localhost", serverAddrPort))
+			}
 		}
 
 		poolEndpoints[i] = endpoints
@@ -1020,13 +1034,21 @@ func CreatePoolEndpoints(serverAddr string, poolsLayout ...poolDisksLayout) ([]E
 		updateDomainIPs(uniqueArgs)
 	}
 
+	erasureType := len(uniqueArgs.ToSlice()) == 1
+
 	for _, endpoints := range poolEndpoints {
 		// Return Erasure setup when all endpoints are path style.
 		if endpoints[0].Type() == PathEndpointType {
 			setupType = ErasureSetupType
+			break
 		}
-		if endpoints[0].Type() == URLEndpointType && setupType != DistErasureSetupType {
-			setupType = DistErasureSetupType
+		if endpoints[0].Type() == URLEndpointType {
+			if erasureType {
+				setupType = ErasureSetupType
+			} else {
+				setupType = DistErasureSetupType
+			}
+			break
 		}
 	}
 


Test patch:
diff --git a/cmd/endpoint_test.go b/cmd/endpoint_test.go
index 88a85befa74a1..5fd31ed8b9ae5 100644
--- a/cmd/endpoint_test.go
+++ b/cmd/endpoint_test.go
@@ -258,7 +258,7 @@ func TestCreateEndpoints(t *testing.T) {
 			Endpoint{URL: &url.URL{Scheme: "http", Host: "localhost:9000", Path: "/d2"}, IsLocal: true},
 			Endpoint{URL: &url.URL{Scheme: "http", Host: "localhost:9000", Path: "/d3"}, IsLocal: true},
 			Endpoint{URL: &url.URL{Scheme: "http", Host: "localhost:9000", Path: "/d4"}, IsLocal: true},
-		}, DistErasureSetupType, nil},
+		}, ErasureSetupType, nil},
 		// DistErasure Setup with URLEndpointType having mixed naming to local host.
 		{"127.0.0.1:10000", []string{"http://localhost/d1", "http://localhost/d2", "http://127.0.0.1/d3", "http://127.0.0.1/d4"}, "", Endpoints{}, -1, fmt.Errorf("all local endpoints should not have different hostnames/ips")},