Repository: minio/minio (language: go)

Issue:
Add a way to disable DNS caching during startup
**Is your feature request related to a problem? Please describe.**

In a Kubernetes environment, if the container starts before the DNS records have been fully propagated, the DNS cache stores a negative result which causes the startup to be delayed by a minute (https://github.com/minio/minio/blob/master/cmd/common-main.go#L120).

During this timeframe each failing lookup logs an error which puts heavy load on the logging system, resulting in ~150000 lines of log messages for a simple 2 Nodes/2 Disks setup.

**Describe the solution you'd like**

Introduce an option to disable the DNS lookup cache

**Describe alternatives you've considered**

Alternatively, introducing an exponential back off approach would at least remediate the log spam

**Additional context**
If this is something desirable, I can implement it myself, I just wanted to open up the issue for discussion.



Discussion / hints:
> During this timeframe each failing lookup logs an error which puts heavy load on the logging system, resulting in ~150000 lines of log messages for a simple 2 Nodes/2 Disks setup.

The logging itself can be turned off for similar-looking logs, however, the DNS will be available if you use a headless service for a stateful set when the pod gets scheduled, and this doesn't change once the "pod has started". So what you are saying doesn't make sense.

I suspect there is a deployment issue here, can you share how you deployed MinIO? share the yaml file of your deployment pattern. 
Slowed down DNS propagation is the only thing which makes sense to me. 

The Kubernetes spec never guarantees that propagation is finished before the pod is started, especially if the Kubernetes DNS resolver utilizes caching itself.

> Depending on how DNS is configured in your cluster, you may not be able to look up the DNS name for a newly-run Pod immediately. This behavior can occur when other clients in the cluster have already sent queries for the hostname of the Pod before it was created. Negative caching (normal in DNS) means that the results of previous failed lookups are remembered and reused, even after the Pod is running, for at least a few seconds.

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id

Modifying the global DNS settings is not an option for us as this would kill performance in other parts of the system.

These are the relevant parts of my deployment, but I think this has to be fixed in software:
```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: minio-auxiliary
spec:
  podManagementPolicy: Parallel
  replicas: 2
  selector:
    matchLabels:
      name: minio-auxiliary
  serviceName: minio-auxiliary-hl
  template:
    metadata:
      labels:
        name: minio-auxiliary
    spec:
      containers:
      - args:
        - server
        - --console-address
        - :9090
        - http://minio-auxiliary-{0...1}.minio-auxiliary-hl:9000/export/data{0...1}
        env:
        - name: MINIO_STORAGE_CLASS_STANDARD
          value: EC:2
        envFrom:
        - secretRef:
            name: minio-auxiliary
        image: quay.io/minio/minio:RELEASE.2022-10-21T22-37-48Z
        imagePullPolicy: IfNotPresent
        name: minio
        ports:
        - containerPort: 9000
          name: http
        - containerPort: 9090
          name: console
        volumeMounts:
        - mountPath: /export/data0
          name: data0
        - mountPath: /export/data1
          name: data1
  updateStrategy:
    type: RollingUpdate
---
apiVersion: v1
kind: Service
metadata:
  name: minio-auxiliary-hl
spec:
  ports:
  - name: http
    port: 9000
    targetPort: 9000
  - name: console
    port: 9090
    targetPort: 9090
  publishNotReadyAddresses: true
  selector:
    name: minio-auxiliary
```

And this is the recurring error I get in the first 60 seconds:
```
API: SYSTEM()
Time: 08:35:53 UTC 01/04/2023
Error: lookup minio-auxiliary-0.minio-auxiliary-hl on 10.76.0.10:53: no such host (*net.DNSError)
       host="minio-auxiliary-0.minio-auxiliary-hl", elapsedTime="1 second elapsed"
       6: internal/logger/logger.go:259:logger.LogIf()
       5: cmd/endpoint.go:478:cmd.Endpoints.UpdateIsLocal()
       4: cmd/endpoint.go:633:cmd.CreateEndpoints()
       3: cmd/endpoint-ellipses.go:373:cmd.createServerEndpoints()
       2: cmd/server-main.go:194:cmd.serverHandleCmdArgs()
       1: cmd/server-main.go:462:cmd.serverMain()
```
I think i understand the problem now @theSuess 

Fix patch:
diff --git a/cmd/common-main.go b/cmd/common-main.go
index b0cec4205fd58..9ec492f487f3b 100644
--- a/cmd/common-main.go
+++ b/cmd/common-main.go
@@ -117,7 +117,8 @@ func init() {
 	go func() {
 		var t *time.Ticker
 		if containers {
-			t = time.NewTicker(1 * time.Minute)
+			// k8s DNS TTL is 30s (Attempt a refresh only after)
+			t = time.NewTicker(30 * time.Second)
 		} else {
 			t = time.NewTicker(10 * time.Minute)
 		}
diff --git a/cmd/endpoint.go b/cmd/endpoint.go
index 4ed51dd333b33..e5cbe262d9538 100644
--- a/cmd/endpoint.go
+++ b/cmd/endpoint.go
@@ -417,6 +417,27 @@ func (endpoints Endpoints) atleastOneEndpointLocal() bool {
 	return false
 }
 
+func hostResolveToLocalhost(endpoint Endpoint) bool {
+	hostIPs, err := getHostIP(endpoint.Hostname())
+	if err != nil {
+		// Log the message to console about the host resolving
+		reqInfo := (&logger.ReqInfo{}).AppendTags(
+			"host",
+			endpoint.Hostname(),
+		)
+		ctx := logger.SetReqInfo(GlobalContext, reqInfo)
+		logger.LogOnceIf(ctx, err, endpoint.Hostname(), logger.Application)
+		return false
+	}
+	var loopback int
+	for _, hostIP := range hostIPs.ToSlice() {
+		if net.ParseIP(hostIP).IsLoopback() {
+			loopback++
+		}
+	}
+	return loopback == len(hostIPs)
+}
+
 // UpdateIsLocal - resolves the host and discovers the local host.
 func (endpoints Endpoints) UpdateIsLocal(foundPrevLocal bool) error {
 	orchestrated := IsDocker() || IsKubernetes()
@@ -452,6 +473,26 @@ func (endpoints Endpoints) UpdateIsLocal(foundPrevLocal bool) error {
 					endpoints[i].Hostname(),
 				)
 
+				if orchestrated && hostResolveToLocalhost(endpoints[i]) {
+					// time elapsed
+					timeElapsed := time.Since(startTime)
+					// log error only if more than a second has elapsed
+					if timeElapsed > time.Second {
+						reqInfo.AppendTags("elapsedTime",
+							humanize.RelTime(startTime,
+								startTime.Add(timeElapsed),
+								"elapsed",
+								"",
+							))
+						ctx := logger.SetReqInfo(GlobalContext,
+							reqInfo)
+						logger.LogOnceIf(ctx, fmt.Errorf("%s resolves to localhost in a containerized deployment, waiting for it to resolve to a valid IP",
+							endpoints[i].Hostname()), endpoints[i].Hostname(), logger.Application)
+					}
+
+					continue
+				}
+
 				// return err if not Docker or Kubernetes
 				// We use IsDocker() to check for Docker environment
 				// We use IsKubernetes() to check for Kubernetes environment
@@ -465,7 +506,7 @@ func (endpoints Endpoints) UpdateIsLocal(foundPrevLocal bool) error {
 				if err != nil {
 					// time elapsed
 					timeElapsed := time.Since(startTime)
-					// log error only if more than 1s elapsed
+					// log error only if more than a second has elapsed
 					if timeElapsed > time.Second {
 						reqInfo.AppendTags("elapsedTime",
 							humanize.RelTime(startTime,
@@ -475,7 +516,7 @@ func (endpoints Endpoints) UpdateIsLocal(foundPrevLocal bool) error {
 							))
 						ctx := logger.SetReqInfo(GlobalContext,
 							reqInfo)
-						logger.LogIf(ctx, err, logger.Application)
+						logger.LogOnceIf(ctx, err, endpoints[i].Hostname(), logger.Application)
 					}
 				} else {
 					resolvedList[i] = true
diff --git a/cmd/server-main.go b/cmd/server-main.go
index 6c0b138f2b50c..98c77c3023784 100644
--- a/cmd/server-main.go
+++ b/cmd/server-main.go
@@ -399,10 +399,14 @@ func initServer(ctx context.Context, newObject ObjectLayer) error {
 		lkctx, err := txnLk.GetLock(ctx, lockTimeout)
 		if err != nil {
 			logger.Info("Waiting for all MinIO sub-systems to be initialized.. trying to acquire lock")
-			waitDuration := time.Duration(r.Float64() * 2 * float64(time.Second))
+			waitDuration := time.Duration(r.Float64() * 5 * float64(time.Second))
 			bootstrapTrace(fmt.Sprintf("lock not available. error: %v. sleeping for %v before retry", err, waitDuration))
 
-			// Sleep 0 -> 2 seconds to average 1 second retry interval.
+			// Sleep 0 -> 5 seconds, provider a higher range such that sleeps()
+			// and retries for lock are more spread out, needed orchestrated
+			// systems take 30s minimum to respond to DNS resolvers.
+			//
+			// Do not change this value.
 			time.Sleep(waitDuration)
 			continue
 		}


Test patch: