Repository: minio/minio (language: go)

Issue:
Failed to fetch last day tier stats
When I use `mc admin tier info`, my cluster throws :

```
API: TierStats()
Time: 15:58:34 UTC 08/29/2022
DeploymentID: f1062927-f415-47dd-affc-cca707c4a193
RequestID: 170FDB368BC435D0
RemoteHost: 10.244.94.63
Host: hot-pool-0-4.hot-hl.minio-hot.svc.cluster.local:9000
UserAgent: MinIO (linux; amd64) madmin-go/0.0.1
Error: failed to fetch last day tier stats: Post "https://hot-pool-0-3.hot-hl.minio-hot.svc.cluster.local:9000/minio/peer/v24/getlastdaytierstats": net/http: timeout awaiting response headers (*fmt.wrapError)
       objectLocation=cmd.auditObjectOp{Name:"buckets/.usage.json", Pool:1, Set:1, Disks:[]string{"https://hot-pool-0-0.hot-hl.minio-hot.svc.cluster.local:9000/export", "https://hot-pool-0-1.hot-hl.minio-hot.svc.cluster.local:9000/export", "https://hot-pool-0-2.hot-hl.minio-hot.svc.cluster.local:9000/export", "https://hot-pool-0-3.hot-hl.minio-hot.svc.cluster.local:9000/export", "https://hot-pool-0-4.hot-hl.minio-hot.svc.cluster.local:9000/export"}}
       4: internal/logger/logger.go:259:logger.LogIf()
       3: cmd/notification.go:1630:cmd.(*NotificationSys).GetLastDayTierStats()
       2: cmd/tier-handlers.go:294:cmd.adminAPIHandlers.TierStatsHandler()
       1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
```

And I get a `504 Gateway Time-out`.
The cluster is healthy and runs version `2022-08-02T23:59:16Z`.
It also impacts the console which can't load the tiers' config.
Another annoying side effect is that some nodes are marked offline temporarily because of this call.


Discussion / hints:
> Error: failed to fetch last day tier stats: Post "https://hot-pool-0-3.hot-hl.minio-hot.svc.cluster.local:9000/minio/peer/v24/getlastdaytierstats": net/http: timeout awaiting response headers (*fmt.wrapError)

Well, indeed the issue is that you have network issues on the setup, we have waited for at least 15mins to get a response here from the peer. 

The entire call to the peer server response is an in-memory call so there is no reason the server will be spending 15mins, mostly your node is not responding. 
Also @hrenard please describe your setup you seem to skim the entire issue template - we cannot imagine your setup. 
Sorry for the template. 
I use the k8s operator `v4.4.27`, I got 1 pool with 5 nodes and 5 disks.
I'm not convinced it's a network issue because `mc admin info` is all green, the Grafana dashboard seems fine, we're using the cluster with no issue and right after the command finally fails, the nodes that are marked offline temporarily get reconnected.
> Sorry for the template. I use the k8s operator `v4.4.27`, I got 1 pool with 5 nodes and 5 disks. I'm not convinced it's a network issue because `mc admin info` is all green, the Grafana dashboard seems fine, we're using the cluster with no issue and right after the command finally fails, the nodes that are marked offline temporarily get reconnected.

can this be reproduced all the time? @hrenard 
Yes at every call (round-robin balanced). It's not always the same node throwing the error nor being marked offline.
I don't know if it helps but `mc admin tier ls` works. 
It seems to be only the `/getlastdaytierstats` endpoint which fails.
okay can you do this? `mc support profile --duration 30 alias/` - at the same time run `mc admin tier info`  - please share the profile.zip once saved locally. 
Yes. [Here it is](https://hot-objects.liiib.re/share/profile.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=hougo%2F20220901%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220901T160502Z&X-Amz-Expires=7200&X-Amz-SignedHeaders=host&X-Amz-Signature=43a26b9efaf6f5ee5bc3078c6046c9a80aae6d389fb00e6b1a049495b5c269ea).

Fix patch:
diff --git a/cmd/tier-last-day-stats.go b/cmd/tier-last-day-stats.go
index cc8b1cbb42b1f..b4bb8199b0a6d 100644
--- a/cmd/tier-last-day-stats.go
+++ b/cmd/tier-last-day-stats.go
@@ -38,6 +38,10 @@ func (l *lastDayTierStats) addStats(ts tierStats) {
 
 // forwardTo moves time to t, clearing entries between last update and t.
 func (l *lastDayTierStats) forwardTo(t time.Time) {
+	if t.IsZero() {
+		t = time.Now()
+	}
+
 	since := t.Sub(l.UpdatedAt).Hours()
 	// within the hour since l.UpdatedAt
 	if since < 1 {
@@ -45,15 +49,17 @@ func (l *lastDayTierStats) forwardTo(t time.Time) {
 	}
 
 	idx, lastIdx := t.Hour(), l.UpdatedAt.Hour()
-	l.UpdatedAt = t
+
+	l.UpdatedAt = t // update to the latest time index
 
 	if since >= 24 {
 		l.Bins = [24]tierStats{}
 		return
 	}
 
-	for ; lastIdx != idx; lastIdx++ {
-		l.Bins[(lastIdx+1)%24] = tierStats{}
+	for lastIdx != idx {
+		lastIdx = (lastIdx + 1) % 24
+		l.Bins[lastIdx] = tierStats{}
 	}
 }
 


Test patch: